Remove duplicates from an array of objects in jsonnet

Question

I have an array of objects, I would like to remove duplicates. My array has a common field name that I would like to use for deduplication.

I am trying to convert the array to a map and then from map back to array but map conversions gives me an error duplicate field name: "a":

local arr = [ 
    { "name": "a", "value": 1234},
    { "name": "b", "value": 555},
    { "name": "c", "value": 0}, 
    { "name": "a", "value": 1234} 
];
local map = { [x.name] : x  for x in arr };

Desired output:

[ 
      { "name": "a", "value": 1234},
      { "name": "b", "value": 555}, 
      { "name": "c", "value": 0} 
]

Aleš · Accepted Answer · 2018-12-06 17:55:41Z

2

As @seh pointed out in ksonnet channel, the latest jsonnet release now allows to use std.set() on objects.

 local arr = [
    { name: "a", value: 1234 },
    { name: "b", value: 555 },
    { name: "c", value: 0 },
    { name: "a", value: 1234 },
  ];
 std.set(arr, function(o) o.name)

The std.set() header is documented in jsonnet's std lib implementation.

edited Dec 6, 2018 at 17:55

answered Nov 28, 2018 at 21:44

Aleš

9,0489 gold badges65 silver badges108 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

jjo Over a year ago

beware that above (using std.set()) sets result to { name: "a", value: 4321 } rather than 1234.

jjo · Accepted Answer · 2018-11-28 20:08:00Z

1

Ignoring original sort order

You can implement it by replacing the comprehension with std.foldl(), do note though the ordering issue:

local arr = [
  { name: "a", value: 4321 },
  { name: "b", value: 555 },
  { name: "c", value: 0 },
  { name: "a", value: 1234 },
];

// Use foldl to iterate from array, can't use comprehension because of dup fields
local map = std.foldl(function(x, y) x { [y.name]: y }, arr, {});

// Re-convert map to array, note that it'll not respect original order
// but fields' (ie 'name' key)
[ map[x] for x in std.objectFields(map)]

Keeping original sort order

If you need to keep original sort order in output array, you can then add an _idx field to use in a final sort():

local arr = [
  { name: "a", value: 4321 },
  { name: "b", value: 555 },
  { name: "c", value: 0 },
  { name: "a", value: 1234 },
];

// Overload array elements with there index (`_idx` field)
local idxArray = std.mapWithIndex(function(i, x) x { _idx:: i }, arr);

// Use foldl to iterate from array, can't use comprehension because of dup fields
local map = std.foldl(function(x, y) x { [y.name]: y }, idxArray, {});

// Re-convert map to array, it'll keep original order via added _idx field
std.sort([map[x] for x in std.objectFields(map)], function(e) e._idx)

edited Nov 28, 2018 at 20:08

answered Nov 28, 2018 at 14:32

jjo

3,1101 gold badge11 silver badges18 bronze badges

1 Comment

Aleš Over a year ago

Thanks, I ended up using this std.foldl() approach in the end. The latest jsonnet release also allows to use std.set(arr, keyF=id) too.

Collectives™ on Stack Overflow

Remove duplicates from an array of objects in jsonnet

2 Answers 2

1 Comment

Ignoring original sort order

Keeping original sort order

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Ignoring original sort order

Keeping original sort order

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related