How to remove duplicate keys and merge other key-values of list using scala?

Question

I am getting following List[JSONObject] structure as a output of some snippet-

List(List({
"groupName": "group1",
"maxSeverity": -1,
"hostCount": 3,
"members": [
    "192.168.20.11",
    "192.168.20.52",
    "192.168.20.53"
]
}),
List(),
List({
"groupName": "group1",
"maxSeverity": -1,
"hostCount": 2,
"members": [
    "192.168.20.20",
    "192.168.20.52"
]
}))

I want to merge whole output to form a list which contains - 1) group name

2) severity - which will be minimum from all list elements

3) hostcout - addition of hostcount from all list elements

4) members - similar array without duplicate values from all list elements.

So output will be somewhat like this-

List({
"groupName": "group1",
"maxSeverity": -1,
"hostCount": 5,
"members": [
    "192.168.20.11",
    "192.168.20.52",
    "192.168.20.53",
    "192.168.20.20",
    "192.168.20.52"
]
})

How do I merge whole list to a single list to get above mentioned output???

Actually I got stuck at this point. I have no clue to do this. Will you help me please? — user3322141
– user3322141, Commented Oct 21, 2014 at 14:16
Use groupBy groupName and then use foldLeft for the different fields. — tuxdna
– tuxdna, Commented Oct 21, 2014 at 14:18

Nate · Accepted Answer · 2014-10-21 19:08:22Z

1

Consider using Jackson to parse these into a case class, and then work with the data that way.

object JsonMerge {
  import com.fasterxml.jackson.databind.ObjectMapper
  import com.fasterxml.jackson.module.scala.DefaultScalaModule
  import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper

  case class ServerGroup(groupName: String, hostCount: Int, maxSeverity: Int, members: Iterable[String])

  def collapseGroup(groups: List[ServerGroup]): ServerGroup = {
    val members = groups.flatMap(_.members).toSet
    ServerGroup(groups.head.groupName, members.size, groups.map(_.maxSeverity).min, members)
  }

  def main(args: Array[String]) {
    val objectMapper = new ObjectMapper with ScalaObjectMapper
    objectMapper.registerModule(DefaultScalaModule)

    val allGroups = objectMapper.readValue[List[ServerGroup]](rawData)
    val output = allGroups.groupBy(_.groupName).values.map(collapseGroup)
    objectMapper.writerWithDefaultPrettyPrinter().writeValue(System.out, output)
  }

  val rawData = """
[{
  "groupName": "group1",
  "hostCount": 3,
  "maxSeverity": -1,
  "members": [
    "192.168.20.11",
    "192.168.20.52",
    "192.168.20.53"
  ]
},{
  "groupName": "group1",
  "hostCount": 2,
  "maxSeverity": -1,
  "members": [
    "192.168.20.20",
    "192.168.20.52"
  ]
},{
  "groupName": "group2",
  "hostCount": 1,
  "maxSeverity": 2,
  "members": [
    "192.168.20.52"
  ]
}]"""

}

This has this output:

[ {
  "groupName" : "group2",
  "hostCount" : 1,
  "maxSeverity" : 2,
  "members" : [ "192.168.20.52" ]
}, {
  "groupName" : "group1",
  "hostCount" : 4,
  "maxSeverity" : -1,
  "members" : [ "192.168.20.11", "192.168.20.52", "192.168.20.53", "192.168.20.20" ]
} ]

edited Oct 21, 2014 at 19:08

answered Oct 21, 2014 at 19:02

Nate

2,20517 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user3322141 Over a year ago

Is it possible without using case classes??

Nate Over a year ago

Absolutely. Jackson is relatively amazing. The DefaultScalaModule has some nice integration with case classes which make these things much cleaner. I try to keep most of my models in a case class for just that reason (and immutability, the copy method, etc). Here is a similar method that doesn't use a case class: gist.github.com/nbauernfeind/ddd71c9ce38c3c2272a9

tuxdna Over a year ago

Beaware of case classes if the arity of members is more than 22. Only scala 2.11 supports that. Still Tuples and Function paramters can be upto 22.

tuxdna · Accepted Answer · 2014-10-21 15:12:29Z

0

This is what you may need:

Code

object JsonMerge {

  def main(args: Array[String]) {
    val l1 = List(List(
      JSON.parseFull(
        """
      {
"groupName": "group1",
"maxSeverity": -1,
"hostCount": 3,
"members": [
    "192.168.20.11",
    "192.168.20.52",
    "192.168.20.53"
]
}
      """).get.asInstanceOf[Map[String, Any]]),
      List(),
      List(
        JSON.parseFull("""
    {
"groupName": "group1",
"maxSeverity": -1,
"hostCount": 2,
"members": [
    "192.168.20.20",
    "192.168.20.52"
]
}
    """).get.asInstanceOf[Map[String, Any]]),
      List(
        JSON.parseFull("""
    {
"groupName": "group2",
"maxSeverity": 2,
"hostCount": 1,
"members": [
    "192.168.20.52"
]
}
    """).get.asInstanceOf[Map[String, Any]])).flatten

    //    println(l1)
    //    println(l1.groupBy(_("groupName")))

    val minimumSeverity = -1

    val mergedList = l1.groupBy(_("groupName")).map { grouped =>

      val folded = grouped._2.foldLeft(Map[String, Any]("maxSeverity" -> minimumSeverity)) { (current, next) =>
        val severity1 = current.get("maxSeverity").map(s => s match {
          case v: Int => v
          case _ => minimumSeverity
        }).getOrElse(minimumSeverity)

        val severity2 = current.get("maxSeverity").map(s => s match {
          case v: Int => v
          case _ => minimumSeverity
        }).getOrElse(minimumSeverity)

        val severity = Math.max(severity1, severity2)

        val m1 = current.get("members").map(n => n match {
          case l: List[_] => l
          case _ => List()
        }).getOrElse(List()).toSet

        val m2 = next.get("members").map(n => n match {
          case l: List[_] => l
          case _ => List()
        }).getOrElse(List()).toSet

        val members = m1 union m2

        Map("maxSeverity" -> severity,
          "hostCount" -> members.size,
          "members" -> members.toList)
      }

      folded + ("groupName" -> grouped._1)
    }

    mergedList foreach println

  }

}

Output

Map(maxSeverity -> -1, hostCount -> 1, members -> List(192.168.20.52), groupName -> group2)
Map(maxSeverity -> -1, hostCount -> 4, members -> List(192.168.20.11, 192.168.20.52, 192.168.20.53, 192.168.20.20), groupName -> group1)

answered Oct 21, 2014 at 15:12

tuxdna

8,5074 gold badges47 silver badges64 bronze badges

4 Comments

Nate Over a year ago

I don't understand why so many people are obsessed and fascinated with foldLeft. I would never want to maintain code that looks like this.

tuxdna Over a year ago

That's because its a list and also its natural to think like that. What else you got ?

user3322141 Over a year ago

@ tuxdna it gives me error at 'val mergedList = l1.groupBy(_("groupName")).map { grouped =>' as it expect int parameter instead of string in group by

tuxdna Over a year ago

This must not be the case. Can you share your code?

Collectives™ on Stack Overflow

How to remove duplicate keys and merge other key-values of list using scala?

2 Answers 2

3 Comments

Code

Output

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Code

Output

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related