I have nested maps with a key -> Map(key1 -> Map(), key2 -> Map()) kinda representation, which basically represent the path structure of a particular HTTP request made.
root/twiki/bin/edit/Main/Double_bounce_sender
root/twiki/bin/rdiff/TWiki/NewUserTemplate
I have stored them in a Map of maps which would give me the hierarchy of the path. Using a parser I read the data off server logs and get the corresponding data needed and then index the data in a sorted map.
val mainList: RDD[List[String]] = requesturl flatMap ( r => r.toString split("\\?") map (x => parser(x.split("/").filter(x => !x.contains("=")).toList).valuesIterator.toList))
def parser(list: List[String]): Map[Int, String]= {
val m = list.zipWithIndex.map(_.swap).toMap
val sM = SortedMap(m.toSeq:_*)
sM.+(0 -> "root")
}
After getting the data in the structure required, I loop through the entire collection to structure the data into a path map which would look like
root - twiki - bin - edit - Main - Double_bounce_sender
-rdiff - TWiki - NewUserTemplate
- oops - etc - local - getInterface
type innerMap = mutable.HashMap[String, Any]
def getData(input: RDD[List[String]]): mutable.HashMap[String, innerMap] ={
var mainMap = new mutable.HashMap[String, innerMap]
for(x <- input){
val z: mutable.HashMap[String, innerMap] = storeData(x.toIterator, mainMap ,x(0).toString)
mainMap = mainMap ++ z
}
mainMap
}
def storeData(list: Iterator[String], map: mutable.HashMap[String, innerMap], root: String): mutable.HashMap[String, innerMap]={
list.hasNext match {
case true =>
val v = list.next()
val y = map contains (root) match {
case true =>
println("Adding when exists: "+v)
val childMap = map.get(v).get match {
case _:HashMap[String, Any] => asInstanceOf[mutable.HashMap[String, innerMap]]
case _ => new mutable.HashMap[String, innerMap]
}
val x = map + (v -> storeData(list, childMap, v))
x
case false =>
val x = map + (v -> storeData(list, new mutable.HashMap[String, innerMap], v))
x
}
y.asInstanceOf[mutable.HashMap[String, innerMap]]
case false =>
new mutable.HashMap[String, innerMap]
}
}
The get data method calls each input list and sends it to the storeData method which builds the map.
I'm stuck at two places.
- The MainMap(HashMap[String, innerMap]) sent recursively to storeData goes as a new empty map every time.
- The second issue is that I'm trying to figure out a way of merging 2 nested Maps that do not have a defined length. Such as merging the maps below.
Map(root -> Map(twiki -> Map(bin -> Map(edit -> Map(Main -> Map(Double -> Map())))))))
Map(root -> Map(twiki -> Map(bin -> Map(rdiff -> Map(TWiki -> Map(NewUser -> Map())))))))
Looking for suggestions on how I could implement this solution and get a final map that contains all the possible paths present in the server log files in one map.
