Retrieving nested data from array of JSON files

Question

I'm new to Clojure and trying to retrieve a list of keys from JSON. The initial data structure is an array of JSON files slurped from a directory, an example of one of these files is below (so its actually a list of the following):

[
    {
        "id": "d588596f-c8ce-41de-85f6-12321a2e1888",
        "lines": [
            {
                "description": "SKU-1079 x 1",
                "price": {
                    "GBP": 14.99
                }
            },
            {
                "description": "Delivery",
                "price": {
                    "GBP": 3.49
                }
            }
        ],
        "date": {
            "date": "2016-09-07T00:53:31.000Z"
        },
        "total": {
            "GBP": 18.48
        },
        "invoice-address": [
            "93",
            "Westhorpe Road",
            "Inverness",
            "IV1 3WU"
        ],
        "delivery-address": [
            "93",
            "Westhorpe Road",
            "Inverness",
            "IV1 3WU"
        ]
    },
    {
        "id": "f1f471b2-5bf7-404e-9345-dcccdfba5c8a",
        "lines": [
            {
                "description": "SKU-1003 x 1",
                "price": {
                    "GBP": 14.99
                }
            },
            {
                "description": "SKU-1015 x 1",
                "price": {
                    "GBP": 14.99
                }
            },
            {
                "description": "SKU-1086 x 1",
                "price": {
                    "GBP": 14.99
                }
            },
            {
                "description": "SKU-1029 x 1",
                "price": {
                    "GBP": 14.99
                }
            },
            {
                "description": "SKU-1074 x 1",
                "price": {
                    "GBP": 14.99
                }
            },
            {
                "description": "Delivery",
                "price": {
                    "GBP": 3.49
                }
            }
        ],
        "date": {
            "date": "2016-09-07T01:15:48.000Z"
        },
        "total": {
            "GBP": 78.44
        },
        "invoice-address": [
            "18",
            "Barbican",
            "East Central London",
            "EC17 4HP"
        ],
        "delivery-address": [
            "18",
            "Barbican",
            "East Central London",
            "EC17 4HP"
        ]
    }
]

I need to get a single flat list of the "Description" values except "Delivery", just "SKU-1074 x 1" for every order in each file. I then use this list in another function to link to product IDs. I had semi-achieved this previously but it appeared to not be iterating over everything on only did one order with the following code that I have been editing to try to achieve this:

(defn getOrders [year month day]
  (let [fs (filter #(.isFile %) (file-seq (clojure.java.io/file (str "data/orders/" year "/" month "/" day))))
        ordersData (map #(json/read-str (slurp %) :key-fn keyword) fs)
        getLines (fn [x] (map :lines x))
        getDescription (fn [x] (map #(get % :description)))]
        ;(for [x (map #(getLines %) ordersData)] (remove #{"Delivery"} (map #(get % :description) x)))
        (->>  (for [x ordersData] (for [y x] (for [z y] z)))
              (map (fn [x] (map (fn [y] y) x))) ;Tried many different maps/fors here
        )
  )
)

Tried many iterations of this code, mostly used nested maps, for, and anonymous functions but all I can ever seem to return is a list of nil values.

The following retrieves the lines for the first order from the ordersData map, but I can't understand how to go about iterating at this level. In an imperative language I'd probably use a nested loop but not completely sure.

(get (first (first (getOrders "2017" "09" "07"))) :lines)

Taylor Wood · Accepted Answer · 2018-04-17 17:20:27Z

2

Here's a fairly straightforward way to do it:

(def my-maps (json/read-str "json goes here"))

Define a function that will operate on each top-level map in your JSON:

(defn descriptions [m]
  (->> (get m "lines")              ;; get the lines from the map
       (map #(get % "description")) ;; get the description values
       (remove #{"Delivery"})))     ;; remove any "Delivery" values

This descriptions function could be more efficient, but I think this is a good explanatory example.

Then map that function over your JSON maps:

(map descriptions my-maps)
=> (("SKU-1079 x 1")
    ("SKU-1003 x 1" "SKU-1015 x 1" "SKU-1086 x 1" "SKU-1029 x 1" "SKU-1074 x 1"))

To get the completely flat list like you want, replace that map with mapcat:

(mapcat descriptions my-maps)
=> ("SKU-1079 x 1" "SKU-1003 x 1" "SKU-1015 x 1" "SKU-1086 x 1" "SKU-1029 x 1" "SKU-1074 x 1")

answered Apr 17, 2018 at 17:20

Taylor Wood

16.2k1 gold badge24 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Alex Over a year ago

The problem im having with this is using it with: ordersData (map #(json/read-str (slurp %) :key-fn keyword) fs as the map data, i've tried modifying your code slightly to do so but im just getting empty lists. I think there needs to be another level of iteration

Taylor Wood Over a year ago

@J.Greenfield I wrote this example using the JSON from your question, which seems to be just one of your files, so yeah I think you just need to add another level of iteration.

Alex Over a year ago

Thanks a lot, managed to get it working by flattening first to avoid the nested iteration, and changed the strings in get to the :keyword.

Collectives™ on Stack Overflow

Retrieving nested data from array of JSON files

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related