Iterate over a large JSON Array with JSONPath

Question

I have a simple Java app that needs to traverse a large JSON array (with around 20K items), and within each array, I parse a sub-array. Each item looks like this:

{"index":0,"f1":[2,16,16,16,16,16,32,16],"f2":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"startTime":0.0}

I am using JSONPath to iterate over each item. What I do is first I read the length, and simply iterate over the whole array. But it is very slow (like, 1 item per second).

int length = JsonPath.read(response, "$.result.length()");
for (int i = 0; i < length; i++) {
    double start_time = JsonPath.read(response, "$.result["+i+"].startTime");
    ArrayList<Integer> f1= JsonPath.read(response, "$.result["+i+"].f1");
    //...other things
}

Is there a way to optimise it?

Yes - first parse your JSON file into a data structure only once, and the you access the data structure. What you currently do is parsing the JSON every time you retrieve a value from it, which is, as you have noticed, very, very slow. See stackoverflow.com/questions/2591098/how-to-parse-json-in-java — Erwin Bolwidt
– Erwin Bolwidt, Commented Mar 26, 2019 at 21:36

Michał Ziober · Accepted Answer · 2019-03-26 23:17:02Z

3

You should minimise number of read operation. First time you scan the whole file and and next you scan n times file partially. Reading from disk is slower than from memory: Latency Numbers Every Programmer Should Know, so you should load the file to memory once and then iterate over items. Also, from JsonPath documentation:

If you only want to read once this is OK. In case you need to read an other path as well this is not the way to go since the document will be parsed every time you call JsonPath.read(...). To avoid the problem you can parse the json first.

String json = "...";
Object document = Configuration.defaultConfiguration().jsonProvider().parse(json);

List<Integer> f10 = JsonPath.read(document, "$.result[0].f1");
List<Integer> f11 = JsonPath.read(document, "$.result[1].f1");

You can improve your JsonPath: $.result and read only what you need by: $.result..['f1','startTime'].

Example app which loads only required fields:

import com.jayway.jsonpath.JsonPath;

import java.io.File;
import java.util.List;
import java.util.Map;

public class JsonPathApp {

    public static void main(String[] args) throws Exception {
        File jsonFile = new File("./resource/test.json").getAbsoluteFile();
        List<Object> array = JsonPath.read(jsonFile, "$.result..['f1','startTime']");
        for (Object item : array) {
            Map<String, Object> map = (Map<String, Object>) item;
            System.out.println(map.get("f1"));
            System.out.println(map.get("startTime"));
        }
    }
}

answered Mar 26, 2019 at 23:17

Michał Ziober

38.9k18 gold badges110 silver badges154 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

angel_30 Over a year ago

Thanks. Yes this is what I do now. Lighting speed now...unbelievable!

Michał Ziober Over a year ago

@TinaJ, I'm glad to hear that. We have more tools than only JsonPath, take a look on similar question with Jackson example: How do I read a data from a JSON file with high efficiency in Java with Jackson?. In case, my answer was helpful take a look on How does accepting an answer work?

Alferd Nobel Over a year ago

fantastic answer @MichałZiober !

angel_30 · Accepted Answer · 2019-03-26 23:10:00Z

1

Got it. Thanks to Erwin, I can parse the whole JSON at once into a HASHMap simply like this:

ArrayList<HashMap> json= JsonPath.read(response, "$.result");

And then we can simply call get(i) to access a specific item in the loop:

double start_time = (double) json.get(i).get("startTime");

answered Mar 26, 2019 at 23:10

angel_30

5,69716 gold badges77 silver badges145 bronze badges

Collectives™ on Stack Overflow

Iterate over a large JSON Array with JSONPath

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related