Hello guys I have a assignment, which I have honestly no idea how to complete. I have a huuuge Json file which is 22 GB big, so I have to get the data from it but it's full with empty spaces and new lines and I have to trimm does before even opening the file. I tried getting the data and trim it but it says it is out of memory I even set the memory limit to -1 if anyone has any idea how is that supposed to happen please I am open to suggestions.
4
-
have you tired chunking your trimming/reading process?madalinivascu– madalinivascu2017-10-12 11:35:40 +00:00Commented Oct 12, 2017 at 11:35
-
Look for a JSON stream parser, allowing you to read the file bit by bit and not in one chunk.deceze– deceze ♦2017-10-12 11:38:19 +00:00Commented Oct 12, 2017 at 11:38
-
Post your code, errors, what you have tried. We will not write the code for you. That being said, @deceze beat me to it, you have to parse the file, not try to load it and process after.Nic3500– Nic35002017-10-12 11:38:59 +00:00Commented Oct 12, 2017 at 11:38
-
So I guess the way is to read the JSON file line by line any idea how can I get each line without reading the whole file ?Stiliyan– Stiliyan2017-10-12 19:29:43 +00:00Commented Oct 12, 2017 at 19:29
Add a comment
|
1 Answer
There is simple and cool parser for parsing large JSON files.
https://github.com/salsify/jsonstreamingparser
Also maybe you will find an answer on the following links
Parse large JSON file
Processing large JSON files in PHP
