Reading Large JSON file into variable in C#.net

Question

I am trying to parse the JSON files and insert into the SQL DB.My parser worked perfectly fine as long as the files are small (less than 5 MB).

I am getting "Out of memory exception" when trying to read the large(> 5MB) files.

if (System.IO.Directory.Exists(jsonFilePath))
                {
                    string[] files = System.IO.Directory.GetFiles(jsonFilePath);
                    foreach (string s in files)
                    {
                        var jsonString = File.ReadAllText(s);
                        fileName = System.IO.Path.GetFileName(s);
                        ParseJSON(jsonString, fileName);

                    }
                }

I tried the JSONReader approach, but no luck on getting the entire JSON into string or variable.Please advise.

I tried the JSONReader approach, but no luck on getting the entire JSON into string or variable. If by "JSONReader" you mean JsonTextReader from Json.NET, can you share what you tried but did not work? — dbc
– dbc, Commented Jul 25, 2016 at 21:39
foreach (string s in files){ fileName = System.IO.Path.GetFileName(s); using (WebClient client = new WebClient()){using (StreamReader sr = new StreamReader(client.OpenRead(jsonFilePath + fileName))) {using (JsonReader reader = new JsonTextReader(sr)){ var jsonString= reader.Value.ToString();ParseJSON(jsonString, fileName) } } } — user1046415
– user1046415, Commented Jul 26, 2016 at 14:06
Can you share a sample of what your JSON looks like, and how you are parsing it? I.e., a minimal reproducible example example of your problem giving an idea of what you are doing inside ParseJSON would make it much more likely for us to give a helpful answer. — dbc
– dbc, Commented Aug 13, 2016 at 17:45

Community · Accepted Answer · 2017-05-23 11:44:25Z

3

+50

Use 64 bit, check RredCat's answer on a similar question:

Newtonsoft.Json - Out of memory exception while deserializing big object

NewtonSoft Jason Performance Tips

Read the article by David Cox about tokenizing:

"The basic approach is to use a JsonTextReader object, which is part of the Json.NET library. A JsonTextReader reads a JSON file one token at a time. It, therefore, avoids the overhead of reading the entire file into a string. As tokens are read from the file, objects are created and pushed onto and off of a stack. When the end of the file is reached, the top of the stack contains one object — the top of a very big tree of objects corresponding to the objects in the original JSON file"

Parsing Big Records with Json.NET

edited May 23, 2017 at 11:44

CommunityBot

11 silver badge

answered Aug 13, 2016 at 3:56

A-J-Bauer

2,9811 gold badge30 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Jeffrey Rennie · Accepted Answer · 2016-07-25 22:32:44Z

0

The json file is too large to fit in memory, in any form.

You must use a JSON reader that accepts a filename or stream as input. It's not clear from your question which JSON Reader you are using. From which library?

If your JSON reader builds the whole JSON tree, you will still run out of memory. As you read the JSON file, either cherry pick the data you are looking for, or write data structures to another on-disk format that can be easily queried, for example, an sqlite database.

edited Jul 25, 2016 at 22:32

answered Jul 25, 2016 at 21:32

Jeffrey Rennie

3,5133 gold badges20 silver badges21 bronze badges

3 Comments

user1046415 Over a year ago

I have to read the entire file and store into SQL DB by forming datasets.

Jeffrey Rennie Over a year ago

Then use a JSON reader that accepts a filename or stream as input.

user1046415 Over a year ago

JsonTextReader reader = new JsonTextReader(new StreamReader(jsonFilePath+fileName)); while (reader.Read()) { ParseJSON(reader.Value.ToString(), fileName); }--- not getting the value as string from the reader

Collectives™ on Stack Overflow

Reading Large JSON file into variable in C#.net

2 Answers 2

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related