Json.net: Preventing OutOfMemoryErrors when handling large objects

Question

My question is basically the same as this one, but unfortunately the answer doesn't work for me.

I'm trying to handle some edge cases, one of which is the case where I receive an unusually large (e.g., 150 million character) json string from my server. I'm using the memory optimization technique described in the json.net docs, but I still get an OutOfMemoryError in the deserialization step.

What is the correct way to handle large json objects with json.net? I'd settle for setting a max length flag, but I'm not sure how to do that.

Here's my code:

    Dim serverResponseStream As Stream = 'GZipWrapperStream response from server

    Using sr As StreamReader = New StreamReader(serverResponseStream), _
        reader As JsonReader = New JsonTextReader(sr)
        Dim serializer As JsonSerializer = New JsonSerializer()

        'Out of memory exception here in Deserialize
        Dim response = serializer.Deserialize(reader) 
    End Using

Edit: as per this answer, I have also tried:

    Dim a = New JArray()
    Using sr As StreamReader = New StreamReader(serverResponseStream), _
            reader As JsonReader = New JsonTextReader(sr)
        While reader.Read()
            If reader.TokenType = JsonToken.StartObject Then

                'Out of memory exception here
                Dim j = JObject.Load(reader)

                a.Add(j)                    
            End If
        End While
    End Using

AFAIK you'll have to use JsonReader to manually "read and unwrap" the top-level objects such that you only Deserialize (and use) smaller parts of the object-graph at once - the use of JsonTextReader around a string is to allow the stream to be consumed directly, which can avoid an intermediate String but it will not lower the overall usage of Deserialize. (Alternatively, bump up the VM and cross yer fingers xD) — user2864740
– user2864740, Commented May 30, 2014 at 4:16
See drdobbs.com/windows/parsing-big-records-with-jsonnet/240165316 — user2864740
– user2864740, Commented May 30, 2014 at 4:19
And you may be able to find some links/information off of stackoverflow.com/questions/14990590/… — user2864740
– user2864740, Commented May 30, 2014 at 4:21
Thanks for the links. I should have included that article in the question--I read it but was hoping not to have to resort to that technique. — ForeverWintr
– ForeverWintr, Commented May 30, 2014 at 4:54
possible duplicate of Deserialize json array stream one item at a time — Brian Rogers
– Brian Rogers, Commented Jun 2, 2014 at 1:26

dekajoo · Accepted Answer · 2014-05-30 08:46:01Z

-1

This might work (depending on the size of json you're deserilizing) :

serializer.MaxJsonLength = Int32.MaxValue;

Edit : my bad this is if you're using JavaScriptSerializer()

answered May 30, 2014 at 8:46

dekajoo

2,1221 gold badge26 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ForeverWintr Over a year ago

Yeah, If there's an equivalent property in a JsonSerializer I'd like to know what it is. Unfortunately I haven't been able to find one.

Collectives™ on Stack Overflow

Json.net: Preventing OutOfMemoryErrors when handling large objects

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related