ASP.NET core REST service returns invalid json

Question

We have an ASP.NET core REST service (.NET5). It works fine. But if we call a GET method from two browsers simultaneously, the returned JSON becomes invalid.

The JSON looks scrambled:

{
   "assemblyId":null,                             ┌─ something invalid was inserted here
   "stepScrapQuantity":0,                         ▼
   "webViewer3dFileUrl":"http://172.20.33.188:8000me":"00:00:00",
   "currentProcessTime":"00:00:00"                
}                   ▲                             
                    └─ it looks like this string was inserted above

The returned DTO is automatically serialized by the built-in JSON-serializer (System.Text.Json) from ASP.NET.
On each GET, the JSON is messed up differently (random). Sometimes keys are messed up, sometimes values are messed up, sometimes it's all fine.
It looks like there's a multithreading issue in System.Text.Json
Our backend runs in a docker container.
It's not reproducible in debug-mode (localhost).
It's only reproducible with large JSON-Data (in our case 1.5 MB)
It only fails on a real notwork interface (on localhost it's all fine). So maybe it's a network issue.

Did anyone notice a similar issue?

EDIT (2022/10/14) The longer we investigate the problem, the stranger it becomes. The logging of the JSON message in our backend-middleware shows a valid JSON. The received JSON on frontend side is invalid. That means the bug is not in our backend. The payload is broken somewhere between backend and frontend. That's only reproducible when the backend runs as docker container and the service is called remotely (not from localhost).

Looking at this layer model, I wonder if it's possible that docker engine can break the JSON:

source: https://docs.mirantis.com/containers/v3.0/dockeree-ref-arch/networking/scalable-container-networks.html

Do you produce the json or you are just serializing an object? — SpyrosLina
– SpyrosLina, Commented Oct 10, 2022 at 15:05
Needs more info, at the least where the json is coming from. — BurnsBA
– BurnsBA, Commented Oct 10, 2022 at 15:23
What type are you returning? Does it have a custom JsonConverter<T>? Can you share a minimal reproducible example? — dbc
– dbc, Commented Oct 10, 2022 at 18:00
Oh this is interesting: looks like Utf8JsonReader doesn't properly enforce that well-formed JSON is being written. If, in a JsonConverter<T>, the WriteJson() method writes two property names in a row, the writer will happily emit such malformed JSON. See dotnetfiddle.net/Lv7gSS which roughly reproduces the JSON you see. Is there any chance that is what is happening here? — dbc
– dbc, Commented Oct 10, 2022 at 18:29

Roland Kwee · Accepted Answer · 2022-10-11 08:49:16Z

1

Your REST service code is not thread-safe. You have global variables that are used by every GET call, so data gets mixed up.

answered Oct 11, 2022 at 8:49

Roland Kwee

5,3027 gold badges59 silver badges91 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Templar_VII Over a year ago

Yes that's exactly how it feels. It reacts like there's a static varible in the serialization code. But in a ASP.NET core REST API, all the serialization is hidden in Microsoft Libraries. There's no code from our side.

Roland Kwee Over a year ago

@Templar_VII please show us your code, perhaps we find the problem/solution

Templar_VII Over a year ago

I'm working on a sample application to reproduce the bug. I can't share my entire project. It's huge and it has to many dependencies. I'll let you know when I have the code ready.

Roland Kwee Over a year ago

@Templar_VII Just select the relevant part of the code. Yes, that may be hard. In your case, usually the method that serves the GET request is just one small method.

Templar_VII · Accepted Answer · 2022-10-20 07:20:27Z

1

This error seems to be beyond our control.

We recorded the traffic with Wireshark. That sowed us that all malformed strings in the JSON are starting points of a TCP package. So it seems the TCP packages are reassembled incorrectly.

I don't know which component is responsible for splitting and assembling the TCP packages, but it is certainly outside our software.

What supports this theory are lots of packages that are "out-of-order":

Solution: Disabling TCP Segmentation Offload (TSO) finally solved the problem. https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-D80AEC2F-E0DA-4172-BFFD-B721BF36C2E8.html

edited Oct 20, 2022 at 7:20

answered Oct 14, 2022 at 14:50

Templar_VII

3012 silver badges11 bronze badges

Collectives™ on Stack Overflow

ASP.NET core REST service returns invalid json

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related