How to get a Flux of a property on top-level JSON object from WebClient

Question

I need to call an API whose response is of the following form, and it promises to return the metadata before the results:

{
  "metadata": {...},
  "results": [...]
}

I'm using WebClient, and I want to end up with a Flux<Result>, so I can process them without blocking for everything to arrive. Is there a way to do that?

I think I'd end up with is something like a Mono<Tuple2<Metadata, Flux<Result>>> so I don't block waiting for the metadata or the results

Brian Clozel · Accepted Answer · 2018-04-06 06:52:50Z

1

There's no way to do that, for several reasons:

getting the metadata before the results is not guaranteed by the JSON spec
the API is probably responding with an "application/json" Content-Type, which is not streaming

This could work if the remote API would actually stream the response results (and only the results!) using "application/stream+json" or "text/event-stream". This way each result can be parsed separately.

Right now the API response you're getting needs to be parsed in one go. If the response is not that big and not streaming, parsing the whole thing in one pass is actually more efficient and you can always do something like:

Mono<ResponseObject> response = //...
Flux<Result> results = result.flatMap(r -> Flux.fromIterable(r.getResults());

answered Apr 6, 2018 at 6:52

Brian Clozel

59.7k15 gold badges180 silver badges184 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

doctau Over a year ago

It's not guaranteed by the JSON spec, but there is a promise to send that key first. It's fine if it was an Optional<Metadata> in case it didn't come first. It's certainly possible to write something to do what I need, using a similar method to the standard Jackson2JsonDecoder but processing part of the token stream myself first and using the normal functionality to turn each element of the results array into an object, but I'd prefer not to :)

Brian Clozel Over a year ago

Do you have a json parser in mind that supports non-blocking partial deserialization?

doctau Over a year ago

I think an heavily altered version of SpringWeb's jackson decoder could do it. Handle the top-level object myself, and then once the "results" key is ready, do the same as the standard version does sending a TokenBuffer per array element for the part that can be streamed (plus dealing with any following top-level object keys). There would be a bunch of work to deal with all the odd cases, but it should be possible. In any case, if it doesn't exist I'll try writing it when I have some time and see how it goes.

Collectives™ on Stack Overflow

How to get a Flux of a property on top-level JSON object from WebClient

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related