How to parse an infinite json array from stdin in go?

Question

I'm trying to write a small replacement for i3status, a small programm that comunicates with i3bar conforming this protocol. They exchange messeages via stdin and stdout.

The stream in both directions is an infinite array of json objects. The start of the stream from i3bar to i3status (which i want to replace) looks like this:

[
{"name": "some_name_1","instance": "some_inst_1","button": 1,"x": 213,"y": 35}
,{"name": "some_name_1","instance": "some_inst_2","button": 2,"x": 687,"y": 354}
,{"name": "some_name_2","instance": "some_inst","button": 1,"x": 786,"y": 637}
,{"name": "some_name_3","instance": "some_inst","button": 3,"x": 768,"y": 67}
...

This is an "array" of objects which represent clicks. The array will never close.

My question is now: What is the right way of parsing this?

Obviously I cannot use the json library because this is not a vaild json object.

There's bound to be a better way, but the easiest is probably just remove the , from the beginning, and then use the json library to parse the rest of the line: that is just a json object, right? (and the better way was probably posted while leaving this comment, so use that instead ;) that's why this is not an answer :) ) — Nanne
– Nanne, Commented Apr 2, 2015 at 20:05
Exactly. The rest is just an json object. I can't do such hacky stuff. I would hate myself ;) — Tristan Storch
– Tristan Storch, Commented Apr 2, 2015 at 20:39
Strictly speaking, this is not valid JSON. A valid JSON array must have a terminating right square bracket (]), meaning it cannot be infinite. I would not expect a JSON parsing library to be able to handle it via the normal means, though you might be able to invoke some of its parsing functionality if it exposes it. — jpmc26
– jpmc26, Commented May 8, 2019 at 21:51

maerics · Accepted Answer · 2015-04-03 01:42:31Z

6

Write a custom reader function (or Decoder) which does a "streaming array parse" like so:

Read and discard leading whitespace.
If the next character is not a [ then return an error (can't be an array).
While true do:
1. Call json.Decoder.Decode into the "next" item.
2. Yield or process the "next" item.
3. Read and discard whitespace.
4. If the next character is:
  1. A comma , then continue the for-loop in #3.
  2. A close bracket ] then exit the for-loop in #3.
  3. Otherwise return an error (invalid array syntax).

answered Apr 3, 2015 at 1:42

maerics

157k47 gold badges277 silver badges299 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

sberry Over a year ago

A basic state machine is the way to go.

Tristan Storch Over a year ago

How do I get the "next" item? Should I just use a scanner and split with the split function on newlines?

maerics Over a year ago

@TristanStorch: just call json.Decoder.Decode(...) - from the docs, it "reads the next JSON-encoded value from its input" and should properly handle any irrelevant whitespace.

Tristan Storch Over a year ago

For the record: I solved it with this approach. There is a tricky bit with json.Decoder.

James Welchman · Accepted Answer · 2019-05-08 10:47:00Z

I'm writing my own handler for click events in i3 as well. That's how I stumbled upon this thread.

The Golang standard library does actually do exactly what is required (Golang 1.12). Not sure it did when you asked the question or not?

// json parser read from stdin
decoder := json.NewDecoder(os.Stdin)

// Get he first token - it should be a '['
tk, err := decoder.Token()
if err != nil {
    fmt.Fprintln(os.Stderr, "couldn't read from stdin")
    return
}
// according to the docs a json.Delim is one [, ], { or }
// Make sure the token is a delim
delim, ok := tk.(json.Delim)
if !ok {
    fmt.Fprintln(os.Stderr, "first token not a delim")
    return
}
// Make sure the value of the delim is '['
if delim != json.Delim('[') {
    fmt.Fprintln(os.Stderr, "first token not a [")
    return
}

// The parser took the [ above
// therefore decoder.More() will return
// true until a closing ] is seen - i.e never 
for decoder.More() {
    // make a Click struct with the relevant json structure tags
    click := &Click{}

    // This will block until we have a complete JSON object
    // on stdin
    err = decoder.Decode(click)
    if err != nil {
            fmt.Fprintln(os.Stderr, "couldn't decode click")
            return
    }
    // Do something with click event
}

Y123 · Accepted Answer · 2015-04-02 20:05:30Z

0

What you are looking for is a Streaming API for JSON. There are many available a quick Google search revealed this project that does list Streaming as one of it's advance features.

answered Apr 2, 2015 at 20:05

Y123

9971 gold badge13 silver badges31 bronze badges

2 Comments

Tristan Storch Over a year ago

Thanks for the quick answer. I googled now myself for Go JSON Streaming APIs, but sadly the main part is just about sending (as your link). The receiving is all about newline delimited lists ({...}\n{...}\n{...}...) and no infinite array. For this I would have just used Scanners and json unmarshaling, which seams not to be the idiomatic way in my case.

Y123 Over a year ago

I am sorry to hear that. Let's assume you have no means of controlling this infinite input as it will come in as an array and not as splits like you need. You can code a "pre-processor" to clean up before you ignoring the start and end box bracket is relatively trivial. Now to ignore the comma consider this create a counter with value 1, for every start { increment a counter and for every } decrement a counter - when the counter is 1 - ignore the incoming comma (and replace with a \n as needed by the API).

Collectives™ on Stack Overflow

How to parse an infinite json array from stdin in go?

3 Answers 3

4 Comments

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related