How to pipe curl into Go program?

Question

I want to write a little Go program I can use to beautify json data. It already works when I use a file. Here is the code:

package main

import (
    "bufio"
    "fmt"
    "github.com/Jeffail/gabs"
    "log"
    "os"
)

func main() {
    info, err := os.Stdin.Stat()
    if err != nil {
        log.Fatal(err)
    }

    if info.Mode()&os.ModeCharDevice != 0 || info.Size() <= 0 {
        fmt.Println("The command is intended to work with pipes.")
        fmt.Println("cat file.json | prettyjson")
        return
    }

    reader := bufio.NewReader(os.Stdin)


    input, err := reader.ReadBytes('\n')
    if err != nil {
        log.Fatal()
    }

    jsonParsed, err := gabs.ParseJSON(input)
    if err != nil {
        log.Fatal("couldn't parse json")
    }

    fmt.Println(fmt.Println(jsonParsed.StringIndent("", "  ")))
}

If I run this code with with curl like this:

curl -s "https://min-api.cryptocompare.com/data/top/exchanges?fsym=BTC&tsym=USD" | prettyjson

I get: (23) Failed writing body

I saw in this post that the pipe is being closed before curl can write all the data, but how do I optimize my Go program to wait until curl is done?

ioutil.ReadAll, json.Indent.

Peter
– Peter

2018-11-02 17:15:08 +00:00
Commented Nov 2, 2018 at 17:15 — Peter
– Peter, Commented Nov 2, 2018 at 17:15

user4466350 · Accepted Answer · 2018-11-02 19:06:42Z

Regarding OP source code i would consider to change the condition to detect the presence of a pipe.

As already provided in https://stackoverflow.com/a/43947435/4466350 the correct condition does not need to check for the length of the input. Thinking about it, this totally makes sense as you might open stdin without writing data on it.

Besides the proposed solution seems uselessly complex for what it tries to achieve, pretty printing of a json input.

I found out that using the standard library was sufficient to fulfill the goal for the given test case.

About the question ...but how do I optimize my Go program to wait until curl is done?, it seems that OP does not understand the way the file descriptors are working. In fact, the question is not even correct, as the process could theoretically remain alive but actively decided to close Stdin. The OP is not interested in process liveliness, instead, he should simply look for EOF signal while reading Stdin, indicating that the interesting data was sent correctly.

Anyways, a simple solution look likes this, wrap stdin with a json decoder, loop until eof or an error occur, for each decoded data, encode it to json with a wrapper of stdout, on error break again.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "os"
)

func main() {
    info, err := os.Stdin.Stat()
    if err != nil {
        log.Fatal(err)
    }

    if info.Mode()&os.ModeCharDevice != 0 {
        fmt.Println("The command is intended to work with pipes.")
        fmt.Println("cat file.json | prettyjson")
        return
    }

    dec := json.NewDecoder(os.Stdin)
    enc := json.NewEncoder(os.Stdout)
    enc.SetIndent("", "  ")

    for {
        data := map[string]interface{}{}
        if err := dec.Decode(&data); err != nil {
            if err == io.EOF {
                break
            }
            log.Fatalf("decode error %v", err)
        }
        if err := enc.Encode(data); err != nil {
            log.Fatalf("encod error %v", err)
        }
    }
}

Thank you! I'm still quite new to Go and dont have a lot of experience with the standard libraries, I did understand where my error was now and how simple the solution was, great Answer :)

Collectives™ on Stack Overflow

How to pipe curl into Go program?

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related