0

I am trying to figure out if there is a way to send a JSON object to a BigQuery table that has a column of type JSON. I know the current practice is to stringify the JSON and send it over which gets parsed and stored as JSON but Im trying to find a way to send an object.

So I know the following works:

type Message struct {
    stuff      int64  `json:"stuff"`
    stuff2     string `json:"stuff2"`
    stuff3     string `json:"stuff3"`
}

stuff3Map := map[string]string{
    "status":        "ok",
    "response_time": "5",
}

stuff3JSON, err := json.Marshal(stuff3Map)
if err != nil {
    log.Fatalf("Failed to marshal stuff: %v", err)
}

msg := Message{
    stuff: 34,
    stuff2: "test",
    stuff3: string(stuff3JSON)
}

However Id like to know if it would be possible to make something like the following work

type Message struct {
    stuff      int64  `json:"stuff"`
    stuff2     string `json:"stuff2"`
    stuff3     map[string]string `json:"stuff3"` // or []byte or string
}

stuff3Map := map[string]string{
    "status":        "ok",
    "response_time": "5",
}

stuff3JSON, err := json.Marshal(stuff3Map)
if err != nil {
    log.Fatalf("Failed to marshal stuff: %v", err)
}

msg := Message{
    stuff: 34,
    stuff2: "test",
    stuff3: stuff3JSON
}

I know that BigQuery does support it (as mentioned here https://docs.cloud.google.com/bigquery/docs/schemas#standard_sql_data_types) and also as mentioned here: https://docs.cloud.google.com/bigquery/docs/schemas#creating_a_JSON_schema_file I can do it by defining a JSON schema but the objects I want to send are not of predetermined structure.

From what I can get from the docs, its a client library issue more than a BigQuery issue but still.

Also this https://docs.cloud.google.com/bigquery/docs/nested-repeated#define_nested_and_repeated_columns exists but also needs predetermined fields.

Finally, the bigquery.JSONValue(mapData) type exists but is that what I need and is there no way without it ? Thank you in advance.

7
  • I do not quite understand what you're after: in your 1st exampe json.Marshal produces a []byte, and you type-convert it to string (which indeed copies the memory), and in your 2nd example you skip the type-conversion, but you are not sending "an object" nonetheless — you're sending the result of marshaling your object into a JSON format (which produces a []byte). Is this really what you're after? Commented Nov 20 at 9:29
  • @kostix Using Go might have been an error because it does not have a "JSON Object" type. But you can easily replace the example with a javsacript script that would send the object: { "status": "ok", "response_time": "5", } its all the same for my purpose. My question is how can I make it so I can send these "pure" json objects to BigQuery without stringifying them. Commented Nov 20 at 11:02
  • [1/3] Ah, I see (no JS doesn't have a "JSON object" type either, let's not digress). I'm afraid you might still be confused a bit; let me explain. Please note that JSON is not a data type; it's a specification of a data format in sense of describing something which is merely a sequence of bytes. IOW, it defines the syntax of making such a sequence of bytes, and the semantics of valid sequences (that is, "how to create" and "what does it mean"). The fact struct types in Go (and in all PLs with ALGOL heritage), and objects in JS look like JSON objects has no direct connection to JSON. Commented Nov 20 at 11:39
  • [2/3] D. Crockford merely invented JSON's syntax in a way values "marked up" using it look exactly like so-called "literals" refresenting the same values in JS look (almost) exactly in the same way. The semantics of JSON have the same properties. But that is all there is to it. Now let's move to BigQuery. Note that BQ is a network-connected service. This means whatever you "use" in a program which uses any PL necessarily sends bytes to some service which processes your request. Hence, even though BQ spec states it supports a data type it calls "JSON", this only means it processes bytes… Commented Nov 20 at 11:44
  • [3/3]…sent by your program, and gives them special interpretation. (Note that processing, say, integers is no different, just the rules of this is way simpler.) So, now you have in your program an in-memory repserentation of what you want to send to BQ as a JSON-typed thing, and these two are quite different, even in the case of JS. So you cannot escape completely serializing your objects to JSON. Some library may make this transparent but won't remove this completely. Commented Nov 20 at 11:48

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.