1

I need to run a query by spark, but I'm not getting it. I tried using the explode command, but the performance is very bad.

How can I browse the struct field by spark?

I would like to run totally on spark dataframe and I must run it in AWS Glue.

Thanks.

#standardSQL

SELECT
    visitNumber,
    visitId,
    fullVisitorId,
    hits.screenname,
    dim.value
FROM
    table,
    UNNEST(hits) AS hits,
    UNNEST(customDimensions) AS dim
LIMIT
    10;

I tried using the explode command, but the performance is very bad.

Improve this question
4
  • Welcome to Stackoverflow! Can you please read this page and edit your question to have better formatting (including code blocks for your code)? That makes it easier for reviewers and future readers to read your question :) Commented Jun 10, 2023 at 7:42
  • 1
    Does this help with your question? stackoverflow.com/a/52803329/1477418 Commented Jun 11, 2023 at 11:34
  • Thank you! I'm going to test it. By the way, how to put this answer as correct? Commented Jun 11, 2023 at 12:29
  • 1
    i think if it helps then you can upvote the other answer and mark this question as duplicate to the other question, or just add a modified solution as an answer to this question Commented Jun 12, 2023 at 17:41
Related questions
673
No matching client found for package name (Google Analytics) - multiple productFlavors & buildTypes
188
How to turn off INFO logging in Spark?
0
How to expand nested JSON into Spark dataframe on AWS glue
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.