1

Anyone know how to call single notebook with different parameters parallel, and all notebooks should appear on spark UI to make the trouble shooting easier? I have one child notebook, calling from other master notebook with objects list dict. Child notebook has database connections, data frames, reading from database and reading different types of files based on objects list dict.

Currently using ThreadPool but have difficulty in troubleshooting, if any notebook fails its name and error doesn't appear in spark UI because its running on driver node.

any simple example would help?

Thanks.

Improve this question
2
  • Is it possible to bring the objects list to any orchestration layer (Synapse or ADF or Airflow). call the parameterized notebook activity from a pipeline with For-Each loop iterating the Objects list in batches of 5 or 10? Got access to livyApi? option is to submit jobs through liviapi RestAPI in parallel. Commented Nov 6 at 17:59
  • Thanks Rej, ADF is used to all parent notebook and then it calls childnotebook using ThreadPoolexecutor. Commented Nov 13 at 9:50
Related questions
4
Synapse Notebook reference - Call Synapse notebook from another with parameters
1
Run all cells above in a Spark notebook in Azure Synapse
0
unable to read csv file from synapse spark scala notebook
Load 4 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.