0

I have a airflow DAG which calls databricks job that has a task level parameters defined as job_run_id (job.run_id) and has a type as python_script. When I try to access it using sys.argv and spark_python_task, it only prints the json that has passed through the airflow job. I want that sys should be able to get both the parameters passed by DAG and databricks job.

We have a use case where we don't want to use anything related to dbutils. Its a python script so we want it to be independent of dbutils.

If i pass like below without passing any parameters from dag

{ "spark_python_task": [] } it does print job id in sys.argv

but when i pass like below with json format parameter from airflow dag

{ "spark_python_task": [{any test json}] }

it only prints json in sys.argv and no job id

I want the python script to print both task level parameters and parameters passed by airflow dag.

Improve this question
1
  • you should use ` to format inline code, and ``` to format block of code. Commented Aug 23 at 23:37
Related questions
0
How to pass named parameters from airflow dag to a databricks spark python task?
424
Raise warning in Python without interrupting program
2
Execute a Databricks Notebook with PySpark code using Apache Airflow
Load 6 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.