Pass parameters to Airflow Experimental REST api when creating dag run

Question

Looks like Airflow has an experimental REST api that allow users to create dag runs with https POST request. This is awesome.

Is there a way to pass parameters via HTTP to the create dag run? Judging from the official docs, found here, it would seem the answer is "no" but I'm hoping I'm wrong.

harscoet · Accepted Answer · 2018-06-14 20:46:56Z

17

I had the same issue. "conf" value must be in string

curl -X POST \
    http://localhost:8080/api/experimental/dags/<DAG_ID>/dag_runs \
    -H 'Cache-Control: no-cache' \
    -H 'Content-Type: application/json' \
    -d '{"conf":"{\"key\":\"value\"}"}'

answered Jun 14, 2018 at 20:46

harscoet

3401 silver badge11 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Nabin Over a year ago

Saved my day :)

mildewey Over a year ago

I still had trouble understanding. For future readers, this does NOT mean that you just have a regular json dictionary as the payload. You must have a json dictionary with one key "conf" and one string value which must be valid json. IE json-in-json. In my own (python) code, that meant something like this: payload = dict(conf=json.dumps(dict(key="value")) requests.post(url, json=payload, headers=headers)

Peter Berg · Accepted Answer · 2020-04-04 03:27:24Z

10

Judging from the source code, it would appear as though parameters can be passed into the dag run.

If the body of the http request contains json, and that json contains a top level key conf the value of the conf key will be passed as configuration to trigger_dag. More on how this works can be found here.

Note the value of the conf key must be a string, e.g.

curl -X POST \
    http://localhost:8080/api/experimental/dags/<DAG_ID>/dag_runs \
    -H 'Cache-Control: no-cache' \
    -H 'Content-Type: application/json' \
    -d '{"conf":"{\"key\":\"value\"}"}'

edited Apr 4, 2020 at 3:27

answered May 1, 2018 at 18:20

Peter Berg

6,2348 gold badges41 silver badges51 bronze badges

Comments

Pankaj · Accepted Answer · 2021-10-08 06:33:14Z

3

This is no longer true with the stable REST API.

You can do something like -

curl --location --request POST 'localhost:8080/api/v1/dags/unpublished/dagRuns' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--data-raw '{
    "dag_run_id": "dag_run_1",
    "conf": {
        "key": "value"
    }
}'

I understand that the question is asked for experimental API but this question is the top search result for airflow REST API.

answered Oct 8, 2021 at 6:33

Pankaj

2,7282 gold badges21 silver badges33 bronze badges

Collectives™ on Stack Overflow

Pass parameters to Airflow Experimental REST api when creating dag run

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related