How to pass parameters from PythonOperator to HttpSensor via XComs in Airflow?

Question

I want to send an HTTP request whose parameters depend on the result of a dependent Python callable. I am trying to use XComs for this purpose. Simplified example:

def get_index():
  return 0

get_index = PythonOperator(
  task_id='get_index',
  python_callable=get_index,
  dag=dag)

http_request = HttpSensor(
  task_id='send_http_request',
  http_conn_id=HTTP_HOST,
  endpoint=ENDPOINT,
  params={
    "index": "{{ ti.xcom_pull('get_index')  }}"
  },
  dag=dag)

get_index >> http_request

Unfortunately, after examining the options of the HTTP request I see the macro is not evaluated properly and instead of 0, {{ ti.xcom_pull('get_index') }} is sent. What might have gone wrong? Should I use the HttpOperator instead of HttpSensor?

Momchil Peychev · Accepted Answer · 2017-08-25 12:03:39Z

1

As faeder mentioned, jinja templates in params are currently not evaluated. I resolved the issue by switching to SimpleHttpOperatrs and putting the template in the data field.

answered Aug 25, 2017 at 12:03

Momchil Peychev

613 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

faeder · Accepted Answer · 2017-08-25 07:12:37Z

0

I think that params are deprecated and not scanned by the jinja templating engine in airflow. Try to use request_params instead of params (it's a a dictionary of string key/value pairs).

answered Aug 25, 2017 at 7:12

faeder

713 bronze badges

3 Comments

Sir Rufo Over a year ago

This does not provide an answer to the question. Once you have sufficient reputation you will be able to comment on any post; instead, provide answers that don't require clarification from the asker. - From Review

faeder Over a year ago

@SirRufo Actually it does answer the question. In the Sensor described, it uses the jinja expression "{{ ti.xcom_pull('get_index') }}" which is never read by airflow as a jinja template and therefore not repalaced by the values from xcom. If however, params is renamed as request_params, airflow interpret it and replaced "{{ ti.xcom_pull('get_index') }}" by the value from XCom. Tested on my machine.

Sir Rufo Over a year ago

An answer with I think and try is more a suggestion that would fit perfect as a comment but not as a qualified answer.

Collectives™ on Stack Overflow

How to pass parameters from PythonOperator to HttpSensor via XComs in Airflow?

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related