How to make a DAG fail when we are raising exception in our python code?

Question

I am new to Airflow DAGs. I have a python script to fetch API calls. I have added a raise Exception in my code, if all API calls failed i.e. status_code!=200, then it should raise exception. However, in airflow logs, it is shown as INFO - [base] An error occurred: All API calls failed for all IDs and DAG succeeds. How do I make the DAG failed if all API calls fail.

Below is the sample code I have:

        response = requests.get(url)
        data = response.json()
        if response.status_code!=200:
            print(f"Error occurred for ID - Response code {response.status_code} {response.reason}")
            break
        if response.status_code==200:
            df1 = pd.concat([df1, pd.json_normalize(data)])
        else:
            print(f'No data to fetch data for {reaction}:{id}')


if not df1.empty:  
    all_accounts_failed = False

if all_accounts_failed:
    raise Exception("All API calls failed for all IDs")
    raise
else:
    try:
      #rest of the code

Expecting the DAG to be failed however DAG succeeded with just showing it as INFO

Snippet from DAG

stackoverflow.com/questions/43111506/…

subram
– subram

2025-01-07 09:50:01 +00:00
Commented Jan 7 at 9:50 — subram
– subram, Commented Jan 7 at 9:50

bcincy · Accepted Answer · 2025-01-21 03:34:00Z

0

There are two paths to solving this:

You can update your code to raise AirflowException. Here is a link to Airflow Exceptions. You'll need to add to the imports from airflow.exceptions import AirflowException to use it. Raising an Airflow Exception will cause the DAG to fail.
You can convert your requests to using the HttpHook, which will throw an AirflowException if the response is not a 200 response. Here is a link to the hook.

answered Jan 21 at 3:34

bcincy

737 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to make a DAG fail when we are raising exception in our python code?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related