Return HTTP response and continue processing - python bottle API

Question

I have an analysis service using Python's bottle that is invoked by posting a url to the raw data.

What is the simplest/cleanest/best way to immediately return a 200 response to the callee and continue processing a potentially long-running logic in the application?

Spawn a new thread/process? Use a queue? Async?

from bottle import post, HTTPResponse, request

@post('/postprocess')
def records():
   data_url = request.json.get('data_url', False)
   postback_url = request.json.get('postback_url', False)

   if not data_url or not postback_url:
       return HTTPResponse(status=400, body="Missing data paramater")

   #Immediately return response to callee
   return HTTPResponse(status=200, body="Complete")

   #Continue processing long-running code
   result = get_and_process_data(data_url)
   #POST result to another endpoint

Take a look at Celery or python-rq to run long background tasks. — iurisilvio
– iurisilvio, Commented Mar 18, 2014 at 14:32
I think that you need to use Threads: Read this: [Creating Threads in python][1] [1]: stackoverflow.com/questions/2905965/creating-threads-in-python — Rui Martins
– Rui Martins, Commented Mar 24, 2014 at 15:53

Heikki Toivonen · Accepted Answer · 2014-03-17 17:59:36Z

3

There is no simplest solution to this. For a production system I would first look into an existing system created for these kinds of situations to determine if that would be a good fit, and if not, only then develop something more suitable to my situation. To that end, I would recommend you take a look at Celery

answered Mar 17, 2014 at 17:59

Heikki Toivonen

31.2k11 gold badges45 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Alyssa Over a year ago

Thanks. I ended up using threads but will look at Celery or python-rq for a more robust future solution.

Ragy Abraham · Accepted Answer · 2020-10-27 03:16:08Z

2

I've recently found a very straightforward solution to this.

from threading import Thread    
@app.route('/')  
def send_response():
    uri = request.args
    ua = request.headers['user-agent']
    token = request.args.get("token")
    my_thread = Thread(target=process_response, args=[uri,
                                                      ua,
                                                      token])
    my_thread.start()
    res = Response(status=200,
                   mimetype='application/json')
    return res
        
def process_response(uri, ua, token):
    print(uri)
    print(ua)
    print(token)
    pass

answered Oct 27, 2020 at 3:16

Ragy Abraham

863 bronze badges

Comments

Simon H · Accepted Answer · 2014-03-22 11:38:58Z

1

I would suggest using some Queuing mechanism to queue the data that needs processing. Then you can implement a pool of workers that can work on processing the Q.

You can then take advantage of Q monitoring tools to iron out any performance issues and your worker pool can scale up as needed.

answered Mar 22, 2014 at 11:38

Simon H

1,74512 silver badges14 bronze badges

Collectives™ on Stack Overflow

Return HTTP response and continue processing - python bottle API

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related