asyncio with synchronous code

Question

I have a module which makes blocking network requests to some TCP server and receive responses. I must integrate it into asyncio application. My module looks like this:

import socket

# class providing transport facilities
# can be implemented in any manner
# for example it can manage asyncio connection
class CustomTransport:
    def __init__(self, host, port):
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.host = host
        self.port = port

    def write(self, data):
        left = len(data)
        while left:
            written = self.sock.send(data.encode('utf-8'))
            left = left - written
            data = data[written:]

    def read(self, sz):
        return self.sock.recv(sz)

    def open(self):
        self.sock.connect((self.host, self.port))

    def close(self):
        self.sock.shutdown(2)


# generated. shouldn't be modified
# however any transport can be passed
class HelloNetClient:
    def __init__(self, transport):
        self.transport = transport

    def say_hello_net(self):
        self.transport.write('hello')
        response = self.transport.read(5)
        return response


# can be modified
class HelloService:
    def __init__(self):
        # create transport for connection to echo TCP server
        self.transport = CustomTransport('127.0.0.1', 6789)
        self.hello_client = HelloNetClient(self.transport)

    def say_hello(self):
        print('Saying hello...')
        return self.hello_client.say_hello_net()

    def __enter__(self):
        self.transport.open()
        return self

    def __exit__(self,exc_type, exc_val, exc_tb):
        self.transport.close()

Usage:

def start_conversation():
    with HelloService() as hs:
        answer = hs.say_hello()
        print(answer.decode('utf-8'))


if __name__ == "__main__":
    start_conversation()

Now I see that only way to turn my module to be compatible with asyncio is to convert everything to coroutines and replace regular socket with asyncio-provided transport. But I don't want to touch generated code (HelloNetClient). Is it possible?

P.S. I want it to be used like this:

async def start_conversation():
    async with HelloService() as hs:
        answer = await hs.say_hello()
        print(answer.decode('utf-8'))


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(start_conversation())

Romuald Brunet · Accepted Answer · 2022-07-26 16:28:44Z

5

HelloService will probably need to use run_in_executor (which manages a thread pool) to run HelloNetClient methods in the background. For example:

async def say_hello(self):
    print('Saying hello...')
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(None, self.hello_client.say_hello_net)

This is not an idiomatic use of asyncio, and you're missing out on some of its features - for example, you won't be able to create thousands of clients that all work in parallel, and you won't get reliable cancellation (ability to cancel() any task you wish). Nonetheless, simple usage will work just fine.

Unfortunately the ability to provide custom transports is not of help here because the middle tier, HelloNetClient, expects synchronous behavior. Even if you were to write a custom transport that hooked into asyncio, methods like say_hello_net would still wait for as long as it takes for the response to arrive, so HelloService would have to schedule them in a separate thread. For this reason your best bet is to use the default transport and connect the code with asyncio with the code in the service as shown above.

edited Jul 26, 2022 at 16:28

Romuald Brunet

5,9194 gold badges40 silver badges34 bronze badges

answered Aug 20, 2018 at 6:32

user4815162342

159k22 gold badges350 silver badges418 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

JaDogg Over a year ago

This is not an idiomatic use of asyncio - most important part. 🤔 So in a real world scenario I would need to call DB & other APIs.. so is it better to actually use Async DB drivers + async http clients? or can we live with async http clients and thread pooled DB calls?

user4815162342 Over a year ago

@bhathiya-perera If you have async libraries, you should definitely use them. run_in_executor is just a helper for executing legacy sync code in an async context.

Collectives™ on Stack Overflow

asyncio with synchronous code

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related