Python - share sockets between processes

Question

Is it possible to share socket objects between 2 running processes?

To simplify the issue, assume that we have two files s1.py and s2.py. Both are tcp servers listening on different ports.

s1.py body

from twisted.internet import reactor, protocol

CLIENTS = []

class Echo(protocol.Protocol):
    def connectionMade(self):
        CLIENTS.append(self)

    def dataReceived(self, data):
        self.transport.write(data)

    def connectionLost(self):
        CLIENTS.remove(self)

def main():
    factory = protocol.ServerFactory()
    factory.protocol = Echo
    reactor.listenTCP(8000, factory)
    reactor.run()

main()

and s2.py

from twisted.internet import reactor, protocol

class Echo(protocol.Protocol):

    def dataReceived(self, data):
        for socket in list_of_serialized_redis_CLIENTS_socket_object:
           socket.transport.write(data)

def main():
    factory = protocol.ServerFactory()
    factory.protocol = Echo
    reactor.listenTCP(8001, factory)
    reactor.run()

main()

Is it possible to share CLIENTS from s1.py with s2.py? Maybe there is some way to serialize CLIENTS and store it in redis or so.

What do you hope to accomplish through this sharing? Your two example programs are problematic because they perform no cross-process synchronization so any data that you try to send in one may end up interleaved with data from the other. Which probably is the same as data being corrupted and unusable at the receiver. — Jean-Paul Calderone
– Jean-Paul Calderone, Commented Jan 16, 2018 at 14:43

notorious.no · Accepted Answer · 2018-01-19 22:05:08Z

1

Using socket.share() might not be as straight forward as you might think. First off, it's a Windows exclusive feature and Windows is the one platform that gets the least attention by Twisted. Getting access to the low-level socket object in Twisted can be tricky because there are a lot of things happening behind the scenes and might cause strange outcomes if changed. So in other words, it's certainly possible to go this route, but it's not recommended.

If you're open to new solutions, I think an RPC-style server will solve your issue. My thinking is, since the connection objects live in s1, simply create a remote process that will "do stuff" to those connections. Other processes can connect to that server and execute functions on objects local to s1. Twisted's Perspective Broker is a ready-made solution you could leverage.

s1.py

from twisted.internet import reactor, protocol, endpoints
from twisted.spread import pb

class Echo(protocol.Protocol):
    def connectionMade(self):
        self.factory.client_set.add(self)

    def connectionLost(self, reason):
        self.factory.client_set.remove(self)

class Remote(pb.Root):
    def __init__(self, client_set):
        self.client_set = client_set

    def remote_s1_write(self, data):
        msg_tmpl = 'From s1...{0}'
        for client in self.client_set:
            response = msg_tmpl.format(data).encode('utf-8')
            client.transport.write(response)

def main():
    client_set = set()      # container will be shared between the 2 servers

    # setup Echo server
    factory = protocol.ServerFactory()
    factory.protocol = Echo
    factory.client_set = client_set
    echo_server = endpoints.TCP4ServerEndpoint(reactor, interface='localhost', port=8000)
    echo_server.listen(factory)

    # setup PB server
    pb_root = Remote(client_set)
    pb_factory = pb.PBServerFactory(pb_root)
    pb_server = endpoints.TCP4ServerEndpoint(reactor, interface='localhost', port=7999)
    pb_server.listen(pb_factory)

    reactor.run()

main()

s2.py

from twisted.internet import reactor, protocol, endpoints
from twisted.spread import pb

class Echo(protocol.Protocol):
    def dataReceived(self, data):
        remote = self.factory.pb_factory.getRootObject()
        remote.addCallback(lambda obj: obj.callRemote('s1_write', data))
        remote.addCallback(lambda echo: 's1 said: {0}'.format(data))
        remote.addErrback(lambda reason: 'error: {0}'.format(str(reason.value)))
        remote.addCallback(lambda response: print(response))
        self.transport.write(data)

def main():
    # connect to s1's PB server
    pb_factory = pb.PBClientFactory()
    pb_connection = endpoints.TCP4ClientEndpoint(reactor, host='localhost', port=7999)
    pb_connection.connect(pb_factory)

    # setup Echo server
    factory = protocol.ServerFactory()
    factory.protocol = Echo
    factory.pb_factory = pb_factory
    echo_server = endpoints.TCP4ServerEndpoint(reactor, interface='localhost', port=8001)
    echo_server.listen(factory)

    reactor.run()

main()

I've taken the liberty and updated the syntax. Try this code and see if it works and ask if you have any issues.

answered Jan 19, 2018 at 22:05

notorious.no

5,1573 gold badges22 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Jean-Paul Calderone Over a year ago

PB is terribly complicated. It's also very narrowly used, therefore has all of the extrinsic disadvantages of an obscure piece of technology. I'd think carefully before recommending it to anyone.

notorious.no Over a year ago

The idea was to use PB to demonstrate the design. In prod, PB could (should?) be replaced by another RPC-style implementation (REST, XML, AMQP, 0MQ, etc). I'll keep PB's obscurity in mind next time. Will it be deprecated or simplified any time soon? For what it's worth I do think it's a solid concept but it could benefit from a simplified redesign.

Jean-Paul Calderone Over a year ago

I think "remote object references" is too complicated a protocol to interact with from Python. At the very least, there's ten man-years of tools development missing to make it feasible. So I would deprecate and then remove twisted.spread. Not sure if the Twisted project itself will take this step (and perhaps that's reasonable: existing software that uses PB and works shouldn't be ruined. And the project is somewhat lacking in a way to explicitly state "this is abandoned but we won't intentionally break it").

Collectives™ on Stack Overflow

Python - share sockets between processes

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related