How to stream data in 4MB chunks using python (through grpc)

Question

I hava a python grpc service that streams large amounts of data to client microservices.

service GeoService {
   rpc GetGeoCoordinates(GetRequest) returns (stream Chunk){}
}

message Chunk {
    bytes data_part = 1;
}

I can not send more that 4MB of data at once because tcp connection has limits. Here is my code (only relevant part):

def GetGeoCoordinates(self, request, context):
    ...
    ...

    dataBytes = geo_pb2.Chunk(data_part=bytes(json.dumps(coordinates["data"]), 'utf-8'))

    yield dataBytes

How can I send this data in 4MB chunks?

Also is it a good practice to json.dumps() large data then stream? Any help is appreciated.

@RomanPerekhrest I fixed it. Now it is being consumed. Now I need to stream it in 4MB chunks. I would really appreciate if you help me on this — ABDULLOKH MUKHAMMADJONOV
– ABDULLOKH MUKHAMMADJONOV, Commented Oct 12, 2023 at 11:56

ABDULLOKH MUKHAMMADJONOV · Accepted Answer · 2023-10-12 13:17:05Z

0

After digging through SO and looking at many answers I finally managed to achieve what I expected:

 def GetGeoCoordinates(self, request, context):
    CHUNK_SIZE = 1024 * 1024 * 4     # 4 mb
    ...
    ...
    
    # convert data to bytes
    dataBytes = bytes(json.dumps(coordinates["data"]), 'utf-8')

    # stream in 4MB chunks
    index = 0
    while index < len(dataBytes):
        yield geo_pb2.Chunk(
            data_part=dataBytes[index:index+CHUNK_SIZE]
        )
        index += CHUNK_SIZE

This answer helped me the most Chunking in gRPC with Python

edited Oct 12, 2023 at 13:17

answered Oct 12, 2023 at 12:56

ABDULLOKH MUKHAMMADJONOV

5,3946 gold badges28 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to stream data in 4MB chunks using python (through grpc)

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related