How to prevent gRPC server in python from logging multiline traceback

Question

I created a grpc server written in python. When ther is an error, I need to raise an exception in the server, this function in grpc/_servery.py fails with the raised exception. This is causing the program to print the traces which just uses print function. Is there a way to make this funciton use jsonlogger which is used for all other logging?

Because of the multiline traces, I am getting multiple lines of log for same error in datadog. How do I fix the multi line traces.

Sample code of the server

class UnaryService(pb2_grpc.UnaryServicer):

    def __init__(self, *args, **kwargs):
        pass

    def GetServerResponse(self, request, context):
        message = request.message
        raise Exception("error raised")

I have tried follwoing approaches:

Returning None: I tried setting status code, error details in the gRPC context and return None when when there is an exception instead of raising. This worked in the sense, it is now not logging the traces but I got an issue with datadog tracing. The request was considered as succeeded even when it fails. I also tried adding tags to the current span but it still was not marked as failed.
Patching print_exc(): Another solution I approaced was to patch the traceback.print_exc() function. This also works but I do not like this solution as if something changes in traceback or grpc library, it could be impacted.

og_print_exc = traceback.print_exc()

def patched_print_exc(*args, **kwargs):
    tb = traceback.format_exc()
    logger.error("error traceback: ", tb)

traceback.print_exc = patched_print_exc

Are there better ways to handle this scenario? Thank You

Péter Szilvási · Accepted Answer · 2024-07-01 07:06:11Z

0

I had a similar issue where the gRPC server in Python logged multiline tracebacks, causing multiple log lines for the same error in Datadog. Here is how I resolved it:

By upgrading grpcio from version 1.59.3 to 1.64.1, the issue was fixed.

In grpcio==1.59.3, the _call_behavior function in grpc._server.py always calls traceback.print_exc() when an exception occurs, which results in the traceback being printed to the terminal. However, in grpcio==1.64.1, traceback.print_exc() is only called when an unprintable Exception occurs.

Upgrading to the newer version should resolve the issue of multiline tracebacks being logged.

edited Jul 1, 2024 at 7:06

Péter Szilvási

2,5323 gold badges30 silver badges55 bronze badges

answered Jun 28, 2024 at 8:53

woongchi

1

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to prevent gRPC server in python from logging multiline traceback

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related