torchserve : batch_size is always 1 even config.properties specify other value

Question

I think my torchserve loaded config.properties correctly because the number of worker is 2 as I set. But the batch_size is 1 instead of 20.

Anyone has an idea what might go wrong ? Thanks !

I have checked and torchserve load config.properties correctly, alas it ignored the batch_size and max_batch_delay specified in config.properties.

Here is my config.properties for the reference

inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
log_file=/ml_server/logs/torchserve.log
default_workers_per_model=2
number_of_netty_threads=32
job_queue_size=1000
batch_size=20
max_batch_delay=10

Below is the log, worker with batchSize: 1

ml-server  | 2024-03-06T00:11:11,091 [INFO ] W-9001-model_1.0-stdout MODEL_LOG - model_name: _model, batchSize: 1
ml-server  | 2024-03-06T00:11:11,091 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - model_name: _model, batchSize: 1

Lu9999 · Accepted Answer · 2024-03-06 20:52:33Z

0

I figured it out. It is because my config.properties was wrong. This is correct.

models={\
  "prompt_injection_model": {\
    "1.0": {\
        "defaultVersion": true,\
        "marName": "prompt_injection_model.mar",\
        "minWorkers": 2,\
        "maxWorkers": 5,\
        "batchSize": 128,\
        "maxBatchDelay": 20,\
        "responseTimeout": 60\
    }\
  }\
}

answered Mar 6, 2024 at 20:52

Lu9999

11

Sign up to request clarification or add additional context in comments.

1 Comment

Community Over a year ago

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.

Collectives™ on Stack Overflow

torchserve : batch_size is always 1 even config.properties specify other value

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related