Celery Error: numpy array is not JSON serializable

Question

I'm using flask server with celery and redis. The error occurred when .apply_async() is called. The numpy array is part of visualizing the output of a keral neural network model. I do know that there is a way to convert keras model to json. My main problem lies in the fact that I do not know when or how celery performs the conversion, and I do not have control over it.

Here is my code:

@celery.task(bind=True)
def celery_createDirectoryAndSaveNNOutput(self, pInput, ID, filename, layersToShow, model):
    layer_outputs = [layer.output for layer in model.layers[1:]]
    viz_model = Model(input=model.input, output=layer_outputs)
    features = viz_model.predict(pInput)

    layerOutputs = {}
    folderName = "static/"+ID+"_"+filename

    if not os.path.exists(folderName):
        os.makedirs(folderName)

    for layerIndex in layersToShow:
        images = getFeatureMapImages(features[int(layerIndex)])
        layerOutputs[layerIndex] = []
        for i in range(0, len(images)):
            path = folderName+"/layer"+str(int(layerIndex))+"_"+str(i)+".jpg"
            cv2.imwrite(path, images[i])
            layerOutputs[layerIndex].append(path)
        self.update_state(state='PROGRESS', meta={'current': 0, 'total': 10,"status":filename})

    return {'current': i, 'total': len(layersToShow),'status': "temp"}


@app.route("/nnvisualisation_uploadMultipleImages", methods=["POST"])
def nnvisualisation_uploadMultipleImages():
    uploaded_files = request.files.getlist("file[]")
    weight = request.form.get("weight")
    ID = request.form.get("ID")

    layersToShow = [5]
    modelName = "VGG16"

    preds = {}
    path = os.path.join(STATIC_PATH, uploaded_files[0].filename)
    uploaded_files[0].save(os.path.join(STATIC_PATH, uploaded_files[0].filename))
    pInput, result = preTrainedModel[modelName](path)
    #ERROR HERE:
    task = celery_createDirectoryAndSaveNNOutput.s( pInput=pInput, ID=ID, filename=uploaded_files[0].filename, layersToShow=layersToShow, model=getModel(modelName)).apply_async(serializer='json')
    ...


    return jsonify({}), 202, {'Location': url_for('taskstatus',task_id=task.id)}

I've tried all available serializer yaml:

EncodeError: cannot represent an object: keras.engine.training.Model object at 0x10fdf26d0>

pickle:

EncodeError: Can't pickle type 'module': attribute lookup builtin.module failed

msgpack:

EncodeError: can't serialize array([[[[-103.93900299, -107.77899933, -123.68000031],... , dtype=float32) (numpy array)

json:

EncodeError: array([[[[-103.93900299, -107.77899933, -123.68000031],... , dtype=float32) (numpy array) is not JSON serializable

Any comment or suggestion is greatly appreciated. Thank you.

json is a string format compatible with javascript. It encodes dictionaries, lists and strings. Other python classes have to have to 'serialize' themselves into one of those structures. numpy arrays don't automatically do that,though there are tools that can help. Do some searching about keras and json. — hpaulj
– hpaulj, Commented Feb 10, 2017 at 17:30
Thank you for your comment. I do know that there is a way to convert keras model to json. My main problem lies in the fact that I do not know when or how celery performs the conversion, and I do not have control over it. — matchifang
– matchifang, Commented Feb 10, 2017 at 20:57

Community · Accepted Answer · 2017-05-23 12:17:05Z

2

My main problem lies in the fact that I do not know when or how celery performs the conversion, and I do not have control over it.

There dose exist a way to control conversion. You can register customized json serializer which can dump numpy array.

Refer this doc Serializers

And there is a good example.

edited May 23, 2017 at 12:17

CommunityBot

11 silver badge

answered Feb 19, 2017 at 12:50

gzc

8,6998 gold badges44 silver badges63 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

hpaulj · Accepted Answer · 2017-02-10 17:50:18Z

-1

https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model

# save as JSON
json_string = model.to_json()

though that saves just the architecture not the weights etc.

In any case you need to explore the methods provided by keras.

answered Feb 10, 2017 at 17:50

hpaulj

233k14 gold badges260 silver badges392 bronze badges

2 Comments

matchifang Over a year ago

Thank you for your comment. The problem is that the conversion is done by celery, so I do not have control over how the model is changed to json.

matchifang Over a year ago

Could you please advise me where I should put that line of code at? I don't know where celery serializes it. Thank you.

Collectives™ on Stack Overflow

Celery Error: numpy array is not JSON serializable

2 Answers 2

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related