How to interpret multi-class deep learning classifier by using SHAP?

Question

I trained a multi-class classifier in Keras on IRIS data set. I want to interpret my deep learning model by using SHAP. I use the following lines of code where model is my trained neural network classifier, X1[train] is my training input, and X1[test] is my test input:

import shap
explainer = shap.DeepExplainer(model, X1[train])
shap_values = explainer.shap_values(X1[test])

However, I get this error (I am fine when my softmax has one output in binary classification. The problem appears when softmax has more than 1 output):

ValueError: Unexpectedly found an instance of type `<class 'numpy.ndarray'>`. Expected a symbolic tensor instance.

During handling of the above exception, another exception occurred:
Layer sequential_96 was called with an input that isn't a symbolic tensor. Received type: <class 'numpy.ndarray'>. Full input:

How can I solve this problem and get SHAP values for a single class in multi class deep learning classifier?

ronnyb · Accepted Answer · 2020-04-02 08:48:13Z

1

How are you importing keras? When doing from keras import... I had the same issue, but changing to from tensorflow.keras import ... the DeepExplainer worked

answered Apr 2, 2020 at 8:48

ronnyb

781 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to interpret multi-class deep learning classifier by using SHAP?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related