How to use Shap with a LSTM neural network?

Question

I am working with keras to generate LSTM neural net model. I want to find Shapley values for each of the model's features using the shap package. The problem, of course, is that the model's LSTM layer requires a three dimensional input (samples, time steps, features), but the shap package requires a two dimensional input. Is there anyway around this problem?

Below I've included some code that reproduces the issue.


import numpy as np
from random import uniform

N=100

#Initlaize input/output vectors
x1=[] 
x2=[] 
x3=[]
y1=[]
y2=[]

#Generate some data
for i in range(N):
    x1.append(i/100+uniform(-.1,.1))
    x2.append(i/100+uniform(-3,5)+2)
    x3.append(uniform(0,1)/np.sqrt(i+1))
    
    y1.append(2*x1[i]-.5*x2[i]+x3[i]+uniform(-1,1))
    y2.append(x1[i]+3*x3[i]+5+uniform(-1,3))

#Convert lists to numpy arrays
x1=np.array(x1).reshape(N,1)
x2=np.array(x2).reshape(N,1)
x3=np.array(x3).reshape(N,1)

y1=np.array(y1).reshape(N,1)

#Assemble into matrices
X = np.hstack((x1, x2, x3))
Y = y1

# reshape input to be [samples, time steps, features]
X = np.reshape(X, (X.shape[0], 1, X.shape[1]))

#Import keras functions
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import LSTM


#Lets build us a neural net!
model=Sequential()
model.add(LSTM(4, input_shape=(1,3)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam',run_eagerly=())
model.fit(X, Y, epochs=100, batch_size=10, verbose=2)


import shap
import tensorflow as tf
tf.compat.v1.disable_eager_execution()

DE=shap.KernelExplainer(model.predict,shap.sample(X,10))
shap_values = DE.shap_values(X) # X is 3d numpy.ndarray

I have tried reshaping X into a two dimensional array in the shap_values function, but that does not work. Similarly, trying to feed a two dimensional array into the LSTM layer causes an error as well.

user2586955 · Accepted Answer · 2023-07-11 11:19:16Z

0

You are using the wrong model from shap package, (KernelExplainer) is used with base machine learning models but not deep models. You need to use DeepExplainer with LSTM and any network from Keras.

answered Jul 11, 2023 at 11:19

user2586955

3992 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to use Shap with a LSTM neural network?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related