Heatmap of Class Activation

In this notebook, I use F. Chollet's heatmap of class activation technique described in Deep Learning with Python's 5th chapter.

Setting Up the Environment

In [43]:
% matplotlib inline
import os
import numpy as np
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input, decode_predictions
import keras.backend as K
import matplotlib.pyplot as plt
import cv2
In [3]:
from keras.applications import VGG16

model = VGG16(weights='imagenet')

Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels.h5
553467904/553467096 [==============================] - 477s 1us/step

Predicting a Picture's Class

In [88]:
img_path = os.path.join('jo_feline','jo','jo2.jpg')

img = image.load_img(img_path, target_size = (224,224))

x = image.img_to_array(img)
y = x.copy()

x = np.expand_dims(x, axis=0)

x = preprocess_input(x)

plt.imshow(y * 255)
Out[88]:
<matplotlib.image.AxesImage at 0x7febed732eb8>
In [89]:
preds = model.predict(x)

print('Predicted:', decode_predictions(preds, top=3)[0])

Predicted: [('n02086646', 'Blenheim_spaniel', 0.46582136), ('n02085782', 'Japanese_spaniel', 0.14177406), ('n02086240', 'Shih-Tzu', 0.071626857)]
In [90]:
np.argmax(preds)
Out[90]:
156

Setting Up the Grad-CAM Algorithm

In [91]:
blenheim_spaniel_output = model.output[:,156]

last_conv_layer = model.get_layer('block5_conv3')

grads = K.gradients(blenheim_spaniel_output, last_conv_layer.output)[0]

pooled_grads = K.mean(grads, axis = (0,1,2))

iterate = K.function([model.input], [pooled_grads, last_conv_layer.output[0]])

pooled_grads_value, conv_layer_output_value = iterate([x])

for i in range(512):
    conv_layer_output_value[:,:,i] *= pooled_grads_value[i]
In [92]:
heatmap = np.mean(conv_layer_output_value, axis = -1)

heatmap = np.maximum(heatmap, 0)
heatmap /= np.max(heatmap)
plt.matshow(heatmap)
Out[92]:
<matplotlib.image.AxesImage at 0x7febec47d160>

Plotting with the Picture

In [93]:
img = cv2.imread(img_path)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

heatmap = cv2.resize(heatmap, (img.shape[1], img.shape[0]))

heatmap = np.uint8(255 * heatmap)

heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)

superimposed_img = heatmap * 0.4 + img

plt.imshow(superimposed_img)
plt.show()
plt.imshow(img)
Out[93]:
<matplotlib.image.AxesImage at 0x7febec0f09e8>