Fine-Tuning Pretrained Networks
Perhaps the most practical, “remember this snippet” worthy section of Chollet’s notebook on using pretrained networks is the section where he outlines how to fine-tune your pre-trained ConvNets for your use case.
Generally, he breaks this practice up into 5 simple steps:
1. Add a custom network on top of an already-trained base network
To do this, we’ll import one of the pre-trained network objects from keras, without the dense layer attache
from keras.applications import VGG16
import warnings
warnings.filterwarnings('ignore')Using TensorFlow backend.
base_model = VGG16(weights='imagenet',
include_top=False, ## <- most important step
input_shape=(150, 150, 3))WARNING:tensorflow:From C:\Users\Nick\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
And so we’ve got a big ol’ summary() printout for the VGG16 model
base_model.summary()_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) (None, 150, 150, 3) 0
_________________________________________________________________
block1_conv1 (Conv2D) (None, 150, 150, 64) 1792
_________________________________________________________________
block1_conv2 (Conv2D) (None, 150, 150, 64) 36928
_________________________________________________________________
block1_pool (MaxPooling2D) (None, 75, 75, 64) 0
_________________________________________________________________
block2_conv1 (Conv2D) (None, 75, 75, 128) 73856
_________________________________________________________________
block2_conv2 (Conv2D) (None, 75, 75, 128) 147584
_________________________________________________________________
block2_pool (MaxPooling2D) (None, 37, 37, 128) 0
_________________________________________________________________
block3_conv1 (Conv2D) (None, 37, 37, 256) 295168
_________________________________________________________________
block3_conv2 (Conv2D) (None, 37, 37, 256) 590080
_________________________________________________________________
block3_conv3 (Conv2D) (None, 37, 37, 256) 590080
_________________________________________________________________
block3_pool (MaxPooling2D) (None, 18, 18, 256) 0
_________________________________________________________________
block4_conv1 (Conv2D) (None, 18, 18, 512) 1180160
_________________________________________________________________
block4_conv2 (Conv2D) (None, 18, 18, 512) 2359808
_________________________________________________________________
block4_conv3 (Conv2D) (None, 18, 18, 512) 2359808
_________________________________________________________________
block4_pool (MaxPooling2D) (None, 9, 9, 512) 0
_________________________________________________________________
block5_conv1 (Conv2D) (None, 9, 9, 512) 2359808
_________________________________________________________________
block5_conv2 (Conv2D) (None, 9, 9, 512) 2359808
_________________________________________________________________
block5_conv3 (Conv2D) (None, 9, 9, 512) 2359808
_________________________________________________________________
block5_pool (MaxPooling2D) (None, 4, 4, 512) 0
=================================================================
Total params: 14,714,688
Trainable params: 14,714,688
Non-trainable params: 0
_________________________________________________________________
And we’re going to make a new model that inherits from this
from keras.models import Sequential
model = Sequential()
model.add(base_model)model.summary()_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
vgg16 (Model) (None, 4, 4, 512) 14714688
=================================================================
Total params: 14,714,688
Trainable params: 14,714,688
Non-trainable params: 0
_________________________________________________________________
You might have noticed looking at the VGG.summary() that this doesn’t do any sort of classification, so we’ll tack on the usual Flatten/Dense steps at the end. So we’ll add that
from keras.layers import Flatten, Densemodel.add(Flatten())
model.add(Dense(256, activation='relu'))
model.add(Dense(1, activation='sigmoid'))model.summary()_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
vgg16 (Model) (None, 4, 4, 512) 14714688
_________________________________________________________________
flatten_1 (Flatten) (None, 8192) 0
_________________________________________________________________
dense_1 (Dense) (None, 256) 2097408
_________________________________________________________________
dense_2 (Dense) (None, 1) 257
=================================================================
Total params: 16,812,353
Trainable params: 16,812,353
Non-trainable params: 0
_________________________________________________________________
2. Freeze the base network
16 million trainable params is entirely too many for a network that was already trained on a better computer than we’ll have access to– especially at the lower-level features. So we’re going to ignore updating those and focus on the classification element with a simple one liner
base_model.trainable = Falsemodel.summary()_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
vgg16 (Model) (None, 4, 4, 512) 14714688
_________________________________________________________________
flatten_1 (Flatten) (None, 8192) 0
_________________________________________________________________
dense_1 (Dense) (None, 256) 2097408
_________________________________________________________________
dense_2 (Dense) (None, 1) 257
=================================================================
Total params: 16,812,353
Trainable params: 2,097,665
Non-trainable params: 14,714,688
_________________________________________________________________
Much better
3. Train the part you added
This is the portion where you’ll do your typical model training code– presented as pseudocode here
train_generator = train_datagen.flow_from_directory()
test_generator = test_datagen.flow_from_directory()
model.compile(loss='binary_crossentropy',
optimizer=optimizers.RMSprop(lr=2e-5),
metrics=['acc'])
history = model.fit_generator(
train_generator,
steps_per_epoch=100,
epochs=30,
validation_data=validation_generator,
validation_steps=50
)4. Unfreeze some layers in the base network
Now that you’ve got a decent classification section of your network, it will likely do you some good to start tailoring the later-layers of your ConvNet to your application.
This is the clever chunk of code you probably opened this notebook up for
base_model.trainable = True
set_trainable = False
for layer in base_model.layers:
if layer.name == 'block5_conv1':
set_trainable = True
if set_trainable:
layer.trainable = True
else:
layer.trainable = FalseThe reason this works is because base_model.layers serves up each layer as they’re constructed, sequentially, within the network
print([layer.name for layer in base_model.layers])['input_1', 'block1_conv1', 'block1_conv2', 'block1_pool', 'block2_conv1', 'block2_conv2', 'block2_pool', 'block3_conv1', 'block3_conv2', 'block3_conv3', 'block3_pool', 'block4_conv1', 'block4_conv2', 'block4_conv3', 'block4_pool', 'block5_conv1', 'block5_conv2', 'block5_conv3', 'block5_pool']
Thus, we loop over each layer, setting layer.trainable = False until we happen upon block5_conv1, near the end, and set the rest to True
5. Jointly train both layers in the part you added
The result is a Network that has a about 4x as many trainable params than before, which should yield better accuracy
model.summary()_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
vgg16 (Model) (None, 4, 4, 512) 14714688
_________________________________________________________________
flatten_1 (Flatten) (None, 8192) 0
_________________________________________________________________
dense_1 (Dense) (None, 256) 2097408
_________________________________________________________________
dense_2 (Dense) (None, 1) 257
=================================================================
Total params: 16,812,353
Trainable params: 9,177,089
Non-trainable params: 7,635,264
_________________________________________________________________