Module 4.4_ Transfer Learning.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Module 4_4.ipynb","provenance":[]},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"6owUrbliYPFg","colab_type":"text"},"source":["## Module 4.4: Transfer Learning\n","\n","In this module we will:\n","- Implement standard transfer learning\n","\n","Sadly because we want to use a Keras dataset we use very small images that are not really suitable for the advanced pre-trained models Keras has available. So we will not achieve good results!\n"]},{"cell_type":"markdown","metadata":{"id":"qlY14rAOYuPK","colab_type":"text"},"source":["Start by importing required libraries."]},{"cell_type":"code","metadata":{"id":"0CDLS2Q3YKEt","colab_type":"code","colab":{}},"source":["from keras.datasets import cifar10\n","from keras.applications.densenet import DenseNet121\n","from keras.layers import Dense, Activation, Flatten, Dropout\n","from keras.models import Model\n","from keras.optimizers import Adam\n","from keras.callbacks import EarlyStopping\n","from keras.utils import np_utils\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"86L2DrxXY7qO","colab_type":"text"},"source":["We load and pre-process the CIFAR10 data. We use this data as an example only - they are really too small to use with the advanced pre-defined Keras networks, and the results are poor."]},{"cell_type":"code","metadata":{"id":"pHCWrbAqY70a","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":51},"outputId":"69a59add-73bd-48d8-e022-ee2c9531cb96","executionInfo":{"status":"ok","timestamp":1572735398294,"user_tz":-60,"elapsed":17752,"user":{"displayName":"Michael Ashcroft","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAaRQqpOuFHR3D_ZulW6qlXPomIq5vZ-wR4ZuIm=s64","userId":"16725792548700883920"}}},"source":["(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n","\n","# Normalize pixel values to be between 0 and 1\n","x_train, x_test = x_train / 255.0, x_test / 255.0\n","\n","# Make versions of the labels that are one-hot vectors\n","y_train=np_utils.to_categorical(y_train, 10)\n","y_test=np_utils.to_categorical(y_test, 10)"],"execution_count":3,"outputs":[{"output_type":"stream","text":["Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n","170500096/170498071 [==============================] - 11s 0us/step\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"vqih3Bjz-d5j","colab_type":"text"},"source":["Although you can define your own advanced CNN architecture, it is common to use proven networks. With such networks, it is also possible to load pre-trained weights. These will have been obtained from a large, general image set - such as the imagenet database which currently contains more than 14 million images and 20000 categories. So we would expect that network trained on such data are good and picking out general features from images that could be useful in many tasks, even those containing categories not seen in the training data. We can then use these weights for the feature extraction component of the network, and concentrate on tuning the final classification layers of the network that estimate class probabilities based on extracted features.\n","\n","Keras provides various pre-defined networks (with pre-trained weights) that make use of the layers we have looked at, including:\n","\n","Inception Networks:\n","- InceptionV3\n","\n","Residual Networks:\n","- ResNet50\n","- ResNet101\n","- ResNet152\n","- ResNet50V2\n","- ResNet101V2\n","- ResNet152V2\n","\n","Residual Inception Networks:\n","- InceptionResNetV2\n","\n","Dense Networks:\n","- DenseNet121\n","- DenseNet169\n","- DenseNet201\n","\n","We will use DenseNet121 for this example, but you are free to replace it with one of the others.\n","\n","Let's load the DenseNet121 network architecture with pre-trained weights based on the imagenet dataset.\n","\n","Note we remove the top, as we will replace the classification component of the network with our own layers and proceed to fine-tune (train) these weights. We also specify the image size we will deal with. Ideally the image size should be similar to the one originally used by the loaded network - too small or large can cause issues (including failure to compile, if dimensionality reduction reduces matrix size to much!). We are using images that are too small for DenseNet121, so will not achieve good results, but will proceed as an example."]},{"cell_type":"code","metadata":{"id":"4qyJ9uiuEP60","colab_type":"code","colab":{}},"source":["base_model=DenseNet121(weights=\"imagenet\",include_top=False,input_shape=(32,32,3))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Ituity7IDVKm","colab_type":"text"},"source":["DenseNet121 has 121 layers. You can look at them in a summary if you desire."]},{"cell_type":"code","metadata":{"id":"g2SmyS0pDVVw","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":1000},"outputId":"991400db-c0ff-4ac6-e624-c848c443e2a7","executionInfo":{"status":"ok","timestamp":1572735472733,"user_tz":-60,"elapsed":834,"user":{"displayName":"Michael Ashcroft","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAaRQqpOuFHR3D_ZulW6qlXPomIq5vZ-wR4ZuIm=s64","userId":"16725792548700883920"}}},"source":["base_model.summary()"],"execution_count":6,"outputs":[{"output_type":"stream","text":["Model: \"densenet121\"\n","__________________________________________________________________________________________________\n","Layer (type)                    Output Shape         Param #     Connected to                     \n","==================================================================================================\n","input_2 (InputLayer)            (None, 32, 32, 3)    0                                            \n","__________________________________________________________________________________________________\n","zero_padding2d_3 (ZeroPadding2D (None, 38, 38, 3)    0           input_2[0][0]                    \n","__________________________________________________________________________________________________\n","conv1/conv (Conv2D)             (None, 16, 16, 64)   9408        zero_padding2d_3[0][0]           \n","__________________________________________________________________________________________________\n","conv1/bn (BatchNormalization)   (None, 16, 16, 64)   256         conv1/conv[0][0]                 \n","__________________________________________________________________________________________________\n","conv1/relu (Activation)         (None, 16, 16, 64)   0           conv1/bn[0][0]                   \n","__________________________________________________________________________________________________\n","zero_padding2d_4 (ZeroPadding2D (None, 18, 18, 64)   0           conv1/relu[0][0]                 \n","__________________________________________________________________________________________________\n","pool1 (MaxPooling2D)            (None, 8, 8, 64)     0           zero_padding2d_4[0][0]           \n","__________________________________________________________________________________________________\n","conv2_block1_0_bn (BatchNormali (None, 8, 8, 64)     256         pool1[0][0]                      \n","__________________________________________________________________________________________________\n","conv2_block1_0_relu (Activation (None, 8, 8, 64)     0           conv2_block1_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv2_block1_1_conv (Conv2D)    (None, 8, 8, 128)    8192        conv2_block1_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block1_1_bn (BatchNormali (None, 8, 8, 128)    512         conv2_block1_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block1_1_relu (Activation (None, 8, 8, 128)    0           conv2_block1_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv2_block1_2_conv (Conv2D)    (None, 8, 8, 32)     36864       conv2_block1_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block1_concat (Concatenat (None, 8, 8, 96)     0           pool1[0][0]                      \n","                                                                 conv2_block1_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block2_0_bn (BatchNormali (None, 8, 8, 96)     384         conv2_block1_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block2_0_relu (Activation (None, 8, 8, 96)     0           conv2_block2_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv2_block2_1_conv (Conv2D)    (None, 8, 8, 128)    12288       conv2_block2_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block2_1_bn (BatchNormali (None, 8, 8, 128)    512         conv2_block2_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block2_1_relu (Activation (None, 8, 8, 128)    0           conv2_block2_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv2_block2_2_conv (Conv2D)    (None, 8, 8, 32)     36864       conv2_block2_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block2_concat (Concatenat (None, 8, 8, 128)    0           conv2_block1_concat[0][0]        \n","                                                                 conv2_block2_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block3_0_bn (BatchNormali (None, 8, 8, 128)    512         conv2_block2_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block3_0_relu (Activation (None, 8, 8, 128)    0           conv2_block3_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv2_block3_1_conv (Conv2D)    (None, 8, 8, 128)    16384       conv2_block3_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block3_1_bn (BatchNormali (None, 8, 8, 128)    512         conv2_block3_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block3_1_relu (Activation (None, 8, 8, 128)    0           conv2_block3_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv2_block3_2_conv (Conv2D)    (None, 8, 8, 32)     36864       conv2_block3_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block3_concat (Concatenat (None, 8, 8, 160)    0           conv2_block2_concat[0][0]        \n","                                                                 conv2_block3_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block4_0_bn (BatchNormali (None, 8, 8, 160)    640         conv2_block3_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block4_0_relu (Activation (None, 8, 8, 160)    0           conv2_block4_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv2_block4_1_conv (Conv2D)    (None, 8, 8, 128)    20480       conv2_block4_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block4_1_bn (BatchNormali (None, 8, 8, 128)    512         conv2_block4_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block4_1_relu (Activation (None, 8, 8, 128)    0           conv2_block4_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv2_block4_2_conv (Conv2D)    (None, 8, 8, 32)     36864       conv2_block4_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block4_concat (Concatenat (None, 8, 8, 192)    0           conv2_block3_concat[0][0]        \n","                                                                 conv2_block4_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block5_0_bn (BatchNormali (None, 8, 8, 192)    768         conv2_block4_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block5_0_relu (Activation (None, 8, 8, 192)    0           conv2_block5_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv2_block5_1_conv (Conv2D)    (None, 8, 8, 128)    24576       conv2_block5_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block5_1_bn (BatchNormali (None, 8, 8, 128)    512         conv2_block5_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block5_1_relu (Activation (None, 8, 8, 128)    0           conv2_block5_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv2_block5_2_conv (Conv2D)    (None, 8, 8, 32)     36864       conv2_block5_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block5_concat (Concatenat (None, 8, 8, 224)    0           conv2_block4_concat[0][0]        \n","                                                                 conv2_block5_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block6_0_bn (BatchNormali (None, 8, 8, 224)    896         conv2_block5_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block6_0_relu (Activation (None, 8, 8, 224)    0           conv2_block6_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv2_block6_1_conv (Conv2D)    (None, 8, 8, 128)    28672       conv2_block6_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block6_1_bn (BatchNormali (None, 8, 8, 128)    512         conv2_block6_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block6_1_relu (Activation (None, 8, 8, 128)    0           conv2_block6_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv2_block6_2_conv (Conv2D)    (None, 8, 8, 32)     36864       conv2_block6_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv2_block6_concat (Concatenat (None, 8, 8, 256)    0           conv2_block5_concat[0][0]        \n","                                                                 conv2_block6_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","pool2_bn (BatchNormalization)   (None, 8, 8, 256)    1024        conv2_block6_concat[0][0]        \n","__________________________________________________________________________________________________\n","pool2_relu (Activation)         (None, 8, 8, 256)    0           pool2_bn[0][0]                   \n","__________________________________________________________________________________________________\n","pool2_conv (Conv2D)             (None, 8, 8, 128)    32768       pool2_relu[0][0]                 \n","__________________________________________________________________________________________________\n","pool2_pool (AveragePooling2D)   (None, 4, 4, 128)    0           pool2_conv[0][0]                 \n","__________________________________________________________________________________________________\n","conv3_block1_0_bn (BatchNormali (None, 4, 4, 128)    512         pool2_pool[0][0]                 \n","__________________________________________________________________________________________________\n","conv3_block1_0_relu (Activation (None, 4, 4, 128)    0           conv3_block1_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block1_1_conv (Conv2D)    (None, 4, 4, 128)    16384       conv3_block1_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block1_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block1_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block1_1_relu (Activation (None, 4, 4, 128)    0           conv3_block1_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block1_2_conv (Conv2D)    (None, 4, 4, 32)     36864       conv3_block1_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block1_concat (Concatenat (None, 4, 4, 160)    0           pool2_pool[0][0]                 \n","                                                                 conv3_block1_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block2_0_bn (BatchNormali (None, 4, 4, 160)    640         conv3_block1_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block2_0_relu (Activation (None, 4, 4, 160)    0           conv3_block2_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block2_1_conv (Conv2D)    (None, 4, 4, 128)    20480       conv3_block2_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block2_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block2_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block2_1_relu (Activation (None, 4, 4, 128)    0           conv3_block2_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block2_2_conv (Conv2D)    (None, 4, 4, 32)     36864       conv3_block2_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block2_concat (Concatenat (None, 4, 4, 192)    0           conv3_block1_concat[0][0]        \n","                                                                 conv3_block2_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block3_0_bn (BatchNormali (None, 4, 4, 192)    768         conv3_block2_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block3_0_relu (Activation (None, 4, 4, 192)    0           conv3_block3_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block3_1_conv (Conv2D)    (None, 4, 4, 128)    24576       conv3_block3_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block3_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block3_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block3_1_relu (Activation (None, 4, 4, 128)    0           conv3_block3_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block3_2_conv (Conv2D)    (None, 4, 4, 32)     36864       conv3_block3_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block3_concat (Concatenat (None, 4, 4, 224)    0           conv3_block2_concat[0][0]        \n","                                                                 conv3_block3_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block4_0_bn (BatchNormali (None, 4, 4, 224)    896         conv3_block3_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block4_0_relu (Activation (None, 4, 4, 224)    0           conv3_block4_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block4_1_conv (Conv2D)    (None, 4, 4, 128)    28672       conv3_block4_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block4_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block4_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block4_1_relu (Activation (None, 4, 4, 128)    0           conv3_block4_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block4_2_conv (Conv2D)    (None, 4, 4, 32)     36864       conv3_block4_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block4_concat (Concatenat (None, 4, 4, 256)    0           conv3_block3_concat[0][0]        \n","                                                                 conv3_block4_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block5_0_bn (BatchNormali (None, 4, 4, 256)    1024        conv3_block4_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block5_0_relu (Activation (None, 4, 4, 256)    0           conv3_block5_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block5_1_conv (Conv2D)    (None, 4, 4, 128)    32768       conv3_block5_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block5_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block5_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block5_1_relu (Activation (None, 4, 4, 128)    0           conv3_block5_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block5_2_conv (Conv2D)    (None, 4, 4, 32)     36864       conv3_block5_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block5_concat (Concatenat (None, 4, 4, 288)    0           conv3_block4_concat[0][0]        \n","                                                                 conv3_block5_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block6_0_bn (BatchNormali (None, 4, 4, 288)    1152        conv3_block5_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block6_0_relu (Activation (None, 4, 4, 288)    0           conv3_block6_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block6_1_conv (Conv2D)    (None, 4, 4, 128)    36864       conv3_block6_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block6_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block6_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block6_1_relu (Activation (None, 4, 4, 128)    0           conv3_block6_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block6_2_conv (Conv2D)    (None, 4, 4, 32)     36864       conv3_block6_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block6_concat (Concatenat (None, 4, 4, 320)    0           conv3_block5_concat[0][0]        \n","                                                                 conv3_block6_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block7_0_bn (BatchNormali (None, 4, 4, 320)    1280        conv3_block6_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block7_0_relu (Activation (None, 4, 4, 320)    0           conv3_block7_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block7_1_conv (Conv2D)    (None, 4, 4, 128)    40960       conv3_block7_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block7_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block7_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block7_1_relu (Activation (None, 4, 4, 128)    0           conv3_block7_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block7_2_conv (Conv2D)    (None, 4, 4, 32)     36864       conv3_block7_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block7_concat (Concatenat (None, 4, 4, 352)    0           conv3_block6_concat[0][0]        \n","                                                                 conv3_block7_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block8_0_bn (BatchNormali (None, 4, 4, 352)    1408        conv3_block7_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block8_0_relu (Activation (None, 4, 4, 352)    0           conv3_block8_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block8_1_conv (Conv2D)    (None, 4, 4, 128)    45056       conv3_block8_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block8_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block8_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block8_1_relu (Activation (None, 4, 4, 128)    0           conv3_block8_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block8_2_conv (Conv2D)    (None, 4, 4, 32)     36864       conv3_block8_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block8_concat (Concatenat (None, 4, 4, 384)    0           conv3_block7_concat[0][0]        \n","                                                                 conv3_block8_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block9_0_bn (BatchNormali (None, 4, 4, 384)    1536        conv3_block8_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block9_0_relu (Activation (None, 4, 4, 384)    0           conv3_block9_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block9_1_conv (Conv2D)    (None, 4, 4, 128)    49152       conv3_block9_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block9_1_bn (BatchNormali (None, 4, 4, 128)    512         conv3_block9_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block9_1_relu (Activation (None, 4, 4, 128)    0           conv3_block9_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv3_block9_2_conv (Conv2D)    (None, 4, 4, 32)     36864       conv3_block9_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block9_concat (Concatenat (None, 4, 4, 416)    0           conv3_block8_concat[0][0]        \n","                                                                 conv3_block9_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block10_0_bn (BatchNormal (None, 4, 4, 416)    1664        conv3_block9_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv3_block10_0_relu (Activatio (None, 4, 4, 416)    0           conv3_block10_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv3_block10_1_conv (Conv2D)   (None, 4, 4, 128)    53248       conv3_block10_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv3_block10_1_bn (BatchNormal (None, 4, 4, 128)    512         conv3_block10_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv3_block10_1_relu (Activatio (None, 4, 4, 128)    0           conv3_block10_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv3_block10_2_conv (Conv2D)   (None, 4, 4, 32)     36864       conv3_block10_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv3_block10_concat (Concatena (None, 4, 4, 448)    0           conv3_block9_concat[0][0]        \n","                                                                 conv3_block10_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv3_block11_0_bn (BatchNormal (None, 4, 4, 448)    1792        conv3_block10_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv3_block11_0_relu (Activatio (None, 4, 4, 448)    0           conv3_block11_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv3_block11_1_conv (Conv2D)   (None, 4, 4, 128)    57344       conv3_block11_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv3_block11_1_bn (BatchNormal (None, 4, 4, 128)    512         conv3_block11_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv3_block11_1_relu (Activatio (None, 4, 4, 128)    0           conv3_block11_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv3_block11_2_conv (Conv2D)   (None, 4, 4, 32)     36864       conv3_block11_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv3_block11_concat (Concatena (None, 4, 4, 480)    0           conv3_block10_concat[0][0]       \n","                                                                 conv3_block11_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv3_block12_0_bn (BatchNormal (None, 4, 4, 480)    1920        conv3_block11_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv3_block12_0_relu (Activatio (None, 4, 4, 480)    0           conv3_block12_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv3_block12_1_conv (Conv2D)   (None, 4, 4, 128)    61440       conv3_block12_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv3_block12_1_bn (BatchNormal (None, 4, 4, 128)    512         conv3_block12_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv3_block12_1_relu (Activatio (None, 4, 4, 128)    0           conv3_block12_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv3_block12_2_conv (Conv2D)   (None, 4, 4, 32)     36864       conv3_block12_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv3_block12_concat (Concatena (None, 4, 4, 512)    0           conv3_block11_concat[0][0]       \n","                                                                 conv3_block12_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","pool3_bn (BatchNormalization)   (None, 4, 4, 512)    2048        conv3_block12_concat[0][0]       \n","__________________________________________________________________________________________________\n","pool3_relu (Activation)         (None, 4, 4, 512)    0           pool3_bn[0][0]                   \n","__________________________________________________________________________________________________\n","pool3_conv (Conv2D)             (None, 4, 4, 256)    131072      pool3_relu[0][0]                 \n","__________________________________________________________________________________________________\n","pool3_pool (AveragePooling2D)   (None, 2, 2, 256)    0           pool3_conv[0][0]                 \n","__________________________________________________________________________________________________\n","conv4_block1_0_bn (BatchNormali (None, 2, 2, 256)    1024        pool3_pool[0][0]                 \n","__________________________________________________________________________________________________\n","conv4_block1_0_relu (Activation (None, 2, 2, 256)    0           conv4_block1_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block1_1_conv (Conv2D)    (None, 2, 2, 128)    32768       conv4_block1_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block1_1_bn (BatchNormali (None, 2, 2, 128)    512         conv4_block1_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block1_1_relu (Activation (None, 2, 2, 128)    0           conv4_block1_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block1_2_conv (Conv2D)    (None, 2, 2, 32)     36864       conv4_block1_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block1_concat (Concatenat (None, 2, 2, 288)    0           pool3_pool[0][0]                 \n","                                                                 conv4_block1_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block2_0_bn (BatchNormali (None, 2, 2, 288)    1152        conv4_block1_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block2_0_relu (Activation (None, 2, 2, 288)    0           conv4_block2_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block2_1_conv (Conv2D)    (None, 2, 2, 128)    36864       conv4_block2_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block2_1_bn (BatchNormali (None, 2, 2, 128)    512         conv4_block2_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block2_1_relu (Activation (None, 2, 2, 128)    0           conv4_block2_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block2_2_conv (Conv2D)    (None, 2, 2, 32)     36864       conv4_block2_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block2_concat (Concatenat (None, 2, 2, 320)    0           conv4_block1_concat[0][0]        \n","                                                                 conv4_block2_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block3_0_bn (BatchNormali (None, 2, 2, 320)    1280        conv4_block2_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block3_0_relu (Activation (None, 2, 2, 320)    0           conv4_block3_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block3_1_conv (Conv2D)    (None, 2, 2, 128)    40960       conv4_block3_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block3_1_bn (BatchNormali (None, 2, 2, 128)    512         conv4_block3_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block3_1_relu (Activation (None, 2, 2, 128)    0           conv4_block3_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block3_2_conv (Conv2D)    (None, 2, 2, 32)     36864       conv4_block3_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block3_concat (Concatenat (None, 2, 2, 352)    0           conv4_block2_concat[0][0]        \n","                                                                 conv4_block3_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block4_0_bn (BatchNormali (None, 2, 2, 352)    1408        conv4_block3_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block4_0_relu (Activation (None, 2, 2, 352)    0           conv4_block4_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block4_1_conv (Conv2D)    (None, 2, 2, 128)    45056       conv4_block4_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block4_1_bn (BatchNormali (None, 2, 2, 128)    512         conv4_block4_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block4_1_relu (Activation (None, 2, 2, 128)    0           conv4_block4_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block4_2_conv (Conv2D)    (None, 2, 2, 32)     36864       conv4_block4_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block4_concat (Concatenat (None, 2, 2, 384)    0           conv4_block3_concat[0][0]        \n","                                                                 conv4_block4_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block5_0_bn (BatchNormali (None, 2, 2, 384)    1536        conv4_block4_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block5_0_relu (Activation (None, 2, 2, 384)    0           conv4_block5_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block5_1_conv (Conv2D)    (None, 2, 2, 128)    49152       conv4_block5_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block5_1_bn (BatchNormali (None, 2, 2, 128)    512         conv4_block5_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block5_1_relu (Activation (None, 2, 2, 128)    0           conv4_block5_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block5_2_conv (Conv2D)    (None, 2, 2, 32)     36864       conv4_block5_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block5_concat (Concatenat (None, 2, 2, 416)    0           conv4_block4_concat[0][0]        \n","                                                                 conv4_block5_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block6_0_bn (BatchNormali (None, 2, 2, 416)    1664        conv4_block5_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block6_0_relu (Activation (None, 2, 2, 416)    0           conv4_block6_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block6_1_conv (Conv2D)    (None, 2, 2, 128)    53248       conv4_block6_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block6_1_bn (BatchNormali (None, 2, 2, 128)    512         conv4_block6_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block6_1_relu (Activation (None, 2, 2, 128)    0           conv4_block6_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block6_2_conv (Conv2D)    (None, 2, 2, 32)     36864       conv4_block6_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block6_concat (Concatenat (None, 2, 2, 448)    0           conv4_block5_concat[0][0]        \n","                                                                 conv4_block6_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block7_0_bn (BatchNormali (None, 2, 2, 448)    1792        conv4_block6_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block7_0_relu (Activation (None, 2, 2, 448)    0           conv4_block7_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block7_1_conv (Conv2D)    (None, 2, 2, 128)    57344       conv4_block7_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block7_1_bn (BatchNormali (None, 2, 2, 128)    512         conv4_block7_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block7_1_relu (Activation (None, 2, 2, 128)    0           conv4_block7_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block7_2_conv (Conv2D)    (None, 2, 2, 32)     36864       conv4_block7_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block7_concat (Concatenat (None, 2, 2, 480)    0           conv4_block6_concat[0][0]        \n","                                                                 conv4_block7_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block8_0_bn (BatchNormali (None, 2, 2, 480)    1920        conv4_block7_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block8_0_relu (Activation (None, 2, 2, 480)    0           conv4_block8_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block8_1_conv (Conv2D)    (None, 2, 2, 128)    61440       conv4_block8_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block8_1_bn (BatchNormali (None, 2, 2, 128)    512         conv4_block8_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block8_1_relu (Activation (None, 2, 2, 128)    0           conv4_block8_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block8_2_conv (Conv2D)    (None, 2, 2, 32)     36864       conv4_block8_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block8_concat (Concatenat (None, 2, 2, 512)    0           conv4_block7_concat[0][0]        \n","                                                                 conv4_block8_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block9_0_bn (BatchNormali (None, 2, 2, 512)    2048        conv4_block8_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block9_0_relu (Activation (None, 2, 2, 512)    0           conv4_block9_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block9_1_conv (Conv2D)    (None, 2, 2, 128)    65536       conv4_block9_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block9_1_bn (BatchNormali (None, 2, 2, 128)    512         conv4_block9_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block9_1_relu (Activation (None, 2, 2, 128)    0           conv4_block9_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv4_block9_2_conv (Conv2D)    (None, 2, 2, 32)     36864       conv4_block9_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block9_concat (Concatenat (None, 2, 2, 544)    0           conv4_block8_concat[0][0]        \n","                                                                 conv4_block9_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block10_0_bn (BatchNormal (None, 2, 2, 544)    2176        conv4_block9_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv4_block10_0_relu (Activatio (None, 2, 2, 544)    0           conv4_block10_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block10_1_conv (Conv2D)   (None, 2, 2, 128)    69632       conv4_block10_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block10_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block10_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block10_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block10_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block10_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block10_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block10_concat (Concatena (None, 2, 2, 576)    0           conv4_block9_concat[0][0]        \n","                                                                 conv4_block10_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block11_0_bn (BatchNormal (None, 2, 2, 576)    2304        conv4_block10_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block11_0_relu (Activatio (None, 2, 2, 576)    0           conv4_block11_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block11_1_conv (Conv2D)   (None, 2, 2, 128)    73728       conv4_block11_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block11_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block11_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block11_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block11_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block11_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block11_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block11_concat (Concatena (None, 2, 2, 608)    0           conv4_block10_concat[0][0]       \n","                                                                 conv4_block11_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block12_0_bn (BatchNormal (None, 2, 2, 608)    2432        conv4_block11_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block12_0_relu (Activatio (None, 2, 2, 608)    0           conv4_block12_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block12_1_conv (Conv2D)   (None, 2, 2, 128)    77824       conv4_block12_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block12_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block12_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block12_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block12_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block12_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block12_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block12_concat (Concatena (None, 2, 2, 640)    0           conv4_block11_concat[0][0]       \n","                                                                 conv4_block12_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block13_0_bn (BatchNormal (None, 2, 2, 640)    2560        conv4_block12_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block13_0_relu (Activatio (None, 2, 2, 640)    0           conv4_block13_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block13_1_conv (Conv2D)   (None, 2, 2, 128)    81920       conv4_block13_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block13_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block13_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block13_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block13_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block13_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block13_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block13_concat (Concatena (None, 2, 2, 672)    0           conv4_block12_concat[0][0]       \n","                                                                 conv4_block13_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block14_0_bn (BatchNormal (None, 2, 2, 672)    2688        conv4_block13_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block14_0_relu (Activatio (None, 2, 2, 672)    0           conv4_block14_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block14_1_conv (Conv2D)   (None, 2, 2, 128)    86016       conv4_block14_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block14_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block14_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block14_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block14_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block14_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block14_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block14_concat (Concatena (None, 2, 2, 704)    0           conv4_block13_concat[0][0]       \n","                                                                 conv4_block14_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block15_0_bn (BatchNormal (None, 2, 2, 704)    2816        conv4_block14_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block15_0_relu (Activatio (None, 2, 2, 704)    0           conv4_block15_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block15_1_conv (Conv2D)   (None, 2, 2, 128)    90112       conv4_block15_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block15_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block15_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block15_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block15_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block15_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block15_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block15_concat (Concatena (None, 2, 2, 736)    0           conv4_block14_concat[0][0]       \n","                                                                 conv4_block15_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block16_0_bn (BatchNormal (None, 2, 2, 736)    2944        conv4_block15_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block16_0_relu (Activatio (None, 2, 2, 736)    0           conv4_block16_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block16_1_conv (Conv2D)   (None, 2, 2, 128)    94208       conv4_block16_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block16_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block16_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block16_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block16_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block16_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block16_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block16_concat (Concatena (None, 2, 2, 768)    0           conv4_block15_concat[0][0]       \n","                                                                 conv4_block16_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block17_0_bn (BatchNormal (None, 2, 2, 768)    3072        conv4_block16_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block17_0_relu (Activatio (None, 2, 2, 768)    0           conv4_block17_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block17_1_conv (Conv2D)   (None, 2, 2, 128)    98304       conv4_block17_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block17_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block17_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block17_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block17_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block17_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block17_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block17_concat (Concatena (None, 2, 2, 800)    0           conv4_block16_concat[0][0]       \n","                                                                 conv4_block17_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block18_0_bn (BatchNormal (None, 2, 2, 800)    3200        conv4_block17_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block18_0_relu (Activatio (None, 2, 2, 800)    0           conv4_block18_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block18_1_conv (Conv2D)   (None, 2, 2, 128)    102400      conv4_block18_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block18_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block18_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block18_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block18_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block18_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block18_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block18_concat (Concatena (None, 2, 2, 832)    0           conv4_block17_concat[0][0]       \n","                                                                 conv4_block18_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block19_0_bn (BatchNormal (None, 2, 2, 832)    3328        conv4_block18_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block19_0_relu (Activatio (None, 2, 2, 832)    0           conv4_block19_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block19_1_conv (Conv2D)   (None, 2, 2, 128)    106496      conv4_block19_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block19_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block19_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block19_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block19_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block19_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block19_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block19_concat (Concatena (None, 2, 2, 864)    0           conv4_block18_concat[0][0]       \n","                                                                 conv4_block19_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block20_0_bn (BatchNormal (None, 2, 2, 864)    3456        conv4_block19_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block20_0_relu (Activatio (None, 2, 2, 864)    0           conv4_block20_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block20_1_conv (Conv2D)   (None, 2, 2, 128)    110592      conv4_block20_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block20_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block20_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block20_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block20_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block20_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block20_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block20_concat (Concatena (None, 2, 2, 896)    0           conv4_block19_concat[0][0]       \n","                                                                 conv4_block20_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block21_0_bn (BatchNormal (None, 2, 2, 896)    3584        conv4_block20_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block21_0_relu (Activatio (None, 2, 2, 896)    0           conv4_block21_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block21_1_conv (Conv2D)   (None, 2, 2, 128)    114688      conv4_block21_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block21_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block21_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block21_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block21_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block21_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block21_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block21_concat (Concatena (None, 2, 2, 928)    0           conv4_block20_concat[0][0]       \n","                                                                 conv4_block21_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block22_0_bn (BatchNormal (None, 2, 2, 928)    3712        conv4_block21_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block22_0_relu (Activatio (None, 2, 2, 928)    0           conv4_block22_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block22_1_conv (Conv2D)   (None, 2, 2, 128)    118784      conv4_block22_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block22_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block22_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block22_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block22_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block22_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block22_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block22_concat (Concatena (None, 2, 2, 960)    0           conv4_block21_concat[0][0]       \n","                                                                 conv4_block22_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block23_0_bn (BatchNormal (None, 2, 2, 960)    3840        conv4_block22_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block23_0_relu (Activatio (None, 2, 2, 960)    0           conv4_block23_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block23_1_conv (Conv2D)   (None, 2, 2, 128)    122880      conv4_block23_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block23_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block23_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block23_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block23_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block23_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block23_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block23_concat (Concatena (None, 2, 2, 992)    0           conv4_block22_concat[0][0]       \n","                                                                 conv4_block23_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block24_0_bn (BatchNormal (None, 2, 2, 992)    3968        conv4_block23_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block24_0_relu (Activatio (None, 2, 2, 992)    0           conv4_block24_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block24_1_conv (Conv2D)   (None, 2, 2, 128)    126976      conv4_block24_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block24_1_bn (BatchNormal (None, 2, 2, 128)    512         conv4_block24_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block24_1_relu (Activatio (None, 2, 2, 128)    0           conv4_block24_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv4_block24_2_conv (Conv2D)   (None, 2, 2, 32)     36864       conv4_block24_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv4_block24_concat (Concatena (None, 2, 2, 1024)   0           conv4_block23_concat[0][0]       \n","                                                                 conv4_block24_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","pool4_bn (BatchNormalization)   (None, 2, 2, 1024)   4096        conv4_block24_concat[0][0]       \n","__________________________________________________________________________________________________\n","pool4_relu (Activation)         (None, 2, 2, 1024)   0           pool4_bn[0][0]                   \n","__________________________________________________________________________________________________\n","pool4_conv (Conv2D)             (None, 2, 2, 512)    524288      pool4_relu[0][0]                 \n","__________________________________________________________________________________________________\n","pool4_pool (AveragePooling2D)   (None, 1, 1, 512)    0           pool4_conv[0][0]                 \n","__________________________________________________________________________________________________\n","conv5_block1_0_bn (BatchNormali (None, 1, 1, 512)    2048        pool4_pool[0][0]                 \n","__________________________________________________________________________________________________\n","conv5_block1_0_relu (Activation (None, 1, 1, 512)    0           conv5_block1_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block1_1_conv (Conv2D)    (None, 1, 1, 128)    65536       conv5_block1_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block1_1_bn (BatchNormali (None, 1, 1, 128)    512         conv5_block1_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block1_1_relu (Activation (None, 1, 1, 128)    0           conv5_block1_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block1_2_conv (Conv2D)    (None, 1, 1, 32)     36864       conv5_block1_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block1_concat (Concatenat (None, 1, 1, 544)    0           pool4_pool[0][0]                 \n","                                                                 conv5_block1_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block2_0_bn (BatchNormali (None, 1, 1, 544)    2176        conv5_block1_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block2_0_relu (Activation (None, 1, 1, 544)    0           conv5_block2_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block2_1_conv (Conv2D)    (None, 1, 1, 128)    69632       conv5_block2_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block2_1_bn (BatchNormali (None, 1, 1, 128)    512         conv5_block2_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block2_1_relu (Activation (None, 1, 1, 128)    0           conv5_block2_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block2_2_conv (Conv2D)    (None, 1, 1, 32)     36864       conv5_block2_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block2_concat (Concatenat (None, 1, 1, 576)    0           conv5_block1_concat[0][0]        \n","                                                                 conv5_block2_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block3_0_bn (BatchNormali (None, 1, 1, 576)    2304        conv5_block2_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block3_0_relu (Activation (None, 1, 1, 576)    0           conv5_block3_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block3_1_conv (Conv2D)    (None, 1, 1, 128)    73728       conv5_block3_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block3_1_bn (BatchNormali (None, 1, 1, 128)    512         conv5_block3_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block3_1_relu (Activation (None, 1, 1, 128)    0           conv5_block3_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block3_2_conv (Conv2D)    (None, 1, 1, 32)     36864       conv5_block3_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block3_concat (Concatenat (None, 1, 1, 608)    0           conv5_block2_concat[0][0]        \n","                                                                 conv5_block3_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block4_0_bn (BatchNormali (None, 1, 1, 608)    2432        conv5_block3_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block4_0_relu (Activation (None, 1, 1, 608)    0           conv5_block4_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block4_1_conv (Conv2D)    (None, 1, 1, 128)    77824       conv5_block4_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block4_1_bn (BatchNormali (None, 1, 1, 128)    512         conv5_block4_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block4_1_relu (Activation (None, 1, 1, 128)    0           conv5_block4_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block4_2_conv (Conv2D)    (None, 1, 1, 32)     36864       conv5_block4_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block4_concat (Concatenat (None, 1, 1, 640)    0           conv5_block3_concat[0][0]        \n","                                                                 conv5_block4_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block5_0_bn (BatchNormali (None, 1, 1, 640)    2560        conv5_block4_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block5_0_relu (Activation (None, 1, 1, 640)    0           conv5_block5_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block5_1_conv (Conv2D)    (None, 1, 1, 128)    81920       conv5_block5_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block5_1_bn (BatchNormali (None, 1, 1, 128)    512         conv5_block5_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block5_1_relu (Activation (None, 1, 1, 128)    0           conv5_block5_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block5_2_conv (Conv2D)    (None, 1, 1, 32)     36864       conv5_block5_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block5_concat (Concatenat (None, 1, 1, 672)    0           conv5_block4_concat[0][0]        \n","                                                                 conv5_block5_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block6_0_bn (BatchNormali (None, 1, 1, 672)    2688        conv5_block5_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block6_0_relu (Activation (None, 1, 1, 672)    0           conv5_block6_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block6_1_conv (Conv2D)    (None, 1, 1, 128)    86016       conv5_block6_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block6_1_bn (BatchNormali (None, 1, 1, 128)    512         conv5_block6_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block6_1_relu (Activation (None, 1, 1, 128)    0           conv5_block6_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block6_2_conv (Conv2D)    (None, 1, 1, 32)     36864       conv5_block6_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block6_concat (Concatenat (None, 1, 1, 704)    0           conv5_block5_concat[0][0]        \n","                                                                 conv5_block6_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block7_0_bn (BatchNormali (None, 1, 1, 704)    2816        conv5_block6_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block7_0_relu (Activation (None, 1, 1, 704)    0           conv5_block7_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block7_1_conv (Conv2D)    (None, 1, 1, 128)    90112       conv5_block7_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block7_1_bn (BatchNormali (None, 1, 1, 128)    512         conv5_block7_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block7_1_relu (Activation (None, 1, 1, 128)    0           conv5_block7_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block7_2_conv (Conv2D)    (None, 1, 1, 32)     36864       conv5_block7_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block7_concat (Concatenat (None, 1, 1, 736)    0           conv5_block6_concat[0][0]        \n","                                                                 conv5_block7_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block8_0_bn (BatchNormali (None, 1, 1, 736)    2944        conv5_block7_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block8_0_relu (Activation (None, 1, 1, 736)    0           conv5_block8_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block8_1_conv (Conv2D)    (None, 1, 1, 128)    94208       conv5_block8_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block8_1_bn (BatchNormali (None, 1, 1, 128)    512         conv5_block8_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block8_1_relu (Activation (None, 1, 1, 128)    0           conv5_block8_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block8_2_conv (Conv2D)    (None, 1, 1, 32)     36864       conv5_block8_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block8_concat (Concatenat (None, 1, 1, 768)    0           conv5_block7_concat[0][0]        \n","                                                                 conv5_block8_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block9_0_bn (BatchNormali (None, 1, 1, 768)    3072        conv5_block8_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block9_0_relu (Activation (None, 1, 1, 768)    0           conv5_block9_0_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block9_1_conv (Conv2D)    (None, 1, 1, 128)    98304       conv5_block9_0_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block9_1_bn (BatchNormali (None, 1, 1, 128)    512         conv5_block9_1_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block9_1_relu (Activation (None, 1, 1, 128)    0           conv5_block9_1_bn[0][0]          \n","__________________________________________________________________________________________________\n","conv5_block9_2_conv (Conv2D)    (None, 1, 1, 32)     36864       conv5_block9_1_relu[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block9_concat (Concatenat (None, 1, 1, 800)    0           conv5_block8_concat[0][0]        \n","                                                                 conv5_block9_2_conv[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block10_0_bn (BatchNormal (None, 1, 1, 800)    3200        conv5_block9_concat[0][0]        \n","__________________________________________________________________________________________________\n","conv5_block10_0_relu (Activatio (None, 1, 1, 800)    0           conv5_block10_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block10_1_conv (Conv2D)   (None, 1, 1, 128)    102400      conv5_block10_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block10_1_bn (BatchNormal (None, 1, 1, 128)    512         conv5_block10_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block10_1_relu (Activatio (None, 1, 1, 128)    0           conv5_block10_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block10_2_conv (Conv2D)   (None, 1, 1, 32)     36864       conv5_block10_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block10_concat (Concatena (None, 1, 1, 832)    0           conv5_block9_concat[0][0]        \n","                                                                 conv5_block10_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block11_0_bn (BatchNormal (None, 1, 1, 832)    3328        conv5_block10_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block11_0_relu (Activatio (None, 1, 1, 832)    0           conv5_block11_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block11_1_conv (Conv2D)   (None, 1, 1, 128)    106496      conv5_block11_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block11_1_bn (BatchNormal (None, 1, 1, 128)    512         conv5_block11_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block11_1_relu (Activatio (None, 1, 1, 128)    0           conv5_block11_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block11_2_conv (Conv2D)   (None, 1, 1, 32)     36864       conv5_block11_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block11_concat (Concatena (None, 1, 1, 864)    0           conv5_block10_concat[0][0]       \n","                                                                 conv5_block11_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block12_0_bn (BatchNormal (None, 1, 1, 864)    3456        conv5_block11_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block12_0_relu (Activatio (None, 1, 1, 864)    0           conv5_block12_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block12_1_conv (Conv2D)   (None, 1, 1, 128)    110592      conv5_block12_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block12_1_bn (BatchNormal (None, 1, 1, 128)    512         conv5_block12_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block12_1_relu (Activatio (None, 1, 1, 128)    0           conv5_block12_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block12_2_conv (Conv2D)   (None, 1, 1, 32)     36864       conv5_block12_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block12_concat (Concatena (None, 1, 1, 896)    0           conv5_block11_concat[0][0]       \n","                                                                 conv5_block12_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block13_0_bn (BatchNormal (None, 1, 1, 896)    3584        conv5_block12_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block13_0_relu (Activatio (None, 1, 1, 896)    0           conv5_block13_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block13_1_conv (Conv2D)   (None, 1, 1, 128)    114688      conv5_block13_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block13_1_bn (BatchNormal (None, 1, 1, 128)    512         conv5_block13_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block13_1_relu (Activatio (None, 1, 1, 128)    0           conv5_block13_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block13_2_conv (Conv2D)   (None, 1, 1, 32)     36864       conv5_block13_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block13_concat (Concatena (None, 1, 1, 928)    0           conv5_block12_concat[0][0]       \n","                                                                 conv5_block13_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block14_0_bn (BatchNormal (None, 1, 1, 928)    3712        conv5_block13_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block14_0_relu (Activatio (None, 1, 1, 928)    0           conv5_block14_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block14_1_conv (Conv2D)   (None, 1, 1, 128)    118784      conv5_block14_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block14_1_bn (BatchNormal (None, 1, 1, 128)    512         conv5_block14_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block14_1_relu (Activatio (None, 1, 1, 128)    0           conv5_block14_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block14_2_conv (Conv2D)   (None, 1, 1, 32)     36864       conv5_block14_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block14_concat (Concatena (None, 1, 1, 960)    0           conv5_block13_concat[0][0]       \n","                                                                 conv5_block14_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block15_0_bn (BatchNormal (None, 1, 1, 960)    3840        conv5_block14_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block15_0_relu (Activatio (None, 1, 1, 960)    0           conv5_block15_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block15_1_conv (Conv2D)   (None, 1, 1, 128)    122880      conv5_block15_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block15_1_bn (BatchNormal (None, 1, 1, 128)    512         conv5_block15_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block15_1_relu (Activatio (None, 1, 1, 128)    0           conv5_block15_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block15_2_conv (Conv2D)   (None, 1, 1, 32)     36864       conv5_block15_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block15_concat (Concatena (None, 1, 1, 992)    0           conv5_block14_concat[0][0]       \n","                                                                 conv5_block15_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block16_0_bn (BatchNormal (None, 1, 1, 992)    3968        conv5_block15_concat[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block16_0_relu (Activatio (None, 1, 1, 992)    0           conv5_block16_0_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block16_1_conv (Conv2D)   (None, 1, 1, 128)    126976      conv5_block16_0_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block16_1_bn (BatchNormal (None, 1, 1, 128)    512         conv5_block16_1_conv[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block16_1_relu (Activatio (None, 1, 1, 128)    0           conv5_block16_1_bn[0][0]         \n","__________________________________________________________________________________________________\n","conv5_block16_2_conv (Conv2D)   (None, 1, 1, 32)     36864       conv5_block16_1_relu[0][0]       \n","__________________________________________________________________________________________________\n","conv5_block16_concat (Concatena (None, 1, 1, 1024)   0           conv5_block15_concat[0][0]       \n","                                                                 conv5_block16_2_conv[0][0]       \n","__________________________________________________________________________________________________\n","bn (BatchNormalization)         (None, 1, 1, 1024)   4096        conv5_block16_concat[0][0]       \n","__________________________________________________________________________________________________\n","relu (Activation)               (None, 1, 1, 1024)   0           bn[0][0]                         \n","==================================================================================================\n","Total params: 7,037,504\n","Trainable params: 6,953,856\n","Non-trainable params: 83,648\n","__________________________________________________________________________________________________\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"hykKIhJNXW_R","colab_type":"text"},"source":["Now we create a wrapper function that will take our base network and create a custom 'top' to it. This function will also set the weights in the base network (the feature extraction weights) to be not be trained."]},{"cell_type":"code","metadata":{"id":"eNZJf2HYQ6wh","colab_type":"code","colab":{}},"source":["def get_final_model(base_model, dropout, class_layers, num_classes):\n","    # Turn of training for all layers in the base model.\n","    for layer in base_model.layers:\n","        layer.trainable = False\n","\n","    # Flatten final layer of network \n","    # Remember the classification component (the dense layers and output layer)\n","    # has been removed. So we are flattening the output of the feature extraction\n","    # part of the full original network.\n","    layer = Flatten()(base_model.output)\n","    \n","    # Add classification layers, or 'top', to the network \n","    # We use dropout regularization\n","    for nodes in class_layers:\n","        layer = Dense(nodes, activation='relu')(layer) \n","        layer = Dropout(dropout)(layer)\n","\n","    # Add output layer\n","    output = Dense(num_classes, activation='softmax')(layer) \n","    \n","    final_model = Model(inputs=base_model.input, outputs=output)\n","\n","    return final_model"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"2vCA713PXj5Y","colab_type":"text"},"source":["Let's get our base model."]},{"cell_type":"code","metadata":{"id":"f_iI_Yy9Q8R7","colab_type":"code","colab":{}},"source":["final_model = get_final_model(base_model, \n","                                      .5, \n","                                      [256], \n","                                      10)\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"aFgM31SRXSff","colab_type":"text"},"source":["We compile the network. Feel free to change the optimizer."]},{"cell_type":"code","metadata":{"id":"wkobk03lSxsk","colab_type":"code","colab":{}},"source":["final_model.compile(optimizer=Adam(),\n","              loss='categorical_crossentropy',\n","              metrics=['accuracy'])"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ec_kvQ89XPYr","colab_type":"text"},"source":["Now we train the network, using early stopping."]},{"cell_type":"code","metadata":{"id":"BvpGW0YWS4nJ","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":1000},"outputId":"eee056a9-a9c8-4f0a-b9dd-d87de7ce5882","executionInfo":{"status":"ok","timestamp":1572561736343,"user_tz":-60,"elapsed":12453862,"user":{"displayName":"Michael Ashcroft","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAaRQqpOuFHR3D_ZulW6qlXPomIq5vZ-wR4ZuIm=s64","userId":"16725792548700883920"}}},"source":["earlyStopping = EarlyStopping(monitor=\"val_loss\", \n","                              patience=10,\n","                              verbose=1,\n","                              restore_best_weights=True)\n","\n","history = final_model.fit(x_train, \n","                    y_train, \n","                    epochs=100, \n","                    shuffle=True, \n","                    callbacks=[earlyStopping],\n","                    validation_data=(x_test,y_test))"],"execution_count":8,"outputs":[{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use tf.where in 2.0, which has the same broadcast rule as np.where\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n","\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1020: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n","\n","Train on 50000 samples, validate on 10000 samples\n","Epoch 1/100\n","50000/50000 [==============================] - 377s 8ms/step - loss: 1.7672 - acc: 0.3723 - val_loss: 1.8947 - val_acc: 0.3913\n","Epoch 2/100\n","50000/50000 [==============================] - 370s 7ms/step - loss: 1.5600 - acc: 0.4494 - val_loss: 1.8088 - val_acc: 0.4239\n","Epoch 3/100\n","50000/50000 [==============================] - 366s 7ms/step - loss: 1.5084 - acc: 0.4682 - val_loss: 1.9516 - val_acc: 0.4094\n","Epoch 4/100\n","50000/50000 [==============================] - 367s 7ms/step - loss: 1.4840 - acc: 0.4780 - val_loss: 1.9122 - val_acc: 0.4235\n","Epoch 5/100\n","50000/50000 [==============================] - 366s 7ms/step - loss: 1.4639 - acc: 0.4849 - val_loss: 1.8974 - val_acc: 0.4203\n","Epoch 6/100\n","50000/50000 [==============================] - 371s 7ms/step - loss: 1.4485 - acc: 0.4866 - val_loss: 1.9554 - val_acc: 0.4033\n","Epoch 7/100\n","50000/50000 [==============================] - 370s 7ms/step - loss: 1.4419 - acc: 0.4927 - val_loss: 1.9223 - val_acc: 0.4262\n","Epoch 8/100\n","50000/50000 [==============================] - 371s 7ms/step - loss: 1.4249 - acc: 0.4986 - val_loss: 1.7454 - val_acc: 0.4529\n","Epoch 9/100\n","50000/50000 [==============================] - 365s 7ms/step - loss: 1.4252 - acc: 0.4969 - val_loss: 1.7993 - val_acc: 0.4429\n","Epoch 10/100\n","50000/50000 [==============================] - 362s 7ms/step - loss: 1.4175 - acc: 0.5038 - val_loss: 1.8107 - val_acc: 0.4434\n","Epoch 11/100\n","50000/50000 [==============================] - 361s 7ms/step - loss: 1.4035 - acc: 0.5062 - val_loss: 1.8689 - val_acc: 0.4356\n","Epoch 12/100\n","50000/50000 [==============================] - 361s 7ms/step - loss: 1.3987 - acc: 0.5074 - val_loss: 1.7463 - val_acc: 0.4550\n","Epoch 13/100\n","50000/50000 [==============================] - 367s 7ms/step - loss: 1.3916 - acc: 0.5085 - val_loss: 1.8278 - val_acc: 0.4382\n","Epoch 14/100\n","50000/50000 [==============================] - 367s 7ms/step - loss: 1.3902 - acc: 0.5105 - val_loss: 1.7755 - val_acc: 0.4492\n","Epoch 15/100\n","50000/50000 [==============================] - 367s 7ms/step - loss: 1.3834 - acc: 0.5112 - val_loss: 1.8317 - val_acc: 0.4414\n","Epoch 16/100\n","50000/50000 [==============================] - 368s 7ms/step - loss: 1.3783 - acc: 0.5154 - val_loss: 1.7200 - val_acc: 0.4621\n","Epoch 17/100\n","50000/50000 [==============================] - 364s 7ms/step - loss: 1.3762 - acc: 0.5177 - val_loss: 1.7445 - val_acc: 0.4546\n","Epoch 18/100\n","50000/50000 [==============================] - 367s 7ms/step - loss: 1.3720 - acc: 0.5182 - val_loss: 1.8576 - val_acc: 0.4236\n","Epoch 19/100\n","50000/50000 [==============================] - 367s 7ms/step - loss: 1.3648 - acc: 0.5205 - val_loss: 1.7643 - val_acc: 0.4511\n","Epoch 20/100\n","50000/50000 [==============================] - 369s 7ms/step - loss: 1.3648 - acc: 0.5213 - val_loss: 1.6564 - val_acc: 0.4688\n","Epoch 21/100\n","50000/50000 [==============================] - 360s 7ms/step - loss: 1.3656 - acc: 0.5168 - val_loss: 1.6891 - val_acc: 0.4748\n","Epoch 22/100\n","50000/50000 [==============================] - 367s 7ms/step - loss: 1.3590 - acc: 0.5263 - val_loss: 1.7368 - val_acc: 0.4550\n","Epoch 23/100\n","50000/50000 [==============================] - 365s 7ms/step - loss: 1.3533 - acc: 0.5233 - val_loss: 1.6630 - val_acc: 0.4728\n","Epoch 24/100\n","50000/50000 [==============================] - 366s 7ms/step - loss: 1.3542 - acc: 0.5236 - val_loss: 1.6126 - val_acc: 0.4757\n","Epoch 25/100\n","50000/50000 [==============================] - 360s 7ms/step - loss: 1.3498 - acc: 0.5250 - val_loss: 1.7703 - val_acc: 0.4538\n","Epoch 26/100\n","50000/50000 [==============================] - 364s 7ms/step - loss: 1.3467 - acc: 0.5286 - val_loss: 1.6791 - val_acc: 0.4812\n","Epoch 27/100\n","50000/50000 [==============================] - 365s 7ms/step - loss: 1.3473 - acc: 0.5263 - val_loss: 1.7013 - val_acc: 0.4687\n","Epoch 28/100\n","50000/50000 [==============================] - 363s 7ms/step - loss: 1.3497 - acc: 0.5265 - val_loss: 1.7283 - val_acc: 0.4668\n","Epoch 29/100\n","50000/50000 [==============================] - 364s 7ms/step - loss: 1.3400 - acc: 0.5293 - val_loss: 1.7805 - val_acc: 0.4549\n","Epoch 30/100\n","50000/50000 [==============================] - 364s 7ms/step - loss: 1.3393 - acc: 0.5280 - val_loss: 1.7373 - val_acc: 0.4660\n","Epoch 31/100\n","50000/50000 [==============================] - 365s 7ms/step - loss: 1.3411 - acc: 0.5301 - val_loss: 1.7427 - val_acc: 0.4658\n","Epoch 32/100\n","50000/50000 [==============================] - 368s 7ms/step - loss: 1.3410 - acc: 0.5289 - val_loss: 1.6792 - val_acc: 0.4649\n","Epoch 33/100\n","50000/50000 [==============================] - 367s 7ms/step - loss: 1.3331 - acc: 0.5297 - val_loss: 1.6583 - val_acc: 0.4662\n","Epoch 34/100\n","50000/50000 [==============================] - 368s 7ms/step - loss: 1.3354 - acc: 0.5322 - val_loss: 1.6855 - val_acc: 0.4672\n","Restoring model weights from the end of the best epoch\n","Epoch 00034: early stopping\n"],"name":"stdout"}]}]}