chapter05_fundamentals-of-ml.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6IJ9hM-RVORh"
      },
      "source": [
        "# Fundamentals of machine learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Huq2jDqjVORi"
      },
      "source": [
        "## Generalization: The goal of machine learning"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "- optimization : trainign set 에서 model을 조정하는 과정 (머신러닝의 학습)\n",
        "- generalizaion : 이전에 본 적 없는 data에서 model이 얼마나 잘 수행되는지  "
      ],
      "metadata": {
        "id": "KDbydVSiQqjj"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oeir4Af1VORi"
      },
      "source": [
        "### Underfitting and overfitting"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "- underfitting : loss of training data가 낮아질수록 loss of test data도 낮아짐  \n",
        "  -> training datta에 있는 관련 특성을 모두 학습하지 못함 -> 개선될 여지 有\n",
        "- overfitting : training data로 반복학습을 거치면 일반화 성능이 더이상 높아지지 않음  \n",
        "  -> sol1) more training data  \n",
        "  -> sol2) regulazaiton"
      ],
      "metadata": {
        "id": "zpeZrxy4Pcoz"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SForidrBVORj"
      },
      "source": [
        "#### Noisy training data"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "overfitting is particularly likely to occur when your data in noisy  \n",
        "In real world datasets, fairly common for some inputs to be invalid"
      ],
      "metadata": {
        "id": "UErIuiTXSkQS"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0eRIIFfMVORj"
      },
      "source": [
        "#### Ambiguous features"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "ex) Model that takes an image of a banana and predicts whether the banana is unripe, ripe or rotten. -> have no objective boundary  \n",
        "\n",
        "MORE ROBUST FIT -> would ignore individual data points and look at the bigger picture"
      ],
      "metadata": {
        "id": "p_8bOwbjTRjj"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sqy4_mwAVORk"
      },
      "source": [
        "#### Rare features and spurious correlations"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "rare feature values가 포함된 dataset으로 훈련된 overffiting 되기 쉽다.\n"
      ],
      "metadata": {
        "id": "Wfgp56EfUmMp"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jkYYN2PIVORl"
      },
      "source": [
        "**Adding white-noise channels or all-zeros channels to MNIST**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T0uS0PN7VORm"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras.datasets import mnist\n",
        "import numpy as np\n",
        "\n",
        "(train_images, train_labels), _ = mnist.load_data()\n",
        "train_images = train_images.reshape((60000, 28 * 28)) \n",
        "train_images = train_images.astype(\"float32\") / 255\n",
        "\n",
        "train_images_with_noise_channels = np.concatenate(\n",
        "    [train_images, np.random.random((len(train_images), 784))], axis=1) # random으로 noise만들기 \n",
        "\n",
        "train_images_with_zeros_channels = np.concatenate(\n",
        "    [train_images, np.zeros((len(train_images), 784))], axis=1) # all zero 만들기 (왜?????)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yMzD38E6VORo"
      },
      "source": [
        "**Training the same model on MNIST data with noise channels or all-zero channels**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rd3wQqU-VORo",
        "outputId": "9272fd5d-9135-42d3-c40a-1bf5f9fd5397",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "375/375 [==============================] - 7s 18ms/step - loss: 0.6232 - accuracy: 0.8098 - val_loss: 0.2871 - val_accuracy: 0.9134\n",
            "Epoch 2/10\n",
            "375/375 [==============================] - 7s 18ms/step - loss: 0.2491 - accuracy: 0.9225 - val_loss: 0.2022 - val_accuracy: 0.9399\n",
            "Epoch 3/10\n",
            "375/375 [==============================] - 7s 17ms/step - loss: 0.1600 - accuracy: 0.9509 - val_loss: 0.1909 - val_accuracy: 0.9366\n",
            "Epoch 4/10\n",
            "375/375 [==============================] - 7s 17ms/step - loss: 0.1135 - accuracy: 0.9645 - val_loss: 0.1449 - val_accuracy: 0.9563\n",
            "Epoch 5/10\n",
            "375/375 [==============================] - 7s 17ms/step - loss: 0.0813 - accuracy: 0.9745 - val_loss: 0.1233 - val_accuracy: 0.9652\n",
            "Epoch 6/10\n",
            "375/375 [==============================] - 7s 18ms/step - loss: 0.0616 - accuracy: 0.9806 - val_loss: 0.1362 - val_accuracy: 0.9614\n",
            "Epoch 7/10\n",
            "375/375 [==============================] - 9s 24ms/step - loss: 0.0450 - accuracy: 0.9855 - val_loss: 0.1526 - val_accuracy: 0.9604\n",
            "Epoch 8/10\n",
            "375/375 [==============================] - 9s 23ms/step - loss: 0.0334 - accuracy: 0.9894 - val_loss: 0.1318 - val_accuracy: 0.9681\n",
            "Epoch 9/10\n",
            "375/375 [==============================] - 7s 17ms/step - loss: 0.0258 - accuracy: 0.9918 - val_loss: 0.1333 - val_accuracy: 0.9683\n",
            "Epoch 10/10\n",
            "375/375 [==============================] - 7s 18ms/step - loss: 0.0192 - accuracy: 0.9935 - val_loss: 0.1286 - val_accuracy: 0.9696\n",
            "Epoch 1/10\n",
            "375/375 [==============================] - 7s 18ms/step - loss: 0.2871 - accuracy: 0.9162 - val_loss: 0.1501 - val_accuracy: 0.9603\n",
            "Epoch 2/10\n",
            "375/375 [==============================] - 7s 17ms/step - loss: 0.1186 - accuracy: 0.9650 - val_loss: 0.1032 - val_accuracy: 0.9685\n",
            "Epoch 3/10\n",
            "375/375 [==============================] - 7s 18ms/step - loss: 0.0778 - accuracy: 0.9766 - val_loss: 0.0999 - val_accuracy: 0.9683\n",
            "Epoch 4/10\n",
            "375/375 [==============================] - 7s 17ms/step - loss: 0.0566 - accuracy: 0.9827 - val_loss: 0.0876 - val_accuracy: 0.9741\n",
            "Epoch 5/10\n",
            "375/375 [==============================] - 7s 17ms/step - loss: 0.0427 - accuracy: 0.9873 - val_loss: 0.0777 - val_accuracy: 0.9778\n",
            "Epoch 6/10\n",
            "375/375 [==============================] - 7s 18ms/step - loss: 0.0324 - accuracy: 0.9906 - val_loss: 0.0843 - val_accuracy: 0.9772\n",
            "Epoch 7/10\n",
            "375/375 [==============================] - 7s 18ms/step - loss: 0.0247 - accuracy: 0.9926 - val_loss: 0.0823 - val_accuracy: 0.9772\n",
            "Epoch 8/10\n",
            "375/375 [==============================] - 7s 17ms/step - loss: 0.0191 - accuracy: 0.9945 - val_loss: 0.0778 - val_accuracy: 0.9787\n",
            "Epoch 9/10\n",
            "375/375 [==============================] - 7s 18ms/step - loss: 0.0143 - accuracy: 0.9960 - val_loss: 0.0806 - val_accuracy: 0.9797\n",
            "Epoch 10/10\n",
            "375/375 [==============================] - 7s 17ms/step - loss: 0.0109 - accuracy: 0.9969 - val_loss: 0.0888 - val_accuracy: 0.9780\n"
          ]
        }
      ],
      "source": [
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "def get_model():\n",
        "    model = keras.Sequential([\n",
        "        layers.Dense(512, activation=\"relu\"),\n",
        "        layers.Dense(10, activation=\"softmax\")\n",
        "    ])\n",
        "    model.compile(optimizer=\"rmsprop\",\n",
        "                  loss=\"sparse_categorical_crossentropy\",\n",
        "                  metrics=[\"accuracy\"])\n",
        "    return model\n",
        "\n",
        "model = get_model()\n",
        "history_noise = model.fit(\n",
        "    train_images_with_noise_channels, train_labels, # noise channel에 대한 label은???????\n",
        "    epochs=10,\n",
        "    batch_size=128,\n",
        "    validation_split=0.2)\n",
        "\n",
        "model = get_model()\n",
        "history_zeros = model.fit(\n",
        "    train_images_with_zeros_channels, train_labels,\n",
        "    epochs=10,\n",
        "    batch_size=128,\n",
        "    validation_split=0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JBtwMZ4BVORq"
      },
      "source": [
        "**Plotting a validation accuracy comparison**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f6-UngpjVORq",
        "outputId": "6cc5b10c-4375-486f-a970-4101ec52b7b2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 313
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7f87b1bb0250>"
            ]
          },
          "metadata": {},
          "execution_count": 7
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3hUVfrA8e9L6AIWQBakKyI1BAJSpAgqKK4IIoqVxV7Wn3UFu6hrgV0RZHVRAcUCVhYFRKUICCKhxEKvEpqAUkKRlPf3x7kTJmGSDCGTO0nez/PMk5lb37kzue/cc849R1QVY4wxJqsSfgdgjDEmOlmCMMYYE5IlCGOMMSFZgjDGGBOSJQhjjDEhWYIwxhgTkiWICBCRZ0Vkl4hs9173FpHNIpIsInE+xhWROESktrfNmPzaZi77GycizxbEvo6HiGwUkQv8jiMvgmMXkUdE5M1wls3DfjqKyKq8xmkKliWIPPD+QQ55J8XA41VvXm3gAaCxqv7FW2UYcLeqVlDVpSewXxWRs04g9HyJIytV/dXbZlp+bdP4R1X/qao358e2sn5nVXWuqjbMj22byCvpdwCF2F9V9ZsQ02sDu1X1t6BpdYBfCiasHEVLHMYUKSJSUlVT/Y4jv9kVRD7yLru/Bmp4VxUfiEgyEAMkisg6b7kaIvKJiOwUkQ0ick/QNmK8S/x1IrJfRBaLSC0RmeMtkuht+6oQ+y8hIo+JyCYR+U1E3hGRk0WkTKg4QqyvInK7iKwRkT0iMkpEJKdte/PqeuuW9F4PEJH1XvwbROTaoH0MFJEVIvKHiEwXkTo5HM/zRGS+F8tmERkQNPtUEZni7WOhiJwZtN4r3vL7vOPXMWjeUyLyoRf/fhH5RUTig+ZvFJEHReRHEdkrIhNFpGzQ/EtFZJkX03wRaZ5N7G1EJMGLYYeI/DuH93mLiKwVkd9FZLKI1AjnM8myjRreVe1pQdPixBV1lhKRM0Vkpojs9qa9JyKnZBPPUyLybtDr673PfbeIPBrifS7wYtsmIq+KSGlv3jHfWRHpIiJJQes3EpHZ3vq/iMhlQfPGee835OccIu6PRGS797nNEZEmQfPKici/vPexV0TmiUg5b17I75kX181B2xggIvOyfDZ3icgaYI03LafvXnb/26NE5F9Z3stkEbkvu/daYFTVHsf5ADYCF2QzrwuQlGWaAmd5z0sAi4EngNJAfWA90N2b/xDwE9AQECAWqJx1O9nseyCw1ttmBeBTYHyoOLJZX4EvgFNwV0I7gR65bRuo661bEjgJ2Ac09OZVB5p4z3t522jkLfsYMD+bWOoA+4H+QCmgMtDCmzcO2A208bbzHjAhaN3rvOVL4or7tgNlvXlPAYeBS3AJ83ng+yyf7Q9ADeA0YAVwuzcvDvgNONdb90Zv+TJZvxfAAuB673kFoG0277MrsAtoCZQBRgJzwvlMQmxrJnBL0OuhwOve87OAC719VAXmAMNDfae9Y/Su97wxkAx08tb9N5AatGwroK13rOt6x+ve7L5zBP1/eJ/rWuAR3P9CV+8zD3x3cvycs/n+V/TiHA4sC5o3CpgNnOF9du295XL6ns0Gbg7axgBgXpb39rX3PSkXxncv5P+29/62AiW85aoAB4Fqvp/r/A6gMD68f6ZkYE/Q45as/wBZvkiBBHEu8GuW+YOBsd7zVUCvbPab2wl+BnBn0OuGQApQMsz1FTgv6PWHwKDcts2xCWIPcEXgnyZonWnATUGvS3j/CHVCxDIY+CybOMcBbwa9vgRYmcP7+gOI9Z4/BXwTNK8xcCjLZ3td0OuXOHqSfQ14Jsu2VwGdg9YNnDjnAE8DVXL5Lr0FvBT0uoJ3XOvm9pmE2NbNwEzvuQCbgU7ZLHs5sDTL+w6VIJ4gc/I9CThC9j+Q7g3+3LJ+58icIDriTqAlguZ/ADyVl885SxynePs+2fueHQp8B47jezab3BNE11ziCP7u5fS/vQK40Ht+NzA1nPcZ6YcVMeXd5ap6StDjjTDXq4MrgtoTeOB+QVXz5tcCQhYBhaEGsCno9SbcSbta6MVD2h70/CDuhBX2tlX1AHAVcDuwzSseOMebXQd4Jeh9/447kZ0RIo7cjkN2ceIVEa3wihL24E4SVXJYt6x4xWO5bLsO8ECWz64W7thkdRNwNrBSRBaJyKXZvI9Mx1VVk3G/moOPSbbvNYtPgHYiUh33iz8dmAsgItVEZIKIbBGRfcC7ZD4m2amBSzSB+A548eFt92wR+cIr2tkH/DPM7WZsW1XTg6ZtIg/v3Su+ecErvtmHS3h4sVQByhL6+3Qi/28QdGy8OHL67uW0r7dxVx94f8efQEz5xhJEwdsMbMiSXCqq6iVB87MtZ83FVtxJLKA2rjhgR97DPf5tq+p0Vb0QV7y0Eggkz83AbVneezlVnR9if3k6Dl6Z7z+AfsCpqnoKsBeXiE7UZuC5LPGXV9UPsi6oqmtUtT9wOvAi8LGInBRim5mOq7dMZWDL8Qanqn8AX+ES9DW4X/7qzf4n7hdvM1WthDsJhXNMtuFObIH4ynvxBbyG+4wbeNt9JMztgnvvtUQk+DxUmzy8d9z77QVcgDsp1w2EjCvCO0zo71NO37MDQPmg138JsUzg+Ibz3ctpX+8CvUQkFlcEOymb5QqUJYiC9wOwX0Qe9irOYkSkqYi09ua/CTwjIg3EaS4igX/IHbg6gOx8ANwnIvVEpALupDBR86d1RVjb9n6p9vJOdH/iiuICvxBfBwYHKg/FVaBfmc3+3gMuEJF+IlJSRCqLSIsw4qyIS1w7gZIi8gRQ6Tjfa3beAG4XkXO9z+YkEekpIhWzLigi14lIVe/X8R5vcnrW5XDH9W8i0kJEyuCO60JV3ZjHGN8HbgD6es8DKuI+i70icgauPDwcHwOXehW5pYEhZD5vVMTVOSV7V4p3ZFk/p+/sQtxVwT/EVaR3Af4KTAgztmAVcd+33biT+j8DM7zPYAzwb3GV+TEi0s473jl9z5YBfUSkvLimujeFEUNO371s/7dVNQlYhLty+ERVD+XhGOQ7SxB597lkvg/is3BWUnevwKVAC2AD7tfNm7hfPeAqAT/E/RLchyujLufNewp42yve6Bdi82NwX7A53rYPA3/Pw3sLJdxtlwDux/06/B3ojHfSUNXPcL+mJ3jFAD8DF4famar+iitzfsDbzjJcpV5upgNfAqtxxRWHyVIMkFeqmgDcAryKK1teiyuXDqUH8Iu41mOvAFeH+qdX11T6cVzx0DbcL8yrTyDMyUADYLuqJgZNfxpXEb4XmIJrZJArVf0FuAuXbLbh3ndS0CIP4n6978cl0IlZNvEU2XxnVfUILiFcjPs/+A9wg6quDCe2LN7Bfd5bgOXA91nmP4irIF6E+z69iKv7yOl79jKuvmUHrgjovVxiyO27l9P/Nt4+mhElxUsAcvQK1BhjjF9EpBOuqKmORsmJ2a4gjDHGZyJSCvg/XKutqEgOYAnCGGN8JSKNcPVU1XH3b0QNK2IyxhgTkl1BGGOMCanIdNZXpUoVrVu3rt9hGGNMobJ48eJdqlo11LwikyDq1q1LQkKC32EYY0yhIiKbsptnRUzGGGNCsgRhjDEmJEsQxhhjQrIEYYwxJqSIJggR6SEiq8SNljUoxPw6IjJD3Ohds0WkZtC8l8SNMLVCREaIHDuKljHGmMiJWIIQkRjcKE4X4wZl6S8ijbMsNgx4R1Wb43qJfN5btz3QAWgONAVa4zp9M8YYU0AieQXRBlirquu9Xhsn4PprD9YYN0wiwKyg+Yob4KM0bljAUuTPmAbGGGPCFMn7IM4gc1e3SbjhNoMlAn1w3SH3BiqKSGVVXSAis3DdCwvwqqquyLoDEbkVuBWgdu3a+f8OjDHFniokJ8POnbBrl/u7cyecfz7UqQN798KmTXDWWVC+fO7bK0z8vlHuQeBVERmAG2dgC5DmDc7RCAjUSXwtIh1VdW7wyqo6GhgNEB8fb51KGWNylZ4Ov/8OpUrBySfDH3/Ahx8ePfEHEsH//R/07Anz58N55x27nYkTXYL49lvo5ZV91KwJZ58NDRrAQw/BmWfCoUNQsqTbX2ETyQSxhaChCnEn+0xDCarqVtwVBN4oZVeo6h4RuQX43hufFxGZBrTDG1/XGGMCVGHLlsy/8HftgmbN3K/8P/5wJ/DAvN9/d0nihRfg4YfdFcDtt7ttVaoEVapA1aqQkuKmNWgAL710dHrgUcMbibxNG5csVq+GNWvc348+cgkGYOxYuOceqFfPbSuQQG64ASoeMxZhdIlYb67iBoFfDXTDJYZFwDXeCFWBZaoAv6tquog8B6Sp6hMichVu5K4euCKmL4Hhqvp5dvuLj49X62rDmKJpxgxYtw42bID1612RzoUXwjPPQFqa+3We9VR2990wciQcOeKWDT65V63qrgri4tz627e7BFCmTP7FrAoi8MMPMHny0eSxZg0cOOASU6VK8PTTLqEEEkfgb8eObv1IE5HFqhofal7EriBUNVVE7sYNwxcDjFHVX0RkCJCgqpOBLsDzIqK4Iqa7vNU/BrrihghU4MuckoMxpnBbtgxWrTqaANavdyfJ115z82+80V0llCrlinXq1nUndICYGBgzxhUXBf/KP/VUN790aVcMlJ2YGDjjjPx/T4GTe5s27hGg6hJSJW+06vr1XVHUypXwxRfuyiVQ9AXw5JMusZx9duYkcsop+R/zMe+hqIwHYVcQxkSvjRvdCTA4AZQpA++/7+a3awffe6NIn366O2l26QLPP++mLV0KlSu7E3lMjB/voGCkpcGvv8K2bdC+vZt2773uCmTTJlc0BtCkCfz8s3v+/PNw550uqeRFTlcQliCMCWHPHvj8c/crL1ABecMN7tdoxYruUamS+2XYqZP7VThnztF5gUf58gVTTOC33bvdr9zgBLB7N/zvf25+v36uGAWgbFlXHt+ixdEEsXixO7b16kGFCv68h2j355/uuK5Z4xLF5Ze76e3awdy5riI8LyxBGBOGnTth0iT49FNX5p2SApdd5k5y6enQuDHs2wf797tmjwD33Qf//rd7HarC8bHHXDn57t2uzLtSpcwJ5NproXt3V5zw7rtHE09gfoMG7pdzWport05Pz/yoVMmdcA8fdvEHpqu6vzVquCS1b5/7ZZp1fsOG7oT8229HTzzBy7Rp4+Zv3AjLl7vy/F9/PZoIJkyAcuXggQfccQioUcNdBcyY4U78S5bAwYNu2l/+AiWsk5+o4UsdhDGFwe7d7gQM7mT99dfuJHbvvXDFFdC6tZtXooQrIglIT3cn7IAyZdzJcP/+zI927Y4u37Tp0QSzY4f726mTm795s2vpktVbb8HAgbBwIXTocOz8jz92cX77LfTocez86dPhoovc++rb99j5333nijKmToW//e3Y+T/+6FoDff555vhOOskdp927XdPOG2+Erl3dtLp1XdII1rLlsds20c+uIEyxs2EDfPKJeyxaBFu3unLvH35wv3ZjYwu+WCgtzV1F7N9/NIns3++SSq1aLsYJE1yiCn507+4qOJOSXDIoUcLFHpjftav7NZ+U5Mr4s87v0AFOO81VAC9ffnR6YJlWrVwy2L7dXTmULOkSQtWqxaPorDiwIiZjcAng9ttdhSe4Jo5XXAG33Xa0RYwxxY0VMZliRxUSE91VQtu27o7YatVcUdDQodCnjysOMcZkzxKEKTJUXVn9p5+6xLB+vSsmeeQRlyDq1IEFC/yO0pjCwxKEKdTS0lzzykaNXJn4zTe71926weDBrolq1ap+R2lM4WQJwhQ6KSkwc6a7Upg0yTWf3LnTNfd8/32oXbtg7jI1pqizBGEKlYkTXUXznj2ufX7Pnq4+IdCuvnlzf+MzpiixBGGiVnKya5//6adwxx3QubPrc/+yy1zro4suclcNxpjIsARhos66dfDss/DBB657gdNPh7/+1c1r1Qreftvf+IwpLixBmKiSkuK6Of7jD3cH8VVXuS4qinIHbcZEK0sQxne//gqjR8NTT7nunN9917VKql7d78iMKd6syyzjmy1b4K67XL3CSy+5Hj3BdQ9hycEY/1mCMAXuwAE3HOOZZ7orh4EDYe1aOPdcvyMzxgSzIiZTYFJSXBFS2bIwaxZcd53rDrtuXb8jM8aEYgnCRNzOna7/o4kT3ShYFStCQoLrOdUYE72siMlEzO7drh+kevXgX/9yYx8cPOjmWXIwJvrZFYSJiKQkNwJbcrJrqvrEE65lkjGm8LArCJNv9u51dz6DG1TmoYfciGQffGDJwZjCKKIJQkR6iMgqEVkrIoNCzK8jIjNE5EcRmS0iNb3p54vIsqDHYRG5PJKxmrzbvx+ee84VJfXpA7t2uemPP+5GRDPGFE4RSxAiEgOMAi4GGgP9RaRxlsWGAe+oanNgCPA8gKrOUtUWqtoC6AocBL6KVKwmb5KT4cUXXWJ47DF3x/P8+TY6mzFFRSSvINoAa1V1vaoeASYAvbIs0xiY6T2fFWI+QF9gmqoejFikJk+2b4dHH4U2bdxwnpMn2+D0xhQlkaykPgPYHPQ6Cch6K1Qi0Ad4BegNVBSRyqq6O2iZq4F/h9qBiNwK3ApQu3btfArbZOfQIfjvf+GXX+CNN9wd0KtX29CdxhS0AwdcTwSbN7sGIeXKQb9++b8fv1sxPQi8KiIDgDnAFiAtMFNEqgPNgOmhVlbV0cBogPj4eI10sMXV4cPw5pvwz3/Ctm2uK4zDh90Nb5YcjMlfycnupB84+Qcewa//+CPzOi1aFL4EsQWoFfS6pjctg6puxV1BICIVgCtUdU/QIv2Az1Q1JYJxmhz88IMbeyEpyfWy+v770KWL31EZUzjt2xf6hB/8eu/eY9erWhVq1XL1fR07ulaCtWq5vzVrwhlnRCbeSCaIRUADEamHSwxXA9cELyAiVYDfVTUdGAyMybKN/t70Ik8V1qxx4x+ULOke9eq5v8nJ7gazmJij82JioEwZNw5zfktJga1boU4dV4zUpAmMHevGeY7E/owp7FTdiT27k39g2v79x65brZo7yTdoAOeff/SkH0gANWr4NzBWxBKEqqaKyN244qEYYIyq/iIiQ4AEVZ0MdAGeFxHFFTHdFVhfROrirkC+jVSM0WLPHjeM5sSJmacnJblfBv/6l+sKO6u9e6FSJXj4YRg+/GjyCCSQ7dvdUJyPPQYffpg5wVSsCHPmuO0895zrGykwb/lyN6bz4sVw2mnw5ZcRPwTG5IkqpKXBkSPukZIS/vPjWTa75/v2Ha0LOHAgc2wi8Je/uJP8OefABRccPfkHEkCNGtHdq0BE6yBUdSowNcu0J4Kefwx8nM26G3EV3UXeL7/ApEnubuPmzSE11T1OO83Nv+QSd4kZmJ6W5v4GflV07uy+jIHpgUdgnOZ69SA+PvO6pUod3X9qqqtTCKxXvz7cd1/BHgNjwpGS4urCXnnFnZCPHIns/kqWdCfw0qXd/0zW5yed5K6we/Q49uRfvXrm/7PCSFSLRt1ufHy8JiQk+B1G2FJT3a/2Cy90r7dtszEQjMnJzz/DjTfCkiVw+eXQsOGxJ+xQJ/G8Pi9VqngUqYrIYlWNDzXP71ZMxdKmTa6r63nzIDHRXTVYcjAmtNRUGDYMnnwSTj4ZPv0Uevf2O6riwRJEAfvoI7jlFkhPh/HjXXIwxoS2apW7ali4EPr2hf/8xxW3moJhnfUVoLvvdm2VGzaEpUvdVYQx5ljp6fDyy659/5o1rsPHDz+05FDQLEEUoCZN3PgI8+a54TaNMcdat87da3P//a6O7uef4eqri0d9QLSxIqYICvwKqlnTjYlwxx1+R2RM9FKF11933cTHxMC4cXDDDZYY/GRXEBGybZtr+vbggzBtmt/RGBPdfv0VLroI7rwTOnQ42mLJkoO/LEFEwJQprvJ53jzXud3YsX5HZPJq926X7E1kqMKYMdCsGSxY4K4gvvzS3Udg/GcJIp8tXQqXXurukExIgFtvtV9BhdUnn0Dduu6z7NDB3a2+eXOuq5kwbd0Kf/0r3HQTxMXBTz/BbbfZ/0s0sQSRTwJ9rMTFuRYXCxe6MZlN4ZOa6srB+/Z1I+I984y7a/e++6B2bWjXDv79b1csYo6fquv0sWlTmDnT3RU9c6a7499EF0sQJyhQsVa7tht/GVyLC7861zInZscO13Jm2DC46y749lvXl9WyZW7si3/+03Wo+MADrjPDtm3dshs3+h154fDbby7xXnut659o2TK4556j3cKY6GIfywnYvduNwXzHHW5UtdNP9zsicyLmz3cj4i1c6G5ifPXVzB2pNWgAgwe7rh7WrIEXXjh6tVGvnvsOvPQSbNjg33uIZp984q4avvjCDVU7dy6cfbbfUZmcWILIo1mzXEX0lCmut9Vp01zPjabwUYWRI12nh+XLw/ff534T41lnuV50ExJcu/0XX3TTH37YdXYYH+8SyLp1kY8/2v3+u7ti6NvXXWkvWQL/+IdrymqimyWIPJo+3XWZvXChu6HHLpELpwMHXDK45x7Xa+6iRcff/Un9+u6E98MP7uph6FB38hs82CWSli1d0dSaNZF5D9FsyhR31fDhhzBkiGup1KSJ31GZcFlvrsdh3TpXrNSmzdH+4E86KaK7NBG0erUbLW/5cnj2WffrPz8T/aZNrljlo4/cVQlAbCxceaX7Nd2wYf7tK9rs3esq9ceOdU1Y337bNeAw0Sen3lztd2+Yxo93/cLcfLMrkihVypJDYfbZZ9C6tRtUafp092s/v68C69RxV5cLFrgWTy+/7L4zjz3mKmibN3ctpFasyN/9+u3rr48mhUcecVdllhwKJ0sQudi71xVB3HCDKyqYMsXaaRdmqakwaJBrXNCwoRs174ILIr/fWrXg3nvhu+/cvRSvvOK6rn7ySdccumlTePppN3hUYZWc7O6EvugilwgXLHCjFZYp43dkJs9UtUg8WrVqpfktKUm1Xj3VmBjVIUNUU1PzfRemAO3Yodq1qyqo3nab6uHDfkekumWL6ogRqp06qYq42Bo1Un3iCdWfflJNT/c7wvB8+61q/fruPTzwgOrBg35HZMKFGwI65HnVriByUL06dOvmxm5+/HFrdVGYff+9uwKcP9+Vi7/+enT8sq1RA/7+d3e/xZYtrmlttWquTqRZM2jUyH33EhNd54/R5tAhV9fQpYu7sp4zx90XUq6c35GZ/GCV1FkkJbkWLa+8Yv3BFAWq8NprrninZk03GlmLFn5HlbsdO1ysH33kkkd6uqsjqVrVJZDTT3d/g58H/z399MgnwO+/dx3qrV7txjp54QWrlyuMbMjRMH36qauEPnLElQVbgijcDh6E2293DQx69nR/Tz3V76jCU62auwHzjjvc3cdffAHr17vE8dtv7u+6de7vwYOht3HKKccmj+wSSsWK4det/fmnqzsZOtQl3W++cVfapuiJaIIQkR7AK0AM8KaqvpBlfh1gDFAV+B24TlWTvHm1gTeBWoACl6jqxkjEeeCAa20yerS7wemDD1z7dVN4rV3rmrD+9JNrf//oo4X3XpXTT4eBA7Off+CASxTBySPr3+XL3c2dv/8eehtly4ZOHlmn7d3rku7PP7sfU//6F1SqFJn3bfwXsQQhIjHAKOBCIAlYJCKTVXV50GLDgHdU9W0R6Qo8D1zvzXsHeE5VvxaRCkDESmCHDIE33nDt4IcMydy9gil8Pv8crr/e1RlNmwbdu/sdUWSddJK7Wa9+/dyXPXIEdu3KOZkkJbm7nX/7zbX6yqpGDZg6FS6+OP/fi4kukbyCaAOsVdX1ACIyAegFBCeIxsD93vNZwCRv2cZASVX9GkBVkyMYJ4884gb3Of/8SO7FRFpamiv6eO45aNUKPv7Ydddtjipd2p3ga9TIfdn0dNizJ3Py2L/fNREuLEV15sREMkGcAQT3np8EnJtlmUSgD64YqjdQUUQqA2cDe0TkU6Ae8A0wSFXTglcWkVuBWwFq166d50BPPtmSQ2G3axdcc427Sevmm13fStaj7okpUQJOO809GjXyOxrjB79LZR8EOovIUqAzsAVIwyWujt781kB9YEDWlVV1tKrGq2p81apVCyxoE10WLXJXDHPmwJtvuuJCSw7GnLhIJogtuArmgJretAyqulVV+6hqHPCoN20P7mpjmaquV9VUXNFTywjGagohVdew4LzzXAuc775zo5MZY/JHJBPEIqCBiNQTkdLA1cDk4AVEpIqIBGIYjGvRFFj3FBEJXBZ0JXPdhSnmDh1yLXtuuw26dnVdZrRq5XdUxhQtEUsQ3i//u4HpwArgQ1X9RUSGiMhl3mJdgFUishqoBjznrZuGK16aISI/AQK8EalYTeGyfj20bw/jxrlK6S++gMqV/Y7KmKLH7qQ2hcrUqW7wGYD33nNjOBhj8s66+zaFXqAJa8+erunq4sWWHIyJNOtqw0S93btdl+tffgkDBsB//mOdwRlTECxBmKi2eLHrMmPbNvjvf+GWW2w8DmMKihUxmaj11lvQoYO7o3fePLj1VksOxhQkSxAmKj37rLsjulMn1y9Q69Z+R2RM8WMJwkSddevcWM1XXeU626tSxe+IjCmeLEGYqPPgg65TuZdftlH8jPGTJQgTVWbOhEmTXA+71av7HY0xxZslCBM1UlPd0KD16rlxjo0x/rJmriZqvPmmGwHu44+tN1ZjooFdQZiosGcPPP44dO7sBqQxxvjPEoSJCkOGuDumhw+3ex2MiRaWIIzvVq92I8DdfDO0aOF3NMaYAEsQxncPPOD6VnrmGb8jMcYEyzVBiMhfgwb1MSZfffWVG8/h8cehWjW/ozHGBAvnxH8VsEZEXhKRcyIdkCk+UlNdc9Yzz4R77vE7GmNMVrk2c1XV60SkEtAfGCciCowFPlDV/ZEO0BRdr78Oy5e7G+PKlPE7GmNMVmEVHanqPuBjYAJQHegNLBGRv0cwNlOE/f67GwCoWze47LLclzfGFLxw6iAuE5HPgNlAKaCNql4MxAIPRDY8U1Q99ZS79+Hll61ZqzHRKpw7qa8AXlbVOcETVfWgiNwUmbBMUbZ8uRsV7rbboFkzv6MxxmQnnATxFLAt8EJEygHVVHWjqs6IVGCmaFKF+8NE0DwAACAASURBVO+HihXdzXHGmOgVTh3ER0B60Os0b1quRKSHiKwSkbUiMijE/DoiMkNEfhSR2SJSM2hemogs8x6Tw9mfiX5Tp8L06a7+wcZ5MCa6hXMFUVJVjwReqOoRESmd20oiEgOMAi4EkoBFIjJZVZcHLTYMeEdV3xaRrsDzwPXevEOqavfVFiFHjrirh4YN4a67/I7GGJObcK4gdopIRjsTEekF7ApjvTbAWlVd7yWYCUCvLMs0BmZ6z2eFmG+KkFGjXLca//43lCrldzTGmNyEkyBuBx4RkV9FZDPwMHBbGOudAWwOep3kTQuWCAT67uwNVBSRyt7rsiKSICLfi8jloXYgIrd6yyTs3LkzjJCMX3btgqefhu7d4eKL/Y7GGBOOcG6UWwe0FZEK3uvkfNz/g8CrIjIAmANswdVxANRR1S0iUh+YKSI/ebEExzYaGA0QHx+v+RiXyWdPPAHJye7qwZq1GlM4hDVgkIj0BJrgftUDoKq5tUHZAtQKel3Tm5ZBVbfiXUF4CegKVd3jzdvi/V0vIrOBOCBTgjCFw08/wX//6+odGjf2OxpjTLjCuVHudVx/TH8HBLgSqBPGthcBDUSknlepfTWQqTWSiFQJ6ghwMDDGm36qiJQJLAN0AIIrt00hoer6WzrlFHdznDGm8AinDqK9qt4A/KGqTwPtgLNzW0lVU4G7genACuBDVf1FRIYEVXp3AVaJyGqgGvCcN70RkCAiibjK6xeytH4yhcTkyTBjhqt/OO00v6MxxhwPUc256F5EflDVNiLyPa44aDfwi6qeVRABhis+Pl4TEhL8DsME+fNPaNLEdcSXmAglbQR0Y6KOiCxW1fhQ88L5l/1cRE4BhgJLAAXeyMf4TBE1YgSsW+dujLPkYEzhk+O/rVc/MMOrOP5ERL4Ayqrq3gKJzhRaO3a4EeIuvRQuusjvaIwxeZFjHYSqpuPuhg68/tOSgwnHY4/B4cPwr3/5HYkxJq/CqaSeISJXiFjrdROepUvhrbfg73+Hs3NtzmCMiVbhJIjbcJ3z/Ski+0Rkv4jsi3BcppBShXvvhcqV3TjTxpjCK5w7qSsWRCCmaPjkE5gzB157zd37YIwpvHJNECLSKdT0rAMIGXP4MDz0kBsE6Oab/Y7GGHOiwml8+FDQ87K4XloXA10jEpEptF5+GTZudDfGWbNWYwq/cIqY/hr8WkRqAcMjFpEplLZtg+eeg8svh67208GYIiGcSuqsknBdYRiT4ZFHICUFhg3zOxJjTH4Jpw5iJO7uaXAJpQXujmpjAEhIgHHj4B//gDPP9DsaY0x+CaekOLiDo1TgA1X9LkLxmEIm0Ky1WjV49FG/ozHG5KdwEsTHwGFVTQM31rSIlFfVg5ENzRQGEyfCd9/Bm29CpUp+R2OMyU9h3UkNlAt6XQ74JjLhmMLk4EFXrBQXBwMG+B2NMSa/hXMFUTZ4mFFVTRaR8hGMyRQSw4bB5s3w3nsQE+N3NMaY/BbOFcQBEWkZeCEirYBDkQvJFAZJSfDii3DlldCxo9/RGGMiIZwriHuBj0RkK27I0b/ghiA1xdigQZCWBi+95HckxphICedGuUUicg7Q0Ju0SlVTIhuWiWbff++KlR55BOrW9TsaY0yk5FrEJCJ3ASep6s+q+jNQQUTujHxoJhqlp7tmrdWrw+DBfkdjjImkcOogbvFGlANAVf8AbolcSCaavf8+LFwIzz8PFSr4HY0xJpLCSRAxwYMFiUgMUDqcjYtIDxFZJSJrRWRQiPl1RGSGiPwoIrNFpGaW+ZVEJElEXg1nfyayDhxwdQ+tW8P11/sdjTEm0sJJEF8CE0Wkm4h0Az4ApuW2kpdIRgEXA42B/iLSOMtiw4B3VLU5MAR4Psv8ZwDrVjxKvPgibNkCw4dDibz04mWMKVTC+Td/GJgJ3O49fiLzjXPZaQOsVdX1qnoEmAD0yrJMY2/bALOC53vNaasBX4WxLxNhmzbB0KHQvz+0b+93NMaYgpBrglDVdGAhsBF30u8KrAhj22cAm4NeJ3nTgiUCfbznvYGKIlJZREoA/wIeDGM/RcKKFdC0KVx3HUyYAH/84XdEmT38MIi4qwhjTPGQbTNXETkb6O89dgETAVT1/Hzc/4PAqyIyAFeUtAVIA+4EpqpqUlD1R6gYbwVuBahdu3Y+hlXwhg2DtWthx46jdya3bw+XXgo9e0Ljxu4E7Yd581yfS08+CbVq+RODMabgiaqGniGSDswFblLVtd609apaP6wNi7QDnlLV7t7rwQCqmrWeIbB8BWClqtYUkfeAjkA6UAFXKf4fVT2mojsgPj5eExISspsd1XbtcifeG2+EUaNg0SL44guYMgWWLXPL1KlzNFl06QLlwinkywfp6dCmjUtcq1ZBeetkxZgiRUQWq2p8qHk5FTH1AbYBs0TkDa+C+nh+wy4CGohIPREpDVwNTM4SWBWvOAlgMDAGQFWvVdXaqloXd5XxTk7JobB78003nvPdd7srh7Zt4dlnYelS19fRf/8LsbEwdixccglUrgx//aubvnlz7ts/EW+/DYsXu6IlSw7GFC/ZXkFkLCByEq7yuD+u/uEd4DNVzbXyWEQuwQ1PGgOMUdXnRGQIkKCqk0WkL67lkuKKmO5S1T+zbGMAEK+qd+e0r8J6BZGaCvXrQ4MGbiznnBw+DN9+e/TqYsMGN715c3dlcemlcO65+ddx3v79cPbZ7m7p+fP9K+IyxkROTlcQuSaILBs6FbgSuEpVu+VTfPmisCaITz6Bvn3hs8/ceM7hUoWVK48mi3nzXN9IlStDjx4uWXTvDqeemvfYBg+GF15wN8a1aZP37Rhjole+JYhoVlgTRJcusHEjrFt3Yr/89+yBr75yCWPaNFevEajoDlxdHE9F9/r10KgRXHUVvPNO3uMyxkS3vNZBmAj78UdXZHTXXSdeLHTKKdCvnzuZb98OCxa4u57373d/mzaFevXcvqZNg0O5dNj+j39AyZKuSw1jTPFkCcJHI0e61kg33ZS/281a0Z2UBKNHu4ruceMyV3S//vqxFd3ffuuKvgYPhjOy3rlijCk2rIjJJ7t3Q82ark+j0aMLbr+5VXT37OmuMvbscTfvFVRzWmOMP6yIKQq99ZY7Wf/97wW737JlXeX1yJGu3mP5cteFxqmnusF/zjsPEhPdc0sOxhRvdgXhg9RUOPNM17x11iy/ozkqUNH922/uKsKatRpT9OV0BRHOkKMmn33+Ofz6q+sVNZoEKrqNMQasiMkXI0ZA7dquktgYY6KVJYgC9tNPMHu2K8IpaddvxpgoZgmigI0c6SqK87tpqzHG5DdLEAXo99/h3XfdmA+VK/sdjTHG5MwSRAF66y13B3NBN201xpi8sARRQNLS4D//gc6d3U1pxhgT7SxBFJAvvnCd8tnVgzGmsLAEUUBGjHCjxvXq5XckxhgTHksQBeCXX2DmTLjzTmvaaowpPCxBFIBA09abb/Y7EmOMCZ8liAj74w8YPx6uuQaqVPE7GmOMCZ8liAgbMwYOHrTKaWNM4WMJIoLS0mDUKOjYEVq08DsaY4w5PpYgIigwIM899/gdiTHGHL+IJggR6SEiq0RkrYgMCjG/jojMEJEfRWS2iNQMmr5ERJaJyC8icnsk44yUkSPdqHGXX+53JMYYc/wiliBEJAYYBVwMNAb6i0jjLIsNA95R1ebAEOB5b/o2oJ2qtgDOBQaJSI1IxRoJy5fDN99Y01ZjTOEVySuINsBaVV2vqkeACUDW28QaAzO957MC81X1iKr+6U0vE+E4I+LVV6FMGbjlFr8jMcaYvInkifcMYHPQ6yRvWrBEoI/3vDdQUUQqA4hILRH50dvGi6q6NesORORWEUkQkYSdO3fm+xvIqz174O23rWmrMaZw8/uX+YNAZxFZCnQGtgBpAKq62St6Ogu4UUSqZV1ZVUeraryqxletWrUg487R2LHWtNUYU/hFMkFsAWoFva7pTcugqltVtY+qxgGPetP2ZF0G+BnoGMFY801amiteOu88iIvzOxpjjMm7SCaIRUADEaknIqWBq4HJwQuISBURCcQwGBjjTa8pIuW856cC5wGrIhhrvpk2Ddavt6sHY0zhF7EEoaqpwN3AdGAF8KGq/iIiQ0TkMm+xLsAqEVkNVAOe86Y3AhaKSCLwLTBMVX+KVKz5acQIOOMM6N3b70iMMebERLQBpqpOBaZmmfZE0POPgY9DrPc1UOiG1VmxAr7+Gp59FkqV8jsaY4w5MX5XUhcpgaatt97qdyTGGHPiLEHkk717XdPWq6+GKGpQZYwxeWYJIp+MHQsHDljltDGm6LAEkQ/S013xUvv20KqV39EYY0z+sASRD6ZNg3XrrNdWY0zRYgkiH4wcCTVqQJ8+uS9rjDGFhSWIE7RqFUyfDnfcYU1bjTFFiyWIE/Tqq1C6tDVtNcYUPZYgTsC+fTBunGvaevrpfkdjjDH5yxLECRg3DpKTrWmrMaZosgSRR+nprnK6XTuIj/c7GmOMyX+WIPJo+nRYu9auHowxRZcliDwaMQL+8he44gq/IzHGmMiwBJEHq1fDl1+6pq2lS/sdjTHGRIYliDx49VV3z4M1bTXGFGWWII5ToGnrVVe5IiZjjCmqIjpgUFH09tuwf79VTgekpKSQlJTE4cOH/Q7FGJODsmXLUrNmTUodR5cPliCOQ6DX1nPPhTZt/I4mOiQlJVGxYkXq1q2LiPgdjjEmBFVl9+7dJCUlUa9evbDXsyKm4/DVV66C2q4ejjp8+DCVK1e25GBMFBMRKleufNxX+pYgjsPIka7e4cor/Y4kulhyMCb65eX/1BJEmNasgalT4bbbrGmrMaZ4iGiCEJEeIrJKRNaKyKAQ8+uIyAwR+VFEZotITW96CxFZICK/ePOuimSc4Rg1yjVtve02vyMxwc4//3ymT5+eadrw4cO54447sl2nS5cuJCQkAHDJJZewZ8+eY5Z56qmnGDZsWI77njRpEsuXL894/cQTT/DNN98cT/jFVuC479mzh//85z8Z02fPns2ll16a7/tLSEjgngiN6BXOdyWSKlSoELFtRyxBiEgMMAq4GGgM9BeRxlkWGwa8o6rNgSHA8970g8ANqtoE6AEMF5FTIhVrbvbvhzFjXNFS9ep+RWFC6d+/PxMmTMg0bcKECfTv3z+s9adOncopp+Ttq5U1QQwZMoQLLrggT9vyS1pami/7DRz3rAkiUuLj4xkxYkTE91PURPIKog2wVlXXq+oRYALQK8syjYGZ3vNZgfmqulpV13jPtwK/AVUjGGuO3nnHJQkbUjRn994LXbrk7+Pee3PeZ9++fZkyZQpHjhwBYOPGjWzdupWOHTtyxx13EB8fT5MmTXjyySdDrl+3bl127doFwHPPPcfZZ5/Neeedx6pVqzKWeeONN2jdujWxsbFcccUVHDx4kPnz5zN58mQeeughWrRowbp16xgwYAAff/wxADNmzCAuLo5mzZoxcOBA/vzzz4z9Pfnkk7Rs2ZJmzZqxcuXKY2LauHEjHTt2pGXLlrRs2ZL58+dnzHvxxRdp1qwZsbGxDBrkLsrXrl3LBRdcQGxsLC1btmTdunXH/BK/++67GTduXEYMDz/8MC1btuSjjz4K+f4AduzYQe/evYmNjSU2Npb58+fzxBNPMHz48IztPvroo7zyyiuZ4h86dGjGyfi+++6ja9euAMycOZNrr70203EfNGgQ69ato0WLFjz00EMAJCcn07dvX8455xyuvfZaVPWYY9SlSxcefvhh2rRpw9lnn83cuXMB12jib3/7G82aNSMuLo5Zs2YBma9Mvv32W1q0aEGLFi2Ii4tj//79GXG3bt2a5s2bZ/t9+fLLL2nZsiWxsbF069YtY/ry5cvp0qUL9evXz5SILr/8clq1akWTJk0YPXp0xvQKFSrw6KOPEhsbS9u2bdmxYwcAAwYM4J577qF9+/bUr18/4/sUTnzbtm2jU6dOtGjRgqZNm2YckxMRyQRxBrA56HWSNy1YIhAYqLM3UFFEKgcvICJtgNLAuqw7EJFbRSRBRBJ27tyZb4EHC/Ta2rq1a95qostpp51GmzZtmDZtGuCuHvr164eI8Nxzz5GQkMCPP/7It99+y48//pjtdhYvXsyECRNYtmwZU6dOZdGiRRnz+vTpw6JFi0hMTKRRo0a89dZbtG/fnssuu4yhQ4eybNkyzjzzzIzlDx8+zIABA5g4cSI//fQTqampvPbaaxnzq1SpwpIlS7jjjjtCFk2cfvrpfP311yxZsoSJEydmFI1MmzaN//3vfyxcuJDExET+8Y9/AHDttddy1113kZiYyPz586kexmVu5cqVWbJkCVdffXXI9wdwzz330LlzZxITE1myZAlNmjRh4MCBvPPOOwCkp6czYcIErrvuukzb7tixY8bJKSEhgeTkZFJSUpg7dy6dOnXKtOwLL7zAmWeeybJlyxg6dCgAS5cuZfjw4Sxfvpz169fz3XffhXwPqamp/PDDDwwfPpynn34agFGjRiEi/PTTT3zwwQfceOONx7TcGTZsGKNGjWLZsmXMnTuXcuXK8dVXX7FmzRp++OEHli1bxuLFi5kzZ06m9Xbu3Mktt9zCJ598QmJiIh999FHGvJUrVzJ9+nR++OEHnn76aVJSUgAYM2YMixcvJiEhgREjRrB7924ADhw4QNu2bUlMTKRTp0688cYbGdvatm0b8+bN44svvsj4ERBOfO+//z7du3dn2bJlJCYm0qJFi5DH7Xj4fR/Eg8CrIjIAmANsATKueUWkOjAeuFFV07OurKqjgdEA8fHxx/7MyAfffOOGFR0/PhJbL1qCflgWqEAxU69evZgwYULGCe7DDz9k9OjRpKamsm3bNpYvX07z5s1DbmPu3Ln07t2b8uXLA3DZZZdlzPv555957LHH2LNnD8nJyXTv3j3HeFatWkW9evU4++yzAbjxxhsZNWoU93qXQ328wctbtWrFp59+esz6KSkp3H333SxbtoyYmBhWr14NwDfffMPf/va3jBhPO+009u/fz5YtW+jduzfgboYKx1VXHa3Wy+79zZw5MyMZxMTEcPLJJ3PyySdTuXJlli5dyo4dO4iLi6Ny5Uy/6WjVqhWLFy9m3759lClThpYtW5KQkMDcuXPDKuZp06YNNWvWBKBFixZs3LiR884775jlgo/jxo0bAZg3bx5/99qhn3POOdSpUyfj+AV06NCB+++/n2uvvZY+ffpQs2ZNvvrqK7766ivi4uIAdxWzZs2aTAnt+++/p1OnThn3EZx22mkZ83r27EmZMmUoU6YMp59+Ojt27KBmzZqMGDGCzz77DIDNmzezZs0aKleuTOnSpTOuaFq1asXXX3+dsa3LL7+cEiVK0Lhx44wri3Dia926NQMHDiQlJYXLL7886hPEFqBW0Oua3rQMXvFRHwARqQBcoap7vNeVgCnAo6r6fQTjzNHIkW60OGvaGr169erFfffdx5IlSzh48CCtWrViw4YNDBs2jEWLFnHqqacyYMCAPN/tPWDAACZNmkRsbCzjxo1j9uzZJxRvmTJlAHfSTU1NPWb+yy+/TLVq1UhMTCQ9PT3sk36wkiVLkp5+9DdV1vd+0kknZTw/3vd38803M27cOLZv387AgQOPmV+qVCnq1avHuHHjaN++Pc2bN2fWrFmsXbuWRo0a5Rp74PhA9scoeLmclgll0KBB9OzZk6lTp9KhQwemT5+OqjJ48GBuy2MrlFAxz549m2+++YYFCxZQvnx5unTpkvE5lCpVKqPZadb4g7cVKF4LJ75OnToxZ84cpkyZwoABA7j//vu54YYb8vR+AiJZxLQIaCAi9USkNHA1MDl4ARGpIiKBGAYDY7zppYHPcBXYH+OTdetgyhS4/XYI+sxMlKlQoQLnn38+AwcOzKic3rdvHyeddBInn3wyO3bsyCiCyk6nTp2YNGkShw4dYv/+/Xz++ecZ8/bv30/16tVJSUnhvffey5hesWLFjPLrYA0bNmTjxo2sXbsWgPHjx9O5c+ew38/evXupXr06JUqUYPz48RkVyRdeeCFjx47NqCP4/fffqVixIjVr1mTSpEkA/Pnnnxw8eJA6deqwfPly/vzzT/bs2cOMGTOy3V92769bt24ZRWNpaWns3bsXgN69e/Pll1+yaNGibK+mOnbsyLBhw+jUqRMdO3bk9ddfJy4u7pi2+Nkdw7zq2LFjxntYvXo1v/76Kw0bNsy0zLp162jWrBkPP/wwrVu3ZuXKlXTv3p0xY8aQnJwMwJYtW/jtt98yrde2bVvmzJnDhg0bAHf8c7J3715OPfVUypcvz8qVK/n++7z/zg0nvk2bNlGtWjVuueUWbr75ZpYsWZLn/QVELEGoaipwNzAdWAF8qKq/iMgQEQlcv3cBVonIaqAa8Jw3vR/QCRggIsu8x4lfLx2nUaMgJsaathYG/fv3JzExMSNBxMbGEhcXxznnnMM111xDhw4dcly/ZcuWXHXVVcTGxnLxxRfTunXrjHnPPPMM5557Lh06dOCcc87JmH711VczdOhQ4uLiWLfuaBVZ2bJlGTt2LFdeeSXNmjWjRIkS3H777WG/lzvvvJO3336b2NhYVq5cmfFrv0ePHlx22WXEx8fTokWLjPqL8ePHM2LECJo3b0779u3Zvn07tWrVol+/fjRt2pR+/fplFE2Ekt37e+WVV5g1axbNmjWjVatWGS22Spcuzfnnn0+/fv2IiYkJuc2OHTuybds22rVrR7Vq1ShbtiwdO3Y8ZrnKlSvToUMHmjZtmlFJfSLuvPNO0tPTadasGVdddRXjxo3L9IscXDPopk2b0rx5c0qVKsXFF1/MRRddxDXXXEO7du1o1qwZffv2PSZxVa1aldGjR9OnTx9iY2MzFdOF0qNHD1JTU2nUqBGDBg2ibdu2eX5f4cQ3e/bsjO/9xIkT+b//+7887y9AQrUQKIzi4+M10LY9PyQnwxlnQM+e8P77+bbZImfFihVhFRuYoiM9PT2jBVSDBg38Dscch1D/ryKyWFVDDpxsd1JnY/x417W39btkzFHLly/nrLPOolu3bpYcigG/WzFFJVVXOR0fDydwVWhMkdO4cWPWr1/vdximgFiCCGHGDFixwo39YP3QGWOKKytiCmHECNe0NZc6KGOMKdIsQWSxfj188YUbb9qathpjijNLEFkEmrYeR6tEY4wpkixBBElOhrfegiuucE1cTfSz7r4Lp4Lu7juSgr9PBS3Sx8sSRJB334W9e63X1sLEuvs+McWlu++A4+mSw1iCyBBo2tqyJbRr53c0hVeoLrsD//8HD4ae7/VCza5dx87LjXX3Xfy6+966dWtGd90tWrQgJiaGTZs2sXPnTq644gpat25N69atM3qBfeqpp7j++uvp0KED119/PRs3bqRr1640b96cbt268euvvwLw0Ucf0bRpU2JjY4/pdTan4x9YN2vX49l9jrNnz6ZLly4h32N2348DBw4wcOBA2rRpQ1xcHP/73/+OiS27bsxPiKoWiUerVq30RHzzjSqojht3QpspdpYvX57pdefOxz5GjXLzDhwIPX/sWDd/585j54WjZ8+eOmnSJFVVff755/WBBx5QVdXdu3erqmpqaqp27txZExMTvRg766JFi1RVtU6dOrpz505NSEjQpk2b6oEDB3Tv3r165pln6tChQ1VVddeuXRn7evTRR3XEiBGqqnrjjTfqRx99lDEv8PrQoUNas2ZNXbVqlaqqXn/99fryyy9n7C+w/qhRo/Smm2465v0cOHBADx06pKqqq1ev1sB3e+rUqdquXTs9cOBApvfXpk0b/fTTT1VV9dChQ3rgwAGdNWuW9uzZM2Obd911l471DnSdOnX0xRdfzJiX3fvr169fRtypqam6Z88e3bBhg8bFxamqalpamtavXz/T+qqqCxYs0L59+6qq6nnnnaetW7fWI0eO6FNPPaWvv/56puO+YcMGbdKkSca6s2bN0kqVKunmzZs1LS1N27Ztq3Pnzj3mGAW8+uqreuWVV6qqav/+/TOW3bRpk55zzjmqqvrkk09qy5Yt9eDBg6qqeumll+o47x/9rbfe0l69eqmqatOmTTUpKUlVVf/4449j9pXd8e/cubPef//9qqo6ZcoU7datm6pm/znm9B6z+34MHjxYx48fnxFbgwYNNDk5OdPnfOmll+q8efNUVXX//v2akpJyzHvI+v+qqgokaDbnVbsPwjNyJFSpYk1bT1ROHYGWL5/z/CpVcp6fHevuu3h29/3dd9/xxhtvMG/evIzjE1zkt2/fvozO7S677DLKlSsHwIIFCzKO+/XXX58xrkaHDh0YMGAA/fr1y/iMgoU6/gGhuh7P7nPM7T2G+n589dVXTJ48OaNe7PDhwxlXPgGhujE/UZYggA0bYPJkeOQRyEPPysZn1t33sYp6d9/btm3jpptuYvLkyRljMqenp/P999+HPF7B7zc7r7/+OgsXLmTKlCkZSS5r8sst5uB4c/occ3qPobalqnzyySfH9EwbGC8CQndjHtz5Yl5YHQSujLxECWvaWlhZd9/Fq7vvlJQUrrzySl588cWMqzRwPZ6OHDky4/WyZctCrt++ffuMhg3vvfdeRi+z69at49xzz2XIkCFUrVqVzZs3Z1ov1PHPSXafY150796dkSNHZtRVLF269JhlQnVjfqKKfYI4cADefNM1bc2HKzLjE+vuu/h09z1//nwSEhJ48sknMyplt27dyogRI0hISKB58+Y0btyY119/PeT6I0eOZOzYsTRv3pzx48dnVLI/9NBDNGvWjKZNm9K+fXtiY2MzrZfd8c9Odp9jXjz++OOkpKTQvHlzmjRpwuOPP37MMqG6MT9Rxb67761b4b77XNPWXM4hJgTr7rv4se6+Cy/r7vs41agBEydacjAmHNbdd/FildTGmLBZd9/FS7G/gjAnrqgUUxpTlOXl/9QShDkhe6WgLAAABTxJREFUZcuWZffu3ZYkjIliqsru3buPu8m0FTGZE1KzZk2SkpLYuXOn36EYY3JQtmzZ4755zhKEOSGBm6KMMUWPFTEZY4wJyRKEMcaYkCxBGGOMCanI3EktIjuBTX7HcYKqALv8DiKK2PHIzI7HUXYsMjuR41FHVauGmlFkEkRRICIJ2d3yXhzZ8cjMjsdRdiwyi9TxsCImY4wxIVmCMMYYE5IliOgy2u8Aoowdj8zseBxlxyKziBwPq4MwxhgTkl1BGGOMCckShDHGmJAsQUQBEaklIrNEZLmI/CIi/+d3TH4TkRgRWSoiX/gdi99E5BQR+VhEVorIChFp53dMfhKR+7z/k59F5AMROb4uSgs5ERkjIr+JyM9B004Tka9FZI3399T82JcliOiQCjygqo2BtsBdItLY55j89n/ACr+DiBKvAF+q6jlALMX4uIjIGcA9QLyqNgVigKv9jarAjQN6ZJk2CJihqg2AGd7rE2YJIgqo6jZVXeI93487AZzhb1T+EZGaQE/gTb9j8ZuInAx0At4CUNUjqrrH36h8VxIoJyIlgfLAVp/jKVCqOgf4PcvkXsDb3vO3gcvzY1+WIKKMiNQF4oCF/kbiq+HAP4B0vwOJAvWAncBYr8jtTRE5ye+g/KKqW4BhwK/ANmCvqn7lb1RRoZqqbvOebweq5cdGLUFEERGpAHwC3Kuq+/yOxw8icinwm6ou9juWKFESaAm8pqpxwAHyqfigMPLK1nvhEmcN4CQRuc7fqKKLunsX8uX+BUsQUUJESuGSw3uq+qnf8fioA3CZiGwEJgBdReRdf0PyVRKQpKqBK8qPcQmjuLoA2KCqO1U1BfgUaO9zTNFgh4hUB/D+/pYfG7UEEQVERHBlzCtU9d9+x+MnVR2sqjVVtS6u8nGmqhbbX4iquh3YLCINvUndgOU+huS3X4G2IlLe+7/pRjGutA8yGbjRe34j8L/82KgliOjQAbge92t5mfe4xO+gTNT4O/CeiPwItAD+6XM8vvGupD4GlgA/4c5hxarbDRH5AFgANBSRJBG5CXgBuFBE1uCusl7Il31ZVxvGGGNCsSsIY4wxIVmCMMYYE5IlCGOMMSFZgjDGGBOSJQhjjDEhWYIwJhcikhbU/HiZiOTbncwiUje4V05joklJvwMwphA4pKot/A7CmIJmVxDG5JGIbBSRl0TkJxH5QUTO8qbXFZGZIvKjiMwQkdre9Goi8pmIJHqPQBcRMSLyhjfGwVciUs5b/h5vjJAfRWSCT2/TFGOWIIzJXbksRUxXBc3bq6rNgFdxvdACjATeVtXmwHvACG/6COBbVY3F9af0ize9ATBKVZsAe4ArvOmDgDhvO7dH6s0Zkx27k9qYXIhIsqpWCDF9I9BVVdd7nS1uV9XKIrILqK6qKd70bapaRUR2AjVV9c+gbdQFvvYGekFEHgZKqeqzIvIlkAxMgv9v7w5xGoiCAAz/D1RVwwG4RW/BARqCqkQQFOk9SDDoHqCyhiBIisD0FhgkpqIZxA50SV/TlGaL+T+zs09sdtXs7GzeMI2Iz44fVfrFCkI6TGyJ97FsxSvWvcEL4IGm2njLATnS0ZggpMMMW8fXjOesx2BeAS8ZPwHX8DNzu7/toqWUE+A8Ip6BMdAHNqoYqUu+kUi79Uopi9b5LCK+f3U9y11Wl8Blrt3QTIC7o5kGN8r1W+Axd99c0SSLd+pOgUkmkQLcO2pUx2YPQvqj7EEMIuLjv+9F6oKfmCRJVVYQkqQqKwhJUpUJQpJUZYKQJFWZICRJVSYISVLVF+w89xXCfNOXAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "val_acc_noise = history_noise.history[\"val_accuracy\"]\n",
        "val_acc_zeros = history_zeros.history[\"val_accuracy\"]\n",
        "epochs = range(1, 11)\n",
        "plt.plot(epochs, val_acc_noise, \"b-\",\n",
        "         label=\"Validation accuracy with noise channels\")\n",
        "plt.plot(epochs, val_acc_zeros, \"b--\",\n",
        "         label=\"Validation accuracy with zeros channels\")\n",
        "plt.title(\"Effect of noise channels on validation accuracy\")\n",
        "plt.xlabel(\"Epochs\")\n",
        "plt.ylabel(\"Accuracy\")\n",
        "plt.legend()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "noise를 추가한 모델은 accuracy 가 떨어짐 (overfitting의 가능성이 커짐)  \n",
        "만약 어떤 feature이 imformative 한지 distracting 한지 모른다면 feature selection 을 한다.  "
      ],
      "metadata": {
        "id": "qpmOjakBZyZb"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kuwzy7-oVORs"
      },
      "source": [
        "### The nature of generalization in deep learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UqQUURhOVORt"
      },
      "source": [
        "**Fitting a MNIST model with randomly shuffled labels**"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "machine learning model 에서 어떤것이든 적합하도록 훈련할 수 있다.  \n",
        "-> MNIST set의 labels를 섞어서 훈련해도 적합이 가능 (물론 generalization은 불가!) "
      ],
      "metadata": {
        "id": "pKuujSqZdmb2"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YFPcUlrDVORu"
      },
      "outputs": [],
      "source": [
        "(train_images, train_labels), _ = mnist.load_data()\n",
        "train_images = train_images.reshape((60000, 28 * 28))\n",
        "train_images = train_images.astype(\"float32\") / 255\n",
        "\n",
        "random_train_labels = train_labels[:]\n",
        "np.random.shuffle(random_train_labels)\n",
        "\n",
        "model = keras.Sequential([\n",
        "    layers.Dense(512, activation=\"relu\"),\n",
        "    layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"sparse_categorical_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "model.fit(train_images, random_train_labels,\n",
        "          epochs=100,\n",
        "          batch_size=128,\n",
        "          validation_split=0.2)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "(train_images, train_labels), _ = mnist.load_data()\n",
        "train_images = train_images.reshape((60000, 28 * 28))\n",
        "train_images = train_images.astype(\"float32\") / 255\n",
        "\n",
        "random_train_labels = train_labels[:]\n",
        "np.random.shuffle(random_train_labels)"
      ],
      "metadata": {
        "id": "IcYWa2Mwbl6Y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dwX1HTbZVORv"
      },
      "source": [
        "#### The manifold hypothesis"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "- 고차원 데이터라 할지라도, 실질적으로 해당 데이터를 나타내주는 저차원 공간인 manifold가 존재한다.   \n",
        "근데 약간 어려움 뭔말인지잘모르겠다 "
      ],
      "metadata": {
        "id": "L7yJRaOLf9zu"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TVK7RU3LVORv"
      },
      "source": [
        "#### Interpolation as a source of generalization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ls9jamjzVORw"
      },
      "source": [
        "#### Why deep learning works"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YYNMh4drVORw"
      },
      "source": [
        "#### Training data is paramount"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ago1495VVORx"
      },
      "source": [
        "## Evaluating machine-learning models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Opftk5gqVORy"
      },
      "source": [
        "### Training, validation, and test sets"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "왜 validation set이 필요할까?  \n",
        "-> layor 수, layor 유닛 등 hyperparameter를 tuning 하기 위함"
      ],
      "metadata": {
        "id": "hUuHebORiD1c"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CURNz36OVORy"
      },
      "source": [
        "#### Simple hold-out validation"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "단순히 training set 과 validation set 을 나누는 것  \n",
        "훈련 세트로 모델 훈련, 검증 세트로 평가 -> 튜닝 -> 훈련 -> 평가 -> ... 반복  \n",
        "hyperparameter 튜닝이 끝나면 test데이터 제외한 data로 최종 모델 훈련 "
      ],
      "metadata": {
        "id": "5wdgSWmHj52i"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2fWlwRobVORz"
      },
      "source": [
        "#### K-fold validation"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "data를 k개의 fold로 나누고 각 분할 i에 대해 남은 k-1개로 training, 1개로 test  \n",
        "-> k번 반복 -> k개 fold 의 평균 검증 점수  \n",
        "hyperparameter 튜닝이 끝나면 test데이터 제외한 data로 최종 모델 훈련 "
      ],
      "metadata": {
        "id": "DCvdR9lTjhgv"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qxZkDcBBVOR0"
      },
      "source": [
        "#### Iterated K-fold validation with shuffling"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "가용 데이터가 적고 정확히 평가하고자 할 때 사용  \n",
        "K-fold cv를 여러번 적용하되 K개의 분할로 나누기 전 매번 데이터를 무작위로 섞음"
      ],
      "metadata": {
        "id": "GD8NLJRajGoF"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_imHpLNgVOR0"
      },
      "source": [
        "### Beating a common-sense baseline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cQ_TnvqpVOR0"
      },
      "source": [
        "### Things to keep in mind about model evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "- 대표성 있는 데이터\n",
        "- 시간의 방향\n",
        "- 데이터 중복"
      ],
      "metadata": {
        "id": "AJInN7kJkTwY"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q9ssEdD5VOR1"
      },
      "source": [
        "## Improving model fit"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NNUZq-TMVOR2"
      },
      "source": [
        "### Tuning key gradient descent parameters"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "when training doesn't get started or stalls too early (loss stuck)  "
      ],
      "metadata": {
        "id": "mah1-xAul58M"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ETUTi6OSVOR2"
      },
      "source": [
        "**Training a MNIST model with an incorrectly high learning rate**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1gatORRnVOR2",
        "outputId": "cb8af81b-a841-4432-e508-6a50d56d9ab9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "375/375 [==============================] - 5s 11ms/step - loss: 865.9658 - accuracy: 0.3365 - val_loss: 2.4330 - val_accuracy: 0.2841\n",
            "Epoch 2/10\n",
            "375/375 [==============================] - 4s 11ms/step - loss: 4.7674 - accuracy: 0.2365 - val_loss: 2.3195 - val_accuracy: 0.2459\n",
            "Epoch 3/10\n",
            "375/375 [==============================] - 7s 18ms/step - loss: 2.8924 - accuracy: 0.2351 - val_loss: 3.3490 - val_accuracy: 0.2449\n",
            "Epoch 4/10\n",
            "375/375 [==============================] - 6s 17ms/step - loss: 2.5750 - accuracy: 0.2297 - val_loss: 2.6816 - val_accuracy: 0.2053\n",
            "Epoch 5/10\n",
            "375/375 [==============================] - 6s 16ms/step - loss: 2.7110 - accuracy: 0.2528 - val_loss: 2.4032 - val_accuracy: 0.2803\n",
            "Epoch 6/10\n",
            "375/375 [==============================] - 6s 15ms/step - loss: 2.6458 - accuracy: 0.2524 - val_loss: 2.2213 - val_accuracy: 0.2567\n",
            "Epoch 7/10\n",
            "375/375 [==============================] - 5s 15ms/step - loss: 2.7842 - accuracy: 0.2665 - val_loss: 2.1184 - val_accuracy: 0.2682\n",
            "Epoch 8/10\n",
            "375/375 [==============================] - 4s 11ms/step - loss: 2.3993 - accuracy: 0.2590 - val_loss: 3.1710 - val_accuracy: 0.3118\n",
            "Epoch 9/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 2.5386 - accuracy: 0.2886 - val_loss: 2.3337 - val_accuracy: 0.3064\n",
            "Epoch 10/10\n",
            "375/375 [==============================] - 4s 11ms/step - loss: 2.3947 - accuracy: 0.2952 - val_loss: 2.1052 - val_accuracy: 0.2336\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f87adaf3e50>"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ],
      "source": [
        "(train_images, train_labels), _ = mnist.load_data()\n",
        "train_images = train_images.reshape((60000, 28 * 28))\n",
        "train_images = train_images.astype(\"float32\") / 255\n",
        "\n",
        "model = keras.Sequential([\n",
        "    layers.Dense(512, activation=\"relu\"),\n",
        "    layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "model.compile(optimizer=keras.optimizers.RMSprop(1.), # 학습률 1 (high)\n",
        "              loss=\"sparse_categorical_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "model.fit(train_images, train_labels,\n",
        "          epochs=10,\n",
        "          batch_size=128,\n",
        "          validation_split=0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pLryyx3TVOR3"
      },
      "source": [
        "**The same model with a more appropriate learning rate**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TXQF7cnRVOR3",
        "outputId": "e249b7d7-5460-49ad-ed87-d3889d3178c3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "375/375 [==============================] - 5s 13ms/step - loss: 0.3846 - accuracy: 0.9104 - val_loss: 0.1419 - val_accuracy: 0.9607\n",
            "Epoch 2/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 0.1380 - accuracy: 0.9651 - val_loss: 0.1859 - val_accuracy: 0.9585\n",
            "Epoch 3/10\n",
            "375/375 [==============================] - 4s 11ms/step - loss: 0.1099 - accuracy: 0.9732 - val_loss: 0.1613 - val_accuracy: 0.9692\n",
            "Epoch 4/10\n",
            "375/375 [==============================] - 4s 11ms/step - loss: 0.0980 - accuracy: 0.9779 - val_loss: 0.2019 - val_accuracy: 0.9680\n",
            "Epoch 5/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 0.0880 - accuracy: 0.9811 - val_loss: 0.1978 - val_accuracy: 0.9703\n",
            "Epoch 6/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 0.0757 - accuracy: 0.9839 - val_loss: 0.2777 - val_accuracy: 0.9702\n",
            "Epoch 7/10\n",
            "375/375 [==============================] - 4s 11ms/step - loss: 0.0707 - accuracy: 0.9863 - val_loss: 0.2389 - val_accuracy: 0.9717\n",
            "Epoch 8/10\n",
            "375/375 [==============================] - 4s 11ms/step - loss: 0.0625 - accuracy: 0.9878 - val_loss: 0.2627 - val_accuracy: 0.9726\n",
            "Epoch 9/10\n",
            "375/375 [==============================] - 4s 11ms/step - loss: 0.0581 - accuracy: 0.9889 - val_loss: 0.2651 - val_accuracy: 0.9751\n",
            "Epoch 10/10\n",
            "375/375 [==============================] - 4s 11ms/step - loss: 0.0473 - accuracy: 0.9905 - val_loss: 0.3131 - val_accuracy: 0.9747\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f87ad809f10>"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ],
      "source": [
        "model = keras.Sequential([\n",
        "    layers.Dense(512, activation=\"relu\"),\n",
        "    layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "model.compile(optimizer=keras.optimizers.RMSprop(1e-2), # 학습률 1e-2 (appropriate)\n",
        "              loss=\"sparse_categorical_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "model.fit(train_images, train_labels,\n",
        "          epochs=10,\n",
        "          batch_size=128,\n",
        "          validation_split=0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "try  \n",
        "- learning rate 조절\n",
        "- batch size 높이기"
      ],
      "metadata": {
        "id": "Ltb3uxNQmUhn"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lNsoqi8aVOR4"
      },
      "source": [
        "### Leveraging better architecture priors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uPh_fgp-VOR4"
      },
      "source": [
        "### Increasing model capacity"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_0ZqDGIiVOR5"
      },
      "source": [
        "**A simple logistic regression on MNIST**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0Z2zmYKEVOR5",
        "outputId": "96f92af7-e911-404b-e497-97930650641a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.6779 - accuracy: 0.8319 - val_loss: 0.3631 - val_accuracy: 0.8998\n",
            "Epoch 2/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.3520 - accuracy: 0.9033 - val_loss: 0.3075 - val_accuracy: 0.9152\n",
            "Epoch 3/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.3158 - accuracy: 0.9118 - val_loss: 0.2897 - val_accuracy: 0.9194\n",
            "Epoch 4/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2997 - accuracy: 0.9163 - val_loss: 0.2815 - val_accuracy: 0.9205\n",
            "Epoch 5/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2896 - accuracy: 0.9191 - val_loss: 0.2764 - val_accuracy: 0.9231\n",
            "Epoch 6/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2830 - accuracy: 0.9211 - val_loss: 0.2712 - val_accuracy: 0.9250\n",
            "Epoch 7/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2781 - accuracy: 0.9222 - val_loss: 0.2698 - val_accuracy: 0.9268\n",
            "Epoch 8/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2742 - accuracy: 0.9232 - val_loss: 0.2669 - val_accuracy: 0.9263\n",
            "Epoch 9/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2715 - accuracy: 0.9246 - val_loss: 0.2656 - val_accuracy: 0.9268\n",
            "Epoch 10/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2686 - accuracy: 0.9257 - val_loss: 0.2645 - val_accuracy: 0.9291\n",
            "Epoch 11/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2665 - accuracy: 0.9261 - val_loss: 0.2639 - val_accuracy: 0.9292\n",
            "Epoch 12/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2646 - accuracy: 0.9266 - val_loss: 0.2648 - val_accuracy: 0.9282\n",
            "Epoch 13/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2631 - accuracy: 0.9274 - val_loss: 0.2630 - val_accuracy: 0.9286\n",
            "Epoch 14/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2617 - accuracy: 0.9276 - val_loss: 0.2640 - val_accuracy: 0.9285\n",
            "Epoch 15/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2604 - accuracy: 0.9285 - val_loss: 0.2614 - val_accuracy: 0.9293\n",
            "Epoch 16/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2594 - accuracy: 0.9291 - val_loss: 0.2633 - val_accuracy: 0.9311\n",
            "Epoch 17/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2585 - accuracy: 0.9288 - val_loss: 0.2611 - val_accuracy: 0.9302\n",
            "Epoch 18/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2573 - accuracy: 0.9300 - val_loss: 0.2630 - val_accuracy: 0.9300\n",
            "Epoch 19/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2563 - accuracy: 0.9298 - val_loss: 0.2607 - val_accuracy: 0.9313\n",
            "Epoch 20/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2555 - accuracy: 0.9298 - val_loss: 0.2624 - val_accuracy: 0.9307\n"
          ]
        }
      ],
      "source": [
        "model = keras.Sequential([layers.Dense(10, activation=\"softmax\")])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"sparse_categorical_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_small_model = model.fit(\n",
        "    train_images, train_labels,\n",
        "    epochs=20,\n",
        "    batch_size=128,\n",
        "    validation_split=0.2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X63bRkM1VOR6",
        "outputId": "5f2c9e01-c5a3-44aa-ce36-20357598734c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 313
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7f87abbad690>"
            ]
          },
          "metadata": {},
          "execution_count": 37
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZgU1dn38e8NDPuiAvoou8ggILI4CG6IiFHE4BYVTFRcg2+IRtyfGOUxMXGLGhSNmriLaExUjChRVMSdJYCiIKuyKJsog4hs9/vHqYGm7R56humuWX6f6+pruva7q2vq7nNO1Slzd0RERJJVizsAEREpn5QgREQkJSUIERFJSQlCRERSUoIQEZGUlCBERCSlKp8gzOwPZrbKzL6Khk82s8Vmts7MusUYV7FxROP3zXIMh5nZ3GhbJ5nZXmb2lpkVmtmfzex/zexvGaznr2b2u2zGmgtmNsTM3s5w3kfM7A/ZjqksmNnLZnZO3HGUFTPrY2ZLEoZnmVmfTOYtxbaycmyb2Qgze6Ks11tSNeIOINvMbBGwF7AlYfQj7j7MzFoClwOt3H1FNO12YJi7v7CL23WgnbvPK+Uqio3D3euXOrjM3Qjc4+5/AYj+EVYBDb0EN9C4+9CyCCb6J3/C3ZuXxfokcPf+Re/NbAhwgbsfHl9EZcvdO5XFelLtm7I6tsurSp8gIj9199dSjG8JrE5IDgCtgFm5CatY5SGO5BhaAZ+UJDmISAXm7pX6BSwC+qUY3w/4HtgKrAOeiv468B0wP5pvH+CfwEpgIXBJwjqqA/8LzAcKgalAC+CthPWsA85Isf1qwHXA58AK4DGgEVArVRwplndgv+j9I8Ao4KUojg+AttE0A+6MtrEW+Ag4IJr2JuEXUdE6hwBvR+/nR/vm+4T9swnYGA33A0YQftEXLX848C7wDbAYGJIQ3x8S5jsBmB7N9y5wYNL3dQUwE/gWeBqoDdRL+r7WAfuk2C+PAPcCL0fzvAP8D3AXsAaYDXRLmL9DtB++ISTDgQnTGgNjo/32IfD7ov0TTd8feBX4GpgDnJ4Uxx9SfXfR9AuBT6Pv6xOgezT+GrYfT58AJyd9P+8A90T7ZjZwdML0cxPWuQD4ZdI2T4z2+9poG8clHgfRvthAKG2vi/ZJD2A5UD1hPacAM9J8rkaEY3kl4di+DqiWeHwRSsdrCP9P/dOs52rg2aRxfwFG7uyzAn2AJanOAUCd6LtZE+3fK5PmTbn/U+2bNMf2hcC86JgYS8IxSvifHQrMjfbtKMDSfP4R7Pi/NZBwfH4TfV8dkvbV0ijmOUXHBHAwMCX6vpcDd5T4/FnWJ+Ty9iJNgkh1ICV8iUUn3mqEk/71QE1g3+hgPDaafiXhhNuecCLuAjROXk+abZ8XHUj7AvWBfwGPp4ojzfLJCWJ1dEDUAJ4ExkTTjo0+w25RjB2AvaNpb5ImQaTadyn+GbYdxITSRSEwGMgjnFy7Ji8HdCMkq56EBHtOtJ1aCdv8kJCY9yCcBIam+75S7JdHCNVgBxESy+uEE9HZ0fb+ALwRzZsXfQf/G32/faPP0D6aPgZ4hpCcDiD8ExYl0HqEJHhutM+7RdvtmGpfJcV4WrSuHtF3sh+hmrNo2j6EY+8Mwo+EvRO+n83AZVHsZxASxR7R9AFA22idRwLr2Z54Do7mPSZadzNg/+TjIPkYiMZ9QsKJHHgOuDzNZ3sMeAFoALQGPgPOT1j3JsJJtDpwMbCMFCdJwvG0HmgQDVcHvgR6ZfBZdzhO2DFB3AxMIhxbLYCPk+bd2f5P3jfbvmfC8bMK6E74oXc38FbS/+y/Cf+LLQlJ9Lg0+3EE2/+38qM4jom+96sIx21NwrlnMVEiivZ50Y/D94Czovf1i/ZdSV5VpZH6eTP7JuF1YYbL9QCauvuN7r7R3RcADwKDoukXANe5+xwPZrj76gzX/XNCRl/g7uuAa4FBZlbaar/n3P1Dd99MSBBdo/GbCP+s+xP+ET919y9LuY3inAm85u5Pufsmd1/t7tNTzHcRcL+7f+DuW9z9UeAHoFfCPCPdfZm7fw28mPBZMvWcu0919w2Ek9kGd3/M3bcQSiRFjf69CP84N0ff7+uEf+DBZlYdOBW43t2/c/ePgUcTtnECsMjdH3b3ze7+X0JJ87QM4rsAuNXdJ0fHzTx3/xzA3f8Rffat7v404dfmwQnLrgDuivbx04RfjAOiZV9y9/nROicC/wGOiJY7H3jI3V+N1r3U3WdnuD8fBX4BYGZ7EH50jE6eKdpng4Br3b3Q3RcBfwbOSpjtc3d/MPouHgX2JrQR7iDaH9OAk6NRfYH17v5+Bp+1OKcDN7n71+6+GBiZtN2d7f/i/Jywj6e5+w+E/+lDzKx1wjw3u/s37v4F8AaZHdtnAC9F390mQgmsDnAooURTC+hoZnnuvsjd50fLbQL2M7Mm7r6uaN+VRFVJECe5+24JrwczXK4VsE9iciH82iw6oFsQiqOlsQ+hCF7kc8Iv0R/9s2Toq4T36wknPqKT3j2E4uwKM3vAzBqWchvFyXRftAIuT9qnLQj7o0jKz1ICyxPef59iuGh9+wCL3X1rwvTPCb+umxK+j8VJ0xI/R8+kz/FzQnXWzqTdV2Z2tplNT1jnAUCThFmWevSTMCGmfaJl+5vZ+2b2dbTs8QnL7sqx+gTwUzOrRzjBTkrzI6MJ4Rdu8nHdLGF423fr7uujt+m+39GEEimEHyDbktJOPmtx9iH9d5rJ/t/ZuretL/rht5o0n5/Mj+3k9W6NPkMzDxfB/IZQ4lhhZmPMrOh/6XxC6WO2mU02sxMy/BzbVJUEUVqLgYVJyaWBux+fML1tKde9jHCSKdKSUH2wPPXspefuI939IKAj4YC5Mpr0HVA3YdZMTm7pZLovFhN+wSXu07ru/lQGy5Z14/gyoIWZJf4ftCRU/6wkfB8tkqYVWQxMTPoc9d394gy2m3JfmVkrQgl1GKGqcjdCFYglzNbMzBKHWwLLzKwWoQRzO7BXtOy4hGUz/X5+tI/dfSmhuuIUQmng8TTLriL8ak0+rpdmsN1U/gH0MbPmhJLEaIAMPmtxviTNd5rB/t/Z8bfD/3SUUBtT+s+fbr1G+AxLAdx9tIcrq1pFMd4SjZ/r7oOBPaNxz0YxZUwJongfAoVmdrWZ1TGz6mZ2gJn1iKb/Dfi9mbWz4EAzaxxNW05oX0jnKeAyM2tjZvWBPwJPR1VEZcbMephZTzPLIySEDYSGXggNlqeYWV0z24/wi6O0ngT6mdnpZlbDzBqbWari84PA0CgmM7N6ZjbAzBpksI3lQGMza7QLcSb6gPAr7iozy4suo/0pof1mC6FdaES0fzoS2kuK/BvIN7OzomXzon3dIYPt/g24wswOivbBftHJqR7hH3wlgJmdS/gFm2hP4JJoe6cR2pTGEeqja0XLbjaz/sBPEpb7O3CumR1tZtXMrJmZ7Z8ituVAczOrmTT+MULdd+dov/xItM+eAW4yswbRZxpOKIGUmLuvJLSPPEz4ofZpNGlnn7U4zwDXmtnuUeL5dcK0ne3/dPumyFOEfdw1SmJ/BD6Iqtp2xTPAgOi7yyNcmv8D8K6ZtTezvtH2NrD9Qg7M7Bdm1jQqcXwTrWtrivWnVVUSxIsWbvYqej2XyULRAX8CoZ5wIeEX0t8IV2oA3EH48v5DuFLg74S6QQhFvkejourpKVb/EOGX2FvRujew48FaVhoSTsprCMXU1cBt0bQ7CVclLSfUBz9Z2o1EdarHEw7erwnJp0uK+aYQGinviWKaR2j8y2Qbswn/hAui/brPzpbZyfo2EhJCf8J3ey9wdkLd/DBCFcBXhMbIhxOWLSSclAYRfuF9RfiVViuD7f4DuInwi7gQeJ7Q0PwJoc7+PcJ30plw1VKiD4B2Ubw3AT/z0N5TCFxCOB7XEKpkxiZs80NCg/qdhMbqiez4S7/I64SrZb4ys1UJ45+L5n8uoWoolV8TfogsIFyxNJpwrJfWaMIVc9uql3b2WXfi/wj/BwsJ/7fbSkMZ7P90+6Zo+deA3xFKN18SSmyDkucrKXefQ2gDupvwvf+UcOn+RsLxdnM0/ivCD4hro0WPA2aZ2TrCFWCD3P37kmzbdqzOFJHyymK+ic3M5hMuJ011T5FUQlWlBCEiu8DMTiVUv7wedyySO1XlTmoRKSUze5NwgcNZSVd8SSWnKiYREUlJVUwiIpJSpaliatKkibdu3TruMEREKpSpU6eucvemqaZVmgTRunVrpkyZEncYIiIVipl9nm6aqphERCQlJQgREUlJCUJERFKqNG0QIpJ7mzZtYsmSJWzYsCHuUGQnateuTfPmzcnLy8t4GSUIESm1JUuW0KBBA1q3bs2OncxKeeLurF69miVLltCmTZuMl1MVk4iU2oYNG2jcuLGSQzlnZjRu3LjEJT0lCBHZJUoOFUNpviclCBERSanKJ4hNm6BfP7j//rgjEZGSOuqooxg/fvwO4+666y4uvjj9g/369Omz7aba448/nm+++eZH84wYMYLbb7+92G0///zzfPLJJ9uGr7/+el57bdd7Qn/zzTc54YQSPx00K6p8gsjLg48/hg8/jDsSESmpwYMHM2bMmB3GjRkzhsGDB6dZYkfjxo1jt912K9W2kxPEjTfeSL9+/Uq1rvKqyicIgPx8+OyzuKMQkZL62c9+xksvvcTGjRsBWLRoEcuWLeOII47g4osvpqCggE6dOnHDDTekXL5169asWhUeDnfTTTeRn5/P4Ycfzpw5c7bN8+CDD9KjRw+6dOnCqaeeyvr163n33XcZO3YsV155JV27dmX+/PkMGTKEZ599FoAJEybQrVs3OnfuzHnnnccPP/ywbXs33HAD3bt3p3PnzsyePfvHQSX4+uuvOemkkzjwwAPp1asXM2fOBGDixIl07dqVrl270q1bNwoLC/nyyy/p3bs3Xbt25YADDmDSpEm7tnPJcoIws+PMbI6ZzTOza1JMH2pmH5nZdDN7O3rub9G0A83sPTObFc1TO1txtm+vBCFSFvr0+fHr3nvDtPXrU09/5JEwfdWqH0/bmT322IODDz6Yl19+GQilh9NPPx0z46abbmLKlCnMnDmTiRMnbju5pjJ16lTGjBnD9OnTGTduHJMnT9427ZRTTmHy5MnMmDGDDh068Pe//51DDz2UgQMHcttttzF9+nTatm27bf4NGzYwZMgQnn76aT766CM2b97Mfffdt216kyZNmDZtGhdffPFOq7FuuOEGunXrxsyZM/njH//I2WefDcDtt9/OqFGjmD59OpMmTaJOnTqMHj2aY489lunTpzNjxgy6dk31SPiSyVqCMLPqwCjC8347AoMTE0BktLt3dveuwK2EZzxjZjUIDzof6u6dgD7ApmzFmp8PK1ZAiqpIESnnEquZEquXnnnmGbp37063bt2YNWvWDtVBySZNmsTJJ59M3bp1adiwIQMHDtw27eOPP+aII46gc+fOPPnkk8yaNavYeObMmUObNm3Iz88H4JxzzuGtt97aNv2UU04B4KCDDmLRokXFruvtt9/mrLPOAqBv376sXr2atWvXcthhhzF8+HBGjhzJN998Q40aNejRowcPP/wwI0aM4KOPPqJBgwbFrjsT2bxR7mBgnrsvADCzMcCJwLZvyd3XJsxfj/BIQwgPg5/p7jOi+VZnMU66dQsN1WvXQimrI0UEePPN9NPq1i1+epMmxU9P58QTT+Syyy5j2rRprF+/noMOOoiFCxdy++23M3nyZHbffXeGDBlS6ru9hwwZwvPPP0+XLl145JFHeLM0QSaoVasWANWrV2fz5s2lWsc111zDgAEDGDduHIcddhjjx4+nd+/evPXWW7z00ksMGTKE4cOHbytxlFY2q5iaAYsThpdE43ZgZr+KHoZ+K3BJNDofcDMbb2bTzOyqVBsws4vMbIqZTVm5cmWpA+3XD159FVq2LPUqRCQm9evX56ijjuK8887bVnpYu3Yt9erVo1GjRixfvnxbFVQ6vXv35vnnn+f777+nsLCQF198cdu0wsJC9t57bzZt2sSTTz65bXyDBg0oLCz80brat2/PokWLmDdvHgCPP/44Rx55ZKk+2xFHHLFtm2+++SZNmjShYcOGzJ8/n86dO3P11VfTo0cPZs+ezeeff85ee+3FhRdeyAUXXMC0adNKtc1EsXe14e6jgFFmdiZwHXAOIa7DgR7AemCCmU119wlJyz4APABQUFCwy89OdQfd8yNS8QwePJiTTz55W1VTly5d6NatG/vvvz8tWrTgsMMOK3b57t27c8YZZ9ClSxf23HNPevTosW3a73//e3r27EnTpk3p2bPntqQwaNAgLrzwQkaOHLmtcRpCn0cPP/wwp512Gps3b6ZHjx4MHTq0VJ9rxIgRnHfeeRx44IHUrVuXRx99FAiX8r7xxhtUq1aNTp060b9/f8aMGcNtt91GXl4e9evX57HHHivVNhNl7ZnUZnYIMMLdj42GrwVw9z+lmb8asMbdG5nZIKC/u58TTfsdsMHdb0u3vYKCAt+VBwYdeyzsuSc8/nipVyFS5Xz66ad06NAh7jAkQ6m+r+jHd0Gq+bNZxTQZaGdmbcysJjAIGJsUWLuEwQHA3Oj9eKCzmdWNGqyPJKHtIhuqVYOdtD2JiFQpWaticvfNZjaMcLKvDjzk7rPM7EZgiruPBYaZWT/CFUprCNVLuPsaM7uDkGQcGOfuL2UrVgiXuk6apGomEZEiWW2DcPdxwLikcdcnvL+0mGWfIFzqmhP5+fDdd7BsGTT7UVO6iKTj7uqwrwIoTXOC7qSORJcs64Y5kRKoXbs2q1evLtXJR3Kn6HkQtWuX7H7j2K9iKi86dYJzz9V9ECIl0bx5c5YsWcKuXGYuuVH0RLmSUIKI7L03PPRQ3FGIVCx5eXklekKZVCyqYkrgDmvWxB2FiEj5oASR4Oc/h1694o5CRKR8UIJI0KYNLFgQHiIkIlLVKUEkyM+HzZth4cK4IxERiZ8SRAJd6ioisp0SRAIlCBGR7ZQgEjRuDL//Peyk40cRkSpB90Ekue66uCMQESkfVIJIsnYtTJ0adxQiIvFTgkhyzz1QUADr1sUdiYhIvJQgkhQ1VM+dW/x8IiKVnRJEEl3JJCISKEEk2W+/8FcJQkSqOiWIJHXrQsuWShAiIrrMNYV77w3df4uIVGVKECkMGBB3BCIi8VMVUworVsCzz4Z7IkREqioliBSmToXTToOZM+OOREQkPkoQKehSVxERJYiUWrWCvDwlCBGp2pQgUqhRI9wPoQQhIlWZEkQa+fkwZ07cUYiIxEeXuaZxyy2hmklEpKpSgkijffu4IxARiZeqmNL4+mu4806YNSvuSERE4qEEkcbGjTB8OLz+etyRiIjEQwkijb32ggYNdCWTiFRdWU0QZnacmc0xs3lmdk2K6UPN7CMzm25mb5tZx6TpLc1snZldkc04UzEL7RBKECJSVWUtQZhZdWAU0B/oCAxOTgDAaHfv7O5dgVuBO5Km3wG8nK0Yd0aXuopIVZbNEsTBwDx3X+DuG4ExwImJM7h7Ynd49QAvGjCzk4CFQGzNxPn5sGQJbNgQVwQiIvHJZoJoBixOGF4SjduBmf3KzOYTShCXROPqA1cD/1fcBszsIjObYmZTVq5cWWaBF7nsMigshNq1y3zVIiLlXuyN1O4+yt3bEhLCddHoEcCd7r5uJ8s+4O4F7l7QtGnTMo+tYUOoU6fMVysiUiFk80a5pUCLhOHm0bh0xgD3Re97Aj8zs1uB3YCtZrbB3e/JSqRpbN0KV14JhxwCP/tZLrcsIhK/bCaIyUA7M2tDSAyDgDMTZzCzdu4+NxocAMwFcPcjEuYZAazLdXIAqFYNnnoK1qxRghCRqidrCcLdN5vZMGA8UB14yN1nmdmNwBR3HwsMM7N+wCZgDXBOtuIprfx8XeoqIlVTVvticvdxwLikcdcnvL80g3WMKPvIMpefD889F2cEIiLxiL2Rurxr3x5WrQp9M4mIVCVKEDuRnw9Nm8KXX8YdiYhIbqm775044QRYsSLuKEREck8liJ0wizsCEZF4KEFkYPhwuOZHXQ2KiFRuShAZ+OwzGD8+7ihERHJLCSIDRfdCbN0adyQiIrmjBJGB/HxYvx6WLYs7EhGR3FGCyEB+fvirO6pFpCpRgshA+/bQrRts2RJ3JCIiuaP7IDLQrBlMmxZ3FCIiuaUShIiIpKQEkaHrroNeveKOQkQkd5QgMmQGU6bAxo1xRyIikhtKEBnKzw+N1AsXxh2JiEhuKEFkSJe6ikhVowSRoXbtwl8lCBGpKpQgMrTHHnDmmdCqVdyRiIjkhu6DKIEnn4w7AhGR3FEJooS+/z7uCEREckMJogTuvhvq1YPCwrgjERHJPiWIEmjWDNxh7ty4IxERyT4liBLQpa4iUpUoQZRA27bhjuo5c+KOREQk+5QgSqBOHWjZUiUIEakadJlrCV1+Oey5Z9xRiIhknxJECf3613FHICKSG6piKqHNm8NVTLofQkQqOyWIEpowIVzNNHly3JGIiGRXVhOEmR1nZnPMbJ6ZXZNi+lAz+8jMppvZ22bWMRp/jJlNjaZNNbO+2YyzJHSpq4hUFVlLEGZWHRgF9Ac6AoOLEkCC0e7e2d27ArcCd0TjVwE/dffOwDnA49mKs6RatoSaNZUgRKTyy2YJ4mBgnrsvcPeNwBjgxMQZ3H1twmA9wKPx/3X3ZdH4WUAdM6uVxVgzVr067LefEoSIVH7ZvIqpGbA4YXgJ0DN5JjP7FTAcqAmkqko6FZjm7j+kWPYi4CKAli1blkHImcnP181yIlL5xX6Zq7uPAkaZ2ZnAdYQqJQDMrBNwC/CTNMs+ADwAUFBQ4NmPNrjkEnXYJyKVXzYTxFKgRcJw82hcOmOA+4oGzKw58BxwtrvPz0qEpXTUUXFHICKSfdlsg5gMtDOzNmZWExgEjE2cwczaJQwOAOZG43cDXgKucfd3shhjqWzYAG+8AV98EXckIiLZk7UE4e6bgWHAeOBT4Bl3n2VmN5rZwGi2YWY2y8ymE9ohiqqXhgH7AddHl8BON7Ny08HF2rXQty8891zckYiIZE9W2yDcfRwwLmnc9QnvL02z3B+AP2Qztl3RtCk0aqQrmUSkctOd1KVgFq5kUoIQkcpMCaKU2rdXghCRyi2jBGFm9cysWvQ+38wGmlledkMr3/LzQyO1Ou0Tkcoq0xLEW0BtM2sG/Ac4C3gkW0FVBGefDR9+CHlVOk2KSGWWaYIwd18PnALc6+6nAZ2yF1b516oV9OgBNWK/1VBEJDsyThBmdgjwc8L9CQDVsxNSxeAOjz8OEyfGHYmISHZkmiB+A1wLPBfdy7Av8Eb2wir/zODaa+Hhh+OOREQkOzKqIHH3icBEgKixepW7X5LNwCoCXeoqIpVZplcxjTazhmZWD/gY+MTMrsxuaOWfEoSIVGaZVjF1jJ7dcBLwMtCGcCVTlZafD6tXh5eISGWTaYLIi+57OAkY6+6biB7uU5Xp8aMiUpllepHm/cAiYAbwlpm1AtYWu0QV0LcvLFsG//M/cUciIlL2Mm2kHgmMTBj1uZlV+aci1K0bXiIilVGmjdSNzOwOM5sSvf5MeIZ0lffAAzBqVNxRiIiUvUzbIB4CCoHTo9daQHcAAC++GJKEiEhlk2kbRFt3PzVh+P+ih/xUefn5MGECbN0K1dQ3rohUIpme0r43s8OLBszsMED9mBISxPffw5IlcUciIlK2Mi1BDAUeM7NG0fAatj8etEpLvNS1Zct4YxERKUuZXsU0A+hiZg2j4bVm9htgZjaDqwjat4eaNWHDhrgjEREpWyXqrDq6m7rIcOCusg2n4tlnn1C91LRp3JGIiJStXWlWtTKLooIrSg7ffhtvHCIiZWlXEkSV72oj0YUXQu/e4TkRIiKVQbEJwswKzWxtilchsE+OYqwQDj8cZs6EV16JOxIRkbJRbIJw9wbu3jDFq4G762GbCQYPhubN4eab445ERKRs6NauMlKzJlx+Obz1Frz3XtzRiIjsOiWIMnTBBbD77nDPPXFHIiKy61RNVIbq14eXXoLOneOORERk1ylBlLFDDgl/3cF0IbCIVGCqYsqC996Drl3VP5OIVGxKEFmw994waxbceWfckYiIlF5WE4SZHWdmc8xsnpldk2L6UDP7yMymm9nbZtYxYdq10XJzzOzYbMZZ1lq3hkGDwnMivv467mhEREonawnCzKoDo4D+QEdgcGICiIx2987u3hW4FbgjWrYjMAjoBBwH3Butr8K46ipYtw7uvTfuSERESiebJYiDgXnuvsDdNwJjgBMTZ0jq/K8e27vvOBEY4+4/uPtCYF60vgrjwAPh+ONh5EhYvz7uaERESi6bVzE1AxYnDC8BeibPZGa/IvQMWxPom7Ds+0nLNkux7EXARQAty+HDGEaMgHnzwk10IiIVTeyN1O4+yt3bAlcD15Vw2QfcvcDdC5qWw/62e/QIXXDU0MXEIlIBZTNBLAVaJAw3j8alMwY4qZTLlls//AB/+hO88ELckYiIlEw2E8RkoJ2ZtTGzmoRG57GJM5hZu4TBAcDc6P1YYJCZ1TKzNkA74MMsxpo1eXnwxBNw/fXqClxEKpasJQh33wwMA8YDnwLPuPssM7vRzAZGsw0zs1lmNp3QDnFOtOws4BngE+AV4FfuviVbsWZTtWpw5ZXqClxEKh7zSvKztqCgwKdMmRJ3GClt3Aht24bXm2/GHY2IyHZmNtXdC1JNi72RuiqoWROGD4eJE+H993c+v4hIeaDra3LkwgvhnXegdu24IxERyYwSRI7Urw/PPht3FCIimVMVU4598QWMHh13FCIiO6cEkWO33w5DhqgrcBEp/5Qgcmz4cNi6VV2Bi0j5pwSRY0Vdgd9/v7oCF5HyTQkiBldfDd99p67ARaR8U4KIQefOcNJJKkGISPmmy1xj8s9/hm44RETKK52iYlKUHP77X9i8Od5YRERSUYKI0dtvQ/fu8MwzcUciIvJjShAxOvRQ6NABbrlFXYGLSPmjBBGjatXgqqtCV+DqhkNEyhsliJideSZ06wZnnQXjxsUdjYjIdkoQMatZE157DQoKwtPnRETKC13mWltqCFcAABNKSURBVA7ssQdMmgRmYfjzz6FVq3hjEhFRCaKcKEoOL74I7drB00/HG4+IiBJEOdOnDxxySGibeOKJuKMRkapMCaKcadAgNFb36QNnnw0PPxx3RCJSVSlBlEP16oWqpmOOgfPPhxkz4o5IRKoiNVKXU3XrwgsvwPPPQ5cucUcjIlWRShDlWO3a4dkRAFOnwqhR8cYjIlWLEkQF8de/wrBhcOutcUciIlWFqpgqiPvug3XrwsOGNm6E666LOyIRqeyUICqIGjXg8cfD3da/+x1s2gQjRmy/f0JEpKwpQVQgNWqEy17z8sJzJLZsCeNERLJBp5cKpnp1ePDB8JChGjWgsBDq11dJQkTKnhqpK6Bq1UInf+vWwRFHwKWX6nkSIlL2lCAqsHr1oG9fuPtuOOEEWLQo7ohEpDLJaoIws+PMbI6ZzTOza1JMH25mn5jZTDObYGatEqbdamazzOxTMxtppkqUZGbw5z/DXXfBxInQqRPcdpuecS0iZSNrCcLMqgOjgP5AR2CwmXVMmu2/QIG7Hwg8C9waLXsocBhwIHAA0AM4MluxVmRmoYrpk0/g6KPD862VSkWkLGSzBHEwMM/dF7j7RmAMcGLiDO7+hruvjwbfB5oXTQJqAzWBWkAesDyLsVZ4LVuGrjleey00ZK9aBVdeCd9+G3dkIlJRZTNBNAMWJwwvicalcz7wMoC7vwe8AXwZvca7+6fJC5jZRWY2xcymrFy5sswCr6jMoFGj8P7VV+GOO6BDB/jHP9SILSIlVy4aqc3sF0ABcFs0vB/QgVCiaAb0NbMjkpdz9wfcvcDdC5o2bZrLkMu9wYPhgw9g773h9NPViC0iJZfNBLEUaJEw3DwatwMz6wf8Fhjo7j9Eo08G3nf3de6+jlCyOCSLsVZKBQUhSdx5Z2jEvvbauCMSkYokmwliMtDOzNqYWU1gEDA2cQYz6wbcT0gOKxImfQEcaWY1zCyP0ED9oyom2bkaNeA3v4FPPw1VTgBz58J778Ubl4iUf1lLEO6+GRgGjCec3J9x91lmdqOZDYxmuw2oD/zDzKabWVECeRaYD3wEzABmuPuL2Yq1KmjRIlQ3AdxwAxx2GPy//wfffBNvXCJSfplXktbLgoICnzJlStxhVAiFhXD99TByJOy5Z7iP4vTTdXmsSFVkZlPdvSDVtHLRSC251aBBaJf48ENo1iw8lOjBB+OOSkTKGyWIKuygg0Ij9l//CmeeGcYtWhS6EhcRUYKo4qpXh1/+MvQIu2kTHHtsSBzvvx93ZCISNyUI2SYvLzzSdM0aOPRQ+NWvdCe2SFWmBCE7OPHE0K/TJZeEqqcOHcJlsSJS9ShByI80aBCubPrgAzjuONh33zB+48Z44xKR3FKCkLQKCuChh7Z3/pefH262U3fiIlWDEoRkZNMm6NwZLr8cDj4YdMuJSOWnBCEZ2XtvGDsWnn0WvvoKevYMz6FQaUKk8lKCkIyZwamnhn6dhg6FZctCX08iUjnp31tKrFEjGDUKtmwJw3PmwDHHQPv2cMAB4dGnBxwAXbpAnTrxxioipacEIaVWvXr4+8UX0LcvfPwx3H8/fP99GD9pEhx+OLz9dqieKkocHTpA3brxxS0imVGCkF12zDHhBbB1a+iu4+OPoWvXMG7GjNAx4A/R0z7MoE2b0OX4nnuGEkjDhtt7mxWR8kFtEFKmqlUL900MHBi674BwR/a6dTB7NvzznzBiBPTqBU2ahOm33x46DezbFx54AFavji18EUmg7r4ldnPmwFNPhddnn4WG7yFD1MOsSC6ou28p19q3D6WK2bNh2jQYPhz22y9M27IFzj8/lDyK2jZEJDfUBiHlhhl06xZeRRYuhH//O9zR3aABnHQSDB4M/fqFzgVFJHtUgpBybb/9YOlSePVVOO20cDXU8cfDG2+E6evWhYZxESl7aoOQCuWHH+A//4H+/UNbxdVXwxNPhMtp27ff/ipIWaMqIsmKa4NQFZNUKLVqwU9/un34yCNh/nyYOjV0A7J1a7hcdtmyMP2GG+DLL0PSyM8Pf9u0UfWUSCaUIKRCO/748IJQupg/f8fLZBcuhJdfDr3RFunVK9yDAfCXv0C9eqEqa999w+W2RTcAilR1ShBSadSqBR077jjuscfC36+/DpfTfvbZjt1//OlPsHz59uG8vPAI1rvvDsN33x2Sxr77hlfDhtn9DCLliRKEVAl77AGHHBJeiZYuDV2FLFiw/dW5c5i2bl14sl6ixo3ht7+Fyy6DDRtC+8e++8I++0DTprD77uFmwbLmHpLcF1+E+GrUgH/9C0aPhsWLQ1y9eoXXUUepCk3KhhKEVGnVq4c2iTZt4Oijd5xWv354Pndi8liwYPsT9hYuhAsv3HGZatVCf1QXXBCqu66+Otwx3rRpeDVpAr17Q/Pm4RkbW7ZA7dqh2/SiZNW1a7ik95VXwpP9vvgivL77Lmxj0SJo1Sokho8/hhYtwvtXXgmlqKLniI8ZA2vXhqTRqVN2qs42bw5tPLvvvv3Oeak8lCBEirHbbtC9e3gly88PSWLBgvCMjJUrw6uoD6pvvw3P9161KrSLFF2O+69/hQQxYUK4Gqt+/XATYFHvuEWdHG7YEJbdf3849tiQFFq2DKUhCM/juPTS7fGsXRuq0WrWDMOPPALjx4f39euHBz317w9XXJHZZ3cPn2fx4vD64osQV/fuMHMmDBgQLgbYujVs8+ij4cYbdQVZZaIEIVJK1atD69bhlUr37iFBQDj5r1kTTvhFnRK2bQs33RROwvXqbU8ABxwQpp90UnhlqmFD6NFj+/DLL4fk9d578P774fXhh9un9+sHe+0VShi1a4cE0KvX9hN/27YhSSX605/C59pzz5AQWrYMbTRz5sALL2x/PsikSTBxYuiTq3PncBNkWVq3DiZPDp/pgw9C/PfdF0oyubRhQ/iR0KFDbrebK7oPQqQKcQ8n602bwh3p7723/ZLgatXgqqtCEti8Ga69NiSAFi22/23SJP3JvuhUYhYS33XXheHWrUOiGDgwtI+UtI1m61aYOzfEUKcO3HNPKDkVlcjy80MJZvr0kLRHjAiJsW/fsL1WrUq6l9JbsSJ0X//uu/DOO+Hy6kaNwngz+OtfQ4w/+UnF6Z24uPsglCBEqrilS0MJZ599yvYJgV99FbpJGTs23AnfpEkopZiFX//t2oUqvGTffRdOwkWlng8+CKWvCRPCSf/DD+Gll0Jpp2fP7VVuRa68Eh59NJTMILQZnXEG/PGPJYt/y5ZQAnznHTj33NC+c8UV8Oc/h/cFBXDYYXDooXDCCSE5desWEhWEB2YdeyycfHKItSxs2RKuxJs6Nbxq1YKbb961dSpBiEis1q8PjfadO4df/s2aheq2Pn1CyaJWrXBC7dkTpkwJVWVmobqt6OqsAQNClVgmtm6FWbNClyyvvx5KP3ffHUo5/fqFKqGjjgrbb9x4+3ILF8Ljj4cSwnvvhXYdCO979Qon59WrQzVbrVqptztzZrhgYPz4kOh++ctQ6tmyBf7+9/DslDZtMvsMX3yxvQrzkkvg4YdD9RpsL6k8/3xm+ySd2BKEmR0H/AWoDvzN3W9Omj4cuADYDKwEznP3z6NpLYG/AS0AB45390XptqUEIVIxbN0aSgZjx4Z2i9mzw/irrw6/hjduDG0YPXqU/X0nhYVw+ulh/d99F5JQ584hefTuDW+9FZLGAQeEkkFRCWHffUvXjlJYGJLjXnuFX/xFDfj5+aF0cdxxYXt164YLASZNCvNNmQL//W9o4ygsDMno7rtDgjrooLCe/fcvmxJfLAnCzKoDnwHHAEuAycBgd/8kYZ6jgA/cfb2ZXQz0cfczomlvAje5+6tmVh/Y6u7r021PCUKkYpo/P/wt7Um4NDZtCtVcr78eShmDBoVLljdtCokjVdXXrnIPbSmvvBJeb7wREsD48aEkcMstcM01ocG9S5eQBA46KMSWzWe7x5UgDgFGuPux0fC1AO7+pzTzdwPucffDzKwj8IC7H57p9pQgRKQi+f77UGLo3TskhaVLQ/VVhw65vdExrgcGNQMWJwwvicalcz7wcvQ+H/jGzP5lZv81s9uiEskOzOwiM5tiZlNWFrVIiYhUAEVtCLVrh+FmzeDAA8vXXfDl4nkQZvYLoAC4LRpVAzgCuALoAewLDElezt0fcPcCdy9o2rRpjqIVEakaspkglhIamIs0j8btwMz6Ab8FBrr7D9HoJcB0d1/g7puB54EU97KKiEi2ZDNBTAbamVkbM6sJDALGJs4QtTvcT0gOK5KW3c3MiooFfYFPEBGRnMlagoh++Q8DxgOfAs+4+ywzu9HMBkaz3QbUB/5hZtPNbGy07BZC9dIEM/sIMODBbMUqIiI/phvlRESqsLiuYhIRkQpMCUJERFJSghARkZQqTRuEma0EPo87jmI0AVbFHUQxFN+uUXy7RvHtml2Jr5W7p7yRrNIkiPLOzKakawgqDxTfrlF8u0bx7ZpsxacqJhERSUkJQkREUlKCyJ0H4g5gJxTfrlF8u0bx7ZqsxKc2CBERSUklCBERSUkJQkREUlKCKCNm1sLM3jCzT8xslpldmmKePmb2bdQx4XQzuz6GOBeZ2UfR9n/UeZUFI81snpnNNLOcdbNuZu0T9s10M1trZr9Jmien+9DMHjKzFWb2ccK4PczsVTObG/3dPc2y50TzzDWzc3IY321mNjv6/p4zs5QP0NzZsZDF+EaY2dKE7/D4NMseZ2ZzomPxmhzG93RCbIvMbHqaZXOx/1KeV3J2DLq7XmXwAvYGukfvGxCex90xaZ4+wL9jjnMR0KSY6ccTnuxnQC/CM8PjiLM68BXhJp7Y9iHQm/Asko8Txt0KXBO9vwa4JcVyewALor+7R+93z1F8PwFqRO9vSRVfJsdCFuMbAVyRwfc/n/CwsJrAjOT/p2zFlzT9z8D1Me6/lOeVXB2DKkGUEXf/0t2nRe8LCV2cF/eI1fLqROAxD94nPJdj7xjiOBqY7+6x3h3v7m8BXyeNPhF4NHr/KHBSikWPBV5196/dfQ3wKnBcLuJz9/946G4f4H3Cw7pikWb/ZeJgYJ6Hh4ZtBMYQ9nuZKi4+MzPgdOCpst5upoo5r+TkGFSCyAIzaw10Az5IMfkQM5thZi+bWaecBhY48B8zm2pmF6WYXtJniWfLINL/Y8a9D/dy9y+j918Be6WYp7zsx/PY/qz3ZDs7FrJpWFQF9lCa6pHysP+OAJa7+9w003O6/5LOKzk5BpUgypiZ1Qf+CfzG3dcmTZ5GqDLpAtxNeJRqrh3u7t2B/sCvzKx3DDEUy8ITCAcC/0gxuTzsw208lOXL5bXiZvZbYDPwZJpZ4joW7gPaAl2BLwnVOOXRYIovPeRs/xV3XsnmMagEUYbMLI/wJT7p7v9Knu7ua919XfR+HJBnZk1yGaO7L43+rgCeIxTlE2X0LPEs6w9Mc/flyRPKwz4ElhdVu0V/V6SYJ9b9aGZDgBOAn0cnkB/J4FjICndf7u5b3H0r4UmRqbYb9/6rAZwCPJ1unlztvzTnlZwcg0oQZSSqr/w78Km735Fmnv+J5sPMDibs/9U5jLGemTUoek9ozPw4abaxwNnR1Uy9gG8TirK5kvaXW9z7MDIWKLoi5BzghRTzjAd+Yma7R1UoP4nGZZ2ZHQdcRXjW+/o082RyLGQrvsQ2rZPTbHenz7TPsn7AbHdfkmpirvZfMeeV3ByD2WyBr0ov4HBCMW8mMD16HQ8MBYZG8wwDZhGuyHgfODTHMe4bbXtGFMdvo/GJMRowinAFyUdAQY5jrEc44TdKGBfbPiQkqi+BTYQ63POBxsAEYC7wGrBHNG8B8LeEZc8D5kWvc3MY3zxC3XPRcfjXaN59gHHFHQs5iu/x6NiaSTjR7Z0cXzR8POGqnfm5jC8a/0jRMZcwbxz7L915JSfHoLraEBGRlFTFJCIiKSlBiIhISkoQIiKSkhKEiIikpAQhIiIpKUGI7ISZbbEde5kts55Fzax1Yk+iIuVJjbgDEKkAvnf3rnEHIZJrKkGIlFL0PIBbo2cCfGhm+0XjW5vZ61FndBPMrGU0fi8Lz2eYEb0OjVZV3cwejPr7/4+Z1YnmvyR6DsBMMxsT08eUKkwJQmTn6iRVMZ2RMO1bd+8M3APcFY27G3jU3Q8kdJQ3Mho/EpjooaPB7oQ7cAHaAaPcvRPwDXBqNP4aoFu0nqHZ+nAi6ehOapGdMLN17l4/xfhFQF93XxB1qPaVuzc2s1WE7iM2ReO/dPcmZrYSaO7uPySsozWhz/520fDVQJ67/8HMXgHWEXqsfd6jTgpFckUlCJFd42nel8QPCe+3sL1tcAChX6zuwOSoh1GRnFGCENk1ZyT8fS96/y6h91GAnwOTovcTgIsBzKy6mTVKt1Izqwa0cPc3gKuBRsCPSjEi2aRfJCI7V8d2fHD9K+5edKnr7mY2k1AKGByN+zXwsJldCawEzo3GXwo8YGbnE0oKFxN6Ek2lOvBElEQMGOnu35TZJxLJgNogREopaoMocPdVcccikg2qYhIRkZRUghARkZRUghARkZSUIEREJCUlCBERSUkJQkREUlKCEBGRlP4/DFCjuCUtz+YAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "val_loss = history_small_model.history[\"val_loss\"]\n",
        "epochs = range(1, 21)\n",
        "plt.plot(epochs, val_loss, \"b--\",\n",
        "         label=\"Validation loss\")\n",
        "plt.title(\"Effect of insufficient model capacity on validation loss\")\n",
        "plt.xlabel(\"Epochs\")\n",
        "plt.ylabel(\"Loss\")\n",
        "plt.legend()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "-> layer을 늘리거나 more parameters가 있는 layer를 사용"
      ],
      "metadata": {
        "id": "cdWNiAvppQzJ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6JjoixmgVOR6",
        "outputId": "8c186a31-0a29-4306-fb2b-49470aabe1c1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "375/375 [==============================] - 3s 6ms/step - loss: 0.3527 - accuracy: 0.9002 - val_loss: 0.1890 - val_accuracy: 0.9463\n",
            "Epoch 2/20\n",
            "375/375 [==============================] - 3s 7ms/step - loss: 0.1552 - accuracy: 0.9537 - val_loss: 0.1271 - val_accuracy: 0.9615\n",
            "Epoch 3/20\n",
            "375/375 [==============================] - 3s 9ms/step - loss: 0.1091 - accuracy: 0.9664 - val_loss: 0.1099 - val_accuracy: 0.9674\n",
            "Epoch 4/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0827 - accuracy: 0.9755 - val_loss: 0.1027 - val_accuracy: 0.9687\n",
            "Epoch 5/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0658 - accuracy: 0.9798 - val_loss: 0.0944 - val_accuracy: 0.9712\n",
            "Epoch 6/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0535 - accuracy: 0.9844 - val_loss: 0.0935 - val_accuracy: 0.9735\n",
            "Epoch 7/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0451 - accuracy: 0.9863 - val_loss: 0.0901 - val_accuracy: 0.9752\n",
            "Epoch 8/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0378 - accuracy: 0.9882 - val_loss: 0.0950 - val_accuracy: 0.9727\n",
            "Epoch 9/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0305 - accuracy: 0.9907 - val_loss: 0.0966 - val_accuracy: 0.9761\n",
            "Epoch 10/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0261 - accuracy: 0.9916 - val_loss: 0.1008 - val_accuracy: 0.9752\n",
            "Epoch 11/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0220 - accuracy: 0.9934 - val_loss: 0.1013 - val_accuracy: 0.9758\n",
            "Epoch 12/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0180 - accuracy: 0.9945 - val_loss: 0.0999 - val_accuracy: 0.9766\n",
            "Epoch 13/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0160 - accuracy: 0.9949 - val_loss: 0.1098 - val_accuracy: 0.9768\n",
            "Epoch 14/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0129 - accuracy: 0.9959 - val_loss: 0.1085 - val_accuracy: 0.9766\n",
            "Epoch 15/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0110 - accuracy: 0.9967 - val_loss: 0.1126 - val_accuracy: 0.9766\n",
            "Epoch 16/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0090 - accuracy: 0.9971 - val_loss: 0.1175 - val_accuracy: 0.9775\n",
            "Epoch 17/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0086 - accuracy: 0.9972 - val_loss: 0.1200 - val_accuracy: 0.9785\n",
            "Epoch 18/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0064 - accuracy: 0.9980 - val_loss: 0.1283 - val_accuracy: 0.9761\n",
            "Epoch 19/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0063 - accuracy: 0.9981 - val_loss: 0.1420 - val_accuracy: 0.9752\n",
            "Epoch 20/20\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 0.0053 - accuracy: 0.9984 - val_loss: 0.1427 - val_accuracy: 0.9764\n"
          ]
        }
      ],
      "source": [
        "model = keras.Sequential([\n",
        "    layers.Dense(96, activation=\"relu\"),\n",
        "    layers.Dense(96, activation=\"relu\"),\n",
        "    layers.Dense(10, activation=\"softmax\"),\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"sparse_categorical_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_large_model = model.fit(\n",
        "    train_images, train_labels,\n",
        "    epochs=20,\n",
        "    batch_size=128,\n",
        "    validation_split=0.2)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "val_loss = history_large_model.history[\"val_loss\"]\n",
        "epochs = range(1, 21)\n",
        "plt.plot(epochs, val_loss, \"b--\",\n",
        "         label=\"Validation loss\")\n",
        "plt.title(\"Effect of insufficient model capacity on validation loss\")\n",
        "plt.xlabel(\"Epochs\")\n",
        "plt.ylabel(\"Loss\")\n",
        "plt.legend()"
      ],
      "metadata": {
        "id": "LrpW7mIupneE",
        "outputId": "cc460366-cf5a-445d-c9b0-64139ee18f9a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 313
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7f87ab9fd8d0>"
            ]
          },
          "metadata": {},
          "execution_count": 40
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5hU5fn/8fdNV4pUjVJkUVGQ7qJGijUKqKjYIEZFDQYMmtgiacoPNYmKiQ0Tu2hUNCpIbNhb1ITyRaQqnRVEBBGQunD//njOwrDM7s6WmbPl87quuXb21HvOnJl7nnKeY+6OiIhIftXiDkBERMonJQgREUlKCUJERJJSghARkaSUIEREJCklCBERSarKJwgzu8XMvjWzr6P/zzKzZWa2wcy6xhhXoXFE09ukOYYeZvZltK8zzWw/M/vAzNab2Z1m9jszeziF7fzDzP6YzlgzwcwGm9lHKS77uJndku6YyoKZvWZmF8cdR1kxs+PMLCfh/1lmdlwqy5ZgX2k5t81spJn9s6y3W1w14g4g3cxsMbAfsD1h8uPuPtzMWgHXAge6+zfRvNHAcHd/qZT7deAQd59fwk0UGoe71ytxcKkbBdzn7ncDRB+Eb4EGXowLaNx9aFkEE33I/+nuLcpiexK4e9+852Y2GPi5u/eML6Ky5e6Hl8V2kh2bsjq3y6tKnyAip7v7W0mmtwJWJyQHgAOBWZkJq1DlIY78MRwIzC5OchCRCszdK/UDWAyclGT6ScAmYAewAXgm+uvAD8CCaLkDgBeAVcAi4KqEbVQHfgcsANYDU4GWwAcJ29kAnJ9k/9WAPwBLgG+AJ4B9gNrJ4kiyvgMHR88fB8YAr0Rx/Bc4KJpnwN+ifawDPgc6RPPeI/wiytvmYOCj6PmC6NhsSjg+24Ct0f8nASMJv+jz1u8JfAysBZYBgxPiuyVhudOA6dFyHwOd8r1f1wEzgO+BZ4E6QN1879cG4IAkx+Vx4H7gtWiZ/wA/Au4CvgPmAl0Tlm8XHYe1hGTYP2FeE2BidNz+B9ycd3yi+YcBbwJrgHnAefniuCXZexfNHwLMid6v2UC3aPoIdp1Ps4Gz8r0//wHui47NXODEhPmXJGxzIfCLfPs8Izru66J99Ek8D6JjsZlQ2t4QHZPuwEqgesJ2BgCfFfC69iGcy6sI5/YfgGqJ5xehdPwd4fPUt4Dt3AA8n2/a3cA9Rb1W4DggJ9l3ALBX9N58Fx3f6/Mtm/T4Jzs2BZzbQ4D50TkxkYRzlPCZHQp8GR3bMYAV8PpHsvtnqz/h/FwbvV/t8h2rr6KY5+WdE8CRwJTo/V4J/LXY359l/YVc3h4UkCCSnUgJb2LeF281wpf+jUAtoE10Mp4Szb+e8IV7KOGLuDPQJP92Ctj3pdGJ1AaoB7wIPJksjgLWz58gVkcnRA3gKWBcNO+U6DU0jGJsB+wfzXuPAhJEsmOX5MOw8yQmlC7WA4OAmoQv1y751wO6EpLVUYQEe3G0n9oJ+/wfITE3JnwJDC3o/UpyXB4nVIMdQUgs7xC+iC6K9ncL8G60bM3oPfhd9P6eEL2GQ6P544DnCMmpA+FDmJdA6xKS4CXRMe8a7bd9smOVL8Zzo211j96TgwnVnHnzDiCce+cTfiTsn/D+5AJXR7GfT0gUjaP5pwIHRds8FtjIrsRzZLTsT6JtNwcOy38e5D8HommzSfgiB8YD1xbw2p4AXgLqA62BL4DLEra9jfAlWh0YBiwnyZck4XzaCNSP/q8OrACOTuG17naesHuC+AvwIeHcagnMzLdsUcc//7HZ+T4Tzp9vgW6EH3r3Ah/k+8y+TPgstiIk0T4FHMeR7PpstY3i+En0vv+GcN7WInz3LCNKRNExz/tx+AlwYfS8Xt6xK86jqjRSTzCztQmPISmu1x1o5u6j3H2ruy8EHgIGRvN/DvzB3ed58Jm7r05x2xcQMvpCd98A/BYYaGYlrfYb7+7/c/dcQoLoEk3fRviwHkb4IM5x9xUl3Edhfgq85e7PuPs2d1/t7tOTLHc58IC7/9fdt7v7WGALcHTCMve4+3J3XwP8O+G1pGq8u091982EL7PN7v6Eu28nlEjyGv2PJnxw/hK9v+8QPsCDzKw6cDZwo7v/4O4zgbEJ+zgNWOzuj7l7rrv/H6GkeW4K8f0cuN3dJ0fnzXx3XwLg7v+KXvsOd3+W8GvzyIR1vwHuio7xs4RfjKdG677i7guibb4PvAH0ita7DHjU3d+Mtv2Vu89N8XiOBX4GYGaNCT86ns6/UHTMBgK/dff17r4YuBO4MGGxJe7+UPRejAX2J7QR7iY6HtOAs6JJJwAb3f3TFF5rYc4DbnX3Ne6+DLgn336LOv6FuYBwjKe5+xbCZ/rHZtY6YZm/uPtad18KvEtq5/b5wCvRe7eNUALbCziGUKKpDbQ3s5ruvtjdF0TrbQMONrOm7r4h79gVR1VJEGe6e8OEx0MprncgcEBiciH82sw7oVsSiqMlcQChCJ5nCeGX6B4flhR9nfB8I+GLj+hL7z5CcfYbM3vQzBqUcB+FSfVYHAhcm++YtiQcjzxJX0sxrEx4vinJ/3nbOwBY5u47EuYvIfy6bkZ4P5blm5f4Oo7K9zouIFRnFaXAY2VmF5nZ9IRtdgCaJizylUc/CRNiOiBat6+ZfWpma6J1+yWsW5pz9Z/A6WZWl/AF+2EBPzKaEn7h5j+vmyf8v/O9dfeN0dOC3t+nCSVSCD9AdialIl5rYQ6g4Pc0leNf1LZ3bi/64beaAl4/qZ/b+be7I3oNzT10gvk1ocTxjZmNM7O8z9JlhNLHXDObbGanpfg6dqoqCaKklgGL8iWX+u7eL2H+QSXc9nLCl0yeVoTqg5XJFy85d7/H3Y8A2hNOmOujWT8AeycsmsqXW0FSPRbLCL/gEo/p3u7+TArrlnXj+HKgpZklfg5aEap/VhHej5b55uVZBryf73XUc/dhKew36bEyswMJJdThhKrKhoQqEEtYrLmZJf7fClhuZrUJJZjRwH7Ruq8mrJvq+7PHMXb3rwjVFQMIpYEnC1j3W8Kv1vzn9Vcp7DeZfwHHmVkLQkniaYAUXmthVlDAe5rC8S/q/NvtMx0l1CaU/PUXtF0jvIavANz9aQ89qw6MYrwtmv6luw8C9o2mPR/FlDIliML9D1hvZjeY2V5mVt3MOphZ92j+w8DNZnaIBZ3MrEk0byWhfaEgzwBXm1mWmdUD/gQ8G1URlRkz625mR5lZTUJC2Exo6IXQYDnAzPY2s4MJvzhK6ingJDM7z8xqmFkTM0tWfH4IGBrFZGZW18xONbP6KexjJdDEzPYpRZyJ/kv4FfcbM6sZdaM9ndB+s53QLjQyOj7tCe0leV4G2prZhdG6NaNj3S6F/T4MXGdmR0TH4ODoy6ku4QO+CsDMLiH8gk20L3BVtL9zCW1KrxLqo2tH6+aaWV/g5IT1HgEuMbMTzayamTU3s8OSxLYSaGFmtfJNf4JQ990xOi57iI7Zc8CtZlY/ek3XEEogxebuqwjtI48RfqjNiWYV9VoL8xzwWzNrFCWeKxPmFXX8Czo2eZ4hHOMuURL7E/DfqKqtNJ4DTo3eu5qErvlbgI/N7FAzOyHa32Z2deTAzH5mZs2iEsfaaFs7kmy/QFUlQfzbwsVeeY/xqawUnfCnEeoJFxF+IT1M6KkB8FfCm/cGoafAI4S6QQhFvrFRUfW8JJt/lPBL7INo25vZ/WQtKw0IX8rfEYqpq4E7onl/I/RKWkmoD36qpDuJ6lT7EU7eNYTk0znJclMIjZT3RTHNJzT+pbKPuYQP4cLouB5Q1DpFbG8rISH0Jby39wMXJdTNDydUAXxNaIx8LGHd9YQvpYGEX3hfE36l1U5hv/8CbiX8Il4PTCA0NM8m1Nl/QnhPOhJ6LSX6L3BIFO+twDke2nvWA1cRzsfvCFUyExP2+T9Cg/rfCI3V77P7L/087xB6y3xtZt8mTB8fLT8+oWoomSsJP0QWEnosPU0410vqaUKPuZ3VS0W91iL8P8LnYBHhc7uzNJTC8S/o2OSt/xbwR0LpZgWhxDYw/3LF5e7zCG1A9xLe99MJXfe3Es63v0TTvyb8gPhttGofYJaZbSD0ABvo7puKs2/bvTpTRMori/kiNjNbQOhOmuyaIqmEqkoJQkRKwczOJlS/vBN3LJI5VeVKahEpITN7j9DB4cJ8Pb6kklMVk4iIJKUqJhERSarSVDE1bdrUW7duHXcYIiIVytSpU79192bJ5lWaBNG6dWumTJkSdxgiIhWKmS0paJ6qmEREJCklCBERSUoJQkREkqo0bRAiknnbtm0jJyeHzZs3xx2KFKFOnTq0aNGCmjVrpryOEoSIlFhOTg7169endevW7D7IrJQn7s7q1avJyckhKysr5fVUxSQiJbZ582aaNGmi5FDOmRlNmjQpdklPCUJESkXJoWIoyfukBCEiIklV+QSxdSuceCI88kjckYhIcR1//PFMmjRpt2l33XUXw4YVfGO/4447budFtf369WPt2rV7LDNy5EhGjx5d6L4nTJjA7Nmzd/5/44038tZbpR8J/b333uO004p9d9C0qPIJolYt+L//g6lT445ERIpr0KBBjBs3brdp48aNY9CgQQWssbtXX32Vhg0blmjf+RPEqFGjOOmkk0q0rfKqyicIgKwsWLQo7ihEpLjOOeccXnnlFbZu3QrA4sWLWb58Ob169WLYsGFkZ2dz+OGHc9NNNyVdv3Xr1nz7bbg53K233krbtm3p2bMn8+bN27nMQw89RPfu3encuTNnn302Gzdu5OOPP2bixIlcf/31dOnShQULFjB48GCef/55AN5++226du1Kx44dufTSS9myZcvO/d10001069aNjh07Mnfu3D2DSrBmzRrOPPNMOnXqxNFHH82MGTMAeP/99+nSpQtdunSha9eurF+/nhUrVtC7d2+6dOlChw4d+PDDD0t3cFGCAKBNGyUIkbJw3HF7Pu6/P8zbuDH5/McfD/O//XbPeUVp3LgxRx55JK+99hoQSg/nnXceZsatt97KlClTmDFjBu+///7OL9dkpk6dyrhx45g+fTqvvvoqkydP3jlvwIABTJ48mc8++4x27drxyCOPcMwxx9C/f3/uuOMOpk+fzkEHHbRz+c2bNzN48GCeffZZPv/8c3Jzc/n73/++c37Tpk2ZNm0aw4YNK7Ia66abbqJr167MmDGDP/3pT1x00UUAjB49mjFjxjB9+nQ+/PBD9tprL55++mlOOeUUpk+fzmeffUaXLsluCV88ShDsKkHs0K1QRCqcxGqmxOql5557jm7dutG1a1dmzZq1W3VQfh9++CFnnXUWe++9Nw0aNKB///47582cOZNevXrRsWNHnnrqKWbNmlVoPPPmzSMrK4u2bdsCcPHFF/PBBx/snD9gwAAAjjjiCBYvXlzotj766CMuvPBCAE444QRWr17NunXr6NGjB9dccw333HMPa9eupUaNGnTv3p3HHnuMkSNH8vnnn1O/fv1Ct50KXSgHdO0KvXrBhg3QoEHc0YhUXO+9V/C8vfcufH7TpoXPL8gZZ5zB1VdfzbRp09i4cSNHHHEEixYtYvTo0UyePJlGjRoxePDgEl/tPXjwYCZMmEDnzp15/PHHea8kQSaoXbs2ANWrVyc3N7dE2xgxYgSnnnoqr776Kj169GDSpEn07t2bDz74gFdeeYXBgwdzzTXX7CxxlJRKEMCgQfDWW0oOIhVRvXr1OP7447n00kt3lh7WrVtH3bp12WeffVi5cuXOKqiC9O7dmwkTJrBp0ybWr1/Pv//9753z1q9fz/7778+2bdt46qmndk6vX78+69ev32Nbhx56KIsXL2b+/PkAPPnkkxx77LElem29evXauc/33nuPpk2b0qBBAxYsWEDHjh254YYb6N69O3PnzmXJkiXst99+DBkyhJ///OdMmzatRPtMpBKEiFR4gwYN4qyzztpZ1dS5c2e6du3KYYcdRsuWLenRo0eh63fr1o3zzz+fzp07s++++9K9e/ed826++WaOOuoomjVrxlFHHbUzKQwcOJAhQ4Zwzz337GychjDm0WOPPca5555Lbm4u3bt3Z+jQoSV6XSNHjuTSSy+lU6dO7L333owdOxYIXXnfffddqlWrxuGHH07fvn0ZN24cd9xxBzVr1qRevXo88cQTJdpnokpzT+rs7Gwv6Q2DcnOhQwcYPBhGjCjbuEQqszlz5tCuXbu4w5AUJXu/zGyqu2cnW15VTECNGrB+PRTR40xEpEpRgojoWggRkd0pQUSUIERKprJUU1d2JXmflCAiWVmQkxPGZhKR1NSpU4fVq1crSZRzefeDqFOnTrHWUy+mSI8eoZF648YwPpOIFK1Fixbk5OSwatWquEORIuTdUa44lCAip5wSHiKSupo1axbrDmVSsaS1isnM+pjZPDObb2Z7dCA1s95mNs3Mcs3snHzzbjezWWY2x8zusQzclcQdojG1RESqvLQlCDOrDowB+gLtgUFm1j7fYkuBwcDT+dY9BugBdAI6AN2Bkl2KmKIdO6BRIxg5Mp17ERGpONJZxXQkMN/dFwKY2TjgDGDniFnuvjial3+YPAfqALUAA2oCK9MYK9WqhbFgFi5M515ERCqOdFYxNQeWJfyfE00rkrt/ArwLrIgek9x9Tv7lzOxyM5tiZlPKopFMw36LiOxSLru5mtnBQDugBSGpnGBmvfIv5+4Punu2u2c3a9as1PvVtRAiIrukM0F8BbRM+L9FNC0VZwGfuvsGd98AvAb8uIzj20NWVrhpSZIBGkVEqpx0JojJwCFmlmVmtYCBwMQU110KHGtmNcysJqGBeo8qprJ2/PFw442wfXu69yQiUv6lrZHa3XPNbDgwCagOPOrus8xsFDDF3SeaWXdgPNAION3M/p+7Hw48D5wAfE5osH7d3f+dfE9l56ijwkNERDTc927cYfXq0OV1333LKDARkXJMw30XQ1YW3Hpr3FGIiMRPCSKBmXoyiYjkUYLIRwlCRCRQgsgnL0FUkqYZEZESU4LIJysLfvghXA8hIlKVKUHkc/LJ8OCDULt23JGIiMRL94PIp1278BARqepUgkhi+nT44ou4oxARiZcSRBInnwyjR8cdhYhIvJQgklBXVxERJYiklCBERJQgkmrTBpYu1aiuIlK1KUEkkZUF27bBV6nevUJEpBJSgkiib194/fVwj2oRkapK10Ek0aJFeIiIVGUqQRTglVfg00/jjkJEJD5KEAW44goYMybuKERE4qMEUQB1dRWRqk4JogBKECJS1SlBFCArC5Yvh02b4o5ERCQeShAFaNMm/F2yJN44RETiogRRgH79YNYsOOiguCMREYmHroMoQOPG4SEiUlWpBFGIhx+Gl1+OOwoRkXgoQRRi9Gh47LG4oxARiYcSRCHU1VVEqjIliEIoQYhIVaYEUYg2bWDt2vAQEalqlCAKkZUV/upaCBGpitTNtRD9+sH330ODBnFHIiKSeUoQhdhrr/AQEamK0lrFZGZ9zGyemc03sxFJ5vc2s2lmlmtm5+Sb18rM3jCzOWY228xapzPWgtxyi7q6ikjVlLYEYWbVgTFAX6A9MMjM2udbbCkwGHg6ySaeAO5w93bAkcA36Yq1MC+8AM8/H8eeRUTilc4qpiOB+e6+EMDMxgFnALPzFnD3xdG8HYkrRomkhru/GS23IY1xFqpNG5g9u+jlREQqm3RWMTUHliX8nxNNS0VbYK2ZvWhm/2dmd0Qlkt2Y2eVmNsXMpqxataoMQt5TVhYsXgzuadm8iEi5VV67udYAegHXAd2BNoSqqN24+4Punu3u2c2aNUtLIFlZsHkzfP11WjYvIlJupTNBfAW0TPi/RTQtFTnAdHdf6O65wASgWxnHl5KsrNDNdeXKOPYuIhKfdLZBTAYOMbMsQmIYCPy0GOs2NLNm7r4KOAGYkp4wC9enT7iS2iyOvYuIxCdtJYjol/9wYBIwB3jO3WeZ2Sgz6w9gZt3NLAc4F3jAzGZF624nVC+9bWafAwY8lK5YC1OtmpKDiFRN5pWk9TU7O9unTElPIeNXv4L994cRe1zJISJSsZnZVHfPTjavvDZSlyuTJ8Mbb8QdhYhIZilBpCArCxYujDsKEZHMUoJIQZs2sGwZbNsWdyQiIpmjBJGCrCzYsSMkCRGRqkIJIgVt28Lhh8P69XFHIiKSORruOwU9e8LMmXFHISKSWSpBiIhIUkoQKbrgArjqqrijEBHJHFUxpWjlSliwIO4oREQyRyWIFGVlwaJFcUchIpI5ShApysqCb76BH36IOxIRkcxQgkhRVlb4u3hxrGGIiGSMEkSK2rcPQ39XkrENRUSKpEbqFHXuDK+9FncUIiKZoxJEMakEISJVhRJEMfTpA+edF3cUIiKZoQRRDDVrwhdfxB2FiEhmKEEUQ961EKpmEpGqQAmiGLKywoiua9bEHYmISPopQRRDmzbhr66oFpGqQAmiGDp0gCFDoF69uCMREUk/XQdRDAcdBA8+GHcUIiKZoRJEMe3YAd9/H3cUIiLppxJEMfXsCfXrw6RJcUciIpJeKkEUU4sWaqQWkapBCaKY2rSBJUtCVZOISGWmBFFMWVmwdSssXx53JCIi6ZVSgjCzumZWLXre1sz6m1nN9IZWPuXdF2LhwnjjEBFJt1QbqT8AeplZI+ANYDJwPnBBugIrrzp2hFGjQluEiEg6uMN338E++0D16jBjBnz0EaxcGe5smfd46y2oXTt9caSaIMzdN5rZZcD97n67mU1PX1jl1/77wx//GHcUIlKZvP463HVX+NLPSwK5ubBgQWj3nDQJfvMbMIOmTWHffcPjhx/KSYIwsx8TSgyXRdOqpyek8m/lyjAm08EHxx2JiFQGW7eGEkPz5tC1a/jy328/aNAgzB8yBC66KCSH6hn85k01Qfwa+C0w3t1nmVkb4N2iVjKzPsDdhGTysLv/Jd/83sBdQCdgoLs/n29+A2A2MMHdh6cYa9qde24oAn74YdyRiEhF9o9/wOmnQ//+4VGQhg0zF1OilBqp3f19d+/v7rdFjdXfuvtVha1jZtWBMUBfoD0wyMza51tsKTAYeLqAzdxMaP8oV/KG/RYRKan//AeGDYMHHog7koKl2ovpaTNrYGZ1gZnAbDO7vojVjgTmu/tCd98KjAPOSFzA3Re7+wxgj6sKzOwIYD9Co3i5kpUVurlu2RJ3JCJSEW3fDlddFaqUbrgh7mgKlup1EO3dfR1wJvAakAVcWMQ6zYFlCf/nRNOKFJVS7gSuK2K5y81siplNWbVqVSqbLhNZWaGKacmSjO1SRCqRRx+FadNg9GioWzfuaAqWaoKoGV33cCYw0d23Aem8r9oVwKvunlPYQu7+oLtnu3t2s2bN0hjO7nRfCBEpqe++g9/9Dnr1gvPPjzuawqXaSP0AsBj4DPjAzA4E1hWxzldAy4T/W0TTUvFjwnUXVwD1gFpmtsHdR6S4flp16ACPPRb+iogUx44doWH6yitDt9XyzLyEN1g2sxrunlvYfOAL4ERCYpgM/NTdZyVZ9nHg5fy9mKJ5g4HsonoxZWdn+5QpU4r1GkREqjozm+ru2cnmpdpIvY+Z/TWvvt/M7gQKrTmLksdwYBIwB3gu6iI7ysz6R9vtbmY5wLnAA2a2R/Iorz7/HD7+OO4oRKSicIfrrgttDxVFSiUIM3uB0HtpbDTpQqCzuw9IY2zFkukSxCmnwJo1MHlyxnYpIhXY+PEwYADcey8MLzdXdRVegki1DeIgdz874f//V1WH2siTlQWq0RKRVGzaBNdcE9othw6NO5rUpdqLaZOZ9cz7x8x6AJvSE1LF0KZNKEGsK6qpXkSqvNGjYfFiuPtuqFGB7uOZaqhDgSfMbJ/o/++Ai9MTUsWQN+z3okXQuXO8sYhI+bVsGfz5z3DOOXDCCXFHUzypDrXxmbt3JoyZ1MnduwIV7KWWrcQEISJSkKZNw3UPo0fHHUnxlaab61J3b1XG8ZRYphupN26ETz6Bbt2gUaOM7VZEpEyVuptrQdstxboV3t57w4knKjmISHK5uXDmmeFeDxVVaRJEOofaqBDeeSd0XRMRye+hh+Cll8JNfSqqQhupzWw9yROBAXulJaIK5N574csv4ayz4o5ERMqTNWvgD3+A448P1z5UVIUmCHevn6lAKqKsLHjjjXCFZHkfU0VEMufGG+H770O31or83VCaKqYqLysrNFZ/803ckYhIeTFvHvz97+FmQB07xh1N6VSgSzbKn8SurvvtF28sIlI+tG0LTzwBffvGHUnpqQRRCroWQkQS7dgRqpQuuAAaN447mtJTgiiFtm1DcbIiN0KJSNnYuDGMqvD003FHUnZUxVQKNWuGJCEictttMHMmtGxZ9LIVhUoQpfTcczBmTNxRiEicFi0KCWLQoHAr0cpCCaKUJkyAO++MOwoRidN110H16nD77XFHUraUIEopKwuWLg2X1YtI1TNzJrz4YhiQr0WLuKMpW2qDKKWsLNi+HXJyoHXruKMRkUzr0AH+858wcGdloxJEKbVpE/6qq6tI1bJ8Obz6anh+zDFQp0688aSDShCllJUV6h43Ven764lULd98E0ZzXrkSFi6Ehg3jjig9lCBKqVUrWLAADjww7khEJBPWrIGf/ASWLAlDeVfW5ACqYiq16tV3JYctW+KNRUTS6/vv4eSTwwWyEydC795xR5ReShBl5MILwz1nRaTyevJJmDEDXngBTjop7mjSTwmijLRtCy+/HLq8iUjl9MtfwrRpcOqpcUeSGUoQZeSXv4S6dSvfhTIiVd2WLTB4MMyeHQbi69Ah7ogyRwmijDRuDEOGhIG6liyJOxoRKQvbtsH558PYsTB1atzRZJ4SRBm65prwC+Puu+OORERKKzcXfvazcF/pMWNCO2NVo26uZahly3DJfc+ecUciIqWxYwdcdlkYjHP0aLjiirgjiocSRBk7/fS4IxCR0tq8OYyxdvPNcO21cUcTH1UxpcEHH4SrLH/4Ie5IRKQ43ENy2HtveOMN+MMf4o4oXkoQaVCjBrzzDjzySNyRiEiq3OG3v931465mzbgjil9aE4SZ9TGzeWY238xGJJnf28ymmVmumZ2TMMhlj7QAABQISURBVL2LmX1iZrPMbIaZnZ/OOMvaMceEdog77wy9IESk/Bs1Ktz0p2PHUIKQNCYIM6sOjAH6Au2BQWbWPt9iS4HBQP67uG4ELnL3w4E+wF1mVqFGPBkxItRhjhsXdyQiUpTbb4eRI+Hii+H++0NvRElvCeJIYL67L3T3rcA44IzEBdx9sbvPAHbkm/6Fu38ZPV8OfAM0S2OsZa5fv3BBzW23hR4RIlI+TZwIN9wQrnd45BGopor3ndLZi6k5sCzh/xzgqOJuxMyOBGoBC5LMuxy4HKBVq1YlizJNzODPfw7DArvHHY2IQPgsTpkSbhV84olwwglw0EHhXtJjx4bBN2WXct3N1cz2B54ELnb3PX6Hu/uDwIMA2dnZ5e5r+LTT4o5ARHbsgHffDUlhwoRw98fq1aFevZAgDj88jIAge0pnYeoroGXC/y2iaSkxswbAK8Dv3f3TMo4tYzZtgrvugv/9L+5IRKqOH34Ig+pBKM1fdlmoPsrODiWFlStDjyUpXDpLEJOBQ8wsi5AYBgI/TWVFM6sFjAeecPfn0xdi+u3YAbfcEno2TZwYdzQildfq1WFE5fHjwzUMdevCihWh2/nLL4fbA6t3UvGkrQTh7rnAcGASMAd4zt1nmdkoM+sPYGbdzSwHOBd4wMxmRaufB/QGBpvZ9OjRJV2xplPdunDllfDvf8OsWUUvLyLFd/fdsN9+YdTVKVPg0ktDD8K83kgdOig5lIR5JWlBzc7O9ilTpsQdRlKrV4dbk55zTijeikjpzZgB++wT7uj46aehhH7WWaEaSd1UU2dmU909O9k8dejKgCZNdg0FvnRp3NGIVGzr14eRk7t1g9//Pkw7+mj405+ge3clh7JUrnsxVSbXXBN+8axbF3ckIhWTO/zrX3D11aFt4fLLQ1KQ9FGCyJBWrcL4TCJSMvfdB1ddBV27hntCH3103BFVfkoQGbZiRbh14Yknxh2JSPm3aRN8/TVkZYUb9tSoEapra+ibKyPUBpFhv/xluKRfQ4GLFO7118PAeWeeGbqLN2wIw4YpOWSSEkSGXXtt6NX06KNxRyJSPi1bFnr89e0bksFf/6rxkeKiw55hPXqEh4YCF9nT5MnQrh288grceit89pmqY+OkBBGDG26AJUvg2WfjjkSkfFi7Nvzt0iVc5DZ7Nvzud1C7drxxVXVKEDE49VTo1AnmzIk7EpF4zZ8fGp/bt4fvvw93cbvnntAoLfFTc08MqlULg/fp15FUVQsWhDHKnnwyJIVf/1q3+CyPlCBikpccFi4Mg4iJVBXz5oUhtmvWDOOU3XAD/OhHcUclyaiKKUYvvRRuVvLJJ3FHIpJeixbtanNr2zZ00li4EP72NyWH8kwJIkYnnQSNG4fbkopURosXhwvb2raFoUPD9T9m8Ktfwf77xx2dFEUJIkZ5Q4G/9BLMnBl3NFJVrVwJ06eHq5bLyvLl8ItfwCGHwBNPhOQwc2Y456XiUIKI2fDhYcjifv3Uq0ky74sv4NBDw/hGdeuGKs+8G1t9912o/szrgpqKvLsHrFsXGqB/8YvQIH3vvdC8ednHL+mlRuqYNW0a7pd79tnhAymSScOGhcbisWNDO8GcOeHGOwAffBCGuYDQTtC+fbiI7dprQzfU3Nxwb2ezcPXzn/8czuFnnoHDDguliIYN43ttUnpKEOVA166hZ0deN7/58+Hgg+ONSaqGJ54IX+7JRkbt2TPcCXH27JA45swJpYLhw8P8Bx8MF7O1bRuueHYP937esSN05VZyqPh0R7lyZvx4OPdc+Mc/4Oc/jzsaqazefhuOOy6UAIoj7+vCDN5/P/RMmjs3VFONGBHu7iYVS2F3lFMJopz5yU/CY8iQMDT4H/6gO2RJ2Zo4Ec44IwyCd/XVxVs38Vw89tjwkMpLjdTlTL164QN80UVw441wxRWwfXvcUUllMXcu/Oxn4b7NQ4fGHY2Ud0oQ5VDNmvD44+EK03/8I1QHiJTW99+HksNee8GLL4a/IoVRFVM5ZQZ/+QsMGABHHhmmuau6SUrussvC1cvvvAMtW8YdjVQEKkGUc3nJ4eOP4aijICcn3nik4rr+enj4YejVK+5IpKJQgqggtmwJ9cfHHKML6qR4li8Pf486Ci6+ON5YpGJRgqggjj8+dCvcujXcke7jj+OOSCqCmTNDF9S//z3uSKQiUoKoQLp2DUMfNG0absM4fXrcEUl59t134UroevVC47RIcamRuoLJyoL//Cfcdatjx7ijkfJq+3b46U9h6VJ47z044IC4I5KKSCWICqhZM7j55nAVbE4O3HXXritcRSBcYPn663DffaHdSqQklCAquIceClfDXnedkoTs0r49XHUVXH553JFIRaYqpgpu5EhYsyYMm1C7Ntx6q66VqMpyc6FGDbjwwvAQKQ2VICo4s9AecfnlYbjlW26JOyIpyrZtsGrV7vdZmDEjdGUujdWroVMn+Ne/SrcdkTxpTRBm1sfM5pnZfDMbkWR+bzObZma5ZnZOvnkXm9mX0UO9twthFroxDh4Mzz9ftncGk9S89FIYHuWuu0Kp7le/gnHjwrxNm0KHgpYtQ4+iWrVg333DfZkBvv4aOncO8zp1CuNw3XlnGAI+Vbm5MHBguDlPq1Zl/eqkqkpbFZOZVQfGAD8BcoDJZjbR3WcnLLYUGAxcl2/dxsBNQDbgwNRoXd1SpwDVqoWrZDdsCGPsaFiO9Nu8GerUCc8vuCDcbznPPvuEKr+BA8MybduGaY0ahfskNGwYLlyDkBjGjQv3VJg+PYy99eSToTPCoYfC55+Hcbk6d4YuXcLfQw7ZfajuESPgrbfgkUd2bVektNLZBnEkMN/dFwKY2TjgDGBngnD3xdG8HfnWPQV4093XRPPfBPoAz6Qx3gqvevXwJbRlC5x3HvTtqxE70+Grr2DUqDCm0cyZIRF8/HH4om/UCBo02P3L2wxeeKHg7dWrB+efHx55vv02bBfC9Qw5OfDmm6GkAOFHwLvvhmQwblwocVxxBVx6adm/Xqm60pkgmgPLEv7PAVL9bZNs3T3uaGtmlwOXA7RSuXons3BXr2HDQnWGvjTKxpo1YQDFe+8N1xn84hchGdeuHaqGylLTprue9+69q41izpxdJY2DDgrzFy4M92X429/KNgaRCt2Lyd0fBB6EcEe5mMMpN2rVCg2VZ5wR7kpXp064aEpK7osvwsCJ69aF3kEjR4aLFjOpdu1QxdSly+5jKg0dGqqYqqnLiZSxdJ5SXwGJgwq3iKale10hJIXx48NtJS+6qPAqDkluyxaYPDk8P+SQcJe/GTNg7NjMJ4fCNG6s5CDpkc7TajJwiJllmVktYCAwMcV1JwEnm1kjM2sEnBxNk2LYe+9wd7p+/XSv4OLYvj00Eh92GJx0UuiOagZ33AEdOsQdnUjmpC1BuHsuMJzwxT4HeM7dZ5nZKDPrD2Bm3c0sBzgXeMDMZkXrrgFuJiSZycCovAZrKZ68W5hmR7ckX7Qo3njKM/dwrDp3DqWuxo3huedCw79IVWReScZnyM7O9ilTpsQdRrn20ENh+IWXXw6jwcruPv88NDa3bRsuODz7bFXdSOVnZlPdPTvZvArdSC3Fc9ZZ4arr/v3DQG5V9c5i7qHnz9SpMGVKKCH8/vfhYrZJk+CEE8JwFSJVnX4fVSFNm4aLqVq2DO0Sn34ad0Tp5w7LEjpM//GP0KQJHHxwuO7g7rtDiWrDhjD/5JOVHETy6KNQxey3X7hS99hjQ5JYsCBc3FVZrFwJH320q3QwdWq4fmHt2lBSaNUKzj0XjjgitMt06BC6BYvInpQgqqDmzcNVwB9/XPGTw6ZN4cK1gQPDl/+LL4YrimvUCFVGAwaERJA37MiQIeEhIkVTgqiiWrXaNajbW2+Fq3MPPnjX9MaNy/9YTi+/HBrdFy0K9z9o1Sq0s2Rnh+SQN06SiJSMEkQVt3UrXHklzJ27+/SePeHDD8PzvCHE85JHq1bQokV8VTNLloTRUl96Cdq1C2MSHXdcmPejH4WHiJSeEkQVV6sWzJ4dBodbunTXI7Hv/7hxMGvW7usNGLDr6uwhQ6BHj10jl6bb8OGhiuy22+DXv1Ybgki66DoIScnmzWFE0SVLQgLZf3/o0yeMLnrwwWF6kyYhWQwbVvb3JHjnnTDcRcuWoUqpenXd90CkLBR2HYS6uUpK6tQJieDEE+GSS0JygNAYvGhR+AI/9li4/fYwTtFrr5XNflesCAMNnnhiKDFA2L6Sg0j6qYpJSs0Mjj8+PJYuDVds512E9/zz4faaF14Yhv1IVW4u3Hcf3HhjaCe56aZw0xwRyRyVIKRMtWoFN9+8KxmMHx+6nTZvHtoLvvwyte3ccgtcfXVoLJ81KwyvvddeaQtbRJJQgpC0+uc/4ZNP4LTT4P77wzhHv/lN8mVXrQr3XYDQEP3ii/DKK7tujCMimaUEIWllBkcfDU89FaqfRo3aVf20ahX89a+wejU88EC4//Ill4ThMZo2Ddc0lPdrMUQqM/Viktg8/nhICGYhKRx3HIwZEy56E5HM0GiuUi4NHhxun/nPf0K3bjBokEoMIuWJEoTEKu8eyyJS/qgNQkREklKCEBGRpJQgREQkKSUIERFJSglCRESSUoIQEZGklCBERCQpJQgREUmq0gy1YWargCVxx1GIpsC3cQdRCMVXOoqvdBRf6ZQmvgPdvVmyGZUmQZR3ZjaloPFOygPFVzqKr3QUX+mkKz5VMYmISFJKECIikpQSROY8GHcARVB8paP4SkfxlU5a4lMbhIiIJKUShIiIJKUEISIiSSlBlBEza2lm75rZbDObZWa/SrLMcWb2vZlNjx43xhDnYjP7PNr/HvdoteAeM5tvZjPMrFsGYzs04dhMN7N1ZvbrfMtk9Bia2aNm9o2ZzUyY1tjM3jSzL6O/jQpY9+JomS/N7OIMxneHmc2N3r/xZtawgHULPRfSGN9IM/sq4T3sV8C6fcxsXnQujshgfM8mxLbYzKYXsG4mjl/S75WMnYPurkcZPID9gW7R8/rAF0D7fMscB7wcc5yLgaaFzO8HvAYYcDTw35jirA58TbiIJ7ZjCPQGugEzE6bdDoyIno8AbkuyXmNgYfS3UfS8UYbiOxmoET2/LVl8qZwLaYxvJHBdCu//AqANUAv4LP/nKV3x5Zt/J3BjjMcv6fdKps5BlSDKiLuvcPdp0fP1wBygebxRlcgZwBMefAo0NLP9Y4jjRGCBu8d6dby7fwCsyTf5DGBs9HwscGaSVU8B3nT3Ne7+HfAm0CcT8bn7G+6eG/37KdCirPebqgKOXyqOBOa7+0J33wqMIxz3MlVYfGZmwHnAM2W931QV8r2SkXNQCSINzKw10BX4b5LZPzazz8zsNTM7PKOBBQ68YWZTzezyJPObA8sS/s8hnkQ3kII/mHEfw/3cfUX0/GtgvyTLlJfjeCmhRJhMUedCOg2PqsAeLaB6pDwcv17ASnf/soD5GT1++b5XMnIOKkGUMTOrB7wA/Nrd1+WbPY1QZdIZuBeYkOn4gJ7u3g3oC/zSzHrHEEOhzKwW0B/4V5LZ5eEY7uShLF8u+4qb2e+BXOCpAhaJ61z4O3AQ0AVYQajGKY8GUXjpIWPHr7DvlXSeg0oQZcjMahLexKfc/cX88919nbtviJ6/CtQ0s6aZjNHdv4r+fgOMJxTlE30FtEz4v0U0LZP6AtPcfWX+GeXhGAIr86rdor/fJFkm1uNoZoOB04ALoi+QPaRwLqSFu6909+3uvgN4qID9xn38agADgGcLWiZTx6+A75WMnINKEGUkqq98BJjj7n8tYJkfRcthZkcSjv/qDMZY18zq5z0nNGbOzLfYROCiqDfT0cD3CUXZTCnwl1vcxzAyEcjrEXIx8FKSZSYBJ5tZo6gK5eRoWtqZWR/gN0B/d99YwDKpnAvpii+xTeusAvY7GTjEzLKiEuVAwnHPlJOAue6ek2xmpo5fId8rmTkH09kCX5UeQE9CMW8GMD169AOGAkOjZYYDswg9Mj4FjslwjG2ifX8WxfH7aHpijAaMIfQg+RzIznCMdQlf+PskTIvtGBIS1QpgG6EO9zKgCfA28CXwFtA4WjYbeDhh3UuB+dHjkgzGN59Q95x3Hv4jWvYA4NXCzoUMxfdkdG7NIHzR7Z8/vuj/foReOwsyGV80/fG8cy5h2TiOX0HfKxk5BzXUhoiIJKUqJhERSUoJQkREklKCEBGRpJQgREQkKSUIERFJSglCpAhmtt12H2W2zEYWNbPWiSOJipQnNeIOQKQC2OTuXeIOQiTTVIIQKaHofgC3R/cE+J+ZHRxNb21m70SD0b1tZq2i6ftZuD/DZ9HjmGhT1c3soWi8/zfMbK9o+aui+wDMMLNxMb1MqcKUIESKtle+KqbzE+Z97+4dgfuAu6Jp9wJj3b0TYaC8e6Lp9wDvexhosBvhClyAQ4Ax7n44sBY4O5o+AugabWdoul6cSEF0JbVIEcxsg7vXSzJ9MXCCuy+MBlT72t2bmNm3hOEjtkXTV7h7UzNbBbRw9y0J22hNGLP/kOj/G4Ca7n6Lmb0ObCCMWDvBo0EKRTJFJQiR0vECnhfHloTn29nVNngqYVysbsDkaIRRkYxRghApnfMT/n4SPf+YMPoowAXAh9Hzt4FhAGZW3cz2KWijZlYNaOnu7wI3APsAe5RiRNJJv0hEiraX7X7j+tfdPa+rayMzm0EoBQyKpl0JPGZm1wOrgEui6b8CHjSzywglhWGEkUSTqQ78M0oiBtzj7mvL7BWJpEBtECIlFLVBZLv7t3HHIpIOqmISEZGkVIIQEZGkVIIQEZGklCBERCQpJQgREUlKCUJERJJSghARkaT+P/3cPgWR8JsAAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "-> 8 ephocs 이후 overfitting 발생"
      ],
      "metadata": {
        "id": "d0UtAXsIp7Oo"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hv8bw94hVOR7"
      },
      "source": [
        "## Improving generalization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6KKYnl0qVOR8"
      },
      "source": [
        "### Dataset curation"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "- enough data\n",
        "- minimize labeling errors\n",
        "- clean your data and deal with missing values\n",
        "- many feature -> feature selection"
      ],
      "metadata": {
        "id": "EUIa8-_6rAh-"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FOLd5vRnVOR9"
      },
      "source": [
        "### Feature engineering"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "특성을 더 간단한 방식으로 표현  \n",
        " ex) 2차원 픽셀 데이터(시계_원본) -> 시계바늘의 좌표 -> 시계 바늘의 각도\n",
        "\n",
        " - good features allow you to solve problems using fewer resources\n",
        " - good features let you slve a problem with far less data"
      ],
      "metadata": {
        "id": "MjDfKArPriPc"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BR_ujAzSVOR-"
      },
      "source": [
        "### Using early stopping"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "모델이 minimum possible training loss 에 도달하기 전에 training을 중단 ---> 왜......???"
      ],
      "metadata": {
        "id": "VqxOYNz3tRiG"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4v9d7jduVOR_"
      },
      "source": [
        "### Regularizing your model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YBSDt1ugVOR_"
      },
      "source": [
        "#### Reducing the network's size"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S3bjEdH2VOSA"
      },
      "source": [
        "**Original model**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dM2SucK2VOSs",
        "outputId": "2caa574c-ad2f-44cf-a0b4-ae341dfd0bd3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
            "17465344/17464789 [==============================] - 0s 0us/step\n",
            "17473536/17464789 [==============================] - 0s 0us/step\n",
            "Epoch 1/20\n",
            "30/30 [==============================] - 2s 48ms/step - loss: 0.5142 - accuracy: 0.7929 - val_loss: 0.4015 - val_accuracy: 0.8511\n",
            "Epoch 2/20\n",
            "30/30 [==============================] - 1s 35ms/step - loss: 0.3114 - accuracy: 0.9027 - val_loss: 0.3077 - val_accuracy: 0.8841\n",
            "Epoch 3/20\n",
            "30/30 [==============================] - 1s 35ms/step - loss: 0.2304 - accuracy: 0.9253 - val_loss: 0.3096 - val_accuracy: 0.8767\n",
            "Epoch 4/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.1799 - accuracy: 0.9419 - val_loss: 0.2733 - val_accuracy: 0.8911\n",
            "Epoch 5/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.1449 - accuracy: 0.9534 - val_loss: 0.2837 - val_accuracy: 0.8892\n",
            "Epoch 6/20\n",
            "30/30 [==============================] - 1s 35ms/step - loss: 0.1210 - accuracy: 0.9629 - val_loss: 0.2902 - val_accuracy: 0.8884\n",
            "Epoch 7/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.1006 - accuracy: 0.9693 - val_loss: 0.3247 - val_accuracy: 0.8803\n",
            "Epoch 8/20\n",
            "30/30 [==============================] - 1s 35ms/step - loss: 0.0834 - accuracy: 0.9756 - val_loss: 0.3321 - val_accuracy: 0.8830\n",
            "Epoch 9/20\n",
            "30/30 [==============================] - 1s 35ms/step - loss: 0.0679 - accuracy: 0.9821 - val_loss: 0.3567 - val_accuracy: 0.8794\n",
            "Epoch 10/20\n",
            "30/30 [==============================] - 1s 35ms/step - loss: 0.0564 - accuracy: 0.9859 - val_loss: 0.3887 - val_accuracy: 0.8773\n",
            "Epoch 11/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.0455 - accuracy: 0.9882 - val_loss: 0.4135 - val_accuracy: 0.8751\n",
            "Epoch 12/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.0387 - accuracy: 0.9915 - val_loss: 0.4469 - val_accuracy: 0.8747\n",
            "Epoch 13/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.0310 - accuracy: 0.9927 - val_loss: 0.4757 - val_accuracy: 0.8719\n",
            "Epoch 14/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.0224 - accuracy: 0.9958 - val_loss: 0.5298 - val_accuracy: 0.8667\n",
            "Epoch 15/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.0203 - accuracy: 0.9965 - val_loss: 0.5494 - val_accuracy: 0.8719\n",
            "Epoch 16/20\n",
            "30/30 [==============================] - 1s 35ms/step - loss: 0.0149 - accuracy: 0.9974 - val_loss: 0.5806 - val_accuracy: 0.8701\n",
            "Epoch 17/20\n",
            "30/30 [==============================] - 1s 35ms/step - loss: 0.0111 - accuracy: 0.9989 - val_loss: 0.6516 - val_accuracy: 0.8658\n",
            "Epoch 18/20\n",
            "30/30 [==============================] - 1s 35ms/step - loss: 0.0105 - accuracy: 0.9983 - val_loss: 0.6700 - val_accuracy: 0.8638\n",
            "Epoch 19/20\n",
            "30/30 [==============================] - 1s 35ms/step - loss: 0.0055 - accuracy: 0.9997 - val_loss: 0.6891 - val_accuracy: 0.8671\n",
            "Epoch 20/20\n",
            "30/30 [==============================] - 1s 34ms/step - loss: 0.0077 - accuracy: 0.9983 - val_loss: 0.7228 - val_accuracy: 0.8665\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.keras.datasets import imdb\n",
        "(train_data, train_labels), _ = imdb.load_data(num_words=10000)\n",
        "\n",
        "def vectorize_sequences(sequences, dimension=10000):\n",
        "    results = np.zeros((len(sequences), dimension))\n",
        "    for i, sequence in enumerate(sequences):\n",
        "        results[i, sequence] = 1.\n",
        "    return results\n",
        "train_data = vectorize_sequences(train_data)\n",
        "\n",
        "model = keras.Sequential([\n",
        "    layers.Dense(16, activation=\"relu\"),\n",
        "    layers.Dense(16, activation=\"relu\"),\n",
        "    layers.Dense(1, activation=\"sigmoid\")\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"binary_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_original = model.fit(train_data, train_labels,\n",
        "                             epochs=20, batch_size=512, validation_split=0.4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sPwgz-ebVOSt"
      },
      "source": [
        "**Version of the model with lower capacity**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tRKdDbZfVOSu",
        "outputId": "0123f44d-5121-41c2-befc-b784d916a80a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "30/30 [==============================] - 2s 48ms/step - loss: 0.6221 - accuracy: 0.6719 - val_loss: 0.5568 - val_accuracy: 0.8101\n",
            "Epoch 2/20\n",
            "30/30 [==============================] - 1s 33ms/step - loss: 0.4971 - accuracy: 0.8549 - val_loss: 0.4648 - val_accuracy: 0.8586\n",
            "Epoch 3/20\n",
            "30/30 [==============================] - 1s 33ms/step - loss: 0.4040 - accuracy: 0.8978 - val_loss: 0.3979 - val_accuracy: 0.8776\n",
            "Epoch 4/20\n",
            "30/30 [==============================] - 1s 32ms/step - loss: 0.3345 - accuracy: 0.9123 - val_loss: 0.3506 - val_accuracy: 0.8835\n",
            "Epoch 5/20\n",
            "30/30 [==============================] - 1s 33ms/step - loss: 0.2827 - accuracy: 0.9233 - val_loss: 0.3318 - val_accuracy: 0.8796\n",
            "Epoch 6/20\n",
            "30/30 [==============================] - 1s 33ms/step - loss: 0.2455 - accuracy: 0.9316 - val_loss: 0.2996 - val_accuracy: 0.8901\n",
            "Epoch 7/20\n",
            "30/30 [==============================] - 1s 32ms/step - loss: 0.2163 - accuracy: 0.9385 - val_loss: 0.2876 - val_accuracy: 0.8900\n",
            "Epoch 8/20\n",
            "30/30 [==============================] - 1s 33ms/step - loss: 0.1925 - accuracy: 0.9456 - val_loss: 0.2808 - val_accuracy: 0.8889\n",
            "Epoch 9/20\n",
            "30/30 [==============================] - 1s 34ms/step - loss: 0.1732 - accuracy: 0.9505 - val_loss: 0.2752 - val_accuracy: 0.8898\n",
            "Epoch 10/20\n",
            "30/30 [==============================] - 1s 33ms/step - loss: 0.1563 - accuracy: 0.9559 - val_loss: 0.2767 - val_accuracy: 0.8903\n",
            "Epoch 11/20\n",
            "30/30 [==============================] - 1s 34ms/step - loss: 0.1420 - accuracy: 0.9607 - val_loss: 0.2751 - val_accuracy: 0.8908\n",
            "Epoch 12/20\n",
            "30/30 [==============================] - 1s 33ms/step - loss: 0.1295 - accuracy: 0.9649 - val_loss: 0.2805 - val_accuracy: 0.8876\n",
            "Epoch 13/20\n",
            "30/30 [==============================] - 1s 33ms/step - loss: 0.1185 - accuracy: 0.9681 - val_loss: 0.2869 - val_accuracy: 0.8868\n",
            "Epoch 14/20\n",
            "30/30 [==============================] - 1s 33ms/step - loss: 0.1090 - accuracy: 0.9697 - val_loss: 0.2890 - val_accuracy: 0.8873\n",
            "Epoch 15/20\n",
            "30/30 [==============================] - 1s 34ms/step - loss: 0.1003 - accuracy: 0.9738 - val_loss: 0.3018 - val_accuracy: 0.8847\n",
            "Epoch 16/20\n",
            "30/30 [==============================] - 1s 34ms/step - loss: 0.0921 - accuracy: 0.9771 - val_loss: 0.3033 - val_accuracy: 0.8860\n",
            "Epoch 17/20\n",
            "30/30 [==============================] - 1s 34ms/step - loss: 0.0845 - accuracy: 0.9785 - val_loss: 0.3145 - val_accuracy: 0.8832\n",
            "Epoch 18/20\n",
            "30/30 [==============================] - 1s 33ms/step - loss: 0.0772 - accuracy: 0.9813 - val_loss: 0.3232 - val_accuracy: 0.8841\n",
            "Epoch 19/20\n",
            "30/30 [==============================] - 1s 33ms/step - loss: 0.0713 - accuracy: 0.9829 - val_loss: 0.3330 - val_accuracy: 0.8831\n",
            "Epoch 20/20\n",
            "30/30 [==============================] - 1s 34ms/step - loss: 0.0650 - accuracy: 0.9860 - val_loss: 0.3446 - val_accuracy: 0.8806\n"
          ]
        }
      ],
      "source": [
        "model = keras.Sequential([\n",
        "    layers.Dense(4, activation=\"relu\"),\n",
        "    layers.Dense(4, activation=\"relu\"),\n",
        "    layers.Dense(1, activation=\"sigmoid\")\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"binary_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_smaller_model = model.fit(\n",
        "    train_data, train_labels,\n",
        "    epochs=20, batch_size=512, validation_split=0.4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SQ1_USDvVOSu"
      },
      "source": [
        "**Version of the model with higher capacity**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DLHKDJvyVOSv",
        "outputId": "27938881-a08e-4305-aa26-d321cc0ecc28",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "30/30 [==============================] - 12s 392ms/step - loss: 0.5435 - accuracy: 0.7607 - val_loss: 0.3344 - val_accuracy: 0.8651\n",
            "Epoch 2/20\n",
            "30/30 [==============================] - 13s 446ms/step - loss: 0.2674 - accuracy: 0.8931 - val_loss: 0.2835 - val_accuracy: 0.8923\n",
            "Epoch 3/20\n",
            "30/30 [==============================] - 9s 291ms/step - loss: 0.1493 - accuracy: 0.9482 - val_loss: 0.3133 - val_accuracy: 0.8851\n",
            "Epoch 4/20\n",
            "30/30 [==============================] - 9s 291ms/step - loss: 0.0843 - accuracy: 0.9740 - val_loss: 0.3790 - val_accuracy: 0.8836\n",
            "Epoch 5/20\n",
            "30/30 [==============================] - 9s 291ms/step - loss: 0.0814 - accuracy: 0.9773 - val_loss: 0.3580 - val_accuracy: 0.8884\n",
            "Epoch 6/20\n",
            "30/30 [==============================] - 13s 437ms/step - loss: 0.0047 - accuracy: 0.9994 - val_loss: 0.5198 - val_accuracy: 0.8873\n",
            "Epoch 7/20\n",
            "30/30 [==============================] - 14s 479ms/step - loss: 0.0010 - accuracy: 0.9998 - val_loss: 0.6264 - val_accuracy: 0.8881\n",
            "Epoch 8/20\n",
            "30/30 [==============================] - 9s 291ms/step - loss: 1.4238e-04 - accuracy: 0.9999 - val_loss: 1.0470 - val_accuracy: 0.8571\n",
            "Epoch 9/20\n",
            "30/30 [==============================] - 9s 294ms/step - loss: 0.3285 - accuracy: 0.9742 - val_loss: 0.6243 - val_accuracy: 0.8818\n",
            "Epoch 10/20\n",
            "30/30 [==============================] - 9s 293ms/step - loss: 1.4063e-04 - accuracy: 1.0000 - val_loss: 0.6347 - val_accuracy: 0.8867\n",
            "Epoch 11/20\n",
            "30/30 [==============================] - 9s 289ms/step - loss: 5.7246e-05 - accuracy: 1.0000 - val_loss: 0.6991 - val_accuracy: 0.8861\n",
            "Epoch 12/20\n",
            "30/30 [==============================] - 10s 323ms/step - loss: 2.1075e-05 - accuracy: 1.0000 - val_loss: 0.7554 - val_accuracy: 0.8865\n",
            "Epoch 13/20\n",
            "30/30 [==============================] - 14s 457ms/step - loss: 7.3227e-06 - accuracy: 1.0000 - val_loss: 0.8349 - val_accuracy: 0.8865\n",
            "Epoch 14/20\n",
            "30/30 [==============================] - 10s 337ms/step - loss: 2.3707e-06 - accuracy: 1.0000 - val_loss: 0.9129 - val_accuracy: 0.8873\n",
            "Epoch 15/20\n",
            "30/30 [==============================] - 13s 424ms/step - loss: 7.7962e-07 - accuracy: 1.0000 - val_loss: 1.0032 - val_accuracy: 0.8860\n",
            "Epoch 16/20\n",
            "30/30 [==============================] - 10s 327ms/step - loss: 2.8238e-07 - accuracy: 1.0000 - val_loss: 1.0641 - val_accuracy: 0.8861\n",
            "Epoch 17/20\n",
            "30/30 [==============================] - 9s 308ms/step - loss: 1.1169e-07 - accuracy: 1.0000 - val_loss: 1.1290 - val_accuracy: 0.8863\n",
            "Epoch 18/20\n",
            "30/30 [==============================] - 10s 322ms/step - loss: 4.8709e-08 - accuracy: 1.0000 - val_loss: 1.1771 - val_accuracy: 0.8861\n",
            "Epoch 19/20\n",
            "30/30 [==============================] - 9s 291ms/step - loss: 2.7179e-08 - accuracy: 1.0000 - val_loss: 1.2106 - val_accuracy: 0.8865\n",
            "Epoch 20/20\n",
            "30/30 [==============================] - 9s 292ms/step - loss: 1.8223e-08 - accuracy: 1.0000 - val_loss: 1.2326 - val_accuracy: 0.8863\n"
          ]
        }
      ],
      "source": [
        "model = keras.Sequential([\n",
        "    layers.Dense(512, activation=\"relu\"),\n",
        "    layers.Dense(512, activation=\"relu\"),\n",
        "    layers.Dense(1, activation=\"sigmoid\")\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"binary_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_larger_model = model.fit(\n",
        "    train_data, train_labels,\n",
        "    epochs=20, batch_size=512, validation_split=0.4)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### graph"
      ],
      "metadata": {
        "id": "zYgE2nctxa1a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "val_loss = history_original.history[\"val_loss\"]\n",
        "val_loss2 = history_smaller_model.history[\"val_loss\"]\n",
        "\n",
        "epochs = range(1, 21)\n",
        "plt.plot(epochs, val_loss, \"b--\",\n",
        "         label=\"origianl\")\n",
        "plt.plot(epochs, val_loss2, \"b-\",\n",
        "         label=\"smaller_model\")\n",
        "plt.title(\"Effect of insufficient model capacity on validation loss\")\n",
        "plt.xlabel(\"Epochs\")\n",
        "plt.ylabel(\"Loss\")\n",
        "plt.legend()\n",
        "\n",
        "# 작은 네트워크가 original 네트워크보다 나중에 overfitting됨"
      ],
      "metadata": {
        "id": "WNFjBdDUwJnl",
        "outputId": "84844c8d-51fd-409d-e7aa-a6e03e525555",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 313
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7f879dbe9290>"
            ]
          },
          "metadata": {},
          "execution_count": 46
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5gUVdbA4d9hGDIikpSMChgAFQcwoQgqScD0oeiiqMga0HV1ESNidl10zQEjKoiroiKiYgBRTAwsuiIIiCBByTkIM5zvj1MDzdAzNMP0VIfzPk8/09NVXXW6urpO1b237hVVxTnnXPoqFXYAzjnnwuWJwDnn0pwnAuecS3OeCJxzLs15InDOuTTnicA559Jc2iQCEblbRJaLyB/B/2eKyAIRWS8iR4UYV6FxBK8fGOcYjheR2cG6zhCRWiIyUUTWiciDInKziDwXw3KeFpHb4hlrSRCRPiLyZYzzviQid8c7puIgIh+IyEVhx1FcRKSdiCyM+H+6iLSLZd4irCsu+7aIDBaRV4t7uXuqdNgBFBcRmQfUAnIjXn5JVfuLSH3geqCBqi4Npg0B+qvqu3u5XgUaq+qcIi6i0DhUtVKRg4vdncDjqvoIQLDDLwf20T240URVLy+OYIIf86uqWrc4lueMqnbOey4ifYC+qnpCeBEVL1U9vDiWE23bFNe+nahSJhEEuqnqJ1Ferw+siEgCAA2A6SUTVqESIY78MTQAftqTJOCcS2KqmhIPYB5wSpTXTwE2AduA9cBrwV8FNgC/BPPVBt4ClgG/AtdELCMDuBn4BVgHTAHqARMjlrMeODfK+ksBtwLzgaXAy0AVoGy0OKK8X4GDg+cvAU8A7wdxfAscFEwT4N/BOtYC/wOaBdMmYGc4ecvsA3wZPP8l2DabIrbPVmBL8P8pwGDsDD3v/ScAXwGrgQVAn4j47o6Y73RgWjDfV0CLfN/XP4AfgDXA60A5oGK+72s9UDvKdnkJeBL4IJhnErA/8DCwCpgJHBUx/6HBdliNJb3uEdOqAaOD7fYdcFfe9gmmHwJ8DKwEfgZ65ovj7mjfXTD9MmBG8H39BLQMXr+RHfvTT8CZ+b6fScDjwbaZCXSImH5xxDLnAn/Nt84ewXZfG6yjU+R+EGyLzdjV8/pgm7QClgAZEcs5C/i+gM9VBduXl2H79q1Aqcj9C7vaXYX9njoXsJyBwJv5XnsEeHR3nxVoByyMdgwAygffzapg+w7IN2/U7R9t2xSwb18GzAn2idFE7KPYb/ZyYHawbZ8ApIDPP5idf1vdsf1zdfB9HZpvWy0KYv45b58AWgPZwfe9BHhoj4+fxX1ADutBAYkg2g4T8WXlHWBLYQf3QUAZ4MBgp+sYTB+AHVibYgfcI4Bq+ZdTwLovCXaYA4FKwCjglWhxFPD+/IlgRfDFlwaGAyODaR2Dz7BvEOOhwAHBtAkUkAiibbsoO/32nRW7WlgH9AIysYPokfnfBxyFJaU2WCK9KFhP2Yh1focl4P2wH/vlBX1fUbbLS1jx1dFYAvkMO+BcGKzvbmB8MG9m8B3cHHy/7YPP0DSYPhL4D5aEmmE/trxEWRFLdhcH2/yoYL2HRdtW+WL8v2BZrYLv5GCseDJvWm1s3zsXOxk4IOL7yQH+HsR+LpYQ9gumdwUOCpZ5ErCRHQmmdTDvqcGy6wCH5N8P8u8DwWs/EXHABt4Gri/gs70MvAtUBhoCs4BLI5a9FTtYZgBXAIuJcjDE9qeNQOXg/wzgd+CYGD7rTvsJOyeC+4EvsH2rHvBjvnl3t/3zb5vt3zO2/ywHWmIndI8BE/P9Zsdgv8X6WLLsVMB2HMyO31aTII5Tg+/9Bmy/LYMdexYQJJxgm+edBH4N9A6eV8rbdnvySLXK4ndEZHXE47IY39cKqKGqd6rqFlWdCzwLnBdM7wvcqqo/q/leVVfEuOwLsAw9V1XXAzcB54lIUYvl3lbV71Q1B0sERwavb8V+lIdgP7gZqvp7EddRmPOBT1T1NVXdqqorVHValPn6Ac+o6reqmquqw4A/gWMi5nlUVRer6krgvYjPEqu3VXWKqm7GDlqbVfVlVc3FrjDyKt+PwX4g9wff72fYD7WXiGQAZwODVHWDqv4IDItYx+nAPFV9UVVzVPW/2JXj/8UQX1/gAVWdHOw3c1R1PoCqvhF89m2q+jp29tg64r1LgYeDbfw6dgbYNXjv+6r6S7DMz4FxQNvgfZcCL6jqx8GyF6nqzBi35zDgLwAish92cjEi/0zBNjsPuElV16nqPOBBoHfEbPNV9dnguxgGHIDV4e0k2B5TgTODl9oDG1X1mxg+a2F6Aveo6kpVXQA8mm+9u9v+hbkA28ZTVfVP7Dd9rIg0jJjnflVdraq/AeOJbd8+F3g/+O62YldU5YHjsCuUssBhIpKpqvNU9ZfgfVuBg0Wkuqquz9t2eyLVEsEZqrpvxOPZGN/XAKgdmUSws8e8HbcedhlZFLWxS+c887Ezy11+FDH6I+L5RuwAR3Bwexy7DF0qIkNFZJ8irqMwsW6LBsD1+bZpPWx75In6WfbAkojnm6L8n7e82sACVd0WMX0+drZcA/s+FuSbFvk52uT7HBdgxVC7U+C2EpELRWRaxDKbAdUjZlmkwSleREy1g/d2FpFvRGRl8N4uEe/dm331VaCbiFTEDqRfFHAyUR07Y82/X9eJ+H/7d6uqG4OnBX2/I7ArTLATje3JZzeftTC1Kfg7jWX7727Z25cXnOCtoIDPT+z7dv7lbgs+Qx21xijXYlcQS0VkpIjk/ZYuxa4mZorIZBE5PcbPsV2qJYKiWgD8mi+JVFbVLhHTDyrishdjB5M89bHL/iXRZy86VX1UVY8GDsN2jAHBpA1AhYhZYzmIFSTWbbEAOyOL3KYVVPW1GN5b3JXUi4F6IhK5v9fHim2WYd9HvXzT8iwAPs/3OSqp6hUxrDfqthKRBtgVZ3+siHFfrOhCImarIyKR/9cHFotIWeyKZAhQK3jv2Ij3xvr97LKNVXURVsxwFnZ2/0oB712OnYXm368XxbDeaN4A2olIXezKYARADJ+1ML9TwHcaw/bf3f630286SJzVKPrnL2i5gn2GRQCqOkKtJVODIMZ/Bq/PVtVeQM3gtTeDmGLmicB8B6wTkYEiUl5EMkSkmYi0CqY/B9wlIo3FtBCRasG0JVj5f0FeA/4uIo1EpBJwL/B6ULRTbESklYi0EZFM7MC/GatwBas4PEtEKojIwdgZRFENB04RkZ4iUlpEqolItMveZ4HLg5hERCqKSFcRqRzDOpYA1USkyl7EGelb7KzsBhHJDJqndsPqV3KxepvBwfY5DKvPyDMGaCIivYP3Zgbb+tAY1vsc8A8ROTrYBgcHB6GK2A95GYCIXIydkUaqCVwTrO//sDqfsVh5cdngvTki0hk4LeJ9zwMXi0gHESklInVE5JAosS0B6opImXyvv4yVTTcPtssugm32H+AeEakcfKbrsCuKPaaqy7D6ixexE7IZwaTdfdbC/Ae4SUSqBgnm6ohpu9v+BW2bPK9h2/jIIFndC3wbFJHtjf8AXYPvLhNr8v4n8JWINBWR9sH6NrOjQQUi8hcRqRFcQawOlrUtyvILlGqJ4D2xm6LyHm/H8qZgxz4dK8f7FTvjeQ5rGQHwEPYljcNq5p/Hyu7ALtWGBZeYPaMs/gXszGpisOzN7LxTFpd9sIPvKuzycgXwr2Dav7FWQEuw8trhRV1JUObZBdtJV2JJ5ogo82VjlYWPBzHNwSrhYlnHTOzHNjfYrrV3957dLG8LduDvjH23TwIXRpSd98cu3f/AKgVfjHjvOuzgcx52xvYHdtZVNob1vgHcg53hrgPewSp8f8LK1L/GvpPmWCuhSN8CjYN47wHOUauPWQdcg+2Pq7CilNER6/wOq9j+N1Zp/Dk7n7nn+QxrnfKHiCyPeP3tYP63I4p0orkaO+GYi7UQGoHt60U1Amuhtr1YaHefdTfuwH4Hv2K/2+1XNzFs/4K2Td77PwFuw65WfseuwM7LP9+eUtWfsTqax7DvvRvWJH4Ltr/dH7z+B3aicFPw1k7AdBFZj7W4Ok9VN+3JumXnYkjnXNgk5Ju9ROQXrJlmtHtyXApKtSsC59xeEJGzsWKTz8KOxZWcVLuz2DlXRCIyAWto0DtfCyuX4rxoyDnn0pwXDTnnXJpLuqKh6tWra8OGDcMOwznnksqUKVOWq2qNaNOSLhE0bNiQ7OzssMNwzrmkIiLzC5rmRUPOOZfmPBE451ya80TgnHNpLunqCKLZunUrCxcuZPPmzWGH4mJQrlw56tatS2ZmZtihOOdIkUSwcOFCKleuTMOGDdm5w0aXaFSVFStWsHDhQho1ahR2OM45UqRoaPPmzVSrVs2TQBIQEapVq+ZXb84lkJRIBIAngSTi35VziSVlEoFzzqWqrVth2bL4Ld8TQQnr0qULq1evLnSeQYMG8cknResBeMKECZx++h6PVOecS1CbNsHZZ8PJJ0O8SlRTorI4GagqqsrYsWN3O++dd95ZAhE55xLdmjXQvTt88QU88QSUKxef9fgVQTF66KGHaNasGc2aNePhhx9m3rx5NG3alAsvvJBmzZqxYMECGjZsyPLlNujRXXfdRdOmTTnhhBPo1asXQ4YMAaBPnz68+eabgCWFVq1a0axZM/r160deb7Ht2rVj4MCBtG7dmiZNmvDFF1+E86Gdc3Hzr3/BV1/B8OFwRSyjZBdRSl4RtGu362s9e8KVV8LGjdCly67T+/Sxx/LlcM45O0+bMGH365wyZQovvvgi3377LapKmzZtOOmkk5g9ezbDhg3jmGOO2Wn+yZMn89Zbb/H999+zdetWWrZsydFHH73Lcvv378+gQYMA6N27N2PGjKFbt24A5OTk8N133zF27FjuuOOOIhcnOecS06BB0LUrHHtsfNfjVwTF5Msvv+TMM8+kYsWKVKpUibPOOosvvviCBg0a7JIEACZNmkSPHj0oV64clStX3n5wz2/8+PG0adOG5s2b89lnnzF9+vTt08466ywAjj76aObNmxeXz+WcK1kzZsCpp9pJaZky8U8CkKJXBIWdwVeoUPj06tVjuwKIVcWKFYv83s2bN3PllVeSnZ1NvXr1GDx48E7t78uWtfHTMzIyyMnJ2etYnXPhmjwZOneG0qVhyRI7HpUEvyIoJm3btuWdd95h48aNbNiwgbfffpu2bdsWOP/xxx/Pe++9x+bNm1m/fj1jxozZZZ68g3716tVZv3799noD51zq+fRTaN8e9tkHJk2Cww8vuXXH9YpARDoBjwAZwHOqen++6f8GTg7+rQDUVNV94xlTvLRs2ZI+ffrQunVrAPr27UvVqlULnL9Vq1Z0796dFi1aUKtWLZo3b06VKlV2mmffffflsssuo1mzZuy///60atUqrp/BOReOceOgWzdo3Nie165dsuuP25jFIpIBzAJOBRYCk4FeqvpTAfNfDRylqpcUttysrCzNPzDNjBkzOPTQQ4sl7pK0fv16KlWqxMaNGznxxBMZOnQoLVu2DDusEpGs35lz8bBkCVx3HTz2GOy3X3zWISJTVDUr2rR4Fg21Buao6lxV3QKMBHoUMn8v4LU4xpNw+vXrx5FHHknLli05++yz0yYJOOfMqFF213CtWtZENF5JYHfiWTRUB1gQ8f9CoE20GUWkAdAI+KyA6f2AfgD169cv3ihDNGLEiLBDcM6FQBVuuQXuuw+efDK+9wjEIlEqi88D3lTV3GgTVXWoqmapalaNGlHHXnbOuaSQm2sH/vvug3797BG2eCaCRUC9iP/rBq9Fcx5pVizknEs/W7bA+efDM8/ATTfB009DRkbYUcU3EUwGGotIIxEpgx3sR+efSUQOAaoCX8cxFuecC92cOfDBB9Z1xL33QqL0yB63OgJVzRGR/sBHWPPRF1R1uojcCWSral5SOA8YqfFqvuSccyH78Uc49FA47DCYPdsqhxNJXOsIVHWsqjZR1YNU9Z7gtUERSQBVHayqN8YzDuecC8Ps2VYU1Lw55LUNSbQkAIlTWeyiqFSpEgDz5s2jWbNmIUezq8ieVPdmHudSzcKFVgl86KHw7rtw882QyMOEpGRfQ+kuJyeH0qX9q3UuDKp2l/D06dbj8c03w/77hx1V4VLuaHHttTBtWvEu88gj4eGHC59nw4YN9OzZk4ULF5Kbm8ttt93GwIED6dWrFx988AGlS5dm6NCh3HTTTcyZM4cBAwZw+eWXs379enr06MGqVavYunUrd999Nz16FHzfXW5uLjfeeCMTJkzgzz//5KqrruKvf/0rEyZM4LbbbqNq1arMnDmTWbNm7fLeefPm0alTJ4455hi++uorWrVqxcUXX8ztt9/O0qVLGT58OK1bt2blypVccsklzJ07lwoVKjB06FBatGjBihUr6NWrF4sWLeLYY48lslrn1Vdf5dFHH2XLli20adOGJ598koxEaA7hXAlYuxYefxyuvhoqV7ZWQTVrQsOGYUcWGy8aKiYffvghtWvX5vvvv+fHH3+kU6dOgN0AN23aNNq2bbt9wJlvvvmG22+/HYBy5crx9ttvM3XqVMaPH8/1119PYfXmzz//PFWqVGHy5MlMnjyZZ599ll9//RWAqVOn8sgjj0RNAnnmzJnD9ddfz8yZM5k5cyYjRozgyy+/ZMiQIdx7770A3H777Rx11FH88MMP3HvvvVx44YUA3HHHHZxwwglMnz6dM888k99++w2w7iJef/11Jk2axLRp08jIyGD48OF7v1GdS3CbNsGQIXDggXaD2Acf2OutWydPEoAUvCLY3Zl7vDRv3pzrr7+egQMHcvrpp2/vebR79+7bp69fv57KlStTuXJlypYty+rVq6lYsSI333wzEydOpFSpUixatIglS5awfwHXkuPGjeOHH37Y3hPpmjVrmD17NmXKlKF169Y0atSo0DgbNWpE8+bNATj88MPp0KEDIkLz5s23j2nw5Zdf8tZbbwHQvn17VqxYwdq1a5k4cSKjRo0CoGvXrts71fv000+ZMmXK9k7xNm3aRM2aNYu6KZ1LeKowdCjceScsXgwdO8Ldd0NW1J58El/KJYKwNGnShKlTpzJ27FhuvfVWOnToAOwYM6BUqVLbn+f9n5OTw/Dhw1m2bBlTpkwhMzOThg0b7jTmQH6qymOPPUbHjh13en3ChAkxjX2QP4bI+Io6poGqctFFF3HfffcV6f3OJRsReOMNO+sfMQJOOinsiPaOFw0Vk8WLF1OhQgX+8pe/MGDAAKZOnRrT+9asWUPNmjXJzMxk/PjxzJ8/v9D5O3bsyFNPPcXWrVsBmDVrFhs2bNjr+CO1bdt2e9HOhAkTqF69Ovvssw8nnnji9v6RPvjgA1atWgVAhw4dePPNN1m6dCkAK1eu3O3ncC6ZqMLo0dCqFQQlorz1Fnz5ZfInAfArgmLzv//9jwEDBlCqVCkyMzN56qmnOCf/4MdRXHDBBXTr1o3mzZuTlZXFIYccUuj8ffv2Zd68ebRs2RJVpUaNGrzzzjvF9TEAGDx4MJdccgktWrSgQoUKDBs2DLC6g169enH44Ydz3HHHbe8A8LDDDuPuu+/mtNNOY9u2bWRmZvLEE0/QoEGDYo3LuZKmCmPHwuDBkJ1t4wX8/jvUrw/5hg9JanEbjyBeUmk8gnTm35lLdDk5cOKJ8PXX0KgR3Hor9O4NmZlhR1Y0YY1H4JxzSUUVpkyx56VLQ4cO8Oyz8PPPcMklyZsEdseLhlLQihUrtldWR/r000+pVq1aCBE5l9hU4eOPrQjo66+tGOjoo+Guu8KOrGSkTCJQVSRRuvILWbVq1ZhW3HfVFaNkK450qUsVPvnEEsBXX0G9etY1dNDCOm2kRCIoV64cK1asoFq1ap4MEpyqsmLFCsqVKxd2KM6xdi2ccw7ssw889RRcfDFEtLBOGymRCOrWrcvChQtZtmxZ2KG4GJQrV466deuGHYZLQ6owfjyMHGndQFSpYlcELVqkZwLIkxKJIDMzc7d31Drn0tvEiXDbbfb3gANg/ny7ISy4IT6teash51zK++YbaNfOxgd45BGYOze5+gKKt5S4InDOucIMGQI1asDMmVYf4HbmicA5l/JefBFmzPAkUBBPBM65lKUK27bZGAGtW4cdTeLyOgLnXMoaNw4OP9zqBlzBPBE451KSqrUS2rTJOolzBfOiIedcShozBiZPhueeS+97BGLhVwTOuZSzbZtdDRx0EAQjrbpC+BWBcy7ljB0L338Pr7ySuj2GFidPBM65lNO5sw0leeaZYUeSHDwROOdSiipkZFhnci42XkfgnEsZOTlw/PHw6qthR5JcPBE451LGK6/YwDKVK4cdSXLxROCcSwlbtsAdd0BWFnTvHnY0ycXrCJxzKeGFF6xr6aefBh+fas/4FYFzLult2QJ33w3HHQcdO4YdTfLxKwLnXNIrUwZGjIDy5f1qoCg8ETjnUsKJJ4YdQfLyoiHnXFJ75BHo39+ajrqi8UTgnEta69bBXXfBnDlQ2ss3iiyuiUBEOonIzyIyR0RuLGCeniLyk4hMF5ER8YzHOZdaHnkEVqywZOCKLm45VEQygCeAU4GFwGQRGa2qP0XM0xi4CTheVVeJSM14xeOcSy2rV8ODD9o9A61ahR1NcovnFUFrYI6qzlXVLcBIoEe+eS4DnlDVVQCqujSO8TjnUshDD1kyuPPOsCNJfvEsVasDLIj4fyHQJt88TQBEZBKQAQxW1Q/zL0hE+gH9AOr7UEPOOaB3b6hZE444IuxIkl/Y1SulgcZAO6AuMFFEmqvq6siZVHUoMBQgKytLSzpI51ziadzYHm7vxbNoaBFQL+L/usFrkRYCo1V1q6r+CszCEoNzzkX1xx/Qs6e1FHLFI56JYDLQWEQaiUgZ4DxgdL553sGuBhCR6lhR0dw4xuScS3L33w+jRtm4A654xC0RqGoO0B/4CJgB/EdVp4vInSKS1zfgR8AKEfkJGA8MUNUV8YrJOZfcFi60TuUuusiLhYqTaJKl1aysLM3Ozg47DOdcCK68Ep57DmbNgoYNw44muYjIFFXNijbN7yx2ziWFefMsCfTt60mguIXdasg552Ky774wYIBdFbji5YnAOZfw1qyxRHDPPWFHkpq8aMg5l7BUrT+hxo1hrrcnjBtPBM65hJSTA9dcA9deCyecAPvvH3ZEqcsTgXMu4axbBz16wOOPwz/+AW++CRUqhB1V6vI6AudcwrnnHvjoI7tn4K9/DTua1OeJwDmXMFRtzOFBg6BrV2jbNuyI0oMXDTnnEsJ778Fxx8HatVYM5Emg5HgicM6F7tFH4YwzrIJ406awo0k/ngicc6HJyYGrr4a//c0qhydMgFq1wo4q/XgicM6FZuBAaxl0/fXwxhtQsWLYEaWntEkE330HN97oXdc6l0j+/nd4/nkYMgQyMsKOJn2lTSLIzoZ//hOmTAk7EufS27RpcMUVkJsLdevCJZeEHZFLm0RwwQVQvjw8+2zYkTiXvsaMsbuE338fFi8OOxqXJ20SQZUqcO65MGIErF8fdjTOpRdVaxnUowcccgh8+y3Uq7f797mSkTaJAKBfP0sCI0eGHYlz6aVfP2sZ1K0bfP45HHBA2BG5SGmVCI45Bg4/HIYODTsS51Lf//4Hq1fb81694Jln4K23vGVQIkqrRCACl10GkyfD99+HHY1zqWn+fBtT+Igj4MEH7bX27e2qwFsGJaa0SgQAvXtD2bJeaexccVu+3JqDNmkCr79uvYb+/e9hR+VikXaJYL/94Jxz4NVXYePGsKNxLnVcdZVVCPfuDXPmwAMP2O/NJb60SwRgl6hr1tidjM65otm6FZ58En75xf6/5x748UcbYL5u3XBjc3smLRNB27bQtKkXDzlXFNu2Wcu7Qw+1q4Dhw+31gw+211zySctEIAJ9+8KkSTB9etjROJc8PvkEsrKsFVCFCnZj2G23hR2V21tpmQjAWjVkZtplrHMuNu++CytXwssvw3//C1262ImVS26iSdYLW1ZWlmZnZxfLss49185wFi2CcuWKZZHOpaR166ByZRs0pmxZe7jkIiJTVDUr2rS0vSIAu6dg5UoYNSrsSJxLXO+8AwcdZMWo++zjSSAVpXUiaN8eDjzQK42dK8jy5TZ4fJ060Lhx2NG4eEnrRFCqlFUaT5gAs2aFHY1ziad/f1i1CoYNgzJlwo7GxUtaJwKAiy+G0qW90ti5/N54w+4Qvv12aNEi7GhcPKV9Ith/f+sR8aWXYMuWsKNxLnF8/LE1FR04MOxIXLzFlAhEpKKIlAqeNxGR7iKSGd/QSs5ll8GyZdY0zjlnnnnGWtWVLh12JC7eYr0imAiUE5E6wDigN/BSvIIqaaedBvXre6WxcwDjxsHs2XZ/QJUqYUfjSkKsiUBUdSNwFvCkqv4fcHj8wipZGRlw6aV2KTx3btjROBee33+H886Dyy8POxJXkmJOBCJyLHAB8H7wWkr1LH7JJdaK6Pnnw47EuXCoWoeMmzZZZ3IufcSaCK4FbgLeVtXpInIgMH53bxKRTiLys4jMEZEbo0zvIyLLRGRa8Oi7Z+EXn7p17Xb5F16wXhWdSzcvv2yDy993n3XK6NLHHncxEVQaV1LVtbuZLwOYBZwKLAQmA71U9aeIefoAWaraP9b1F2cXE/mNHm2Da7/9NpxxRlxW4VxCWrgQmjWzZqITJtjVsUste93FhIiMEJF9RKQi8CPwk4gM2M3bWgNzVHWuqm4BRgI99iTwktalC9Su7ZXGLv1UrWrFoy+84EkgHcX6lR8WXAGcAXwANMJaDhWmDrAg4v+FwWv5nS0iP4jImyJSL9qCRKSfiGSLSPayZctiDHnPlS5tP4YPPoDffovbapxLKKo2oPxDD9mYAi79xJoIMoP7Bs4ARqvqVqA4ui19D2ioqi2Aj4Fh0WZS1aGqmqWqWTVq1CiG1Rbs0kvt7wsvxHU1ziWE+fOhdWuYNi3sSFyYYk0EzwDzgIrARBFpABRaRwAsAiLP8OsGr22nqitU9c/g3+eAo2OMJ24aNrT7Cp5/HnJzw47GufhRtROfmTNh333DjsaFKaZEoKqPqmodVe2iZj5w8m7eNhloLCKNRKQMcK627pIAAB37SURBVB4wOnIGETkg4t/uwIw9iD1uLrvMKs8+/DDsSJyLn6efhk8/hQcftBMgl75irSyuIiIP5ZXTi8iD2NVBgVQ1B+gPfIQd4P8TND29U0S6B7NdIyLTReR74BqgT5E/STHq3h1q1fJKY5e65s6FAQPg1FPtxMelt1h7EXkBay3UM/i/N/AidqdxgVR1LDA232uDIp7fhN2fkFAyM6FPHxgyBBYvtpZEzqWSRx6xO+qff96HmnSx1xEcpKq3B01B56rqHcCB8QwsbH37Wh3Biy+GHYlzxe+hh+CLL6Be1HZ6Lt3Emgg2icgJef+IyPHApviElBgOPthGMHvuOdi2LexonCse8+dbT7sZGT7GgNsh1kRwOfCEiMwTkXnA48Bf4xZVgrjsMpg3z7ridS7Z5ebC+efDCSd4izi3s1hbDX2vqkcALYAWqnoU0D6ukSWAM8+EatW80tilhn//G776Cm691a4InMuzRzeTq+raiD6GrotDPAmlbFm46CJ45x1YsiTsaJwruo8+sgTQowf85S9hR+MSzd70KpIWbQ369oWcHBu827lkNHYsdO4MTZrYqGPeSsjltzeJoDi6mEh4hx4Kbdta8dAedtTqXEJo396uBr7+2u6PcS6/QhOBiKwTkbVRHuuAtGldf9llMGeOdc/rXDKYOdNujFyzBsqVgzvvtI7lnIum0ESgqpVVdZ8oj8qqmjZDWp9zjvXF4pXGLhm8/bZ1JPfNNz70qouN9zweg/LloXdveOstWLRo9/M7F4bcXLjlFjjrLCvSnDoVjjoq7KhcMvBEEKO//93+3n57uHE4V5CbboJ777UGDhMn2vCrzsXCE0GMGjWCq66yLiemTw87Gud2yGvE8Le/2Tgazz5rTZ+di5Ungj1wyy1QuTLceGPYkThnhg+3OqzcXKhTBy6+OOyIXDLyRLAHqlWzJDBmDHz+edjRuHS2dStce63dHLZ8OaxbF3ZELpl5IthDf/ublb3ecIPfV+DC8ccf0KGDdSV97bXWF5aPMOb2hieCPVS+vLXJ/u47ePPNsKNx6UbVuonIzrZioX//28bPcG5viCbZaW1WVpZmZ2eHGkNuLhx5JGzebBXHZcqEGo5LA6r2KFUKJk+2fe6II8KOyiUTEZmiqlnRpvkVQRFkZMA//2l3Gw8dGnY0LtVt2wb/+AcMHGj/t2rlScAVL08ERdS5M7RrZ8VEa9fudnbnimTrVmsJ9NBDdgWaZBfwLkl4IigiEXjgARvtaciQsKNxqWjjRrtL+OWX4Y474NFHvedQFx+eCPZCq1Zw7rnw4IPw++9hR+NSiSp06wbvvw9PPgmDBnkScPHjiWAv3XOPXb4PHhx2JC6ViMDll8PIkXDFFWFH41KdJ4K9dNBB9oN9/nmYMSPsaFyymzPHRsQD+L//g549w43HpQdPBMXgttugQgXr9Mu5opo2zQaWv/JK2LAh7GhcOvFEUAxq1LCmfe++C5MmhR2NS0affw4nnWT3B3z2mQ8i40qWJ4Jicu21cMABMGCAN/Fze+bdd6FjR+s0btIkOOSQsCNy6cYTQTGpWNHuKfj6axshyrlYffed3SD2xRdQr17Y0bh05F1MFKOcHGjRwrqg+PFH7wPGFW7JEhtMXtVuFitfPuyIXCrzLiZKSOnScP/9MGuWtSJyLppt26wIsXlzWLjQmop6EnBh8kRQzLp1g7Zt7b6C9evDjsYlmq1b4ZJL7G70nj2hdu2wI3LOE0Gxy+t6YskSu+PYuTybNlmXEcOG2YnCY49Zb6LOhc13wzg45hg4+2z4178sITgHcPfd1mXEE0/A7bd7lxEucXgiiJN774U//7TOwlx6y+ud9pZb4IMP7IYx5xKJJ4I4adIE+vWz8QpmzQo7GheG1avhwgvh2GOtVVCFCna/gHOJxhNBHA0aZK1Bbr457EhcSfv4Y2sVNGKEFRN6XYBLZHHdPUWkk4j8LCJzROTGQuY7W0RURKK2cU1WtWpZM8G33rIbzVzq27TJin5OOw0qVbLv/c47fThTl9jilghEJAN4AugMHAb0EpHDosxXGfgb8G28YgnTdddZQrjhBu96Ih2ULm0Dy193HUydamNWOJfo4nlF0BqYo6pzVXULMBLoEWW+u4B/ApvjGAuLFsH118PixfFcy64qVbKmgl9+Ce+9V7LrdiVj82Y761+50u4m/+ILazrsN4m5ZBHPRFAHWBDx/8Lgte1EpCVQT1XfL2xBItJPRLJFJHvZsmVFCmbjRhv3ddiwIr19r1x6KTRtaj2UbtlS8ut38TN1KmRlWXPQvHEEypYNNybn9lRoVVgiUgp4CLh+d/Oq6lBVzVLVrBo1ahRpfY0bWze/zz9f8kU0mZl2hjhzpl2VuOS3datdBbRpY1cCY8faHcPOJaN4JoJFQGRfinWD1/JUBpoBE0RkHnAMMDqeFcZ9+8Ivv1jf7yWta1crN378cRt+0CW3m26yq4CePa2Dwc6dw47IuaKLW++jIlIamAV0wBLAZOB8VZ1ewPwTgH+oaqFdi+5N76ObNtmYAd26wSuvFGkRe2XrVmjfHv77X+t6+LBdqs5dItu2DdasgapVra7pq6/gnHPCjsq52ITS+6iq5gD9gY+AGcB/VHW6iNwpIt3jtd7ClC8Pl11mFbhhyMyE11+3sQvOOcc7pUsmixfDySdbX0HbtllncZ4EXKrw8QhCMH48nHKKFSuMGOF9ziS66dOt6GfVKuso7qKL/DtzycfHI4hixozw1n3yydYB2ciR1gGZS1zjx8Pxx9ugQxMnQp8+ngRc6knLRDBihJXP//e/4cUwcCCcfrpVIH/zTXhxuILl5MAVV0DduvYdHXVU2BE5Fx9pmQg6d7a23mGOIlaqFLz8sh1kevaE5cvDi8XtTNWGGy1dGsaMsZsB69cPOyrn4ictE0HVqtYR2KuvWkuiMON4801YuhQuuMAOPi5cOTlw+eX2UIWDD4Z99w07KufiKy0TAdg9BWvWwKhR4cbRsqVVQI4bB3fdFW4s6W79eujRw7oOr1kz7GicKzlpmwhOOgkOPBCGDw87EktKF11kd6p++GHY0aSnP/6wfeKjj+CZZ+Cee7xS2KWP0mEHEJZSpeDtty0ZhE0EnnzSKq8vuMD+epl0ycnNtea88+bB6NHQpUvYETlXstL2igCgRYvwbi7Lr0IFqy/IyYH/+z8b5tKVjIwM6wvq8889Cbj0lNaJAKxr6FNPtQNw2Bo3hhdftO4nvHO6+HvtNXj6aXvesSMcfXS48TgXlrRPBLm58MknNqh4IjjrLEsCTzxh9zu44qcK//wnnH8+/Oc/1mWEc+ks7RNB1642gliY9xTkd999cMIJ1i/STz+FHU1qycmBq66CG2+E886zEwAfT9ilu7T/CWRmWoudMWOs5UgiyOucrnJlu99h3bqwI0oN27bZ9nzqKbuze/hwH0TGOfBEANiAIrm54YxeVpData0volmz7MogyfoGTEilSkHbtlbsdv/9fiXgXB7/KWDDSF57LTRrFnYkO2vXztqzv/66DWjj9pwqvPACvB8MhvqPf8CVV4Ybk3OJxhNB4N//tvqCRHPDDTaQznXXwddfhx1Nclm82Dr2u/RS69fJORedJ4IIf/yx48wxUZQqZUVW9epZ+fZbb3krl91RtfL/Zs2sG+lHHrGmos656DwRRLjjDruZa82asCPZWdWq8M47sM8+NipWVpYNlu71BtF99hn85S9w6KEwbRpcc43XBzhXGP95RLj0UuuNNBHPHlu0sEHShw2D1autGOv44+2g58y8efa3fXu7S3viRGjSJNSQnCsWqnaj6dKl8Vm+J4IIRx9tB9znngs7kuhKl4YLL4Sff7aO0RYsgA4d7MA3aVLY0YVnxQq7J6BZM0sGIlaMlpERdmTOFV1OjhVtXn219T3Wpk38TlI9EUQQsZ5Ap0yxIoVElZkJ/frB7NlW/v3TT3YDWpcuFns6GT0aDj/cuhO/6SYb6Me5ZLV5s9VTXnop7L+/neQ995wVB7/8sp0IxoMngnwuuADKl0+OM+xy5az8+5dfrMuEb7+1Heass6wYKZVt22bjB/foYT+YyZPhllvsqsm5ZLJunXV1ct55UKOGtXR7803o1Mkahyxfbj0l9+5t9YXxIJpkNY5ZWVmanZ0d13WsWhW/DR5Pa9fCww9bT5rr1tmONXhw6paTX3ONfU+33AJlyoQdjXOxW7HCrmZHjYKPP7behmvWhDPOsBO5k08u/n1aRKaoalbUaZ4ICpaTk5xnmCtXwpAhVmy0ebN1oTFoEDRsGHZke2ftWrshrG9faN3aKtB88BiXLBYuhHfftYP/559bbwYNGtiB/8wz4bjj4luv5YmgCK64wopcxo2L+6riZskS60rhqaesKKVfP/jXv6zoK1nMmWOtJbKz7XJ50SK7+e+aa8KOzLnCbdhgB/yPP7bH9On2+qGH2sH/rLPgqKNK7mSmsESQhOe7JaNOHeur/pdf4KCDwo6maGrVsoPm9ddbVxVPPglTp9olafXqYUe3M1Vr8ZOdbTf2XX21vX7hhXZHdbly0KqVdbdx7LGhhupcVLm51lgj78D/1Vewdavtu23b2pV5t25wyCFhR7orvyIowMKFdtl2001w991xX12JGDXKKsPr1bPul8NKcJFFOiNG2L0R2dlWpAVQpYqVoWZkWBKoUAEOO8xaSzmXSObO3XHg/+wzq18EO9M/9VR7nHCCJYOwedFQEXXtCt9/b2eqyVhXEM2kSdC9ux1k33/fzrJLwsqV8NhjO4p5fvoJqlWDBx6wttFZWTsezZp599AuMa1aZQf8vIP/3Ln2er16Ow78HTpY659E44mgiEaNshuTxoxJzA7piurnn6FzZ6tDeP11a64WT3/8YT+Q6dPtzD4rC+66y348ziWylSvt5GniRHtkZ1t9W6VK1rLntNNs327SJPEbLngdQRGdfrpVrmZF3XTJq2lTK3Lp2tXa4T/5JPz1r/FZV04OnHIK/PqrnUF16BCf9ThXHBYtgi++sMfEiTvuxylTxq6eb73VDvxt2qRWUaUngkKUKWPNFVNRrVowYQKcey5cfrl1V3HXXcV/VlO6tN3LULu2NY9zLlGoWqu0iRN3HPzzinoqVbL99dxz4cQTrblyIpTzx4sngt1QteKTjAzrmTSVVKpk7ZqvvNJaFf32m93OXhw3svz4o/3IzjjDekx1Lmy5ufC//+184F+yxKZVr24te/r3t79HHpk69YKxSKOPWjQiNjrYsmV2QEv0csA9Vbq0dWBXvz7cdpsN5vLWW9Zyp6iys6FjR+s2u3Nnr/h14VmzBj780E54PvjAeu4F299POcXO9tu2tSadqfbb3hOeCGLQty9cfLFVGp1wQtjRFD8RK/usV88+64kn2ngHders+bK++MLqHqpXh08+8STgSt5vv9m9MqNHW/Hn1q3WiueMM6yOqm1baxrudvBWQzHYsAEOOMBuA3/xRRvkZP16a32zdq3165P3t3Nn68rhv/+FRx/dedq++1rRS1EOsCXl44+tpVSVKnYGtSfjOI8bZz+2Bg0sCSTy53SpQ9V+b6NH25l/Xs/BTZtaY4ju3eGYY7xbcm81tJcqVrQO3J591pqLXXCB3aF70km7zjtqlCWClSvh00+hcmUrIqlc2Sqi1q0r8fD3yKmnWhlqly529fPOO9CuXWzvnTDBfnwffWQdaDkXL1u22P6Wd+a/YIFd2R53nN2b0r277YsuNnG9IhCRTsAjQAbwnKren2/65cBVQC6wHuinqj8VtswwrgjAmpU9/7zdIn7UUXag//LLHQf5vL/77VdwZWteJ3aqVlaZyD2c/vabXd3MmQMvvQS9ehU877p19tlV7eqpUqUSC9OlCVW7mSuyvH/dOrvr/LTT7MB/+umJeSNXogjlhjIRyQBmAacCC4HJQK/IA72I7KOqa4Pn3YErVbVTYcsNKxEUp9tus7tpJ0xI7IFUVq2y4rDPP7fxDgYM2LVC7ZlnbKznSZOgUaNw4nSJb+lSO4ivWmUnCxs37vyI9lr+1/MOVbVq2QlZjx5W5p9MnSiGKayiodbAHFWdGwQxEugBbE8EeUkgUBFIrgqLIura1eoP2rWzoegS9Q7bqlWtmKdPHxg40Iq2brttR9n/kCGWHE4/3epQnIu0dq0VLY4YYXVGubk7polYkWuFCvaIfF6jxq6vVahgV515bfpL+ZBaxSqeiaAOsCDi/4VAm/wzichVwHVAGaB9tAWJSD+gH0D9+vWLPdCSdswxVrF62ml2m3oiJ4OyZWH4cIvvX/+yK4DjjrPin3Hj7IabV15JrbssXdFt3mzFNq+9Bu+9Z/83bAg33AA9e1pDggoVrPg0nZtrJpp4Fg2dA3RS1b7B/72BNqrav4D5zwc6qupFhS03FYqG8nz7rSWDAw6wG10S/WD68882JsDTT1vvrGC32vfsafdYpECOdkWQm2vFnCNG2D0oa9bYWf2558L559uJjx/0wxdWHcGxwGBV7Rj8fxOAqt5XwPylgFWqWuitTKmUCMCSweLFVhafLDZsgPvus7LZN9/c0VyvTRu7+/qcc7yddqpTtXGiX3sNRo60jgUrV7bBVs4/3wZdT6c7c5NBWImgNFZZ3AFYhFUWn6+q0yPmaayqs4Pn3YDbCwo0T6olgkhjxkCLFol3Zq1qrYheegmuu85+8JHmzLGE8MYb1qwWrIOuvKTglcipY8YMO/iPGGGDNpUpY3Ve559vf73iNnGF1g21iHQBHsaaj76gqveIyJ1AtqqOFpFHgFOArcAqoH9koogmVRPBunVw4IHWDHX8+PCSwbZtVhG3ZIl1FvfDD9Zv0NqgWn/4cPvRF+SXX3YkhSlT7LWsLEsIZ50FBx/sxQSJYNs2+07XrLGmzGvW7PyI9tqCBdaVeKlSdsZ//vl2JbvvvmF/GhcLH48gSUyebDd07beflbnGMxmoWr1E5OOHH6yF0F132Q+/USO7s7h5c3u0aWP3UMRq7lwrM37jDftsYC2RjjjCOvXK+3vYYcXT0Z3bYds2G1Dpxx93fL8zZ9r9L6tXx3ZjY7lydod5lSp2sN9vP+jUyeqEvJVY8vFEkETikQxyc225EybYcvv1s0RQpYodEDIzbUDt5s13DKoNOw8pubfmzbN25NOm2eOHH2DTJpuWt/4jj9yRII44wkYwc7u3bJkd6CMP+tOnWzcoeRo1sm1cs+bOB/e859H+936iUosngiSTlwxuvNEeRfXOO3Y2/tFHNgYwWJv/996z5+PG2TgBTZuWfIul3FyrW5g2zYYDzfu7ePGOeerW3TU5NGqUvpWQGzfaAT7/QT+vK2Wwzv6aN9/5Su7ww3et13HpxxNBEvrtN2u7LxLbmfm2bdb985dfwt//bvNfcIF1Itepk/UddOqpiX+WvXSpJYTI5DBjxo6bkTIzrZ6hadNdH4n+2WKRmwvz58OsWTseP/9sf3/7bcd85cvbAT7/Qb9WLa+DcdF5IkhiM2fCJZdYS438TTLz+l4ZO9bO+pcts4PAr7/avKtXW+Vzst+FuXmznQn/8IMdFPMec+ZYF8N5qlXbNTkccohVwu+uDkLVllVQlwcbNsCff9oBuFIlu+u1YsUdzytViv0mKVX7rqId7OfMsQ7V8uyzj32OJk3sb97B/8ADvTdNt2c8ESSxKVNsAI1997XWRCtXWkXdAQfYyGnnnWfFAR072ln/aafZ/+kgJ8fqHiKTQ97jjz92zJeRYQfOhg3tPQUd6CO7QCiKjIydE0P+ZFG6tCXpWbN2DJACO65y8g72TZrseNSs6Wf4rnh4Ikhyeclg7VorAnrgAevjZ+1aKzbJyvKzw/zWrNlxpp33mD/fKkCj9W8Ty/9ly1oF94YNVhG7p3+3bLFklP+A36BB+tZ7uJLjiSAFTJsGL7xgB/1Onby/f+fcnvGBaVLAkUdaj6XOOVfckrwa0Tnn3N7yROCcc2nOE4FzzqU5TwTOOZfmPBE451ya80TgnHNpzhOBc86lOU8EzjmX5pLuzmIRWQbMDzuOAlQHlocdRCE8vr2T6PFB4sfo8e2dvYmvgarWiDYh6RJBIhOR7N2NuRwmj2/vJHp8kPgxenx7J17xedGQc86lOU8EzjmX5jwRFK+hYQewGx7f3kn0+CDxY/T49k5c4vM6AuecS3N+ReCcc2nOE4FzzqU5TwR7SETqich4EflJRKaLyN+izNNORNaIyLTgMaiEY5wnIv8L1r3LcG5iHhWROSLyg4i0LMHYmkZsl2kislZErs03T4lvPxF5QUSWisiPEa/tJyIfi8js4G/VAt57UTDPbBG5qIRi+5eIzAy+v7dFZN8C3lvovhDnGAeLyKKI77FLAe/tJCI/B/vjjSUY3+sRsc0TkWkFvDeu27CgY0qJ7n+q6o89eAAHAC2D55WBWcBh+eZpB4wJMcZ5QPVCpncBPgAEOAb4NqQ4M4A/sBtdQt1+wIlAS+DHiNceAG4Mnt8I/DPK+/YD5gZ/qwbPq5ZAbKcBpYPn/4wWWyz7QpxjHAz8I4Z94BfgQKAM8H3+31O84ss3/UFgUBjbsKBjSknuf35FsIdU9XdVnRo8XwfMAOqEG9Ue6wG8rOYbYF8ROSCEODoAv6hq6HeKq+pEYGW+l3sAw4Lnw4Azory1I/Cxqq5U1VXAx0CneMemquNUNSf49xugbnGuc08VsP1i0RqYo6pzVXULMBLb7sWqsPhERICewGvFvd5YFHJMKbH9zxPBXhCRhsBRwLdRJh8rIt+LyAcicniJBgYKjBORKSLSL8r0OsCCiP8XEk4yO4+Cf3xhbr88tVT19+D5H0CtKPMkwra8BLvCi2Z3+0K89Q+Kr14ooGgjEbZfW2CJqs4uYHqJbcN8x5QS2/88ERSRiFQC3gKuVdW1+SZPxYo7jgAeA94p4fBOUNWWQGfgKhE5sYTXv1siUgboDrwRZXLY228XatfhCdfWWkRuAXKA4QXMEua+8BRwEHAk8DtW/JKIelH41UCJbMPCjinx3v88ERSBiGRiX9hwVR2Vf7qqrlXV9cHzsUCmiFQvqfhUdVHwdynwNnb5HWkRUC/i/7rBayWpMzBVVZfknxD29ouwJK/ILPi7NMo8oW1LEekDnA5cEBwodhHDvhA3qrpEVXNVdRvwbAHrDnVfFJHSwFnA6wXNUxLbsIBjSontf54I9lBQnvg8MENVHypgnv2D+RCR1th2XlFC8VUUkcp5z7FKxR/zzTYauDBoPXQMsCbiErSkFHgWFub2y2c0kNcK4yLg3SjzfAScJiJVg6KP04LX4kpEOgE3AN1VdWMB88SyL8Qzxsh6pzMLWPdkoLGINAquEs/DtntJOQWYqaoLo00siW1YyDGl5Pa/eNWEp+oDOAG7RPsBmBY8ugCXA5cH8/QHpmMtIL4BjivB+A4M1vt9EMMtweuR8QnwBNZa439AVglvw4rYgb1KxGuhbj8sKf0ObMXKWS8FqgGfArOBT4D9gnmzgOci3nsJMCd4XFxCsc3Byobz9sGng3lrA2ML2xdKcPu9EuxfP2AHtQPyxxj83wVrKfNLvGKMFl/w+kt5+13EvCW6DQs5ppTY/uddTDjnXJrzoiHnnEtzngiccy7NeSJwzrk054nAOefSnCcC55xLc54InAuISK7s3DNqsfWEKSINI3u+dC6RlA47AOcSyCZVPTLsIJwraX5F4NxuBP3RPxD0Sf+diBwcvN5QRD4LOlX7VETqB6/XEhsj4PvgcVywqAwReTboc36ciJQP5r8m6Iv+BxEZGdLHdGnME4FzO5TPVzR0bsS0NaraHHgceDh47TFgmKq2wDp9ezR4/VHgc7VO81pid6QCNAaeUNXDgdXA2cHrNwJHBcu5PF4fzrmC+J3FzgVEZL2qVory+jygvarODToH+0NVq4nIcqzbhK3B67+ranURWQbUVdU/I5bREOs3vnHw/0AgU1XvFpEPgfVYL6vvaNDhnnMlxa8InIuNFvB8T/wZ8TyXHXV0XbG+n1oCk4MeMZ0rMZ4InIvNuRF/vw6ef4X1lglwAfBF8PxT4AoAEckQkSoFLVRESgH1VHU8MBCoAuxyVeJcPPmZh3M7lJedBzD/UFXzmpBWFZEfsLP6XsFrVwMvisgAYBlwcfD634ChInIpduZ/BdbzZTQZwKtBshDgUVVdXWyfyLkYeB2Bc7sR1BFkqerysGNxLh68aMg559KcXxE451ya8ysC55xLc54InHMuzXkicM65NOeJwDnn0pwnAuecS3P/Dw7IaY59ea9FAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CjFPbGeNVOSv"
      },
      "source": [
        "#### Adding weight regularization"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "simple model -> parameter values less entropy   \n",
        "- L1 norm : weight의 절댓값에 비례하는 비용 추가\n",
        "- L2 norm : weight의 제곱에 비례하는 비용 추가 "
      ],
      "metadata": {
        "id": "sENrVTxLxjqC"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y81MYEDlVOSw"
      },
      "source": [
        "**Adding L2 weight regularization to the model**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d0Cn5XvcVOSw",
        "outputId": "542f5083-b6ae-4721-c60e-d0de139a9fdc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "30/30 [==============================] - 2s 50ms/step - loss: 0.5873 - accuracy: 0.7808 - val_loss: 0.4614 - val_accuracy: 0.8629\n",
            "Epoch 2/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.3962 - accuracy: 0.8941 - val_loss: 0.3860 - val_accuracy: 0.8837\n",
            "Epoch 3/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.3329 - accuracy: 0.9127 - val_loss: 0.3609 - val_accuracy: 0.8904\n",
            "Epoch 4/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.3001 - accuracy: 0.9231 - val_loss: 0.3544 - val_accuracy: 0.8880\n",
            "Epoch 5/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.2766 - accuracy: 0.9317 - val_loss: 0.3677 - val_accuracy: 0.8801\n",
            "Epoch 6/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.2643 - accuracy: 0.9369 - val_loss: 0.3642 - val_accuracy: 0.8820\n",
            "Epoch 7/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.2566 - accuracy: 0.9385 - val_loss: 0.3566 - val_accuracy: 0.8855\n",
            "Epoch 8/20\n",
            "30/30 [==============================] - 1s 38ms/step - loss: 0.2438 - accuracy: 0.9466 - val_loss: 0.3581 - val_accuracy: 0.8831\n",
            "Epoch 9/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.2373 - accuracy: 0.9465 - val_loss: 0.4094 - val_accuracy: 0.8645\n",
            "Epoch 10/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.2319 - accuracy: 0.9465 - val_loss: 0.3728 - val_accuracy: 0.8796\n",
            "Epoch 11/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.2248 - accuracy: 0.9515 - val_loss: 0.4164 - val_accuracy: 0.8685\n",
            "Epoch 12/20\n",
            "30/30 [==============================] - 1s 38ms/step - loss: 0.2240 - accuracy: 0.9501 - val_loss: 0.3829 - val_accuracy: 0.8779\n",
            "Epoch 13/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.2183 - accuracy: 0.9531 - val_loss: 0.3994 - val_accuracy: 0.8743\n",
            "Epoch 14/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.2180 - accuracy: 0.9518 - val_loss: 0.3991 - val_accuracy: 0.8756\n",
            "Epoch 15/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.2095 - accuracy: 0.9566 - val_loss: 0.3916 - val_accuracy: 0.8781\n",
            "Epoch 16/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.2073 - accuracy: 0.9571 - val_loss: 0.3930 - val_accuracy: 0.8777\n",
            "Epoch 17/20\n",
            "30/30 [==============================] - 1s 38ms/step - loss: 0.2085 - accuracy: 0.9552 - val_loss: 0.4177 - val_accuracy: 0.8726\n",
            "Epoch 18/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.2035 - accuracy: 0.9587 - val_loss: 0.4463 - val_accuracy: 0.8648\n",
            "Epoch 19/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.2020 - accuracy: 0.9587 - val_loss: 0.4010 - val_accuracy: 0.8774\n",
            "Epoch 20/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.1995 - accuracy: 0.9590 - val_loss: 0.4042 - val_accuracy: 0.8771\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.keras import regularizers\n",
        "model = keras.Sequential([\n",
        "    layers.Dense(16,\n",
        "                 kernel_regularizer=regularizers.l2(0.002),\n",
        "                 activation=\"relu\"),\n",
        "    layers.Dense(16,\n",
        "                 kernel_regularizer=regularizers.l2(0.002),\n",
        "                 activation=\"relu\"),\n",
        "    layers.Dense(1, activation=\"sigmoid\")\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"binary_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_l2_reg = model.fit(\n",
        "    train_data, train_labels,\n",
        "    epochs=20, batch_size=512, validation_split=0.4)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "val_loss = history_original.history[\"val_loss\"]\n",
        "val_loss2 = history_l2_reg.history[\"val_loss\"]\n",
        "\n",
        "epochs = range(1, 21)\n",
        "plt.plot(epochs, val_loss, \"b--\",\n",
        "         label=\"origianl\")\n",
        "plt.plot(epochs, val_loss2, \"b-\",\n",
        "         label=\"l2-regularizied\")\n",
        "plt.title(\"Effect of insufficient model capacity on validation loss\")\n",
        "plt.xlabel(\"Epochs\")\n",
        "plt.ylabel(\"Loss\")\n",
        "plt.legend()\n",
        "\n",
        "# l2규제를 사용한 model이 기본 모델보다 overfitting에 잘 버팀 "
      ],
      "metadata": {
        "id": "ZraMuJQQy7nZ",
        "outputId": "b0aa35b9-da6e-4b13-d6b5-a8421ab71087",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 313
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7f87a4e498d0>"
            ]
          },
          "metadata": {},
          "execution_count": 49
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5gUVdbA4d8hScYAKhIEFZGowjCoGDALKmbFgIIiJgy76qqfyiKK67rqGldFRFAxp0VExQCiGGBwkSWohEUdkuSMTDjfH6cGmqFnpid013T3eZ+nn5nuqq46XV1dp+reW/eKquKccy59VQk7AOecc+HyROCcc2nOE4FzzqU5TwTOOZfmPBE451ya80TgnHNpLm0SgYjcJyIrRGRp8PwsEflNRDaIyKEhxlVsHMHr+8U5hm4iMjdY15kispeITBKR9SLysIj8n4gMj2E5z4jI3fGMNRFEpK+IfBXjvCNF5L54x1QRRORDEbks7Dgqioh0F5HsiOezRKR7LPOWYV1x2bdFZLCIvFzRyy2tamEHUFFEZCGwF5AX8fJIVR0oIs2Bm4F9VfX3YNpDwEBV/Xc516tAK1WdV8ZFFBuHqtYtc3CxGwI8qaqPAQQ7/AqgvpbiRhNVvboiggl+zC+ratOKWJ4zqtqj4H8R6Qv0V9Ujw4uoYqlqu4pYTrRtU1H7dmWVMokgcLqqfhrl9ebAyogkALAvMCsxYRWrMsRROIZ9gdmlSQLOuSSmqinxABYCJ0R5/QRgM5APbABeDf4qsBGYH8y3D/A2sBz4H3BDxDKqAv8HzAfWA9OAZsCkiOVsAC6Isv4qwF3AL8DvwItAA2CXaHFEeb8CBwT/jwSeAj4I4vgO2D+YJsA/g3WsA/4LtA+mTcTOcAqW2Rf4Kvh/frBtNkdsnxxga/D8BGAwdoZe8P4jga+BNcBvQN+I+O6LmO80YHow39dAx0Lf1y3ADGAt8DpQE6hT6PvaAOwTZbuMBP4FfBjMMxnYG3gUWA38CBwaMX+bYDuswZJer4hpewBjgu02Bbi3YPsE0w8CPgFWAT8B5xeK475o310w/UpgTvB9zQY6Ba/fzvb9aTZwVqHvZzLwZLBtfgSOj5jeL2KZC4CrCq3zjGC7rwvWcUrkfhBsiy3Y1fOGYJt0AZYBVSOWczbwQxGfqwG2Ly/H9u27gCqR+xd2tbsa+z31KGI5twFvFXrtMeDxkj4r0B3IjnYMAGoF383qYPveWmjeqNs/2rYpYt++EpgX7BNjiNhHsd/s1cDcYNs+BUgRn38wO/62emH755rg+2pTaFstCmL+qWCfADKBrOD7XgY8UurjZ0UfkMN6UEQiiLbDRHxZBQfYKtjBfRBQA9gv2OlODqbfih1YW2MH3IOBPQovp4h1Xx7sMPsBdYF3gJeixVHE+wsngpXBF18NGA28Fkw7OfgMuwYxtgEaB9MmUkQiiLbtouz023ZW7GphPXAhUB07iB5S+H3AoVhS6ool0suC9ewSsc4pWALeHfuxX13U9xVlu4zEiq86Ywnkc+yAc2mwvvuACcG81YPv4P+C7/e44DO0Dqa/BryBJaH22I+tIFHWwZJdv2CbHxqst220bVUoxvOCZXUJvpMDsOLJgmn7YPveBdjJQOOI7ycX+FMQ+wVYQtg9mH4qsH+wzGOATWxPMJnBvCcGy24CHFR4Pyi8DwSvzSbigA28C9xcxGd7Efg3UA9oAfwMXBGx7BzsYFkVuAZYTJSDIbY/bQLqBc+rAkuAw2L4rDvsJ+yYCB4AvsT2rWbAzELzlrT9C2+bbd8ztv+sADphJ3RPAJMK/WbHYr/F5liyPKWI7TiY7b+tA4M4Tgy+979g+20N7NjzG0HCCbZ5wUngN0Cf4P+6BduuNI9Uqyx+T0TWRDyujPF9XYBGqjpEVbeq6gLgOaB3ML0/cJeq/qTmB1VdGeOyL8Yy9AJV3QDcAfQWkbIWy72rqlNUNRdLBIcEr+dgP8qDsB/cHFVdUsZ1FOci4FNVfVVVc1R1papOjzLfAOBZVf1OVfNUdRTwB3BYxDyPq+piVV0FvB/xWWL1rqpOU9Ut2EFri6q+qKp52BVGQeX7YdgP5IHg+/0c+6FeKCJVgXOAQaq6UVVnAqMi1nEasFBVX1DVXFX9D3bleF4M8fUHHlTVqcF+M09VfwFQ1TeDz56vqq9jZ4+ZEe/9HXg02MavY2eApwbv/UBV5wfL/AIYDxwVvO8KYISqfhIse5Gq/hjj9hwFXAIgIrtjJxevFJ4p2Ga9gTtUdb2qLgQeBvpEzPaLqj4XfBejgMZYHd4Ogu3xPXBW8NJxwCZV/TaGz1qc84GhqrpKVX8DHi+03pK2f3Euxrbx96r6B/abPlxEWkTM84CqrlHVX4EJxLZvXwB8EHx3OdgVVS3gCOwKZRegrYhUV9WFqjo/eF8OcICINFTVDQXbrjRSLRGcqaq7Rjyei/F9+wL7RCYR7OyxYMdthl1GlsU+2KVzgV+wM8udfhQxWhrx/ybsAEdwcHsSuwz9XUSGiUj9Mq6jOLFui32Bmwtt02bY9igQ9bOUwrKI/zdHeV6wvH2A31Q1P2L6L9jZciPs+/it0LTIz9G10Oe4GCuGKkmR20pELhWR6RHLbA80jJhlkQaneBEx7RO8t4eIfCsiq4L39ox4b3n21ZeB00WkDnYg/bKIk4mG2Blr4f26ScTzbd+tqm4K/i3q+30Fu8IEO9HYlnxK+KzF2Yeiv9NYtn9Jy962vOAEbyVFfH5i37cLLzc/+AxN1Bqj3IRdQfwuIq+JSMFv6QrsauJHEZkqIqfF+Dm2SbVEUFa/Af8rlETqqWrPiOn7l3HZi7GDSYHm2GX/suizl52qPq6qnYG22I5xazBpI1A7YtZYDmJFiXVb/IadkUVu09qq+moM763oSurFQDMRidzfm2PFNsux76NZoWkFfgO+KPQ56qrqNTGsN+q2EpF9sSvOgVgR465Y0YVEzNZERCKfNwcWi8gu2BXJQ8BewXvHRbw31u9np22sqouwYoazsbP7l4p47wrsLLTwfr0ohvVG8ybQXUSaYlcGrwDE8FmLs4QivtMYtn9J+98Ov+kgce5B2T9/UcsV7DMsAlDVV9RaMu0bxPj34PW5qnohsGfw2ltBTDHzRGCmAOtF5DYRqSUiVUWkvYh0CaYPB+4VkVZiOorIHsG0ZVj5f1FeBf4kIi1FpC5wP/B6ULRTYUSki4h0FZHq2IF/C1bhClZxeLaI1BaRA7AziLIaDZwgIueLSDUR2UNEol32PgdcHcQkIlJHRE4VkXoxrGMZsIeINChHnJG+w87K/iIi1YPmqadj9St5WL3N4GD7tMXqMwqMBQ4UkT7Be6sH27pNDOsdDtwiIp2DbXBAcBCqg/2QlwOISD/sjDTSnsANwfrOw+p8xmHlxbsE780VkR7ASRHvex7oJyLHi0gVEWkiIgdFiW0Z0FREahR6/UWsbLpDsF12EmyzN4ChIlIv+Ex/xq4oSk1Vl2P1Fy9gJ2RzgkklfdbivAHcISK7BQnm+ohpJW3/orZNgVexbXxIkKzuB74LisjK4w3g1OC7q441ef8D+FpEWovIccH6trC9QQUicomINAquINYEy8qPsvwipVoieF/spqiCx7uxvCnYsU/DyvH+h53xDMdaRgA8gn1J47Ga+eexsjuwS7VRwSXm+VEWPwI7s5oULHsLO+6UFaU+dvBdjV1ergT+EUz7J9YKaBlWXju6rCsJyjx7YjvpKizJHBxlviyssvDJIKZ5WCVcLOv4EfuxLQi26z4lvaeE5W3FDvw9sO/2X8ClEWXnA7FL96VYpeALEe9djx18emNnbEuxs65dYljvm8BQ7Ax3PfAeVuE7GytT/wb7TjpgrYQifQe0CuIdCpyrVh+zHrgB2x9XY0UpYyLWOQWr2P4nVmn8BTueuRf4HGudslREVkS8/m4w/7sRRTrRXI+dcCzAWgi9gu3rZfUK1kJtW7FQSZ+1BPdgv4P/Yb/bbVc3MWz/orZNwfs/Be7GrlaWYFdgvQvPV1qq+hNWR/ME9r2fjjWJ34rtbw8Ery/FThTuCN56CjBLRDZgLa56q+rm0qxbdiyGdM6FTUK+2UtE5mPNNKPdk+NSUKpdETjnykFEzsGKTT4POxaXOKl2Z7FzroxEZCLW0KBPoRZWLsV50ZBzzqU5Lxpyzrk0l3RFQw0bNtQWLVqEHYZzziWVadOmrVDVRtGmJV0iaNGiBVlZWWGH4ZxzSUVEfilqmhcNOedcmvNE4Jxzac4TgXPOpbmkqyOIJicnh+zsbLZs2RJ2KK4ENWvWpGnTplSvXj3sUJxzgZRIBNnZ2dSrV48WLVqwY4eNrjJRVVauXEl2djYtW7YMOxznXCAlioa2bNnCHnvs4UmgkhMR9thjD79yc66SSYlEAHgSSBL+PTlX+aRMInDOuVSVkwPLl8dv+Z4IEqxnz56sWbOm2HkGDRrEp5+WrQfgiRMnctpppR6pzjlXSW3eDOecA8ceC/EqVU2JyuJkoKqoKuPGjStx3iFDhiQgIudcZbd2LfTqBV9+CU89BTVrxmc9fkVQgR555BHat29P+/btefTRR1m4cCGtW7fm0ksvpX379vz222+0aNGCFSts0KN7772X1q1bc+SRR3LhhRfy0EMPAdC3b1/eeustwJJCly5daN++PQMGDKCgt9ju3btz2223kZmZyYEHHsiXX34Zzod2zsXNP/4BX38No0fDNbGMkl1GKXlF0L37zq+dfz5cey1s2gQ9e+48vW9fe6xYAeeeu+O0iRNLXue0adN44YUX+O6771BVunbtyjHHHMPcuXMZNWoUhx122A7zT506lbfffpsffviBnJwcOnXqROfOnXda7sCBAxk0aBAAffr0YezYsZx++ukA5ObmMmXKFMaNG8c999xT5uIk51zlNGgQnHoqHH54fNfjVwQV5KuvvuKss86iTp061K1bl7PPPpsvv/ySfffdd6ckADB58mTOOOMMatasSb169bYd3AubMGECXbt2pUOHDnz++efMmjVr27Szzz4bgM6dO7Nw4cK4fC7nXGLNmQMnnmgnpTVqxD8JQIpeERR3Bl+7dvHTGzaM7QogVnXq1Cnze7ds2cK1115LVlYWzZo1Y/DgwTu0wd9lFxs/vWrVquTm5pY7VudcuKZOhR49oFo1WLbMjkeJ4FcEFeSoo47ivffeY9OmTWzcuJF3332Xo446qsj5u3Xrxvvvv8+WLVvYsGEDY8eO3WmegoN+w4YN2bBhw7Z6A+dc6vnsMzjuOKhfHyZPhnbtErfuuF4RiMgpwGNAVWC4qj5QaPo/gWODp7WBPVV113jGFC+dOnWib9++ZGZmAtC/f3922223Iufv0qULvXr1omPHjuy111506NCBBg0a7DDPrrvuypVXXkn79u3Ze++96dKlS1w/g3MuHOPHw+mnQ6tW9v8++yR2/XEbs1hEqgI/AycC2cBU4EJVnV3E/NcDh6rq5cUtNyMjQwsPTDNnzhzatGlTIXEn0oYNG6hbty6bNm3i6KOPZtiwYXTq1CnssOIuWb8v5+Jl2TL485/hiSdg993jsw4RmaaqGdGmxbNoKBOYp6oLVHUr8BpwRjHzXwi8Gsd4Kp0BAwZwyCGH0KlTJ84555y0SALOue3eecfuGt5rL2siGq8kUJJ4Fg01AX6LeJ4NdI02o4jsC7QEPi9i+gBgAEDz5s0rNsoQvfLKK2GH4JwLgSrceSf87W/wr3/F9x6BWFSWyuLewFuqmhdtoqoOU9UMVc1o1Cjq2MvOOZcU8vLswP+3v8GAAfYIWzwTwSKgWcTzpsFr0fQmzYqFnHPpZ+tWuOgiePZZuOMOeOYZqFo17KjimwimAq1EpKWI1MAO9mMKzyQiBwG7Ad/EMRbnnAvdvHnw4YfWdcT990Nl6ZU9bnUEqporIgOBj7HmoyNUdZaIDAGyVLUgKfQGXtN4NV9yzrmQzZwJbdpA27Ywd65VDlcmca0jUNVxqnqgqu6vqkOD1wZFJAFUdbCq3h7POBKhbt26AEyfPp3DDz+cdu3a0bFjR15//fWExzJy5EgGDhxYqvdkZWVxww03lHl6NN27d6dwU1/n0sncuVYU1KEDFLQNqWxJAFK0i4kw1a5dmxdffJFWrVqxePFiOnfuzMknn8yuuxZ9n1xubi7VqoX3VeTm5pKRkUFGRtQmxgAlTnfObZedDUOGwIgRsMsu8H//B5V5mJDK0mooZRx44IG0atUKgH322Yc999yT5VGGFpo4cSJHHXUUvXr1om3btuTl5XHrrbfSpUsXOnbsyLPPPgtAfn4+1157LQcddBAnnngiPXv23NbVRGSX1llZWXSP0u3q+++/T9euXTn00EM54YQTWLZsGQCDBw+mT58+dOvWjT59+uwwoE3Pnj055JBDOOSQQ2jQoAGjRo3aYfrGjRu5/PLLyczM5NBDD+Xf//43AJs3b6Z37960adOGs846i82bN1fglnUuOajaXcIjR1qPx/Pnw9ChUExHA6FLuSuCm26C6dMrdpmHHAKPPlr6902ZMoWtW7ey//77R53+/fffM3PmTFq2bMmwYcNo0KABU6dO5Y8//qBbt26cdNJJTJs2jYULFzJ79mx+//132rRpw+WXF3vz9Q6OPPJIvv32W0SE4cOH8+CDD/Lwww8DMHv2bL766itq1arFxIie9goGz5k2bRr9+vXjzDPP5D//+c+26UOHDuW4445jxIgRrFmzhszMTE444QSeffZZateuzZw5c5gxY4bfIOfSxrp18OSTcP31UK+etQrac09o0SLsyGKTcomgsliyZAl9+vRh1KhRVKkS/cIrMzOTli1bAjB+/HhmzJix7Wx/7dq1zJ07l6+++orzzjuPKlWqsPfee3PsscdGXVZRsrOzueCCC1iyZAlbt27dtj6AXr16UatWrajvW7FiBX369OGNN97YqQ+k8ePHM2bMmG0D6WzZsoVff/2VSZMmbatH6NixIx07dixVrM4lm82bbeSwBx6AlSvhgANs7JOgy7GkkXKJoCxn7hVt3bp1nHrqqQwdOnTbWATfffcdV111FWCjjtWvX3+HLqpVlSeeeIKTTz55h2UVN7RltWrVyM/PB9ihe+pI119/PX/+85/p1asXEydOZPDgwdumFdVFdl5eHr1792bQoEG0b99+p+mqyttvv03r1q2LjM25VKYKw4ZZPcDixXDyyXDffZCs1WheR1DBtm7dyllnncWll17KuRFDnXXt2pXp06czffp0evXqtdP7Tj75ZJ5++mlycnIA+Pnnn9m4cSPdunXj7bffJj8/n2XLlu1QhNOiRQumTZsGwNtvvx01nrVr19KkSRMARo0aFdNnuP322+nYsSO9e/eOOv3kk0/miSee2DZsZkGx0dFHH72t24yZM2cyY8aMmNbnXLIRgTfftKKfiRPho4+SNwmAJ4IK98YbbzBp0iRGjhy5rcJ1egyVFv3796dt27Z06tSJ9u3bc9VVV5Gbm8s555xD06ZNadu2LZdccgmdOnXaVlTz17/+lRtvvJGMjAyqFnF74uDBgznvvPPo3LkzDWMc5eKhhx5i/Pjx2+IfM2bH+wDvvvtucnJy6NixI+3atePuu+8G4JprrmHDhg20adOGQYMGRR1607lkpApjxkCXLvDrr/ba22/DV1/BMceEG1tFiFs31PGSSt1Qx6qgu+qVK1eSmZnJ5MmT2XvvvcMOq8xS/ftyqUMVxo2DwYMhK8vGC3jpJegatfvMyq24bqhTro4gFZ122mmsWbOGrVu3cvfddyd1EnAuWeTmwtFHwzffQMuW8Pzz0KcPVK8edmQVzxNBEphYkYMoO+eKpArffw+dO9u4wccfD5dfDpddlpoJoEDK1BEkWxFXuvLvyVVGqjZEZLduVukbtMHg3nuhf//UTgKQIomgZs2arFy50g8ylZyqsnLlSmrWrBl2KM4BlgA++QSOPNKagGZnW9fQHTqEHVlipUTRUNOmTcnOzo7alYOrXGrWrEnTpk3DDsM5wO4IPvdcqF8fnn4a+vWzvoHSTUokgurVq+9wx6xzzkWjChMmwGuvWTcQDRrAp59Cx47pmQAKpETRkHPOlWTSJOje3SqAx46FX36x17t0Se8kAJ4InHNp4NtvLQnMnQuPPQYLFiRPh3CJkBJFQ845V5yHHoJGjeDHH60+wO3IE4FzLuW98ALMmeNJoCieCJxzKUsV8vNtjIBk6xo6kbyOwDmXssaPh3btrG7AFc0TgXMuJanC3Xfb4DHNm4cdTeXmRUPOuZQ0dixMnQrDh3vz0JL4FYFzLuXk59vVwP77w6WXhh1N5edXBM65lDNuHPzwg40dkOodxlUETwTOuZTTo4cNJXnWWWFHkhw8ETjnUooqVK1qncm52HgdgXMuZeTm2pgCL78cdiTJxROBcy5lvPSSDS1Zr17YkSQXTwTOuZSwdSvcc4+NMNarV9jRJBevI3DOpYQRI6xr6WeeAZGwo0kufkXgnEt6W7fCfffBEUfYkJOudPyKwDmX9GrUgFdegVq1/GqgLDwROOdSwtFHhx1B8vKiIedcUnvsMRg40JqOurLxROCcS1rr18O998K8eVDNyzfKLK6JQEROEZGfRGSeiNxexDzni8hsEZklIq/EMx7nXGp57DFYudKSgSu7uOVQEakKPAWcCGQDU0VkjKrOjpinFXAH0E1VV4vInvGKxzmXWtasgYcftnsGunQJO5rkFs8rgkxgnqouUNWtwGvAGYXmuRJ4SlVXA6jq73GMxzmXQh55xJLBkCFhR5L84lmq1gT4LeJ5NtC10DwHAojIZKAqMFhVPyq8IBEZAAwAaO5DDTnngD59YM894eCDw44k+YVdvVINaAV0B5oCk0Skg6quiZxJVYcBwwAyMjI00UE65yqfVq3s4covnkVDi4BmEc+bBq9FygbGqGqOqv4P+BlLDM45F9XSpXD++dZSyFWMeCaCqUArEWkpIjWA3sCYQvO8h10NICINsaKiBXGMyTmX5B54AN55x8YdcBUjbolAVXOBgcDHwBzgDVWdJSJDRKSgb8CPgZUiMhuYANyqqivjFZNzLrllZ1uncpdd5sVCFUk0ydJqRkaGZmVlhR2Gcy4E114Lw4fDzz9DixZhR5NcRGSaqmZEm+Z3FjvnksLChZYE+vf3JFDRwm415JxzMdl1V7j1VrsqcBXLE4FzrtJbu9YSwdChYUeSmrxoyDlXaalaf0KtWsECb08YN54InHOVUm4u3HAD3HQTHHkk7L132BGlLk8EzrlKZ/16OOMMePJJuOUWeOstqF077KhSl9cROOcqnaFD4eOP7Z6Bq64KO5rU54nAOVdpqNqYw4MGwamnwlFHhR1RevCiIedcpfD++3DEEbBunRUDeRJIHE8EzrnQPf44nHmmVRBv3hx2NOnHE4FzLjS5uXD99XDjjVY5PHEi7LVX2FGlH08EzrnQ3HabtQy6+WZ4802oUyfsiNKTVxY750Lzpz9Bu3Zw+eVhR5Le/IrAOZdQ06fDNddAXh40bepJoDLwROCcS5ixY+0u4Q8+gMWLw47GFfBE4JyLO1VrGXTGGXDQQfDdd9CsWcnvc4nhicA5F3cDBljLoNNPhy++gMaNw47IRfLKYudcXPz3v3bWv+uucOGF0KULXHEFVK0admSuML8icM5VqF9+sTGFDz4YHn7YXjvuOLsq8CRQOfkVgXOuQqxYYZ3F/etf1l/QLbdY81BX+XkicM5ViOuus+6i+/WDwYOtaahLDl405Jwrk5wcO/ufP9+eDx0KM2faAPOeBJKLJwLnXKnk58Nrr0GbNnYVMHq0vX7AAfaaSz6eCJxzMfv0U8jIsFZAtWvbjWF33x12VK68vI7AORezf/8bVq2CF1+Eiy7yVkCpQlQ17BhKJSMjQ7OyssIOw7m0sn491Ktng8bssos9XHIRkWmqmhFtmhcNOeeK9d57sP/+MGsW1K/vSSAVeSJwzhVpxQobPL5JE2jVKuxoXLx4HYFzrkgDB8Lq1fDJJ1CjRtjRuHjxROCci+rNN+H11+G++6Bjx7CjcfHkRUPOuag++cSait52W9iRuHiL6YpAROoAm1U1X0QOBA4CPlTVnLhG55wLzbPPWiuhal5ukPJivSKYBNQUkSbAeKAPMDJeQTnnwjN+PMydax3HNWgQdjQuEWJNBKKqm4CzgX+p6nlAu/iF5ZwLw5Il0Ls3XH112JG4RIo5EYjI4cDFwAfBa35PoXMpRNXGDNi82TqTc+kj1kRwE3AH8K6qzhKR/YAJJb1JRE4RkZ9EZJ6I3B5lel8RWS4i04NH/9KF75yrKC++aIPL/+1v0Lp12NG4RCp1FxMiUgWoq6rrSpivKvAzcCKQDUwFLlTV2RHz9AUyVHVgrOv3Liacq3jZ2dC+vTUTnTgRqnh7wpRT7i4mROQVEakftB6aCcwWkVtLeFsmME9VF6jqVuA14IzSBO6cS4zddoPLL4cRIzwJpKNYv/K2wRXAmcCHQEus5VBxmgC/RTzPDl4r7BwRmSEib4lIs2gLEpEBIpIlIlnLly+PMWTnXCxUoU4deOQRG1PApZ9YE0F1EamOJYIxwf0DFdFt6ftAC1XtCHwCjIo2k6oOU9UMVc1o1KhRBazWOQc20HxmJkyfHnYkLkyxJoJngYVAHWCSiOwLFFtHACwCIs/wmwavbaOqK1X1j+DpcKBzjPE458pJFa64An78EXbdNexoXJhiSgSq+riqNlHVnmp+AY4t4W1TgVYi0lJEagC9gTGRM4hI44invYA5pYjdOVcOzzwDn30GDz8MLVqEHY0LU6xdTDQA/gocHbz0BTAEWFvUe1Q1V0QGAh9j9xyMCJqeDgGyVHUMcIOI9AJygVVA37J+EOdc7BYsgFtvhRNPhCuvDDsaF7aYmo+KyNtYa6GCMvw+wMGqenYcY4vKm486V3433ggjR8LMmdAsahMNl2qKaz4aa3dS+6vqORHP7xERr15yLkk98ojVD3gScBB7ZfFmETmy4ImIdAM2xyck51y8/PILLF9ug877GAOuQKxXBFcDLwZ1BQCrgcviE5JzLh7y8uCii2z4ydmzLRk4BzEmAlX9AThYROoHz9eJyE3AjHgG55yrODGjSfcAAB9CSURBVP/8J3z9tfUp5EnARSrVzeSqui6ij6E/xyEe51wcfPwx3HUXnHEGXHJJ2NG4yqY8vYpIhUXhnIubceOgRw848EAbdUz8l+sKKU8iqIguJpxzcXbccXY18M03sNdeYUfjKqNiE4GIrBeRdVEe64F9EhSjc66UfvwRevWCtWuhZk0YMsQ6lnMummIri1W1XqICcc5VjHffhcsuswSwYAEcemjYEbnKznsedy5F5OXBnXfC2WdDmzbw/feeBFxs0iYRTJ4M111nPS46l4ruuAPuvx/694dJk6Bp07AjcskibRLBnDk2IPcnn4QdiXMVq+Dk5sYbbYSx556DXXYJNyaXXNImEVx6qZ0h3Xdf2JE4V3FGj4Zzz7VioSZNoF+/sCNyyShtEkGNGvCXv8CXX9pls3PJLCcHbrrJbg5bsQLWrw87IpfM0iYRgJWd7rknDB0adiTOld3SpXD88fDYY5YMPv3URxhz5ZNWiaBWLbjlFhg/HqZMCTsa50pP1bqJyMqyYqF//hOqVw87Kpfs0ioRAFx9Ney+u18VuOSiCvn51j3Ek0/aXcIXXRR2VC5VpF0iqFfPWleMGQMzvO9UlwTy8+1K9rbb7HmXLnDwweHGlG4+/9zuyXj//bAjiY+0SwQA119vCeH++8OOxLni5eRYS6BHHoEtW/w+mERThQcftLGdZ8ywq7BZs8KOquKlZSLYbTcYOBDeeAN++insaJyLbtMmu0v4xRfhnnvg8ce959BEWrfOmubedpt9D7NmWX9NZ54Jq1eHHV3FSstEAPCnP1lfLH/7W9iROLczVTj9dPjgA7sRctAgTwKJNGcOdO0K770H//iHnTQedBC8/bYN93nRRXbvRqpI20TQqBFcdRW8/DIsXBh2NM7tSMQaNrz2GlxzTdjRpJe334bMTFi50prm3nLL9iTcrZtV1n/0kXXtnSrSNhGAfcFVq8Lf/x52JM6ZefPsLBTgvPPg/PPDjSed5ObaTafnngvt2lmnfcceu/N8AwbYSeQDD9iVQipI60RQcEv+iBGwaFHY0bh0N306HHkkXHstbNwYdjTp5fff4aSTrBjo6qvhiy+K77Tv8cft6qBfP/jhh8TFGS9pnQjAKoLy8uChh8KOxKWzL76AY46xrlA+/9wHkUmk776Dzp3t3owXXoCnny65074aNeCtt+yO7jPPtGKkZJb2iaBlS+uv5dlnYfnysKNx6ejf/4aTT7Yr1MmTrVLSxZ+q/e6PPhqqVYOvv4a+fWN//9572yBAixfDBRdY0VKySvtEANaP+5Ytdru+c4k2ZYrdIPbll9CsWdjRpIfNm+GKK6wY6NhjrcuOsgzik5kJzzwDn322/Ya/ZOSJAGjd2irmnnwy9doHu8pr2TL7e999MHEi7LFHqOGkjYULrS7mhRfg7rutiW55tn2/fnaT6iOPWCvEZOSJIHDnndaV7xNPhB2JS3X5+XDrrdChA2RnW9PEWrXCjio9fPyx1QfMn2/dzAwZYi0Hy+vhh62O58orYdq08i8v0TwRBDp2hF69rGtf79vdxUtODlx+uTVOOP982GefsCNKD3l51tFkjx5WF5OVZTfsVZTq1eHNN62b+7POslZIycQTQYQ774RVq6zMz7mKNGsWnHaajSMwahQMHmxXn1X8Fxh38+dbPcBdd8GFF1rroAMOqPj1NGpklcfLl1tRc05Oxa8jXnw3jJCZaZ1LPfywVSY5VxGWL7ck8MEHViF8003w1796lxHxpmpNQQ8+2DqMGznSyvDj2TS3UycYPtxGQfzzn+O3normiaCQu+6ySrznnw87EpcK/vjDBpJZuhRefRWaN7crzjFjwo4stf36q90gdu21cMQR8N//wmWXJSb5Xnwx3HyzNT4ZMSL+66sInggKOfpoa1Hw97/D1q1hR+Mqyv/+Z90CZGcnbp2rV1tXBd98A8OGQe/eVpHYoYP1ZjlqVOJiSReq1hqoQwfb7s88YxXEiW6W+8ADcMIJ1k/Ud98ldt1loqpJ9ejcubPG20cfqYLqc8/FfVUuAbZuVc3MtO/0gANUs7Pjv87x41UbNLB1Hn206h9/bJ+2bp3qCSfYtIcfjn8s6WLxYtXTTtu+zefPDzeeFStUW7ZU3Wcfiy1sQJYWcVyN60EbOAX4CZgH3F7MfOcACmSUtMxEJIL8fNWMDNX99lPNyYn76lyc3XWX7el33aVar55qq1bxSwabNqlec42tD+yAn5e383xbtqiee67Nc8cdts+5ssnPV331VdXdd1etWVP1n/+Mvs3D8MMPqrVrqx5+uH3npbFpk+rCharffaf6/vuqzz+vOnt22WMJJREAVYH5wH5ADeAHoG2U+eoBk4BvK0siUFV9913bOi+/nJDVuTiZNEm1ShXVfv3s+ddfb08GixZV/Pq2blVt1061enXVTp1UN24set7cXNWrrrL9bMAAe+5K5/fftyfUrl1Vf/wx7Ih29vrrFt+VV1q8M2eqfvaZJa/HHlO9806b1quX6mGH2Qlo3brbTyYiH088UfY4iksEYtMrnogcDgxW1ZOD53cERVF/KzTfo8AnwK3ALaqaVdxyMzIyNCur2FmiWrTI7vy7+ebY2m7n51trg7w8mDnTm/klozVr7DusXh3+8x8bnhSsT5mTT7b9YMKE8rfl37LFhjMcOND6m+nSxf5OmWJt1oujane3Dh1q3R+//HLJHZ6Vxdat1rHdH3/Yvly1aun/1qgB++5r/fJUBu+9Z/U+q1fbjWG33FJ5Yivsjjus3iCaKlWs6emee8Jee9nfyEfka3vvbQNqlYWITFPVjKgTi8oQ5X0A5wLDI573AZ4sNE8n4O3g/4kUcUUADACygKzmzZuXKRv+/LNl1Pvvj/09r75q73nrrTKt0oXsootUq1ZV/fbbnad99ZWddbVuXb7y22nT7AoAVJ95RvWII1Rr1VKdOrV0y3nkEVvG8cdbHUJFWb1a9YEHrJw62hlmaR+1a6t266Z6002qo0er/vRT4othVq1S7dPH4jnkENUZMxK7/rLIzVUdPlz18cdVX3tN9fPPVWfNUl2+PHHbj5CuCM4FTlHV/sHzPkBXVR0YPK8CfA70VdWFIjKROF4RAHTvbq1G5s6NrRlZXh60aQN161prD2/3nTxGj7ZeZYcMsTPuaL76Ck45xVqUTJhgZ1uxysmxYU7vvdfO5oYPh9dft/GF33jDbigqrRdftLuOO3WCceOgYcPSL6PAL7/Ao49aXBs2WAuWG26Axo3tajcvb/vfyP+L+7txo7XHnzrVrrAK7rVp0MC6bejSxR4ZGdZMNh6/l48+ss7ili2zG0DvvNOuVFzJwroiOBz4OOL5HcAdEc8bACuAhcFjC7CYEuoJylNH8NJLdhYxYULs7xkxwt7zwQdlXq1LsP/9T7V+fTtzLamy/8svVevUUT3oINUlS2Jfx803235x0UWqK1faWTeo3nNPuULXMWOswvOgg1R//bX07582TfXCC+1KqFo11UsuUf3Pf8oXUzQ5OVYROny46tVXq3bubPUiBVcOjRqp9uihOmiQVXRG27Y5Oapr19q0efPszP7bb+1seexYK1t/4QXVp55S/cc/7LOAatu2pb/icsVfEcQzEVQDFgAt2V5Z3K6Y+SeWlAS0nIlg0yZr0nfJJbG/Z+tW1ebNrRLHW3ZUfjk5lgDq17eEEItJkywZtGmjunRp0fPl5VmxhKpVNL/5pv3/3nuqIqq9e1fMPvLFFxZ/s2aqc+aUPH9+vp2oHHus/aLr1VO95ZayJZLy2LJFdcoU1X/9yyrnO3SwivqC5LDXXqqNG9tvMDJpxPqoVk311ltVN29O7OdKFcUlgrgVDQGISE/gUawF0QhVHSoiQ4KAxhSadyJxLhoC6/Vxwwa79TxWTz9tdyh+9hkcd1yZV+0S4L77rCjo5ZftDs9YTZpkHZK1aGEjhO21147TFy+2fmqqVLH9oKDxwA8/2JCFbdtaZWxF9SI6fbpVaOfnw4cfWnFLYX/8Aa+8Yh3YzZ5tFdM33WQ9YDZoUDFxlNfGjfZZpk61u3urVoXatYt+1KlT9LRatSpvZXAyCKVoKF6PRDUfjbR5s53JHHNM5Wmf7Hb27bdWJHLhhWV7/8SJVhnatq3qsmXbX585087O69a1ooqCs/6lS+31Jk3i0xR17lzVFi1svZ99tv31Vaus0cPee9uZ8sEHW7Fn5E1rzhVGWDeUxeNRUYmgtDdmPPGEba1jjom9yMElzvr1qvvvb8V4q1eXfTkTJlirn3btLBl8/rkVZTRurPr999vn27zZbhKqVcvK5eNl0SLV9u1Va9RQHTZM9cYbrRgLVE86ye5g9iJLFwtPBIWMHm2fPPKHXZL8fDsbrFfPHs8/7z/AyuTyy608etKk8i/r88+3J4P997e/v/yyfXp+/vaKy0Q0LV61ypqlFpSTX3qpVdQ6VxqeCApZtUp1l11Ur7uu9O9duFC1e3fbcr16FV+56BLjzTft+7jzzopZXn6+nWnXrGn3Gcydu+P0+++39d17b8WsLxYbNljfV7/9lrh1utRSXCKIa2VxPJS3srjAxRdb//BLlpS+gi8/Hx5/HG6/3e5WHTbMRiVyiZedbaPLHXAATJ5sdxGXR24uXHedfccXXGCjWLVqZRXIDRvawCNnn20Vx6NH+70lLnkUV1mcth0n9O8Pa9fCO++U/r1VqljrjO+/txtnzj7b+jpfu7bi43RFy8+HSy+17hNGjy5/EtiwwcYOGDbMbuc//nh4/327AfH44+HTT+0mtcxMG6/Ck4BLFWmbCI45Bvbbzw4gZdW2LXz7rTVXHD3a+kD/7LOKi9EV7+GH7Y7gxx6zs/byWLrU9omPP4Znn7W+f0TsjtwxY+Cnn2z0ut13tz5ufLB5l0rSNhFUqWKX+W+8Ub7lVK9u3RhMnmwHhxNOsKsFH+oyvv7zH+te4OyzrVuG8sjLs+/tp5/soD9gwI7TTzzRXs/MtL+NG5dvfc5VNmlbRxAPmzZZvcETT8BBB1nfMV26hB2VWbTIRmzaYw87kDVuDPXrJ2fxxqZN1rfNunXW980ee5R/mR9/bHUAnTuXf1nOVUbF1RGk/X16779vFb8fflj+uxZr17Zl9eoF/frB4YfbGMh33ln+8uuy2LDBrnpefNGKrArn/Fq1rKO1xo23/432/5572h2hlcUtt8CPP1qZfXmSwKuvWr3O1VfbXbzOpau0TwR5eXZA+fBDayFSEU44wW6nv/56uOceGDsWXnrJejKNt7w8O+i/9JJVhG/aBC1bWkI67TS75X/JEnssXbr9/x9/tPL21at3XmZBf+nNmsGhh1oRSZcuNh5vom/5f/996/Lj5putArcsVG38gNtvh2OPtaIgH2/CpbO0LxrKybED3GGHWSVgRXvrLTvj3LgRBg2ybo/btCn74BJFmTHDDv6vvGL94jRoAOefb61qunWLvQhoyxbr4jdasli4ELKybMAXsCuKTp22J4bMTKuAj1dx09KlViHfpIkNCF6WAVxyc6075qeftsHkR46Mz0AwzlU2xRUNpX0iALjtNmuBkp1duj7pY7V0qXUENnasPa9aFQ480Nq/d+xoB7eOHUvfh/uSJXbgf+kl6/ysWjXo2RP69LGz/4pONmBn0/PmWSdiU6bY3++/twQC1qqmoF/6ggRR1m2qah2rbdxoj6uugokTbWyItm1Lv7z8fLvfY8wY+87vv9+vBFz68ERQgp9+ssrdBx6wA0Q8qMLPP9uZe+Rj4cLt89Svv3NyaN/eXi+wcaOV+7/0khVp5efbAbdPH7sBqlGj+MRfnJwcmDVre2KYMsWG98zPt+nNmllC6NDBzsgLDuyxPAqWUeCpp6wn2LJ66CGryynPMpxLRp4IYvCnP1nZ/qmnVviii7VunR00CyeI9eu3z9OypR1E69Sxs9mNG23s2EsusQTQunViY47Fxo3WxDPyymH+fDsDr1MntkdBt8QFj+bNrSlnaa6aVOGFF6xb6UR/t85VJp4Ikowq/PrrzslhxQo480w7+B95ZPIVa+TkWPFVopqsLl5sRXLjxll9yeuvJ2a9zlVG3nw0RkuXWvlz2GeOInbGv+++FdeSqTJIVBNaVas7uf56q7t47DEYODAx63YuGSXZOWV83XOPDTrufQYlt88/t2KzNm1sdKwbbki+qyfnEsl/HhGuuMK6hnj11bAjcWVRUPF+3HHWbHfSJGud5ZwrnieCCJ07W0ud4cPDjsSVxsqVdk9A+/aWDETgnHMq193QzlVmnggiiFj31NOmWZGCq/zGjLE7nN95B+64A5o2DTsi55KPJ4JCLr7Y7pidPDnsSFxx8vOhb18bP2Dvva156p13Jr7LC+dSgf9sCtl9d+upc7fdwo7EFadKFbvRbtAgSwA1aoQdkXPJyxNBFAVJIDfXzzArk3XrrOfR/v3tburHHkvObrSdq2z8MFeEa66xO2HHjw87kvQ2b57dmZyVZS2BFi2ySuHMTE8CzlUUTwRFaNIEnnnGksH++4cdTepT3d676dKldjMYWO+p33xjHeh16WJ3Bx9+eKihOpdyvLK4CH37Wjn0Cy+EHUnqiezV5JVXbFCYhg2tC+vzz7cxoPPybPrDD1sLrnXr7L4ATwLOVTxPBEVo2tTGDhg50uoKXPmsWmV3bp96qrXyWbnSXs/Oht9/t7GHn3nGrgiWLdt+D8Dhh8PBB4czwptz6cKLhopxxRV2Y9LHH4ff/1AyW7rUeg2dNcvGEejRw0ZO22MP+Mtf7OGcC48ngmKcdhr84x+QEbW/PheL3Fzr3vt//4NPPin78JLOufjxRFCMGjWsuaIru2rVYPBg2GcfOOKIsKNxzkXjiaAEqtZSpWpV65nUxWbmTGv6eeaZcO65YUfjnCuOJ4ISiMCTT8Ly5XZA87brJcvKspZA9etbfYAPDu9c5eathmLQv7+NN+z9D5Xsyy+tG+gGDeCzzzwJOJcM/IogBuedZ4ObPPeclXNXqQIbNtig9+vW2fjCBX979IAWLWy83scf33HarrtaF9dNmoT9ieJj/HgrCtp3X/j009T9nM6lGk8EMahTx/q7f+45OOkk66H0++/hmGN2nveddywRrFplZ8T16lkRSb16sGDBjoPSp5qJE6F1a2tuu+eeYUfjnItVXAevF5FTgMeAqsBwVX2g0PSrgeuAPGADMEBVZxe3zLAGr1+0CJ5/3sYQPvRQO9B/9dX2g3zB3913L7onzIJO7FRhzZrU6eF0/Xr77KqwcSPUrRt2RM65woobvD5uiUBEqgI/AycC2cBU4MLIA72I1FfVdcH/vYBrVfWU4pYbViKoSHffbcNhTpyY/AOpPPus3TE8eTK0bBl2NM65ohSXCOJZWZwJzFPVBaq6FXgNOCNyhoIkEKgDxO/ypBI59VRrhdS9O/z2W9jRlN1DD8HVV9sQn40bhx2Nc66s4pkImgCRh7ns4LUdiMh1IjIfeBC4IdqCRGSAiGSJSNby5cvjEmwiHXaYVawuXw7HHpt8yUAV/vpXuPVWuOACqxepWTPsqJxzZRV681FVfUpV9wduA+4qYp5hqpqhqhmNGjVKbIBx0rXr9mRw4omQkxN2RLEbPhyGDIHLL4fRo71DOOeSXTxbDS0CmkU8bxq8VpTXgKfjGE+lU5AMFi9OroPpRRfB5s0wcKA1pXXOJbd4/oynAq1EpKWI1AB6A2MiZxCRVhFPTwXmxjGeSqlrVzjrLPt/7Fj49ddw44lGFX75xSqF16+35rQ33OBJwLlUEbcrAlXNFZGBwMdY89ERqjpLRIYAWao6BhgoIicAOcBq4LJ4xVPZrV8P/fpZM9QJE6B583DiyM+3A/yyZdZZ3IwZ1m/QuqBav1UruyJwzqWOuN5HEA+p0Hy0KFOnWn3B7rtb09J4JgNV+O9/d3zMmGEjs917L6xda81B27eHDh3s0bWr3UPhnEs+xTUf9TuLK5EuXazP/hNPtKalFZUM8vIsyUycaElmwAB7/cgj7UqkenVo08bulC440DdoYKOIeSd7zqU+TwSVTGQyeOUVuP32si/rvffgzTety4eCoSFPO80SgQi89ZaNE9C6dfTKak8CzqUHTwSVUJcuVkzTLGhzpVryQTk/37p//uor+NOfbP4337Sk0rOnPU480YaHLHDSSfH7DM655OGJoJIqKBL68Udrr//qq9arZ6TVq+Gjj2DcODvrX77cEsA559i8Tz1llc/eusc5VxxPBJXcxo0wZ47VGUyYYJ3dNW5sj/HjrQVPw4Y2EEzPnnaW37ChvXfXXUMN3TmXJDwRVHKdO1vf/iecAPvvb0VADz5o3Tv06AHffgsZGTaUpnPOlYUngiTQubNdDYwYYQf9U4L+WevXtyadzjlXHp4IksQhh9iIZ845V9G8GtE559KcJwLnnEtzngiccy7NeSJwzrk054nAOefSnCcC55xLc54InHMuzXkicM65NJd0A9OIyHLgl7DjKEJDYEXYQRTD4yufyh4fVP4YPb7yKU98+6pqo2gTki4RVGYiklXUCECVgcdXPpU9Pqj8MXp85ROv+LxoyDnn0pwnAuecS3OeCCrWsLADKIHHVz6VPT6o/DF6fOUTl/i8jsA559KcXxE451ya80TgnHNpzhNBKYlIMxGZICKzRWSWiNwYZZ7uIrJWRKYHj0EJjnGhiPw3WHdWlOkiIo+LyDwRmSEinRIYW+uI7TJdRNaJyE2F5kn49hORESLyu4jMjHhtdxH5RETmBn93K+K9lwXzzBWRyxIU2z9E5Mfg+3tXRKKOUF3SvhDnGAeLyKKI77FnEe89RUR+CvbH2xMY3+sRsS0UkelFvDeu27CoY0pC9z9V9UcpHkBjoFPwfz3gZ6BtoXm6A2NDjHEh0LCY6T2BDwEBDgO+CynOqsBS7EaXULcfcDTQCZgZ8dqDwO3B/7cDf4/yvt2BBcHf3YL/d0tAbCcB1YL//x4ttlj2hTjHOBi4JYZ9YD6wH1AD+KHw7yle8RWa/jAwKIxtWNQxJZH7n18RlJKqLlHV74P/1wNzgCbhRlVqZwAvqvkW2FVEGocQx/HAfFUN/U5xVZ0ErCr08hnAqOD/UcCZUd56MvCJqq5S1dXAJ8Ap8Y5NVceram7w9FugaUWus7SK2H6xyATmqeoCVd0KvIZt9wpVXHwiIsD5wKsVvd5YFHNMSdj+54mgHESkBXAo8F2UyYeLyA8i8qGItEtoYKDAeBGZJiIDokxvAvwW8TybcJJZb4r+8YW5/QrspapLgv+XAntFmacybMvLsSu8aEraF+JtYFB8NaKIoo3KsP2OApap6twipidsGxY6piRs//NEUEYiUhd4G7hJVdcVmvw9VtxxMPAE8F6CwztSVTsBPYDrROToBK+/RCJSA+gFvBllctjbbydq1+GVrq21iNwJ5AKji5glzH3haWB/4BBgCVb8UhldSPFXAwnZhsUdU+K9/3kiKAMRqY59YaNV9Z3C01V1napuCP4fB1QXkYaJik9VFwV/fwfexS6/Iy0CmkU8bxq8lkg9gO9VdVnhCWFvvwjLCorMgr+/R5kntG0pIn2B04CLgwPFTmLYF+JGVZepap6q5gPPFbHuUPdFEakGnA28XtQ8idiGRRxTErb/eSIopaA88Xlgjqo+UsQ8ewfzISKZ2HZemaD46ohIvYL/sUrFmYVmGwNcGrQeOgxYG3EJmihFnoWFuf0KGQMUtMK4DPh3lHk+Bk4Skd2Coo+TgtfiSkROAf4C9FLVTUXME8u+EM8YI+udzipi3VOBViLSMrhK7I1t90Q5AfhRVbOjTUzENizmmJK4/S9eNeGp+gCOxC7RZgDTg0dP4Grg6mCegcAsrAXEt8ARCYxvv2C9PwQx3Bm8HhmfAE9hrTX+C2QkeBvWwQ7sDSJeC3X7YUlpCZCDlbNeAewBfAbMBT4Fdg/mzQCGR7z3cmBe8OiXoNjmYWXDBfvgM8G8+wDjitsXErj9Xgr2rxnYQa1x4RiD5z2xljLz4xVjtPiC10cW7HcR8yZ0GxZzTEnY/uddTDjnXJrzoiHnnEtzngiccy7NeSJwzrk054nAOefSnCcC55xLc54InAuISJ7s2DNqhfWEKSItInu+dK4yqRZ2AM5VIptV9ZCwg3Au0fyKwLkSBP3RPxj0ST9FRA4IXm8hIp8Hnap9JiLNg9f3Ehsj4IfgcUSwqKoi8lzQ5/x4EakVzH9D0Bf9DBF5LaSP6dKYJwLntqtVqGjogohpa1W1A/Ak8Gjw2hPAKFXtiHX69njw+uPAF2qd5nXC7kgFaAU8partgDXAOcHrtwOHBsu5Ol4fzrmi+J3FzgVEZIOq1o3y+kLgOFVdEHQOtlRV9xCRFVi3CTnB60tUtaGILAeaquofEctogfUb3yp4fhtQXVXvE5GPgA1YL6vvadDhnnOJ4lcEzsVGi/i/NP6I+D+P7XV0p2J9P3UCpgY9YjqXMJ4InIvNBRF/vwn+/xrrLRPgYuDL4P/PgGsARKSqiDQoaqEiUgVopqoTgNuABsBOVyXOxZOfeTi3XS3ZcQDzj1S1oAnpbiIyAzurvzB47XrgBRG5FVgO9AtevxEYJiJXYGf+12A9X0ZTFXg5SBYCPK6qayrsEzkXA68jcK4EQR1BhqquCDsW5+LBi4accy7N+RWBc86lOb8icM65NOeJwDnn0pwnAuecS3OeCJxzLs15InDOuTT3/1qf65ys7TOzAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Nq-r756VOSx"
      },
      "source": [
        "**Different weight regularizers available in Keras**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yKamhE0LVOSx",
        "outputId": "0b9ea5de-2088-40de-dab9-559ccbd526ae",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.regularizers.L1L2 at 0x7f87a4e19990>"
            ]
          },
          "metadata": {},
          "execution_count": 50
        }
      ],
      "source": [
        "from tensorflow.keras import regularizers\n",
        "regularizers.l1(0.001)\n",
        "regularizers.l1_l2(l1=0.001, l2=0.001)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sY8s81KCVOSx"
      },
      "source": [
        "#### Adding dropout"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "- 무작위로 일부 출력 특성 제외  \n",
        "ex) [0.2,0.5,1.2,0.8,1.1] -> [0,0.5,1.2,0,1.1]\n",
        "- text 단계에서는 어떤 unit도 dropout되지 않음\n",
        "- test 단계에서는 dropout 비율에 비례해 출력을 줄여 주어야 함\n",
        "\n",
        "\n",
        "```\n",
        "layor_output *= 0.5 #(at test)\n",
        "```\n",
        "\n",
        "```\n",
        "layor_output *= random.randint(0,high=2,size=layer_output.shape)\n",
        "layer_output /=0.5 #(in training)\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "e6000B491qoD"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8HwHV_YPVOSy"
      },
      "source": [
        "**Adding dropout to the IMDB model**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H7EW7OE0VOSz",
        "outputId": "eb46b058-bc04-4bcc-e9c4-224f6010d61b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "30/30 [==============================] - 2s 49ms/step - loss: 0.6497 - accuracy: 0.6053 - val_loss: 0.5773 - val_accuracy: 0.6931\n",
            "Epoch 2/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.5581 - accuracy: 0.7647 - val_loss: 0.5132 - val_accuracy: 0.8478\n",
            "Epoch 3/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.5014 - accuracy: 0.8461 - val_loss: 0.4747 - val_accuracy: 0.8371\n",
            "Epoch 4/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.4522 - accuracy: 0.8827 - val_loss: 0.4560 - val_accuracy: 0.8568\n",
            "Epoch 5/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.4199 - accuracy: 0.9017 - val_loss: 0.4358 - val_accuracy: 0.8763\n",
            "Epoch 6/20\n",
            "30/30 [==============================] - 1s 39ms/step - loss: 0.3886 - accuracy: 0.9151 - val_loss: 0.4287 - val_accuracy: 0.8812\n",
            "Epoch 7/20\n",
            "30/30 [==============================] - 1s 38ms/step - loss: 0.3607 - accuracy: 0.9249 - val_loss: 0.4276 - val_accuracy: 0.8807\n",
            "Epoch 8/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.3342 - accuracy: 0.9314 - val_loss: 0.4162 - val_accuracy: 0.8826\n",
            "Epoch 9/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.3126 - accuracy: 0.9406 - val_loss: 0.4329 - val_accuracy: 0.8806\n",
            "Epoch 10/20\n",
            "30/30 [==============================] - 1s 35ms/step - loss: 0.2913 - accuracy: 0.9443 - val_loss: 0.4340 - val_accuracy: 0.8798\n",
            "Epoch 11/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.2738 - accuracy: 0.9499 - val_loss: 0.4664 - val_accuracy: 0.8746\n",
            "Epoch 12/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.2529 - accuracy: 0.9563 - val_loss: 0.4537 - val_accuracy: 0.8774\n",
            "Epoch 13/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.2430 - accuracy: 0.9567 - val_loss: 0.4784 - val_accuracy: 0.8762\n",
            "Epoch 14/20\n",
            "30/30 [==============================] - 1s 36ms/step - loss: 0.2265 - accuracy: 0.9632 - val_loss: 0.5113 - val_accuracy: 0.8730\n",
            "Epoch 15/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.2109 - accuracy: 0.9659 - val_loss: 0.4844 - val_accuracy: 0.8716\n",
            "Epoch 16/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.2053 - accuracy: 0.9665 - val_loss: 0.5462 - val_accuracy: 0.8706\n",
            "Epoch 17/20\n",
            "30/30 [==============================] - 1s 41ms/step - loss: 0.1927 - accuracy: 0.9688 - val_loss: 0.5526 - val_accuracy: 0.8719\n",
            "Epoch 18/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.1823 - accuracy: 0.9714 - val_loss: 0.5954 - val_accuracy: 0.8690\n",
            "Epoch 19/20\n",
            "30/30 [==============================] - 1s 40ms/step - loss: 0.1702 - accuracy: 0.9743 - val_loss: 0.6354 - val_accuracy: 0.8675\n",
            "Epoch 20/20\n",
            "30/30 [==============================] - 1s 37ms/step - loss: 0.1650 - accuracy: 0.9748 - val_loss: 0.5843 - val_accuracy: 0.8679\n"
          ]
        }
      ],
      "source": [
        "model = keras.Sequential([\n",
        "    layers.Dense(16, activation=\"relu\"),\n",
        "    layers.Dropout(0.5),\n",
        "    layers.Dense(16, activation=\"relu\"),\n",
        "    layers.Dropout(0.5),\n",
        "    layers.Dense(1, activation=\"sigmoid\")\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"binary_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_dropout = model.fit(\n",
        "    train_data, train_labels,\n",
        "    epochs=20, batch_size=512, validation_split=0.4)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Summary"
      ],
      "metadata": {
        "id": "xYQpds1U3OTV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "overfitting을 줄이기 위한 방법  \n",
        "- 훈련 데이터를 더 모은다.\n",
        "- network 용량을 감소시킨다.\n",
        "- 가중치 규제를 추가한다.\n",
        "- dropout을 추가한다."
      ],
      "metadata": {
        "id": "nX4b9F4e3Qc9"
      }
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "chapter05_fundamentals-of-ml.i",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}