chapter11_part03_transformer.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/oneoclockc/deeplearning-for-AI/blob/main/chapter11_part03_transformer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NZHhiyttCKHJ"
      },
      "source": [
        "## The Transformer architecture"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KWzxu5l5CKHL"
      },
      "source": [
        "### Understanding self-attention"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "self-attention -> can make features *context-aware*  \n",
        "\"I'll **see** you later soon\"?  \n",
        "- I'll **see** this project to its end\n",
        "- I **see** what you mean  \n",
        "\n",
        "smart embedding space -> provide different vector representation for a word depending on the other words \n"
      ],
      "metadata": {
        "id": "d1oCqfOfUBG8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "https://github.com/eubinecto/k4ji_ai/issues/42"
      ],
      "metadata": {
        "id": "0peQq1-fEJRB"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No5RLZ_TCKHM"
      },
      "source": [
        "#### Generalized self-attention: the query-key-value model"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "```\n",
        "outputs = sum(input c * pairwise_scores(input a, input b)\n",
        "```\n",
        "input a = query  \n",
        "input b = key  \n",
        "input c = value  \n",
        "쿼리의 각 요소에 대해 각 요소가 모든 키와 얼마나 관련이 있는지 계산하고 이 점수를 사용하여 값의 합에 가중치를 부여한다\n"
      ],
      "metadata": {
        "id": "ShriIuE7Ycq5"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6bQL8fmkCKHM"
      },
      "source": [
        "### Multi-head attention"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PQ4aaMF5CKHM"
      },
      "source": [
        "### The Transformer encoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SizrpwXGCKHN"
      },
      "source": [
        "**Getting the data**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "29cZ7bA_CKHN",
        "outputId": "b6f04af9-b545-463b-88cc-2c48217ba5e8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100 80.2M  100 80.2M    0     0  28.3M      0  0:00:02  0:00:02 --:--:-- 28.3M\n"
          ]
        }
      ],
      "source": [
        "!curl -O https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
        "!tar -xf aclImdb_v1.tar.gz\n",
        "!rm -r aclImdb/train/unsup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A7Wp6BYFCKHO"
      },
      "source": [
        "**Preparing the data**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "GBs1G5qvCKHP",
        "outputId": "1da78e06-939c-4b06-f97f-b6be94083aea",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 20000 files belonging to 2 classes.\n",
            "Found 5000 files belonging to 2 classes.\n",
            "Found 25000 files belonging to 2 classes.\n"
          ]
        }
      ],
      "source": [
        "import os, pathlib, shutil, random\n",
        "from tensorflow import keras\n",
        "batch_size = 32\n",
        "base_dir = pathlib.Path(\"aclImdb\")\n",
        "val_dir = base_dir / \"val\"\n",
        "train_dir = base_dir / \"train\"\n",
        "for category in (\"neg\", \"pos\"):\n",
        "    os.makedirs(val_dir / category)\n",
        "    files = os.listdir(train_dir / category)\n",
        "    random.Random(1337).shuffle(files)\n",
        "    num_val_samples = int(0.2 * len(files))\n",
        "    val_files = files[-num_val_samples:]\n",
        "    for fname in val_files:\n",
        "        shutil.move(train_dir / category / fname,\n",
        "                    val_dir / category / fname)\n",
        "\n",
        "train_ds = keras.utils.text_dataset_from_directory(\n",
        "    \"aclImdb/train\", batch_size=batch_size\n",
        ")\n",
        "val_ds = keras.utils.text_dataset_from_directory(\n",
        "    \"aclImdb/val\", batch_size=batch_size\n",
        ")\n",
        "test_ds = keras.utils.text_dataset_from_directory(\n",
        "    \"aclImdb/test\", batch_size=batch_size\n",
        ")\n",
        "text_only_train_ds = train_ds.map(lambda x, y: x)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PtU3rdXiCKHP"
      },
      "source": [
        "**Vectorizing the data**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "gXB7pQdICKHQ"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras import layers\n",
        "\n",
        "max_length = 600\n",
        "max_tokens = 20000\n",
        "text_vectorization = layers.TextVectorization(\n",
        "    max_tokens=max_tokens,\n",
        "    output_mode=\"int\",\n",
        "    output_sequence_length=max_length,\n",
        ")\n",
        "text_vectorization.adapt(text_only_train_ds)\n",
        "\n",
        "int_train_ds = train_ds.map(\n",
        "    lambda x, y: (text_vectorization(x), y),\n",
        "    num_parallel_calls=4)\n",
        "int_val_ds = val_ds.map(\n",
        "    lambda x, y: (text_vectorization(x), y),\n",
        "    num_parallel_calls=4)\n",
        "int_test_ds = test_ds.map(\n",
        "    lambda x, y: (text_vectorization(x), y),\n",
        "    num_parallel_calls=4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KIHq08xeCKHQ"
      },
      "source": [
        "**Transformer encoder implemented as a subclassed `Layer`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "TL0ovYkFCKHR"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "class TransformerEncoder(layers.Layer):\n",
        "    def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):\n",
        "        super().__init__(**kwargs)\n",
        "        self.embed_dim = embed_dim # size of input token vectors\n",
        "        self.dense_dim = dense_dim # size of inner dense layer\n",
        "        self.num_heads = num_heads # number of attention heads\n",
        "        self.attention = layers.MultiHeadAttention(\n",
        "            num_heads=num_heads, key_dim=embed_dim)\n",
        "        self.dense_proj = keras.Sequential(\n",
        "            [layers.Dense(dense_dim, activation=\"relu\"),\n",
        "             layers.Dense(embed_dim),]\n",
        "        )\n",
        "        self.layernorm_1 = layers.LayerNormalization()\n",
        "        self.layernorm_2 = layers.LayerNormalization()\n",
        "\n",
        "    def call(self, inputs, mask=None):\n",
        "        if mask is not None:\n",
        "            mask = mask[:, tf.newaxis, :]\n",
        "        attention_output = self.attention(\n",
        "            inputs, inputs, attention_mask=mask)\n",
        "        proj_input = self.layernorm_1(inputs + attention_output)\n",
        "        proj_output = self.dense_proj(proj_input)\n",
        "        return self.layernorm_2(proj_input + proj_output)\n",
        "\n",
        "    def get_config(self):\n",
        "        config = super().get_config()\n",
        "        config.update({\n",
        "            \"embed_dim\": self.embed_dim,\n",
        "            \"num_heads\": self.num_heads,\n",
        "            \"dense_dim\": self.dense_dim,\n",
        "        })\n",
        "        return config"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eh7qpiHXCKHR"
      },
      "source": [
        "**Using the Transformer encoder for text classification**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "_Equ8kuhCKHS",
        "outputId": "f97b528d-01f4-4d26-c986-04fc49c31388",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_2\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_3 (InputLayer)        [(None, None)]            0         \n",
            "                                                                 \n",
            " embedding_5 (Embedding)     (None, None, 256)         5120000   \n",
            "                                                                 \n",
            " transformer_encoder_2 (Tran  (None, None, 256)        543776    \n",
            " sformerEncoder)                                                 \n",
            "                                                                 \n",
            " global_max_pooling1d_2 (Glo  (None, 256)              0         \n",
            " balMaxPooling1D)                                                \n",
            "                                                                 \n",
            " dropout_2 (Dropout)         (None, 256)               0         \n",
            "                                                                 \n",
            " dense_12 (Dense)            (None, 1)                 257       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 5,664,033\n",
            "Trainable params: 5,664,033\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "vocab_size = 20000\n",
        "embed_dim = 256\n",
        "num_heads = 2\n",
        "dense_dim = 32\n",
        "\n",
        "inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
        "x = layers.Embedding(vocab_size, embed_dim)(inputs)\n",
        "x = TransformerEncoder(embed_dim, dense_dim, num_heads)(x) ###\n",
        "x = layers.GlobalMaxPooling1D()(x)\n",
        "x = layers.Dropout(0.5)(x)\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model = keras.Model(inputs, outputs)\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"binary_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yfXPQxtYCKHS"
      },
      "source": [
        "**Training and evaluating the Transformer encoder based model**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "RXFWYiX8CKHS",
        "outputId": "99ec3427-c46d-4300-c810-71f3bc1e9f5a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "625/625 [==============================] - 49s 67ms/step - loss: 0.4922 - accuracy: 0.7697 - val_loss: 0.3240 - val_accuracy: 0.8620\n",
            "Epoch 2/20\n",
            "625/625 [==============================] - 42s 68ms/step - loss: 0.3197 - accuracy: 0.8623 - val_loss: 0.2767 - val_accuracy: 0.8854\n",
            "Epoch 3/20\n",
            "625/625 [==============================] - 42s 68ms/step - loss: 0.2483 - accuracy: 0.8992 - val_loss: 0.3854 - val_accuracy: 0.8578\n",
            "Epoch 4/20\n",
            "625/625 [==============================] - 43s 69ms/step - loss: 0.1933 - accuracy: 0.9245 - val_loss: 0.3755 - val_accuracy: 0.8716\n",
            "Epoch 5/20\n",
            "625/625 [==============================] - 43s 69ms/step - loss: 0.1581 - accuracy: 0.9401 - val_loss: 0.3854 - val_accuracy: 0.8620\n",
            "Epoch 6/20\n",
            "625/625 [==============================] - 42s 68ms/step - loss: 0.1330 - accuracy: 0.9496 - val_loss: 0.3645 - val_accuracy: 0.8896\n",
            "Epoch 7/20\n",
            "625/625 [==============================] - 44s 71ms/step - loss: 0.1148 - accuracy: 0.9571 - val_loss: 0.4519 - val_accuracy: 0.8842\n",
            "Epoch 8/20\n",
            "625/625 [==============================] - 45s 72ms/step - loss: 0.0987 - accuracy: 0.9634 - val_loss: 0.4820 - val_accuracy: 0.8878\n",
            "Epoch 9/20\n",
            "625/625 [==============================] - 44s 70ms/step - loss: 0.0823 - accuracy: 0.9704 - val_loss: 0.4956 - val_accuracy: 0.8830\n",
            "Epoch 10/20\n",
            "625/625 [==============================] - 44s 70ms/step - loss: 0.0678 - accuracy: 0.9746 - val_loss: 0.5375 - val_accuracy: 0.8814\n",
            "Epoch 11/20\n",
            "625/625 [==============================] - 44s 70ms/step - loss: 0.0548 - accuracy: 0.9805 - val_loss: 0.6794 - val_accuracy: 0.8784\n",
            "Epoch 12/20\n",
            "625/625 [==============================] - 44s 70ms/step - loss: 0.0458 - accuracy: 0.9836 - val_loss: 0.7270 - val_accuracy: 0.8518\n",
            "Epoch 13/20\n",
            "625/625 [==============================] - 44s 70ms/step - loss: 0.0377 - accuracy: 0.9872 - val_loss: 0.8070 - val_accuracy: 0.8710\n",
            "Epoch 14/20\n",
            "625/625 [==============================] - 44s 70ms/step - loss: 0.0310 - accuracy: 0.9901 - val_loss: 0.6353 - val_accuracy: 0.8714\n",
            "Epoch 15/20\n",
            "625/625 [==============================] - 44s 70ms/step - loss: 0.0219 - accuracy: 0.9934 - val_loss: 0.9016 - val_accuracy: 0.8624\n",
            "Epoch 16/20\n",
            "625/625 [==============================] - 44s 70ms/step - loss: 0.0177 - accuracy: 0.9948 - val_loss: 1.0087 - val_accuracy: 0.8600\n",
            "Epoch 17/20\n",
            "625/625 [==============================] - 44s 70ms/step - loss: 0.0212 - accuracy: 0.9944 - val_loss: 0.7973 - val_accuracy: 0.8666\n",
            "Epoch 18/20\n",
            "625/625 [==============================] - 45s 72ms/step - loss: 0.0129 - accuracy: 0.9961 - val_loss: 1.3049 - val_accuracy: 0.8498\n",
            "Epoch 19/20\n",
            "625/625 [==============================] - 44s 70ms/step - loss: 0.0142 - accuracy: 0.9961 - val_loss: 1.1459 - val_accuracy: 0.8654\n",
            "Epoch 20/20\n",
            "625/625 [==============================] - 43s 69ms/step - loss: 0.0130 - accuracy: 0.9961 - val_loss: 1.3378 - val_accuracy: 0.8588\n",
            "782/782 [==============================] - 21s 26ms/step - loss: 0.2967 - accuracy: 0.8741\n",
            "Test acc: 0.874\n"
          ]
        }
      ],
      "source": [
        "callbacks = [\n",
        "    keras.callbacks.ModelCheckpoint(\"transformer_encoder.keras\",\n",
        "                                    save_best_only=True)\n",
        "]\n",
        "model.fit(int_train_ds, validation_data=int_val_ds, epochs=20, callbacks=callbacks)\n",
        "model = keras.models.load_model(\n",
        "    \"transformer_encoder.keras\",\n",
        "    custom_objects={\"TransformerEncoder\": TransformerEncoder})\n",
        "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P6sAUGg6CKHT"
      },
      "source": [
        "#### Using positional encoding to re-inject order information"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "모델이 순서 정보에 접근할 수 있도록 단어 위치 추가"
      ],
      "metadata": {
        "id": "FIcNyDTBbLtW"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jOPuaUT1CKHT"
      },
      "source": [
        "**Implementing positional embedding as a subclassed layer**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "fwAp24irCKHU"
      },
      "outputs": [],
      "source": [
        "class PositionalEmbedding(layers.Layer):\n",
        "    def __init__(self, sequence_length, input_dim, output_dim, **kwargs):\n",
        "        super().__init__(**kwargs)\n",
        "        self.token_embeddings = layers.Embedding(\n",
        "            input_dim=input_dim, output_dim=output_dim)\n",
        "        self.position_embeddings = layers.Embedding(\n",
        "            input_dim=sequence_length, output_dim=output_dim)\n",
        "        self.sequence_length = sequence_length\n",
        "        self.input_dim = input_dim\n",
        "        self.output_dim = output_dim\n",
        "\n",
        "    def call(self, inputs):\n",
        "        length = tf.shape(inputs)[-1]\n",
        "        positions = tf.range(start=0, limit=length, delta=1)\n",
        "        embedded_tokens = self.token_embeddings(inputs)\n",
        "        embedded_positions = self.position_embeddings(positions)\n",
        "        return embedded_tokens + embedded_positions\n",
        "\n",
        "    def compute_mask(self, inputs, mask=None):\n",
        "        return tf.math.not_equal(inputs, 0)\n",
        "\n",
        "    def get_config(self):\n",
        "        config = super().get_config()\n",
        "        config.update({\n",
        "            \"output_dim\": self.output_dim,\n",
        "            \"sequence_length\": self.sequence_length,\n",
        "            \"input_dim\": self.input_dim,\n",
        "        })\n",
        "        return config"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Rsq6f_QCKHU"
      },
      "source": [
        "#### Putting it all together: A text-classification Transformer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H5z5nbzzCKHU"
      },
      "source": [
        "**Combining the Transformer encoder with positional embedding**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "mjMued1kCKHU",
        "outputId": "dde3ff69-b513-494c-fe0d-1bda7b48b366",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_1\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_2 (InputLayer)        [(None, None)]            0         \n",
            "                                                                 \n",
            " positional_embedding (Posit  (None, None, 256)        5273600   \n",
            " ionalEmbedding)                                                 \n",
            "                                                                 \n",
            " transformer_encoder_1 (Tran  (None, None, 256)        543776    \n",
            " sformerEncoder)                                                 \n",
            "                                                                 \n",
            " global_max_pooling1d_1 (Glo  (None, 256)              0         \n",
            " balMaxPooling1D)                                                \n",
            "                                                                 \n",
            " dropout_1 (Dropout)         (None, 256)               0         \n",
            "                                                                 \n",
            " dense_7 (Dense)             (None, 1)                 257       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 5,817,633\n",
            "Trainable params: 5,817,633\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/20\n",
            "625/625 [==============================] - 47s 72ms/step - loss: 0.4724 - accuracy: 0.7902 - val_loss: 0.2672 - val_accuracy: 0.8856\n",
            "Epoch 2/20\n",
            "625/625 [==============================] - 44s 70ms/step - loss: 0.2343 - accuracy: 0.9091 - val_loss: 0.2869 - val_accuracy: 0.8900\n",
            "Epoch 3/20\n",
            "625/625 [==============================] - 46s 73ms/step - loss: 0.1777 - accuracy: 0.9335 - val_loss: 0.3402 - val_accuracy: 0.8878\n",
            "Epoch 4/20\n",
            "625/625 [==============================] - 45s 71ms/step - loss: 0.1454 - accuracy: 0.9467 - val_loss: 0.3685 - val_accuracy: 0.8918\n",
            "Epoch 5/20\n",
            "625/625 [==============================] - 45s 71ms/step - loss: 0.1188 - accuracy: 0.9574 - val_loss: 0.3782 - val_accuracy: 0.8808\n",
            "Epoch 6/20\n",
            "625/625 [==============================] - 45s 71ms/step - loss: 0.1057 - accuracy: 0.9623 - val_loss: 0.3874 - val_accuracy: 0.8940\n",
            "Epoch 7/20\n",
            "625/625 [==============================] - 45s 71ms/step - loss: 0.0912 - accuracy: 0.9676 - val_loss: 0.4408 - val_accuracy: 0.8858\n",
            "Epoch 8/20\n",
            "625/625 [==============================] - 45s 71ms/step - loss: 0.0813 - accuracy: 0.9712 - val_loss: 0.4546 - val_accuracy: 0.8846\n",
            "Epoch 9/20\n",
            "625/625 [==============================] - 44s 71ms/step - loss: 0.0693 - accuracy: 0.9766 - val_loss: 0.5208 - val_accuracy: 0.8838\n",
            "Epoch 10/20\n",
            "625/625 [==============================] - 44s 71ms/step - loss: 0.0620 - accuracy: 0.9798 - val_loss: 0.5897 - val_accuracy: 0.8710\n",
            "Epoch 11/20\n",
            "625/625 [==============================] - 45s 71ms/step - loss: 0.0576 - accuracy: 0.9807 - val_loss: 0.5790 - val_accuracy: 0.8826\n",
            "Epoch 12/20\n",
            "625/625 [==============================] - 45s 71ms/step - loss: 0.0480 - accuracy: 0.9840 - val_loss: 0.6971 - val_accuracy: 0.8458\n",
            "Epoch 13/20\n",
            "625/625 [==============================] - 46s 73ms/step - loss: 0.0399 - accuracy: 0.9868 - val_loss: 0.5955 - val_accuracy: 0.8826\n",
            "Epoch 14/20\n",
            "625/625 [==============================] - 44s 69ms/step - loss: 0.0370 - accuracy: 0.9888 - val_loss: 0.5647 - val_accuracy: 0.8798\n",
            "Epoch 15/20\n",
            "625/625 [==============================] - 45s 71ms/step - loss: 0.0308 - accuracy: 0.9905 - val_loss: 0.8191 - val_accuracy: 0.8748\n",
            "Epoch 16/20\n",
            "625/625 [==============================] - 44s 69ms/step - loss: 0.0283 - accuracy: 0.9909 - val_loss: 0.7610 - val_accuracy: 0.8726\n",
            "Epoch 17/20\n",
            "625/625 [==============================] - 45s 71ms/step - loss: 0.0234 - accuracy: 0.9924 - val_loss: 0.7302 - val_accuracy: 0.8766\n",
            "Epoch 18/20\n",
            "625/625 [==============================] - 45s 71ms/step - loss: 0.0228 - accuracy: 0.9927 - val_loss: 0.7670 - val_accuracy: 0.8774\n",
            "Epoch 19/20\n",
            "625/625 [==============================] - 45s 71ms/step - loss: 0.0215 - accuracy: 0.9934 - val_loss: 0.9392 - val_accuracy: 0.8714\n",
            "Epoch 20/20\n",
            "625/625 [==============================] - 45s 72ms/step - loss: 0.0137 - accuracy: 0.9958 - val_loss: 1.3506 - val_accuracy: 0.8364\n",
            "782/782 [==============================] - 21s 27ms/step - loss: 0.2883 - accuracy: 0.8806\n",
            "Test acc: 0.881\n"
          ]
        }
      ],
      "source": [
        "vocab_size = 20000\n",
        "sequence_length = 600\n",
        "embed_dim = 256\n",
        "num_heads = 2\n",
        "dense_dim = 32\n",
        "\n",
        "inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
        "x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(inputs)\n",
        "x = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\n",
        "x = layers.GlobalMaxPooling1D()(x)\n",
        "x = layers.Dropout(0.5)(x)\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model = keras.Model(inputs, outputs)\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"binary_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "model.summary()\n",
        "\n",
        "callbacks = [\n",
        "    keras.callbacks.ModelCheckpoint(\"full_transformer_encoder.keras\",\n",
        "                                    save_best_only=True)\n",
        "]\n",
        "model.fit(int_train_ds, validation_data=int_val_ds, epochs=20, callbacks=callbacks)\n",
        "model = keras.models.load_model(\n",
        "    \"full_transformer_encoder.keras\",\n",
        "    custom_objects={\"TransformerEncoder\": TransformerEncoder,\n",
        "                    \"PositionalEmbedding\": PositionalEmbedding})\n",
        "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m23PLdGmCKHV"
      },
      "source": [
        "### When to use sequence models over bag-of-words models?"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "chapter11_part03_transformer.i",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.0"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}