Neural machine translation with attention_latest.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Machine Translation\n",
    "\n",
    "Welcome to your first programming assignment for this week! \n",
    "\n",
    "You will build a Neural Machine Translation (NMT) model to translate human readable dates (\"25th of June, 2009\") into machine readable dates (\"2009-06-25\"). You will do this using an attention model, one of the most sophisticated sequence to sequence models. \n",
    "\n",
    "This notebook was produced together with NVIDIA's Deep Learning Institute. \n",
    "\n",
    "Let's load all the packages you will need for this assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Bidirectional, Concatenate, Permute, Dot, Input, LSTM, Multiplys\n",
    "from keras.layers import RepeatVector, Dense, Activation, Lambda\n",
    "from keras.optimizers import Adam\n",
    "from keras.utils import to_categorical\n",
    "from keras.models import load_model, Model\n",
    "import keras.backend as K\n",
    "import numpy as np\n",
    "\n",
    "from faker import Faker\n",
    "import random\n",
    "from tqdm import tqdm\n",
    "from babel.dates import format_date\n",
    "from nmt_utils import *\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulting to user installation because normal site-packages is not writeable\n",
      "Collecting faker\n",
      "  Downloading Faker-33.1.0-py3-none-any.whl.metadata (15 kB)\n",
      "Requirement already satisfied: python-dateutil>=2.4 in /Users/msatidas/Library/Python/3.9/lib/python/site-packages (from faker) (2.8.2)\n",
      "Requirement already satisfied: typing-extensions in /Users/msatidas/Library/Python/3.9/lib/python/site-packages (from faker) (4.5.0)\n",
      "Requirement already satisfied: six>=1.5 in /Users/msatidas/Library/Python/3.9/lib/python/site-packages (from python-dateutil>=2.4->faker) (1.16.0)\n",
      "Downloading Faker-33.1.0-py3-none-any.whl (1.9 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.9/1.9 MB\u001b[0m \u001b[31m10.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: faker\n",
      "Successfully installed faker-33.1.0\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49m/Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install faker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 - Translating human readable dates into machine readable dates\n",
    "\n",
    "The model you will build here could be used to translate from one language to another, such as translating from English to Hindi. However, language translation requires massive datasets and usually takes days of training on GPUs. To give you a place to experiment with these models even without using massive datasets, we will instead use a simpler \"date translation\" task. \n",
    "\n",
    "The network will input a date written in a variety of possible formats (*e.g. \"the 29th of August 1958\", \"03/30/1968\", \"24 JUNE 1987\"*) and translate them into standardized, machine readable dates (*e.g. \"1958-08-29\", \"1968-03-30\", \"1987-06-24\"*). We will have the network learn to output dates in the common machine-readable format YYYY-MM-DD. \n",
    "\n",
    "\n",
    "\n",
    "<!-- \n",
    "Take a look at [nmt_utils.py](./nmt_utils.py) to see all the formatting. Count and figure out how the formats work, you will need this knowledge later. !--> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 - Dataset\n",
    "\n",
    "We will train the model on a dataset of 10000 human readable dates and their equivalent, standardized, machine readable dates. Let's run the following cells to load the dataset and print some examples. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10000/10000 [00:00<00:00, 55284.96it/s]\n"
     ]
    }
   ],
   "source": [
    "m = 10000\n",
    "dataset, human_vocab, machine_vocab, inv_machine_vocab = load_dataset(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([('29 nov 1992', '1992-11-29'),\n",
       "  ('24.07.70', '1970-07-24'),\n",
       "  ('5/19/15', '2015-05-19'),\n",
       "  ('wednesday june 4 1986', '1986-06-04'),\n",
       "  ('thursday april 5 1990', '1990-04-05'),\n",
       "  ('monday august 25 1980', '1980-08-25'),\n",
       "  ('thursday february 15 2001', '2001-02-15'),\n",
       "  ('22 nov 1978', '1978-11-22'),\n",
       "  ('31 oct 1976', '1976-10-31'),\n",
       "  ('friday october 22 1993', '1993-10-22')],\n",
       " {' ': 0,\n",
       "  '.': 1,\n",
       "  '/': 2,\n",
       "  '0': 3,\n",
       "  '1': 4,\n",
       "  '2': 5,\n",
       "  '3': 6,\n",
       "  '4': 7,\n",
       "  '5': 8,\n",
       "  '6': 9,\n",
       "  '7': 10,\n",
       "  '8': 11,\n",
       "  '9': 12,\n",
       "  'a': 13,\n",
       "  'b': 14,\n",
       "  'c': 15,\n",
       "  'd': 16,\n",
       "  'e': 17,\n",
       "  'f': 18,\n",
       "  'g': 19,\n",
       "  'h': 20,\n",
       "  'i': 21,\n",
       "  'j': 22,\n",
       "  'l': 23,\n",
       "  'm': 24,\n",
       "  'n': 25,\n",
       "  'o': 26,\n",
       "  'p': 27,\n",
       "  'r': 28,\n",
       "  's': 29,\n",
       "  't': 30,\n",
       "  'u': 31,\n",
       "  'v': 32,\n",
       "  'w': 33,\n",
       "  'y': 34,\n",
       "  '<unk>': 35,\n",
       "  '<pad>': 36},\n",
       " {'-': 0,\n",
       "  '0': 1,\n",
       "  '1': 2,\n",
       "  '2': 3,\n",
       "  '3': 4,\n",
       "  '4': 5,\n",
       "  '5': 6,\n",
       "  '6': 7,\n",
       "  '7': 8,\n",
       "  '8': 9,\n",
       "  '9': 10},\n",
       " {0: '-',\n",
       "  1: '0',\n",
       "  2: '1',\n",
       "  3: '2',\n",
       "  4: '3',\n",
       "  5: '4',\n",
       "  6: '5',\n",
       "  7: '6',\n",
       "  8: '7',\n",
       "  9: '8',\n",
       "  10: '9'})"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[:10], human_vocab, machine_vocab, inv_machine_vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You've loaded:\n",
    "- `dataset`: a list of tuples of (human readable date, machine readable date)\n",
    "- `human_vocab`: a python dictionary mapping all characters used in the human readable dates to an integer-valued index \n",
    "- `machine_vocab`: a python dictionary mapping all characters used in machine readable dates to an integer-valued index. These indices are not necessarily consistent with `human_vocab`. \n",
    "- `inv_machine_vocab`: the inverse dictionary of `machine_vocab`, mapping from indices back to characters. \n",
    "\n",
    "Let's preprocess the data and map the raw text data into the index values. We will also use Tx=30 (which we assume is the maximum length of the human readable date; if we get a longer input, we would have to truncate it) and Ty=10 (since \"YYYY-MM-DD\" is 10 characters long). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X.shape: (10000, 30)\n",
      "Y.shape: (10000, 10)\n",
      "Xoh.shape: (10000, 30, 37)\n",
      "Yoh.shape: (10000, 10, 11)\n"
     ]
    }
   ],
   "source": [
    "Tx = 30\n",
    "Ty = 10\n",
    "X, Y, Xoh, Yoh = preprocess_data(dataset, human_vocab, machine_vocab, Tx, Ty)\n",
    "\n",
    "print(\"X.shape:\", X.shape)\n",
    "print(\"Y.shape:\", Y.shape)\n",
    "print(\"Xoh.shape:\", Xoh.shape)\n",
    "print(\"Yoh.shape:\", Yoh.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You now have:\n",
    "- `X`: a processed version of the human readable dates in the training set, where each character is replaced by an index mapped to the character via `human_vocab`. Each date is further padded to $T_x$ values with a special character (< pad >). `X.shape = (m, Tx)`\n",
    "- `Y`: a processed version of the machine readable dates in the training set, where each character is replaced by the index it is mapped to in `machine_vocab`. You should have `Y.shape = (m, Ty)`. \n",
    "- `Xoh`: one-hot version of `X`, the \"1\" entry's index is mapped to the character thanks to `human_vocab`. `Xoh.shape = (m, Tx, len(human_vocab))`\n",
    "- `Yoh`: one-hot version of `Y`, the \"1\" entry's index is mapped to the character thanks to `machine_vocab`. `Yoh.shape = (m, Tx, len(machine_vocab))`. Here, `len(machine_vocab) = 11` since there are 11 characters ('-' as well as 0-9). \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets also look at some examples of preprocessed training examples. Feel free to play with `index` in the cell below to navigate the dataset and see how source/target dates are preprocessed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source date: tuesday november 28 2000\n",
      "Target date: 2000-11-28\n",
      "\n",
      "Source after preprocessing (indices): [30 31 17 29 16 13 34  0 25 26 32 17 24 14 17 28  0  5 11  0  5  3  3  3\n",
      " 36 36 36 36 36 36]\n",
      "Target after preprocessing (indices): [3 1 1 1 0 2 2 0 3 9]\n",
      "\n",
      "Source after preprocessing (one-hot): [[0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " ...\n",
      " [0. 0. 0. ... 0. 0. 1.]\n",
      " [0. 0. 0. ... 0. 0. 1.]\n",
      " [0. 0. 0. ... 0. 0. 1.]]\n",
      "Target after preprocessing (one-hot): [[0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]]\n"
     ]
    }
   ],
   "source": [
    "index = 10\n",
    "print(\"Source date:\", dataset[index][0])\n",
    "print(\"Target date:\", dataset[index][1])\n",
    "print()\n",
    "print(\"Source after preprocessing (indices):\", X[index])\n",
    "print(\"Target after preprocessing (indices):\", Y[index])\n",
    "print()\n",
    "print(\"Source after preprocessing (one-hot):\", Xoh[index])\n",
    "print(\"Target after preprocessing (one-hot):\", Yoh[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - Neural machine translation with attention\n",
    "\n",
    "If you had to translate a book's paragraph from French to English, you would not read the whole paragraph, then close the book and translate. Even during the translation process, you would read/re-read and focus on the parts of the French paragraph corresponding to the parts of the English you are writing down. \n",
    "\n",
    "The attention mechanism tells a Neural Machine Translation model where it should pay attention to at any step. \n",
    "\n",
    "\n",
    "### 2.1 - Attention mechanism\n",
    "\n",
    "In this part, you will implement the attention mechanism presented in the lecture videos. Here is a figure to remind you how the model works. The diagram on the left shows the attention model. The diagram on the right shows what one \"Attention\" step does to calculate the attention variables $\\alpha^{\\langle t, t' \\rangle}$, which are used to compute the context variable $context^{\\langle t \\rangle}$ for each timestep in the output ($t=1, \\ldots, T_y$). \n",
    "\n",
    "<table>\n",
    "<td> \n",
    "<img src=\"images/attn_model.png\" style=\"width:500;height:500px;\"> <br>\n",
    "</td> \n",
    "<td> \n",
    "<img src=\"images/attn_mechanism.png\" style=\"width:500;height:500px;\"> <br>\n",
    "</td> \n",
    "</table>\n",
    "<caption><center> **Figure 1**: Neural machine translation with attention</center></caption>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Here are some properties of the model that you may notice: \n",
    "\n",
    "- There are two separate LSTMs in this model (see diagram on the left). Because the one at the bottom of the picture is a Bi-directional LSTM and comes *before* the attention mechanism, we will call it *pre-attention* Bi-LSTM. The LSTM at the top of the diagram comes *after* the attention mechanism, so we will call it the *post-attention* LSTM. The pre-attention Bi-LSTM goes through $T_x$ time steps; the post-attention LSTM goes through $T_y$ time steps. \n",
    "\n",
    "- The post-attention LSTM passes $s^{\\langle t \\rangle}, c^{\\langle t \\rangle}$ from one time step to the next. In the lecture videos, we were using only a basic RNN for the post-activation sequence model, so the state captured by the RNN output activations $s^{\\langle t\\rangle}$. But since we are using an LSTM here, the LSTM has both the output activation $s^{\\langle t\\rangle}$ and the hidden cell state $c^{\\langle t\\rangle}$. However, unlike previous text generation examples (such as Dinosaurus in week 1), in this model the post-activation LSTM at time $t$ does will not take the specific generated $y^{\\langle t-1 \\rangle}$ as input; it only takes $s^{\\langle t\\rangle}$ and $c^{\\langle t\\rangle}$ as input. We have designed the model this way, because (unlike language generation where adjacent characters are highly correlated) there isn't as strong a dependency between the previous character and the next character in a YYYY-MM-DD date. \n",
    "\n",
    "- We use $a^{\\langle t \\rangle} = [\\overrightarrow{a}^{\\langle t \\rangle}; \\overleftarrow{a}^{\\langle t \\rangle}]$ to represent the concatenation of the activations of both the forward-direction and backward-directions of the pre-attention Bi-LSTM. \n",
    "\n",
    "- The diagram on the right uses a `RepeatVector` node to copy $s^{\\langle t-1 \\rangle}$'s value $T_x$ times, and then `Concatenation` to concatenate $s^{\\langle t-1 \\rangle}$ and $a^{\\langle t \\rangle}$ to compute $e^{\\langle t, t'}$, which is then passed through a softmax to compute $\\alpha^{\\langle t, t' \\rangle}$. We'll explain how to use `RepeatVector` and `Concatenation` in Keras below. \n",
    "\n",
    "Lets implement this model. You will start by implementing two functions: `one_step_attention()` and `model()`.\n",
    "\n",
    "**1) `one_step_attention()`**: At step $t$, given all the hidden states of the Bi-LSTM ($[a^{<1>},a^{<2>}, ..., a^{<T_x>}]$) and the previous hidden state of the second LSTM ($s^{<t-1>}$), `one_step_attention()` will compute the attention weights ($[\\alpha^{<t,1>},\\alpha^{<t,2>}, ..., \\alpha^{<t,T_x>}]$) and output the context vector (see Figure  1 (right) for details):\n",
    "$$context^{<t>} = \\sum_{t' = 1}^{T_x} \\alpha^{<t,t'>}a^{<t'>}\\tag{1}$$ \n",
    "\n",
    "Note that we are denoting the attention in this notebook $context^{\\langle t \\rangle}$. In the lecture videos, the context was denoted $c^{\\langle t \\rangle}$, but here we are calling it $context^{\\langle t \\rangle}$ to avoid confusion with the (post-attention) LSTM's internal memory cell variable, which is sometimes also denoted $c^{\\langle t \\rangle}$. \n",
    "  \n",
    "**2) `model()`**: Implements the entire model. It first runs the input through a Bi-LSTM to get back $[a^{<1>},a^{<2>}, ..., a^{<T_x>}]$. Then, it calls `one_step_attention()` $T_y$ times (`for` loop). At each iteration of this loop, it gives the computed context vector $c^{<t>}$ to the second LSTM, and runs the output of the LSTM through a dense layer with softmax activation to generate a prediction $\\hat{y}^{<t>}$. \n",
    "\n",
    "\n",
    "\n",
    "**Exercise**: Implement `one_step_attention()`. The function `model()` will call the layers in `one_step_attention()` $T_y$ using a for-loop, and it is important that all $T_y$ copies have the same weights. I.e., it should not re-initiaiize the weights every time. In other words, all $T_y$ steps should have shared weights. Here's how you can implement layers with shareable weights in Keras:\n",
    "1. Define the layer objects (as global variables for examples).\n",
    "2. Call these objects when propagating the input.\n",
    "\n",
    "We have defined the layers you need as global variables. Please run the following cells to create them. Please check the Keras documentation to make sure you understand what these layers are: [RepeatVector()](https://keras.io/layers/core/#repeatvector), [Concatenate()](https://keras.io/layers/merge/#concatenate), [Dense()](https://keras.io/layers/core/#dense), [Activation()](https://keras.io/layers/core/#activation), [Dot()](https://keras.io/layers/merge/#dot)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31mInit signature:\u001b[0m  \u001b[0mDense\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mSource:\u001b[0m        \n",
      "\u001b[0;34m@\u001b[0m\u001b[0mkeras_export\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"keras.layers.Dense\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;32mclass\u001b[0m \u001b[0mDense\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mLayer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;34m\"\"\"Just your regular densely-connected NN layer.\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m    `Dense` implements the operation:\u001b[0m\n",
      "\u001b[0;34m    `output = activation(dot(input, kernel) + bias)`\u001b[0m\n",
      "\u001b[0;34m    where `activation` is the element-wise activation function\u001b[0m\n",
      "\u001b[0;34m    passed as the `activation` argument, `kernel` is a weights matrix\u001b[0m\n",
      "\u001b[0;34m    created by the layer, and `bias` is a bias vector created by the layer\u001b[0m\n",
      "\u001b[0;34m    (only applicable if `use_bias` is `True`).\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m    Note: If the input to the layer has a rank greater than 2, `Dense`\u001b[0m\n",
      "\u001b[0;34m    computes the dot product between the `inputs` and the `kernel` along the\u001b[0m\n",
      "\u001b[0;34m    last axis of the `inputs` and axis 0 of the `kernel` (using `tf.tensordot`).\u001b[0m\n",
      "\u001b[0;34m    For example, if input has dimensions `(batch_size, d0, d1)`, then we create\u001b[0m\n",
      "\u001b[0;34m    a `kernel` with shape `(d1, units)`, and the `kernel` operates along axis 2\u001b[0m\n",
      "\u001b[0;34m    of the `input`, on every sub-tensor of shape `(1, 1, d1)` (there are\u001b[0m\n",
      "\u001b[0;34m    `batch_size * d0` such sub-tensors). The output in this case will have\u001b[0m\n",
      "\u001b[0;34m    shape `(batch_size, d0, units)`.\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m    Args:\u001b[0m\n",
      "\u001b[0;34m        units: Positive integer, dimensionality of the output space.\u001b[0m\n",
      "\u001b[0;34m        activation: Activation function to use.\u001b[0m\n",
      "\u001b[0;34m            If you don't specify anything, no activation is applied\u001b[0m\n",
      "\u001b[0;34m            (ie. \"linear\" activation: `a(x) = x`).\u001b[0m\n",
      "\u001b[0;34m        use_bias: Boolean, whether the layer uses a bias vector.\u001b[0m\n",
      "\u001b[0;34m        kernel_initializer: Initializer for the `kernel` weights matrix.\u001b[0m\n",
      "\u001b[0;34m        bias_initializer: Initializer for the bias vector.\u001b[0m\n",
      "\u001b[0;34m        kernel_regularizer: Regularizer function applied to\u001b[0m\n",
      "\u001b[0;34m            the `kernel` weights matrix.\u001b[0m\n",
      "\u001b[0;34m        bias_regularizer: Regularizer function applied to the bias vector.\u001b[0m\n",
      "\u001b[0;34m        activity_regularizer: Regularizer function applied to\u001b[0m\n",
      "\u001b[0;34m            the output of the layer (its \"activation\").\u001b[0m\n",
      "\u001b[0;34m        kernel_constraint: Constraint function applied to\u001b[0m\n",
      "\u001b[0;34m            the `kernel` weights matrix.\u001b[0m\n",
      "\u001b[0;34m        bias_constraint: Constraint function applied to the bias vector.\u001b[0m\n",
      "\u001b[0;34m        lora_rank: Optional integer. If set, the layer's forward pass\u001b[0m\n",
      "\u001b[0;34m            will implement LoRA (Low-Rank Adaptation)\u001b[0m\n",
      "\u001b[0;34m            with the provided rank. LoRA sets the layer's kernel\u001b[0m\n",
      "\u001b[0;34m            to non-trainable and replaces it with a delta over the\u001b[0m\n",
      "\u001b[0;34m            original kernel, obtained via multiplying two lower-rank\u001b[0m\n",
      "\u001b[0;34m            trainable matrices. This can be useful to reduce the\u001b[0m\n",
      "\u001b[0;34m            computation cost of fine-tuning large dense layers.\u001b[0m\n",
      "\u001b[0;34m            You can also enable LoRA on an existing\u001b[0m\n",
      "\u001b[0;34m            `Dense` layer by calling `layer.enable_lora(rank)`.\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m    Input shape:\u001b[0m\n",
      "\u001b[0;34m        N-D tensor with shape: `(batch_size, ..., input_dim)`.\u001b[0m\n",
      "\u001b[0;34m        The most common situation would be\u001b[0m\n",
      "\u001b[0;34m        a 2D input with shape `(batch_size, input_dim)`.\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m    Output shape:\u001b[0m\n",
      "\u001b[0;34m        N-D tensor with shape: `(batch_size, ..., units)`.\u001b[0m\n",
      "\u001b[0;34m        For instance, for a 2D input with shape `(batch_size, input_dim)`,\u001b[0m\n",
      "\u001b[0;34m        the output would have shape `(batch_size, units)`.\u001b[0m\n",
      "\u001b[0;34m    \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0munits\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mactivation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0muse_bias\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mkernel_initializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"glorot_uniform\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mbias_initializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"zeros\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mkernel_regularizer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mbias_regularizer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mactivity_regularizer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mkernel_constraint\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mbias_constraint\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mlora_rank\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mactivity_regularizer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mactivity_regularizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munits\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0munits\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactivation\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mactivations\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mactivation\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0muse_bias\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0muse_bias\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_initializer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minitializers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkernel_initializer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias_initializer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minitializers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbias_initializer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_regularizer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mregularizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkernel_regularizer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias_regularizer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mregularizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbias_regularizer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_constraint\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconstraints\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkernel_constraint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias_constraint\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconstraints\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbias_constraint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_rank\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlora_rank\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_enabled\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minput_spec\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mInputSpec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmin_ndim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msupports_masking\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mbuild\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_shape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0minput_dim\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minput_shape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantization_mode\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantized_build\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_shape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantization_mode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantization_mode\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;34m\"int8\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;31m# If the layer is quantized to int8, `self._kernel` will be added\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;31m# in `self._int8_build`. Therefore, we skip it here.\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_kernel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"kernel\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mshape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_dim\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munits\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0minitializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_initializer\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mregularizer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_regularizer\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mconstraint\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_constraint\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0muse_bias\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"bias\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mshape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munits\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0minitializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias_initializer\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mregularizer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias_regularizer\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mconstraint\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias_constraint\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minput_spec\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mInputSpec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmin_ndim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxes\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0minput_dim\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuilt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_rank\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menable_lora\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_rank\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mkernel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuilt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m\"You must build the layer before accessing `kernel`.\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_enabled\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_kernel\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_kernel_a\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_kernel_b\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_kernel\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mcall\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtraining\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactivation\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactivation\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mcompute_output_shape\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_shape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0moutput_shape\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_shape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0moutput_shape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munits\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput_shape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0menable_lora\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrank\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ma_initializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"he_uniform\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mb_initializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"zeros\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_constraint\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m\"Lora is incompatible with kernel constraints. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m\"In order to enable lora on this layer, remove the \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m\"`kernel_constraint` argument.\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuilt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m\"Cannot enable lora on a layer that isn't yet built.\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_enabled\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m\"lora is already enabled. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m\"This can only be done once per layer.\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_tracker\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munlock\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_kernel_a\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"lora_kernel_a\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mshape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrank\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0minitializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitializers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma_initializer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mregularizer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_regularizer\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_kernel_b\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"lora_kernel_b\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mshape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrank\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0minitializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitializers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb_initializer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mregularizer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_regularizer\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_kernel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainable\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_tracker\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlock\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_enabled\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_rank\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrank\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0msave_own_variables\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstore\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# Do nothing if the layer isn't yet built\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuilt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# The keys of the `store` will be saved as determined because the\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# default ordering will change after quantization\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mkernel_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_kernel_with_merged_lora\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mtarget_variables\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mkernel_value\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0muse_bias\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantization_mode\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantization_mode\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"int8\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkernel_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantization_mode\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"float8\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputs_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputs_amax_history\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_amax_history\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs_grad_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs_grad_amax_history\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_quantization_mode_error\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantization_mode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvariable\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtarget_variables\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mstore\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvariable\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mload_own_variables\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstore\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_enabled\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_load_own_variables\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstore\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# Do nothing if the layer isn't yet built\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuilt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# The keys of the `store` will be saved as determined because the\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# default ordering will change after quantization\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mtarget_variables\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_kernel\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0muse_bias\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantization_mode\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantization_mode\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"int8\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantization_mode\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"float8\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputs_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputs_amax_history\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_amax_history\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs_grad_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mtarget_variables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs_grad_amax_history\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_quantization_mode_error\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantization_mode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvariable\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtarget_variables\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mvariable\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0massign\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstore\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_enabled\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_kernel_a\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0massign\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_kernel_a\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_kernel_b\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0massign\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_kernel_b\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mget_config\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mbase_config\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_config\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mconfig\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"units\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munits\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"activation\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mactivations\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mserialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactivation\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"use_bias\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0muse_bias\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"kernel_initializer\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0minitializers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mserialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_initializer\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"bias_initializer\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0minitializers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mserialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias_initializer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"kernel_regularizer\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mregularizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mserialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_regularizer\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"bias_regularizer\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mregularizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mserialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias_regularizer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"kernel_constraint\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mconstraints\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mserialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_constraint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"bias_constraint\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mconstraints\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mserialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias_constraint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_rank\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mconfig\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"lora_rank\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_rank\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mbase_config\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_check_load_own_variables\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstore\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mall_vars\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trainable_variables\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_non_trainable_variables\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mall_vars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mall_vars\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuilt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34mf\"Layer '{self.name}' was never built \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"and thus it doesn't have any variables. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34mf\"However the weights file lists {len(store.keys())} \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"variables for this layer.\\n\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"In most cases, this error indicates that either:\\n\\n\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"1. The layer is owned by a parent layer that \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"implements a `build()` method, but calling the \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"parent's `build()` method did NOT create the state of \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34mf\"the child layer '{self.name}'. A `build()` method \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"must create ALL state for the layer, including \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"the state of any children layers.\\n\\n\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"2. You need to implement \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"the `def build_from_config(self, config)` method \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34mf\"on layer '{self.name}', to specify how to rebuild \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"it during loading. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"In this case, you might also want to implement the \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"method that generates the build config at saving time, \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"`def get_build_config(self)`. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"The method `build_from_config()` is meant \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"to create the state \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"of the layer (i.e. its variables) upon deserialization.\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34mf\"Layer '{self.name}' expected {len(all_vars)} variables, \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m\"but received \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34mf\"{len(store.keys())} variables during loading. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34mf\"Expected: {[v.name for v in all_vars]}\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;31m# Quantization-related (int8 and float8) methods\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mquantized_build\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_shape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"int8\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0minput_dim\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minput_shape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mkernel_shape\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0minput_dim\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munits\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_int8_build\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkernel_shape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"float8\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_float8_build\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_quantization_mode_error\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_int8_build\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mkernel_shape\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mkernel_initializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"zeros\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mkernel_scale_initializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"ones\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputs_quantizer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mquantizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mAbsMaxQuantizer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_kernel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"kernel\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mshape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkernel_shape\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0minitializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkernel_initializer\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"int8\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mtrainable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"kernel_scale\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mshape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munits\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0minitializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkernel_scale_initializer\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mtrainable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_quantized\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_float8_build\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mfrom\u001b[0m \u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype_policies\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mQuantizedFloat8DTypePolicy\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# If `self.dtype_policy` is not QuantizedFloat8DTypePolicy, then set\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# `amax_history_length` to its default value.\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mamax_history_length\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype_policy\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"amax_history_length\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mQuantizedFloat8DTypePolicy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdefault_amax_history_length\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# We set `trainable=True` because we will use the gradients to overwrite\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# these variables\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mscale_kwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"shape\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"initializer\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m\"ones\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"dtype\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m\"float32\"\u001b[0m\u001b[0;34m,\u001b[0m  \u001b[0;31m# Always be float32\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"trainable\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"autocast\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mamax_history_kwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"shape\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mamax_history_length\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"initializer\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m\"zeros\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"dtype\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m\"float32\"\u001b[0m\u001b[0;34m,\u001b[0m  \u001b[0;31m# Always be float32\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"trainable\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"autocast\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputs_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"inputs_scale\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mscale_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputs_amax_history\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"inputs_amax_history\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mamax_history_kwargs\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"kernel_scale\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mscale_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_amax_history\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"kernel_amax_history\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mamax_history_kwargs\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs_grad_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"outputs_grad_scale\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mscale_kwargs\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs_grad_amax_history\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"outputs_grad_amax_history\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mamax_history_kwargs\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# We need to set `overwrite_with_gradient=True` to instruct the\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# optimizer to directly overwrite these variables with their computed\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# gradients during training\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputs_scale\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moverwrite_with_gradient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputs_amax_history\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moverwrite_with_gradient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_scale\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moverwrite_with_gradient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_amax_history\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moverwrite_with_gradient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs_grad_scale\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moverwrite_with_gradient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs_grad_amax_history\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moverwrite_with_gradient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_quantized\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_int8_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtraining\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m@\u001b[0m\u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcustom_gradient\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mdef\u001b[0m \u001b[0mmatmul_with_inputs_gradient\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mdef\u001b[0m \u001b[0mgrad_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mupstream\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mupstream\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m(\u001b[0m\u001b[0mupstream\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mfloat_kernel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdivide\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkernel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_dtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0mkernel_scale\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0minputs_grad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mupstream\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranspose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfloat_kernel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;32mreturn\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0minputs_grad\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minputs_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputs_quantizer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;31m# De-scale outputs\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_dtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdivide\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmultiply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs_scale\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mreturn\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_fn\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmatmul_with_inputs_gradient\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconvert_to_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_kernel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconvert_to_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_enabled\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mlora_x\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_kernel_a\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mlora_x\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlora_x\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_kernel_b\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlora_x\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactivation\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactivation\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_float8_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtraining\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_enabled\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m\"Currently, `_float8_call` doesn't support LoRA\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m@\u001b[0m\u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcustom_gradient\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mdef\u001b[0m \u001b[0mquantized_dequantize_inputs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscale\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mamax_history\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mtraining\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mnew_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mquantizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_float8_scale\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mamax_history\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0mscale\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                        \u001b[0mfloat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mml_dtypes\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfinfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"float8_e4m3fn\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"float32\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mnew_amax_history\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mquantizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_float8_amax_history\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mamax_history\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mnew_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mnew_amax_history\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mqdq_inputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mquantizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantize_and_dequantize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscale\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"float8_e4m3fn\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_dtype\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mdef\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mupstream\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvariables\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mupstream\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m(\u001b[0m\u001b[0mupstream\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;32mreturn\u001b[0m \u001b[0mupstream\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnew_scale\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnew_amax_history\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mreturn\u001b[0m \u001b[0mqdq_inputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m@\u001b[0m\u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcustom_gradient\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mdef\u001b[0m \u001b[0mquantized_dequantize_outputs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscale\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mamax_history\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m\"\"\"Quantize-dequantize the output gradient but not the output.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mdef\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mupstream\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvariables\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mupstream\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m(\u001b[0m\u001b[0mupstream\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mnew_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mquantizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_float8_scale\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mamax_history\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0mscale\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                        \u001b[0mfloat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mml_dtypes\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfinfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"float8_e5m2\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"float32\"\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mqdq_upstream\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mquantizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantize_and_dequantize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0mupstream\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscale\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"float8_e5m2\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_dtype\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mnew_amax_history\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mquantizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_float8_amax_history\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0mupstream\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mamax_history\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;32mreturn\u001b[0m \u001b[0mqdq_upstream\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnew_scale\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnew_amax_history\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mreturn\u001b[0m \u001b[0moutputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mquantized_dequantize_inputs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconvert_to_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputs_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconvert_to_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputs_amax_history\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mquantized_dequantize_inputs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconvert_to_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_kernel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconvert_to_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconvert_to_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_amax_history\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# `quantized_dequantize_outputs` is placed immediately after\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# `ops.matmul` for the sake of pattern matching in gemm_rewrite. That\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# way, the qdq will be adjacent to the corresponding matmul_bprop in the\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# bprop.\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mquantized_dequantize_outputs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconvert_to_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs_grad_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconvert_to_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs_grad_amax_history\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;31m# Under non-mixed precision cases, F32 bias has to be converted to\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;31m# BF16 first to get the biasAdd fusion support. ref. PR\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;31m# https://github.com/tensorflow/tensorflow/pull/60306\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mbias\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype_policy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_dtype\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"float32\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mbias_bf16\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"bfloat16\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mbias\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbias_bf16\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactivation\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactivation\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mquantize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtype_check\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# Prevent quantization of the subclasses\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mtype_check\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mDense\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_not_implemented_error\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantize\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"int8\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;31m# Quantize `self._kernel` to int8 and compute corresponding scale\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mkernel_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mquantizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mabs_max_quantize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_kernel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mto_numpy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mkernel_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkernel_scale\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mkernel_shape\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_kernel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_kernel\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;31m# Utilize a lambda expression as an initializer to prevent adding a\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;31m# large constant to the computation graph.\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_int8_build\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkernel_shape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"float8\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_float8_build\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_quantization_mode_error\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;31m# Set new dtype policy\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype_policy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantization_mode\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mpolicy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdtype_policies\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{mode}_from_{self.dtype_policy.name}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype_policy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpolicy\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_get_kernel_with_merged_lora\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype_policy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquantization_mode\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mkernel_value\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_kernel\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0mkernel_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel_scale\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_enabled\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;31m# Dequantize & quantize to merge lora weights into int8 kernel\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;31m# Note that this is a lossy compression\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mkernel_value\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdivide\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkernel_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel_scale\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mkernel_value\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0mkernel_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_kernel_a\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlora_kernel_b\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mkernel_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mquantizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mabs_max_quantize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                    \u001b[0mkernel_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mto_numpy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m                \u001b[0mkernel_scale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkernel_scale\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m            \u001b[0;32mreturn\u001b[0m \u001b[0mkernel_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel_scale\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkernel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFile:\u001b[0m           ~/Library/Python/3.9/lib/python/site-packages/keras/src/layers/core/dense.py\n",
      "\u001b[0;31mType:\u001b[0m           type\n",
      "\u001b[0;31mSubclasses:\u001b[0m     "
     ]
    }
   ],
   "source": [
    "?? Dense"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Defined shared layers as global variables\n",
    "\n",
    "repeator = RepeatVector(Tx)\n",
    "concatenator = Concatenate(axis=-1)\n",
    "densor1 = Dense(10, activation = \"tanh\")\n",
    "densor2 = Dense(1, activation = \"relu\")\n",
    "activator = Activation(softmax, name='attention_weights') # We are using a custom softmax(axis = 1) loaded in this notebook\n",
    "dotor = Dot(axes = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can use these layers to implement `one_step_attention()`. In order to propagate a Keras tensor object X through one of these layers, use `layer(X)` (or `layer([X,Y])` if it requires multiple inputs.), e.g. `densor(X)` will propagate X through the `Dense(1)` layer defined above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# # GRADED FUNCTION: one_step_attention\n",
    "\n",
    "# def one_step_attention(a, s_prev):\n",
    "#     \"\"\"\n",
    "#     Performs one step of attention: Outputs a context vector computed as a dot product of the attention weights\n",
    "#     \"alphas\" and the hidden states \"a\" of the Bi-LSTM.\n",
    "    \n",
    "#     Arguments:\n",
    "#     a -- hidden state output of the Bi-LSTM, numpy-array of shape (m, Tx, 2*n_a)\n",
    "#     s_prev -- previous hidden state of the (post-attention) LSTM, numpy-array of shape (m, n_s)\n",
    "    \n",
    "#     Returns:\n",
    "#     context -- context vector, input of the next (post-attetion) LSTM cell\n",
    "#     \"\"\"\n",
    "    \n",
    "#     ### START CODE HERE ###\n",
    "#     # Use repeator to repeat s_prev to be of shape (m, Tx, n_s) so that you can concatenate it with all hidden states \"a\" (≈ 1 line)\n",
    "#     s_prev = repeator(s_prev) \n",
    "#     # Use concatenator to concatenate a and s_prev on the last axis (≈ 1 line)\n",
    "#     concat = concatenator([a,s_prev]) \n",
    "#     # Use densor1 to propagate concat through a small fully-connected neural network to compute the \"intermediate energies\" variable e. (≈1 lines)\n",
    "#     e = densor1(concat) \n",
    "#     # Use densor2 to propagate e through a small fully-connected neural network to compute the \"energies\" variable energies. (≈1 lines)\n",
    "#     energies = densor2(e) \n",
    "#     # Use \"activator\" on \"energies\" to compute the attention weights \"alphas\" (≈ 1 line)\n",
    "#     alphas = activator(energies) \n",
    "#     # Use dotor together with \"alphas\" and \"a\" to compute the context vector to be given to the next (post-attention) LSTM-cell (≈ 1 line)\n",
    "#     context = dotor([alphas,a]) \n",
    "#     ### END CODE HERE ###\n",
    "    \n",
    "#     return context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GRADED FUNCTION: one_step_attention\n",
    "\n",
    "def one_step_attention(a, s_prev):\n",
    "    \"\"\"\n",
    "    Performs one step of attention: Outputs a context vector computed as a dot product of the attention weights\n",
    "    \"alphas\" and the hidden states \"a\" of the Bi-LSTM.\n",
    "    \n",
    "    Arguments:\n",
    "    a -- hidden state output of the Bi-LSTM, numpy-array of shape (m, Tx, 2*n_a)\n",
    "    s_prev -- previous hidden state of the (post-attention) LSTM, numpy-array of shape (m, n_s)\n",
    "    \n",
    "    Returns:\n",
    "    context -- context vector, input of the next (post-attetion) LSTM cell\n",
    "    \"\"\"\n",
    "    \n",
    "    ### START CODE HERE ###\n",
    "    # Use repeator to repeat s_prev to be of shape (m, Tx, n_s) so that you can concatenate it with all hidden states \"a\" (≈ 1 line)\n",
    "    s_prev = repeator(s_prev)\n",
    "    print(s_prev.shape)\n",
    "    # Use concatenator to concatenate a and s_prev on the last axis (≈ 1 line)\n",
    "    s_curr = concatenator([a, s_prev])\n",
    "    \n",
    "    # Use densor1 to propagate concat through a small fully-connected neural network to compute the \"intermediate energies\" variable e. (≈1 lines)\n",
    "    e = densor1(s_curr)\n",
    "    # Use densor2 to propagate e through a small fully-connected neural network to compute the \"energies\" variable energies. (≈1 lines)\n",
    "    energies = densor2(e)\n",
    "    # Use \"activator\" on \"energies\" to compute the attention weights \"alphas\" (≈ 1 line)\n",
    "    alphas = activator(energies)\n",
    "    # Use dotor together with \"alphas\" and \"a\" to compute the context vector to be given to the next (post-attention) LSTM-cell (≈ 1 line)\n",
    "    context = dotor([alphas, a])\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will be able to check the expected output of `one_step_attention()` after you've coded the `model()` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Implement `model()` as explained in figure 2 and the text above. Again, we have defined global layers that will share weights to be used in `model()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_a = 32\n",
    "n_s = 64\n",
    "post_activation_LSTM_cell = LSTM(n_s, return_state = True)\n",
    "output_layer = Dense(len(machine_vocab), activation=softmax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can use these layers $T_y$ times in a `for` loop to generate the outputs, and their parameters will not be reinitialized. You will have to carry out the following steps: \n",
    "\n",
    "1. Propagate the input into a [Bidirectional](https://keras.io/layers/wrappers/#bidirectional) [LSTM](https://keras.io/layers/recurrent/#lstm)\n",
    "2. Iterate for $t = 0, \\dots, T_y-1$: \n",
    "    1. Call `one_step_attention()` on $[\\alpha^{<t,1>},\\alpha^{<t,2>}, ..., \\alpha^{<t,T_x>}]$ and $s^{<t-1>}$ to get the context vector $context^{<t>}$.\n",
    "    2. Give $context^{<t>}$ to the post-attention LSTM cell. Remember pass in the previous hidden-state $s^{\\langle t-1\\rangle}$ and cell-states $c^{\\langle t-1\\rangle}$ of this LSTM using `initial_state= [previous hidden state, previous cell state]`. Get back the new hidden state $s^{<t>}$ and the new cell state $c^{<t>}$.\n",
    "    3. Apply a softmax layer to $s^{<t>}$, get the output. \n",
    "    4. Save the output by adding it to the list of outputs.\n",
    "\n",
    "3. Create your Keras model instance, it should have three inputs (\"inputs\", $s^{<0>}$ and $c^{<0>}$) and output the list of \"outputs\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# # GRADED FUNCTION: model\n",
    "\n",
    "# def model(Tx, Ty, n_a, n_s, human_vocab_size, machine_vocab_size):\n",
    "#     \"\"\"\n",
    "#     Arguments:\n",
    "#     Tx -- length of the input sequence\n",
    "#     Ty -- length of the output sequence\n",
    "#     n_a -- hidden state size of the Bi-LSTM\n",
    "#     n_s -- hidden state size of the post-attention LSTM\n",
    "#     human_vocab_size -- size of the python dictionary \"human_vocab\"\n",
    "#     machine_vocab_size -- size of the python dictionary \"machine_vocab\"\n",
    "\n",
    "#     Returns:\n",
    "#     model -- Keras model instance\n",
    "#     \"\"\"\n",
    "    \n",
    "#     # Define the inputs of your model with a shape (Tx,)\n",
    "#     # Define s0 and c0, initial hidden state for the decoder LSTM of shape (n_s,)\n",
    "#     X = Input(shape=(Tx, human_vocab_size))\n",
    "#     s0 = Input(shape=(n_s,), name='s0')\n",
    "#     c0 = Input(shape=(n_s,), name='c0')\n",
    "#     s = s0\n",
    "#     c = c0\n",
    "    \n",
    "#     # Initialize empty list of outputs\n",
    "#     outputs = []\n",
    "    \n",
    "#     ### START CODE HERE ###\n",
    "    \n",
    "#     # Step 1: Define your pre-attention Bi-LSTM. Remember to use return_sequences=True. (≈ 1 line)\n",
    "#     a = Bidirectional(LSTM(n_a, return_sequences=True),input_shape=(m, Tx, n_a*2))(X)  \n",
    "    \n",
    "#     # Step 2: Iterate for Ty steps\n",
    "#     for t in range(Ty):\n",
    "    \n",
    "#         # Step 2.A: Perform one step of the attention mechanism to get back the context vector at step t (≈ 1 line)\n",
    "#         context = one_step_attention(a, s)\n",
    "        \n",
    "#         # Step 2.B: Apply the post-attention LSTM cell to the \"context\" vector.\n",
    "#         # Don't forget to pass: initial_state = [hidden state, cell state] (≈ 1 line)\n",
    "#         s, _, c = post_activation_LSTM_cell(context,initial_state = [s, c] ) \n",
    "        \n",
    "#         # Step 2.C: Apply Dense layer to the hidden state output of the post-attention LSTM (≈ 1 line)\n",
    "#         out = output_layer(s)\n",
    "        \n",
    "#         # Step 2.D: Append \"out\" to the \"outputs\" list (≈ 1 line)\n",
    "#         outputs.append(out)\n",
    "    \n",
    "#     # Step 3: Create model instance taking three inputs and returning the list of outputs. (≈ 1 line)\n",
    "#     model = Model(inputs=[X,s0,c0],outputs=outputs)\n",
    "    \n",
    "#     ### END CODE HERE ###\n",
    "    \n",
    "#     return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GRADED FUNCTION: model\n",
    "\n",
    "def model(Tx, Ty, n_a, n_s, human_vocab_size, machine_vocab_size):\n",
    "    \"\"\"\n",
    "    Arguments:\n",
    "    Tx -- length of the input sequence\n",
    "    Ty -- length of the output sequence\n",
    "    n_a -- hidden state size of the Bi-LSTM\n",
    "    n_s -- hidden state size of the post-attention LSTM\n",
    "    human_vocab_size -- size of the python dictionary \"human_vocab\"\n",
    "    machine_vocab_size -- size of the python dictionary \"machine_vocab\"\n",
    "\n",
    "    Returns:\n",
    "    model -- Keras model instance\n",
    "    \"\"\"\n",
    "    \n",
    "    # Define the inputs of your model with a shape (Tx,)\n",
    "    # Define s0 and c0, initial hidden state for the decoder LSTM of shape (n_s,)\n",
    "    X = Input(shape=(Tx, human_vocab_size))\n",
    "    s0 = Input(shape=(n_s,), name='s0')\n",
    "    c0 = Input(shape=(n_s,), name='c0')\n",
    "    s = s0\n",
    "    c = c0\n",
    "    \n",
    "    # Initialize empty list of outputs\n",
    "    outputs = []\n",
    "    \n",
    "    ### START CODE HERE ###\n",
    "    \n",
    "    # Step 1: Define your pre-attention Bi-LSTM. Remember to use return_sequences=True. (≈ 1 line)\n",
    "    \n",
    "    a = Bidirectional(LSTM(n_a, return_sequences=True), input_shape=(m, Tx, 2*n_a))(X)\n",
    "    \n",
    "    # Step 2: Iterate for Ty steps\n",
    "    for t in range(Ty):\n",
    "    \n",
    "        # Step 2.A: Perform one step of the attention mechanism to get back the context vector at step t (≈ 1 line)\n",
    "        context = one_step_attention(a=a, s_prev=s)\n",
    "        # Step 2.B: Apply the post-attention LSTM cell to the \"context\" vector.\n",
    "        # Don't forget to pass: initial_state = [hidden state, cell state] (≈ 1 line)\n",
    "        \n",
    "        s, _, c = post_activation_LSTM_cell(context, initial_state=[s, c])\n",
    "        \n",
    "        # Step 2.C: Apply Dense layer to the hidden state output of the post-attention LSTM (≈ 1 line)\n",
    "        outputs.append(output_layer(s))\n",
    "        \n",
    "        # Step 2.D: Append \"out\" to the \"outputs\" list (≈ 1 line)\n",
    "        \n",
    "    \n",
    "    # Step 3: Create model instance taking three inputs and returning the list of outputs. (≈ 1 line)\n",
    "   \n",
    "    model = Model(inputs = [X, s0, c0], outputs=outputs)\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to create your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(None, 30, 64)\n",
      "(None, 30, 64)\n",
      "(None, 30, 64)\n",
      "(None, 30, 64)\n",
      "(None, 30, 64)\n",
      "(None, 30, 64)\n",
      "(None, 30, 64)\n",
      "(None, 30, 64)\n",
      "(None, 30, 64)\n",
      "(None, 30, 64)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/msatidas/Library/Python/3.9/lib/python/site-packages/keras/src/layers/rnn/bidirectional.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.\n",
      "  super().__init__(**kwargs)\n"
     ]
    }
   ],
   "source": [
    "model = model(Tx, Ty, n_a, n_s, len(human_vocab), len(machine_vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get a summary of the model to check if it matches the expected output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"functional\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"functional\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)        </span>┃<span style=\"font-weight: bold\"> Output Shape      </span>┃<span style=\"font-weight: bold\">    Param # </span>┃<span style=\"font-weight: bold\"> Connected to      </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩\n",
       "│ input_layer_2       │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">37</span>)    │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                 │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │                   │            │                   │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ s0 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)        │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                 │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ bidirectional_2     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)    │     <span style=\"color: #00af00; text-decoration-color: #00af00\">17,920</span> │ input_layer_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]… │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Bidirectional</span>)     │                   │            │                   │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ repeat_vector       │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)    │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ s0[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],         │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">RepeatVector</span>)      │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]        │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ concatenate         │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">128</span>)   │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Concatenate</span>)       │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">9</span>]… │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dense (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)       │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">10</span>)    │      <span style=\"color: #00af00; text-decoration-color: #00af00\">1,290</span> │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">9</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>] │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dense_1 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>)     │         <span style=\"color: #00af00; text-decoration-color: #00af00\">11</span> │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">9</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ attention_weights   │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>)     │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Activation</span>)        │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">9</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]     │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dot (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dot</span>)           │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)     │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ c0 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)        │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                 │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ lstm (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">LSTM</span>)         │ [(<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>),      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">33,024</span> │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>),       │            │ s0[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],         │\n",
       "│                     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)]       │            │ c0[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],         │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">9</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>]        │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dense_2 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">11</span>)        │        <span style=\"color: #00af00; text-decoration-color: #00af00\">715</span> │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">9</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]        │\n",
       "└─────────────────────┴───────────────────┴────────────┴───────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)       \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape     \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m   Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to     \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩\n",
       "│ input_layer_2       │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m37\u001b[0m)    │          \u001b[38;5;34m0\u001b[0m │ -                 │\n",
       "│ (\u001b[38;5;33mInputLayer\u001b[0m)        │                   │            │                   │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ s0 (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m64\u001b[0m)        │          \u001b[38;5;34m0\u001b[0m │ -                 │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ bidirectional_2     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m64\u001b[0m)    │     \u001b[38;5;34m17,920\u001b[0m │ input_layer_2[\u001b[38;5;34m0\u001b[0m]… │\n",
       "│ (\u001b[38;5;33mBidirectional\u001b[0m)     │                   │            │                   │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ repeat_vector       │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m64\u001b[0m)    │          \u001b[38;5;34m0\u001b[0m │ s0[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],         │\n",
       "│ (\u001b[38;5;33mRepeatVector\u001b[0m)      │                   │            │ lstm[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m]        │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ concatenate         │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m128\u001b[0m)   │          \u001b[38;5;34m0\u001b[0m │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│ (\u001b[38;5;33mConcatenate\u001b[0m)       │                   │            │ repeat_vector[\u001b[38;5;34m0\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m1\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m2\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m3\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m4\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m5\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m6\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m7\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m8\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m9\u001b[0m]… │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dense (\u001b[38;5;33mDense\u001b[0m)       │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m10\u001b[0m)    │      \u001b[38;5;34m1,290\u001b[0m │ concatenate[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m9\u001b[0m][\u001b[38;5;34m0\u001b[0m] │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dense_1 (\u001b[38;5;33mDense\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m1\u001b[0m)     │         \u001b[38;5;34m11\u001b[0m │ dense[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m9\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ attention_weights   │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m1\u001b[0m)     │          \u001b[38;5;34m0\u001b[0m │ dense_1[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│ (\u001b[38;5;33mActivation\u001b[0m)        │                   │            │ dense_1[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m9\u001b[0m][\u001b[38;5;34m0\u001b[0m]     │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dot (\u001b[38;5;33mDot\u001b[0m)           │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m1\u001b[0m, \u001b[38;5;34m64\u001b[0m)     │          \u001b[38;5;34m0\u001b[0m │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ c0 (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m64\u001b[0m)        │          \u001b[38;5;34m0\u001b[0m │ -                 │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ lstm (\u001b[38;5;33mLSTM\u001b[0m)         │ [(\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m64\u001b[0m),      │     \u001b[38;5;34m33,024\u001b[0m │ dot[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m64\u001b[0m),       │            │ s0[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],         │\n",
       "│                     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m64\u001b[0m)]       │            │ c0[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],         │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m9\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m2\u001b[0m]        │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dense_2 (\u001b[38;5;33mDense\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m11\u001b[0m)        │        \u001b[38;5;34m715\u001b[0m │ lstm[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m9\u001b[0m][\u001b[38;5;34m0\u001b[0m]        │\n",
       "└─────────────────────┴───────────────────┴────────────┴───────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">52,960</span> (206.88 KB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m52,960\u001b[0m (206.88 KB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">52,960</span> (206.88 KB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m52,960\u001b[0m (206.88 KB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output**:\n",
    "\n",
    "Here is the summary you should see\n",
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            **Total params:**\n",
    "        </td>\n",
    "        <td>\n",
    "         52,960\n",
    "        </td>\n",
    "    </tr>\n",
    "        <tr>\n",
    "        <td>\n",
    "            **Trainable params:**\n",
    "        </td>\n",
    "        <td>\n",
    "         52,960\n",
    "        </td>\n",
    "    </tr>\n",
    "            <tr>\n",
    "        <td>\n",
    "            **Non-trainable params:**\n",
    "        </td>\n",
    "        <td>\n",
    "         0\n",
    "        </td>\n",
    "    </tr>\n",
    "                    <tr>\n",
    "        <td>\n",
    "            **bidirectional_1's output shape **\n",
    "        </td>\n",
    "        <td>\n",
    "         (None, 30, 64)  \n",
    "        </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>\n",
    "            **repeat_vector_1's output shape **\n",
    "        </td>\n",
    "        <td>\n",
    "         (None, 30, 64) \n",
    "        </td>\n",
    "    </tr>\n",
    "                <tr>\n",
    "        <td>\n",
    "            **concatenate_1's output shape **\n",
    "        </td>\n",
    "        <td>\n",
    "         (None, 30, 128) \n",
    "        </td>\n",
    "    </tr>\n",
    "            <tr>\n",
    "        <td>\n",
    "            **attention_weights's output shape **\n",
    "        </td>\n",
    "        <td>\n",
    "         (None, 30, 1)  \n",
    "        </td>\n",
    "    </tr>\n",
    "        <tr>\n",
    "        <td>\n",
    "            **dot_1's output shape **\n",
    "        </td>\n",
    "        <td>\n",
    "         (None, 1, 64)\n",
    "        </td>\n",
    "    </tr>\n",
    "           <tr>\n",
    "        <td>\n",
    "            **dense_3's output shape **\n",
    "        </td>\n",
    "        <td>\n",
    "         (None, 11) \n",
    "        </td>\n",
    "    </tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, after creating your model in Keras, you need to compile it and define what loss, optimizer and metrics you want to use. Compile your model using `categorical_crossentropy` loss, a custom [Adam](https://keras.io/optimizers/#adam) [optimizer](https://keras.io/optimizers/#usage-of-optimizers) (`learning rate = 0.005`, $\\beta_1 = 0.9$, $\\beta_2 = 0.999$, `decay = 0.01`)  and `['accuracy']` metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/msatidas/Library/Python/3.9/lib/python/site-packages/keras/src/optimizers/base_optimizer.py:86: UserWarning: Argument `decay` is no longer supported and will be ignored.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "### START CODE HERE ### (≈2 lines)\n",
    "opt = Adam(learning_rate=0.005, beta_1=0.9, beta_2=0.999,decay=0.01) \n",
    "model.compile(loss='categorical_crossentropy', optimizer=opt,metrics=['accuracy']*10)\n",
    "### END CODE HERE ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last step is to define all your inputs and outputs to fit the model:\n",
    "- You already have X of shape $(m = 10000, T_x = 30)$ containing the training examples.\n",
    "- You need to create `s0` and `c0` to initialize your `post_attention_LSTM_cell` with 0s.\n",
    "- Given the `model()` you coded, you need the \"outputs\" to be a list of 11 elements of shape (m, T_y). So that: `outputs[i][0], ..., outputs[i][Ty]` represent the true labels (characters) corresponding to the $i^{th}$ training example (`X[i]`). More generally, `outputs[i][j]` is the true label of the $j^{th}$ character in the $i^{th}$ training example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s0 = np.zeros((m, n_s))\n",
    "c0 = np.zeros((m, n_s))\n",
    "outputs = list(Yoh.swapaxes(0,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now fit the model and run it for one epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "\u001b[1m100/100\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 40ms/step - dense_2_accuracy: 1.0000 - dense_2_accuracy_1: 1.0000 - dense_2_accuracy_2: 0.9998 - dense_2_accuracy_3: 0.9981 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 1.0000 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 1.0000 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 0.0010 - loss: 0.0244\n",
      "Epoch 2/10\n",
      "\u001b[1m100/100\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 40ms/step - dense_2_accuracy: 0.9999 - dense_2_accuracy_1: 0.9999 - dense_2_accuracy_2: 0.9995 - dense_2_accuracy_3: 0.9966 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 1.0000 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 1.0000 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 9.0653e-04 - loss: 0.0315\n",
      "Epoch 3/10\n",
      "\u001b[1m100/100\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 40ms/step - dense_2_accuracy: 0.9998 - dense_2_accuracy_1: 0.9999 - dense_2_accuracy_2: 0.9988 - dense_2_accuracy_3: 0.9973 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 0.9997 - dense_2_accuracy_6: 0.9996 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 0.9975 - dense_2_accuracy_9: 0.9990 - dense_2_loss: 0.0038 - loss: 0.0394\n",
      "Epoch 4/10\n",
      "\u001b[1m100/100\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 41ms/step - dense_2_accuracy: 0.9968 - dense_2_accuracy_1: 0.9975 - dense_2_accuracy_2: 0.9781 - dense_2_accuracy_3: 0.9817 - dense_2_accuracy_4: 0.9997 - dense_2_accuracy_5: 0.9896 - dense_2_accuracy_6: 0.9882 - dense_2_accuracy_7: 0.9997 - dense_2_accuracy_8: 0.9610 - dense_2_accuracy_9: 0.9928 - dense_2_loss: 0.0390 - loss: 0.4622\n",
      "Epoch 5/10\n",
      "\u001b[1m100/100\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 41ms/step - dense_2_accuracy: 0.9997 - dense_2_accuracy_1: 0.9997 - dense_2_accuracy_2: 0.9995 - dense_2_accuracy_3: 0.9979 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 0.9998 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 0.9999 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 0.0014 - loss: 0.0414\n",
      "Epoch 6/10\n",
      "\u001b[1m100/100\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 41ms/step - dense_2_accuracy: 0.9999 - dense_2_accuracy_1: 0.9999 - dense_2_accuracy_2: 0.9998 - dense_2_accuracy_3: 0.9979 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 0.9999 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 0.9998 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 8.5868e-04 - loss: 0.0283\n",
      "Epoch 7/10\n",
      "\u001b[1m100/100\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 44ms/step - dense_2_accuracy: 0.9998 - dense_2_accuracy_1: 0.9998 - dense_2_accuracy_2: 0.9998 - dense_2_accuracy_3: 0.9974 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 1.0000 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 1.0000 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 6.9754e-04 - loss: 0.0281\n",
      "Epoch 8/10\n",
      "\u001b[1m100/100\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 42ms/step - dense_2_accuracy: 0.9999 - dense_2_accuracy_1: 0.9999 - dense_2_accuracy_2: 0.9997 - dense_2_accuracy_3: 0.9979 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 1.0000 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 1.0000 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 5.5270e-04 - loss: 0.0207\n",
      "Epoch 9/10\n",
      "\u001b[1m100/100\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 43ms/step - dense_2_accuracy: 0.9999 - dense_2_accuracy_1: 0.9999 - dense_2_accuracy_2: 0.9999 - dense_2_accuracy_3: 0.9973 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 1.0000 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 1.0000 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 4.4554e-04 - loss: 0.0223\n",
      "Epoch 10/10\n",
      "\u001b[1m100/100\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 44ms/step - dense_2_accuracy: 1.0000 - dense_2_accuracy_1: 1.0000 - dense_2_accuracy_2: 0.9999 - dense_2_accuracy_3: 0.9979 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 1.0000 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 1.0000 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 3.9239e-04 - loss: 0.0164\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.history.History at 0x36b84e6a0>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit([Xoh, s0, c0], outputs, epochs=10, batch_size=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save_weights(\".weights.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While training you can see the loss as well as the accuracy on each of the 10 positions of the output. The table below gives you an example of what the accuracies could be if the batch had 2 examples: \n",
    "\n",
    "<img src=\"images/table.png\" style=\"width:700;height:200px;\"> <br>\n",
    "<caption><center>Thus, `dense_2_acc_8: 0.89` means that you are predicting the 7th character of the output correctly 89% of the time in the current batch of data. </center></caption>\n",
    "\n",
    "\n",
    "We have run this model for longer, and saved the weights. Run the next cell to load our weights. (By training a model for several minutes, you should be able to obtain a model of similar accuracy, but loading our model will save you time.) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.load_weights('.weights.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"functional\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"functional\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)        </span>┃<span style=\"font-weight: bold\"> Output Shape      </span>┃<span style=\"font-weight: bold\">    Param # </span>┃<span style=\"font-weight: bold\"> Connected to      </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩\n",
       "│ input_layer_2       │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">37</span>)    │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                 │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │                   │            │                   │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ s0 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)        │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                 │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ bidirectional_2     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)    │     <span style=\"color: #00af00; text-decoration-color: #00af00\">17,920</span> │ input_layer_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]… │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Bidirectional</span>)     │                   │            │                   │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ repeat_vector       │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)    │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ s0[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],         │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">RepeatVector</span>)      │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]        │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ concatenate         │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">128</span>)   │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Concatenate</span>)       │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>]… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ repeat_vector[<span style=\"color: #00af00; text-decoration-color: #00af00\">9</span>]… │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dense (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)       │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">10</span>)    │      <span style=\"color: #00af00; text-decoration-color: #00af00\">1,290</span> │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
       "│                     │                   │            │ concatenate[<span style=\"color: #00af00; text-decoration-color: #00af00\">9</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>] │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dense_1 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>)     │         <span style=\"color: #00af00; text-decoration-color: #00af00\">11</span> │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],      │\n",
       "│                     │                   │            │ dense[<span style=\"color: #00af00; text-decoration-color: #00af00\">9</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ attention_weights   │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>)     │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Activation</span>)        │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],    │\n",
       "│                     │                   │            │ dense_1[<span style=\"color: #00af00; text-decoration-color: #00af00\">9</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]     │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dot (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dot</span>)           │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)     │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ c0 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)        │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                 │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ lstm (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">LSTM</span>)         │ [(<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>),      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">33,024</span> │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>),       │            │ s0[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],         │\n",
       "│                     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)]       │            │ c0[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],         │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>],       │\n",
       "│                     │                   │            │ dot[<span style=\"color: #00af00; text-decoration-color: #00af00\">9</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>]        │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dense_2 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">11</span>)        │        <span style=\"color: #00af00; text-decoration-color: #00af00\">715</span> │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">2</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">4</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">5</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">7</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
       "│                     │                   │            │ lstm[<span style=\"color: #00af00; text-decoration-color: #00af00\">9</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]        │\n",
       "└─────────────────────┴───────────────────┴────────────┴───────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)       \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape     \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m   Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to     \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩\n",
       "│ input_layer_2       │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m37\u001b[0m)    │          \u001b[38;5;34m0\u001b[0m │ -                 │\n",
       "│ (\u001b[38;5;33mInputLayer\u001b[0m)        │                   │            │                   │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ s0 (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m64\u001b[0m)        │          \u001b[38;5;34m0\u001b[0m │ -                 │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ bidirectional_2     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m64\u001b[0m)    │     \u001b[38;5;34m17,920\u001b[0m │ input_layer_2[\u001b[38;5;34m0\u001b[0m]… │\n",
       "│ (\u001b[38;5;33mBidirectional\u001b[0m)     │                   │            │                   │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ repeat_vector       │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m64\u001b[0m)    │          \u001b[38;5;34m0\u001b[0m │ s0[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],         │\n",
       "│ (\u001b[38;5;33mRepeatVector\u001b[0m)      │                   │            │ lstm[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m]        │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ concatenate         │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m128\u001b[0m)   │          \u001b[38;5;34m0\u001b[0m │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│ (\u001b[38;5;33mConcatenate\u001b[0m)       │                   │            │ repeat_vector[\u001b[38;5;34m0\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m1\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m2\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m3\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m4\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m5\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m6\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m7\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m8\u001b[0m]… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ repeat_vector[\u001b[38;5;34m9\u001b[0m]… │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dense (\u001b[38;5;33mDense\u001b[0m)       │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m10\u001b[0m)    │      \u001b[38;5;34m1,290\u001b[0m │ concatenate[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m… │\n",
       "│                     │                   │            │ concatenate[\u001b[38;5;34m9\u001b[0m][\u001b[38;5;34m0\u001b[0m] │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dense_1 (\u001b[38;5;33mDense\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m1\u001b[0m)     │         \u001b[38;5;34m11\u001b[0m │ dense[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m],      │\n",
       "│                     │                   │            │ dense[\u001b[38;5;34m9\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ attention_weights   │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m30\u001b[0m, \u001b[38;5;34m1\u001b[0m)     │          \u001b[38;5;34m0\u001b[0m │ dense_1[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│ (\u001b[38;5;33mActivation\u001b[0m)        │                   │            │ dense_1[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m],    │\n",
       "│                     │                   │            │ dense_1[\u001b[38;5;34m9\u001b[0m][\u001b[38;5;34m0\u001b[0m]     │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dot (\u001b[38;5;33mDot\u001b[0m)           │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m1\u001b[0m, \u001b[38;5;34m64\u001b[0m)     │          \u001b[38;5;34m0\u001b[0m │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "│                     │                   │            │ attention_weight… │\n",
       "│                     │                   │            │ bidirectional_2[\u001b[38;5;34m…\u001b[0m │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ c0 (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m64\u001b[0m)        │          \u001b[38;5;34m0\u001b[0m │ -                 │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ lstm (\u001b[38;5;33mLSTM\u001b[0m)         │ [(\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m64\u001b[0m),      │     \u001b[38;5;34m33,024\u001b[0m │ dot[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m64\u001b[0m),       │            │ s0[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],         │\n",
       "│                     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m64\u001b[0m)]       │            │ c0[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],         │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m2\u001b[0m],       │\n",
       "│                     │                   │            │ dot[\u001b[38;5;34m9\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m2\u001b[0m]        │\n",
       "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
       "│ dense_2 (\u001b[38;5;33mDense\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m11\u001b[0m)        │        \u001b[38;5;34m715\u001b[0m │ lstm[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m2\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m3\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m4\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m5\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m6\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m7\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m8\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
       "│                     │                   │            │ lstm[\u001b[38;5;34m9\u001b[0m][\u001b[38;5;34m0\u001b[0m]        │\n",
       "└─────────────────────┴───────────────────┴────────────┴───────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">158,882</span> (620.64 KB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m158,882\u001b[0m (620.64 KB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">52,960</span> (206.88 KB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m52,960\u001b[0m (206.88 KB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Optimizer params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">105,922</span> (413.76 KB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Optimizer params: \u001b[0m\u001b[38;5;34m105,922\u001b[0m (413.76 KB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 (30, 37) <class 'numpy.ndarray'> ('29 nov 1992', '1992-11-29')\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
      "['1', '9', '9', '2', '-', '1', '1', '-', '2', '9']\n",
      "1 (30, 37) <class 'numpy.ndarray'> ('24.07.70', '1970-07-24')\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 25ms/step\n",
      "['1', '9', '7', '0', '-', '0', '7', '-', '2', '4']\n",
      "2 (30, 37) <class 'numpy.ndarray'> ('5/19/15', '2015-05-19')\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 24ms/step\n",
      "['2', '0', '1', '5', '-', '0', '5', '-', '1', '9']\n",
      "3 (30, 37) <class 'numpy.ndarray'> ('wednesday june 4 1986', '1986-06-04')\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
      "['1', '9', '8', '6', '-', '0', '6', '-', '0', '4']\n",
      "4 (30, 37) <class 'numpy.ndarray'> ('thursday april 5 1990', '1990-04-05')\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 23ms/step\n",
      "['1', '9', '9', '0', '-', '0', '4', '-', '0', '5']\n",
      "5 (30, 37) <class 'numpy.ndarray'> ('monday august 25 1980', '1980-08-25')\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/s5/8nm2j6k14rzf8q74p7p16zch001k3l/T/ipykernel_60262/1108836454.py:11: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)\n",
      "  output = [inv_machine_vocab[int(i)] for i in prediction]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
      "['1', '9', '8', '0', '-', '0', '8', '-', '2', '5']\n",
      "6 (30, 37) <class 'numpy.ndarray'> ('thursday february 15 2001', '2001-02-15')\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
      "['2', '0', '0', '1', '-', '0', '2', '-', '1', '5']\n",
      "7 (30, 37) <class 'numpy.ndarray'> ('22 nov 1978', '1978-11-22')\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
      "['1', '9', '7', '8', '-', '1', '1', '-', '2', '2']\n",
      "8 (30, 37) <class 'numpy.ndarray'> ('31 oct 1976', '1976-10-31')\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
      "['1', '9', '7', '6', '-', '1', '0', '-', '3', '1']\n",
      "9 (30, 37) <class 'numpy.ndarray'> ('friday october 22 1993', '1993-10-22')\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
      "['1', '9', '9', '3', '-', '1', '0', '-', '2', '2']\n",
      "10 (30, 37) <class 'numpy.ndarray'> ('tuesday november 28 2000', '2000-11-28')\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
      "['2', '0', '0', '0', '-', '1', '1', '-', '2', '8']\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "for index, x in enumerate(Xoh):\n",
    "    if i > 10:\n",
    "        continue\n",
    "    i = i + 1\n",
    "    print(index, x.shape, type(x), dataset[index])\n",
    "    s0 = np.zeros((1, n_s))\n",
    "    c0 = np.zeros((1, n_s))\n",
    "    prediction = model.predict([x.reshape(1, Tx, len(human_vocab)), s0, c0])\n",
    "    prediction = np.argmax(prediction, axis = -1)\n",
    "    output = [inv_machine_vocab[int(i)] for i in prediction]\n",
    "    print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now see the results on new examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(30, 37)\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 26ms/step\n",
      "source: 22rd maY 1994\n",
      "output: 1994-05-22\n",
      "(30, 37)\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
      "source: 5 April 09\n",
      "output: 2009-04-05\n",
      "(30, 37)\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
      "source: 21th of August 2016\n",
      "output: 2016-08-21\n",
      "(30, 37)\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
      "source: Tue 10 Jul 2007\n",
      "output: 2007-07-10\n",
      "(30, 37)\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 23ms/step\n",
      "source: Saturday May 9 2018\n",
      "output: 2018-05-09\n",
      "(30, 37)\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 13ms/step"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/s5/8nm2j6k14rzf8q74p7p16zch001k3l/T/ipykernel_60262/3302219116.py:15: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)\n",
      "  output = [inv_machine_vocab[int(i)] for i in prediction]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
      "source: March 3 2001\n",
      "output: 2001-03-03\n",
      "(30, 37)\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 21ms/step\n",
      "source: March 3rd 2001\n",
      "output: 2001-03-03\n",
      "(30, 37)\n",
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 23ms/step\n",
      "source: 1 March 2001\n",
      "output: 2001-03-01\n"
     ]
    }
   ],
   "source": [
    "EXAMPLES = ['22rd maY 1994', '5 April 09', '21th of August 2016', 'Tue 10 Jul 2007', 'Saturday May 9 2018', 'March 3 2001', 'March 3rd 2001', '1 March 2001']\n",
    "for example in EXAMPLES:\n",
    "    \n",
    "    source = string_to_int(example, Tx, human_vocab)\n",
    "    source = np.array(list(map(lambda x: to_categorical(x, num_classes=len(human_vocab)), source)))\n",
    "    \n",
    "    print(source.shape)\n",
    "    \n",
    "    s0 = np.zeros((1, n_s))\n",
    "    c0 = np.zeros((1, n_s))\n",
    "    prediction = model.predict([source.reshape(1, Tx, len(human_vocab)), s0, c0])\n",
    "    \n",
    "    prediction = np.argmax(prediction, axis = -1)\n",
    "    \n",
    "    output = [inv_machine_vocab[int(i)] for i in prediction]\n",
    "    \n",
    "    print(\"source:\", example)\n",
    "    print(\"output:\", ''.join(output))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also change these examples to test with your own examples. The next part will give you a better sense on what the attention mechanism is doing--i.e., what part of the input the network is paying attention to when generating a particular output character. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 - Visualizing Attention (Optional / Ungraded)\n",
    "\n",
    "Since the problem has a fixed output length of 10, it is also possible to carry out this task using 10 different softmax units to generate the 10 characters of the output. But one advantage of the attention model is that each part of the output (say the month) knows it needs to depend only on a small part of the input (the characters in the input giving the month). We can  visualize what part of the output is looking at what part of the input.\n",
    "\n",
    "Consider the task of translating \"Saturday 9 May 2018\" to \"2018-05-09\". If we visualize the computed $\\alpha^{\\langle t, t' \\rangle}$ we get this: \n",
    "\n",
    "<img src=\"images/date_attention.png\" style=\"width:600;height:300px;\"> <br>\n",
    "<caption><center> **Figure 8**: Full Attention Map</center></caption>\n",
    "\n",
    "Notice how the output ignores the \"Saturday\" portion of the input. None of the output timesteps are paying much attention to that portion of the input. We see also that 9 has been translated as 09 and May has been correctly translated into 05, with the output paying attention to the parts of the input it needs to to make the translation. The year mostly requires it to pay attention to the input's \"18\" in order to generate \"2018.\" \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 - Getting the activations from the network\n",
    "\n",
    "Lets now visualize the attention values in your network. We'll propagate an example through the network, then visualize the values of $\\alpha^{\\langle t, t' \\rangle}$. \n",
    "\n",
    "To figure out where the attention values are located, let's start by printing a summary of the model ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Navigate through the output of `model.summary()` above. You can see that the layer named `attention_weights` outputs the `alphas` of shape (m, 30, 1) before `dot_2` computes the context vector for every time step $t = 0, \\ldots, T_y-1$. Lets get the activations from this layer.\n",
    "\n",
    "The function `attention_map()` pulls out the attention values from your model and plots them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 21ms/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqYAAAJICAYAAACzNSZtAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB9iElEQVR4nO3dd3gU1foH8O9sTSEFDCVAIECAhC5SDB1EuaCI5QoKl6bipamIoiAiAcTLRRG9ilhQQX4g2OB6FUFEiTQRkABC6EFa6JCeTXb3/P4IWbNpcybsspPk+3kenmVnzrzzTtu8O7t7jiKEECAiIiIi8jGDrxMgIiIiIgJYmBIRERGRTrAwJSIiIiJdYGFKRERERLrAwpSIiIiIdIGFKRERERHpAgtTIiIiItIFk68TuBFOpxNnz55FUFAQFEXxdTpEREREVIgQAmlpaahduzYMhtLviZbrwvTs2bOIiIjwdRpEREREpOLUqVOoW7duqW3KdWEaFBQEADiadApBwcEltrPn5iL+px/QvdddMJnNHlm3lpi5dqdcTHsutsb/iE7de8NkKj2m2aT+LQxfb7c3Yp66nKkaz2G341jCJjRq0xVGk/opHnFLgEdzlB1MzZ6bi19+Xo9uPe9UjZmSmasaz+HIRcK2jWgT2wNGo/qxCbSq7xst52SuQ/4837H5J7Tv0ks15pqDyeoBhQNVz+/H1ZrNAcWo2vy+FnWkctyy8Ud07qG+3Saj3DeitJxDiWdSVeM5HXacS9yGWjGxMBjVj+Xkb/artrEoToxucA3vJYUiR5S+XbtXfqUaDwD8LQa8+2x3jJ0Xj6yc0s+RzV/OUI3ncNhxPGEzGrbpAqPEdtcLC5TKk4i8Jy01FVENIlx1W2nKdWGa//F9UHAwglUK04CAAAQHB3u0mJKNqaUwdcX0UGHqy+32RswqOeqnrMNuR0BAAKoEBUsVpsHBcoWpbI5aClPZmE6TRGF6/fwJCgqGUeX8AYAqkoWp7DmZo6EwDQgIQJBETP8q6eoBnQ4EBAQgOzAIMKgXpqW9VhTOUWa7tRSm0ue5el0Kp+Ov81ymMDX5qRdoJsWJgIAcmPwC4VQpTBWjRT1JAIrJiICAACgmKxSHo9S2VYLUj42jwHbLFKbBwSxMifRC5muX/PETEREREekCC1MiIiIi0gUWpkRERESkCyxMiYiIiEgXfFqY/vLLL+jfvz9q164NRVGwevVqX6ZDRERERD7k08I0IyMDrVu3xoIFC3yZBhERERHpgE+7i+rbty/69u3ryxSIiIiISCfKVT+mNpsNNpvN9Tw1Na+zP3tuLuy5JffzaLfnuj16gpaYdun+He1uj6VRVPoYzIvj2+32RkyHxL5xOOxuj6rrL+XccbXRkKN0P6aatlt+38jsIwCwG9Xz1HJOeuM8h7P0fi/d2si0heQ1qylHyX5MNRxvp8S5m99Gpi2Q13m+ehtR4LH09v5W9T5jAcDPYnR7LI3MNat1u2WubyLyLi3XoSJk/4p6maIoWLVqFe67774S28TFxWHGjKIjgyxfvhwBAeqdpBMRERHRzZWZmYnBgwcjJSVFdZCTclWYFnfHNCIiAmfOXSp95Cd7LuJ/Wo/uve5UHcVFlpaY8kM12gsM/1j6zWyzxGgzvt5ub8Q8eTlLNV7ekIWb0LBNV7khC2/x92iOWu6Y/vLzj+jWU33Yy5QsuTumCb9uRJvbe0iNeCU3JKn8OSk/8pMdOzf/hHZdeqnGXHPwnHpApwPVLuzHlRrNpUZ+ur9Fbakc/xqStPQcTQb5O6ay51Di2TTVeJqHJP3vH6ptLIrA6IbX8N7xUOSI0kdo2f253JCkfhYjFj7bHWPmxSM7p/S72pu+nKkaz1lgSFKZ7a4vMeQwEXlXamoq6tQKkypMy9VH+VarFVartch0k9ksNTymySTXTguZmELiIzT3mCb1YRAlhiT9q61vttsbMY0SQ3O62hpNUgWalu2QOt4a3+tJbbeGTyONJpPUkKRqBVfhtmrnpNML57lMoenWVqK9ljdUUtei5JCkBdevdrxlCq6CbWXa50h8/Sf/4/scoai2z7LJfXUiX3aOQ3UZmTeS+QxGk1R7T79OEZF2Wq5D9mNKRERERLrg0zum6enpOHr0qOt5UlISEhISUK1aNdSrV8+HmRERERHRzebTwnTnzp3o2bOn6/nEiRMBAMOHD8fixYt9lBURERER+YJPC9MePXpo/j4eEREREVVM/I4pEREREekCC1MiIiIi0gUWpkRERESkC+WqH9PyyizZ52j+MKNmo0FTP6WVSf0w9c6y7bm5OIK8jvN90YehopTeMXnhdoqiqC5TNdCiGs+emxcjNMDise3Wck7Knud2Q973ygMtJpjMpb8EPXKreu8c9txcbDi3Fw+1quu54319mFGTwaC5n1JPaF639A6ogbztPvsHEFM7SGq7/96hjmobRTiAq1cxoF1tCKX0PmF3/p9qOM0iqweqtrHn5uIo8jrOZx+lRBUPqx8iIiIi0gUWpkRERESkCyxMiYiIiEgXWJgSERERkS6wMCUiIiIiXfBpYZqWloYJEyagfv368Pf3R6dOnbBjxw5fpkREREREPuLTwvTxxx/H+vXrsXTpUuzbtw933XUXevfujTNnzvgyLSIiIiLyAZ8VpllZWfjqq68wd+5cdOvWDVFRUYiLi0NUVBQWLlzoq7SIiIiIyEd81sG+3W6Hw+GAn5+f23R/f39s3ry52GVsNhtsNpvreWpqal6s3FzYc3NLWVeu26MnlIeY5SFHb8QsDzl6I2Z5yNEbMctDjt6IqTWeIhzSbWTa+ltL74A/n5/F6PZYmtJex11tvHBsiMi7ZK7tfIoQQngxl1J16tQJFosFy5cvR82aNfHZZ59h+PDhiIqKwqFDh4q0j4uLw4wZM4pMX758OQIC1EcEIiIiIqKbKzMzE4MHD0ZKSgqCg0sf2c6nhemxY8fw6KOP4pdffoHRaETbtm3RpEkT7Nq1C4mJiUXaF3fHNCIiAmfOXSp1Q+32XMT/tB7de90Jk8kzQ9iVh5jlIUdvxCwPOXojZnnI0Rsxy0OO3oipNd7HO/5UbaMIB+pdO4iTodGqQ5LGPf+2VJ5+FiMWPtsdY+bFIzun9DuxJ39+XTWeN44NEXlXamoq6tQKkypMffZRPgA0atQI8fHxyMjIQGpqKsLDwzFo0CA0bNiw2PZWqxVWq7XIdJPZLDVmsskk106L8hCzPOTojZjlIUdvxCwPOXojZnnI0RsxZeOpFZqF26q1z7Kpf9xfUHaOQ3UZLfvFG8eGiLxDy7Wqi35MAwMDER4ejqtXr2LdunUYMGCAr1MiIiIiopvMp3dM161bByEEmjZtiqNHj2LSpEmIjo7GyJEjfZkWEREREfmAT++YpqSkYNy4cYiOjsawYcPQpUsXrFu3DmZ+PENERERU6fj0junAgQMxcOBAX6ZARERERDqhi++YEhERERGxMCUiIiIiXWBhSkRERES64NPvmBIR0c3x7rdFR9MrzGoUmNURWLT2CGwO5SZk5W7813+otjHBib5W4Ln/JcIucW/l4Za1PJGaS8cG1TwaDwDMJt4jIsrHq4GIiIiIdIGFKRERERHpAgtTIiIiItKFMhemOTk5OHToEOx2uyfzISIiIqJKSnNhmpmZicceewwBAQFo3rw5Tp48CQB48sknMWfOHI8nSERERESVg+bCdMqUKdizZw82btwIPz8/1/TevXtj5cqVHk2OiIiIiCoPzYXp6tWr8c4776BLly5QlL+6E2nevDmOHTumKVZaWhomTJiA+vXrw9/fH506dcKOHTu0pkREREREFYDmwvTixYuoUaNGkekZGRluhaqMxx9/HOvXr8fSpUuxb98+3HXXXejduzfOnDmjNS0iIiIiKuc0F6bt2rXDd99953qeX4wuWrQIsbGx0nGysrLw1VdfYe7cuejWrRuioqIQFxeHqKgoLFy4UGtaRERERFTOaR756dVXX0Xfvn1x4MAB2O12vPXWWzhw4AC2bt2K+Ph46Th2ux0Oh8Pte6oA4O/vj82bNxe7jM1mg81mcz1PTU3Ni5WbC3tubinrynV79ITyELM85OiNmOUhR2/ELA85eiNmecjRGzG1xrMahXQbmbb+VqPUev0sRrfH0pjglG4j0xYAhMOzPcd48pzIpwj23EgVW2k1WmGKEEL9FaiQY8eOYc6cOdizZw/S09PRtm1bvPDCC2jZsqWmOJ06dYLFYsHy5ctRs2ZNfPbZZxg+fDiioqJw6FDR4fPi4uIwY8aMItOXL1+OgIAArZtBRERERF6WmZmJwYMHIyUlBcHBwaW2LVNh6inHjh3Do48+il9++QVGoxFt27ZFkyZNsGvXLiQmJhZpX9wd04iICJw5d6nUDbXbcxH/03p073UnTCazR3IvDzHLQ47eiFkecvRGzPKQozdiloccvRFTa7xOs35UbWM1CrzULhev7DTD5ij9NwOnf1orlaefxYiFz3bHmHnxyM5xlNp24MTHVOOZ4MSd1j+x3lYfdolvoz3YrOhvIm5Eu8iqHo0HAGYj75hSxZaamoo6tcKkClPNH+WvWbMGRqMRffr0cZu+bt06OJ1O9O3bVzpWo0aNEB8fj4yMDKSmpiI8PByDBg1Cw4YNi21vtVphtVqLTDeZzTCZ1V+YTSa5dlqUh5jlIUdvxCwPOXojZnnI0Rsxy0OO3ogpG0+t0CzcVq19lq30IrOw7ByH6jIyhWbBtjLtFaPmP3Ol8tQbFfeYLEypYtPymqf5apg8eTIcjqIvLkIITJ48WWs4AEBgYCDCw8Nx9epVrFu3DgMGDChTHCIiIiIqvzS/lTxy5AiaNWtWZHp0dDSOHj2qKda6desghEDTpk1x9OhRTJo0CdHR0Rg5cqTWtIiIiIionNN8xzQkJATHjx8vMv3o0aMIDAzUFCslJQXjxo1DdHQ0hg0bhi5dumDdunUwe/ijMyIiIiLSP813TAcMGIAJEyZg1apVaNSoEYC8ovTZZ5/FvffeqynWwIEDMXDgQK0pEBEREVEFpPmO6dy5cxEYGIjo6Gg0aNAADRo0QExMDG655Ra8/vrr3siRiIiIiCoBzXdMQ0JCsHXrVqxfvx579uyBv78/WrVqhW7dunkjPyIiIiKqJMrUj4aiKLjrrrtw1113eTofIiIiIqqkylSYbtiwARs2bMCFCxfgdLoPC/fxxx97JLGKJNcuN3Se3ZHXLtfhhFBKX8ZcSfu9k9mXWvYjANid6mNM2O15XaRl5TpgUhk+MCVTbug1hz1vqMTzKTYYTdr6hPREPKtZ/RxyXB9+8VpmDoym0veTbCfh9ut5ptvsMKn0lWnLVd+O/O2+kp4Do0niWlPU+/PM3+4rGerbfUsVi/o6NbqSnqPaJj/HqxI5AsDaST0kYtqxf/sGfP1UVxhNpf95OPNYR9V4AOB02HH18Fas/uQlGFT6FL1z0DTVeP5WI/pO6YXP3/hIqi/V3+9/QCpPWcN7NvBoPABoVyvU4zHDQ/3UG2lQPahoH+I3qrL+HaPSaS5MZ8yYgZkzZ6Jdu3YIDw+HIvEiT0RERESkRnNh+t5772Hx4sUYOnSoN/IhIiIiokpK8330nJwcdOrUyRu5EBEREVElprkwffzxx7F8+XJv5EJERERElZjmj/Kzs7PxwQcf4Mcff0SrVq2KjNL0xhtveCw5IiIiIqo8NBeme/fuRZs2bQAAf/zxh9s8/hCKiIiIiMpKc2H6888/e2zlCxcuxMKFC3HixAkAQPPmzfHyyy+jb9++HlsHEREREZUPZe5E7OjRo1i3bh2ysrIAAEKo96NXWN26dTFnzhzs2rULO3fuRK9evTBgwADs37+/rGkRERERUTmluTC9fPky7rjjDjRp0gT9+vVDcnIyAOCxxx7Ds88+qylW//790a9fPzRu3BhNmjTB7NmzUaVKFfz6669a0yIiIiKick7zR/nPPPMMzGYzTp48iZiYGNf0QYMGYeLEiZg3b16ZEnE4HPjiiy+QkZGB2NjYYtvYbDbYbDbX89TUVACAPTcX9tySR9uxXx8hJf/RE7TEzB+JSD2m3e2xNIrK6EN5cXy73d6IKbMvtexHQHbkJ/mYDsn15reTbe/peA6pEZDkY8qck4DWfal+vDXvR4mvwmuJac+V+269lvPcIdVG4/G2q5/nWmI6HXLrFdfbCYcdakfT32pUjednMbo9qrEYtH+aVxpF3PgobYXJ7kstPPW6ks9u9/woTbKvGVT+lVajFaYIjZ/B16pVC+vWrUPr1q0RFBSEPXv2oGHDhjh+/DhatWqF9PR0Tcnu27cPsbGxyM7ORpUqVbB8+XL069ev2LZxcXGYMWNGkenLly9HQECApvUSERERkfdlZmZi8ODBSElJQXBwcKltNd8xzcjIKLYIvHLlCqxW7WPpNm3aFAkJCUhJScGXX36J4cOHIz4+Hs2aNSvSdsqUKZg4caLreWpqKiIiItC9112lbqjdnov4n9aje687YTKZS2ynhZaYuRrumG6N/xGduveGSWWcaplxyX293d6IKbMvtexHQP6O6W+bNqBD1ztUY6Zmyt8xTdwVj5jbuquOS+6NeFaz3B3ThF83os3tPVRjmiTOSSBvX+7a8hNu69xLdV/m5MrdMd3720a06qCeIwDpO6Z7t29Eq47qMasFWtQDQtt5fjUjRyrHPds3orVEjgBgk7xjenBXPKIlzqGzV7NU4wF5d0qvHfsNoY06QDGWHvO+ka+oxvOzGLHw2e4YMy8e2Tnqdy+b3HOvVJ6yBner79F4AHBrjRCPx6wV4ufReGFBcue5FjJ/x6hiyP+EW4bmv4Zdu3bFp59+ilmzZgHI6yLK6XRi7ty56Nmzp9ZwsFgsiIqKAgDcdttt2LFjB9566y28//77RdpardZii1+T2QyTWb1IMpnk2mkhE1MocoXpXzFNqn+4TCb5C9pX2+2NmFr2pcx+BABIFKZaYqr87S2mvckjhanWeEYN51BeTJVzUuMfGZl96dDwsan0ftTQrZ3Udmu8DmTOc6NJ/pyUyREAjKofpBeOWfq+NEie6PlrVYwm1WWybPLHOzvHIdU+x+nZbgyFIvcVAi1k96UWnnxNAeCxGxHuMVmYVhZaXic1n7lz587FHXfcgZ07dyInJwfPP/889u/fjytXrmDLli1awxXhdDrdvkdKRERERJWD5sK0RYsWOHz4MN555x0EBQUhPT0dDzzwAMaNG4fw8HBNsaZMmYK+ffuiXr16SEtLw/Lly7Fx40asW7dOa1pEREREVM6V6V5/SEgIpk6desMrv3DhAoYNG4bk5GSEhISgVatWWLduHe68884bjk1ERERE5YvmwvSXX34pdX63bt2kY3300UdaV09EREREFZTmwrRHjx5FpikFfkzgcHi+jzciIiIiqvg0/yTu6tWrbv8uXLiAtWvXon379vjhhx+8kSMRERERVQKa75iGhBTtb+3OO++ExWLBxIkTsWvXLo8kRkRERESVi+aRn0py8OBBtGvXTvPITzciNTUVISEhmPVdAvwCg0psZxAONEo5gGMhzeCU6IMuW6JTb4NwoHnmQewPiFaNaTLI9aNnEA40TU/EoSoxqjFD/dW3Q3E6UOvSHzgX1gLCoN5epqN5xelA3Sv7cbpac9WYqdlyX+swCAdiMg4iMVB9X564kq0azwQn+liSsC6nAewSHwpEhKp3RG0QDrTIOog//NVz3PXnNdV4AGBWnPhHWDL+71I4clWG5kvLUh/OzaI4MTriEt47FYYciaH+/CWGdDQrToyoeR6Lz9dUzTFVolP4/DyfbHgVbx+vqprn6dPqnTJbjQJzutgxebMJNof6tWY0qrexGgVmx+Zi6jazasx69UJV4wHajs+ZM3LbPbODDS//ZpXa7j/X/U+1jb/ViE+m9MLIf/2k3kdocHXVeADgbzHgk/GtMPKdvcjKKf21deRTD6nGM8KB3qbj+NHeEA6on8PT72wslacsDd3gysf0fEgYPJyoQfLvmBZmiWtRK639KVPxZAb50CItNRVN61X3zshPe/fudXsuhEBycjLmzJmDNm3aaA1HRERERASgDIVpmzZtoCgKCt9ovf322/Hxxx97LDEiIiIiqlw0F6ZJSUluzw0GA6pXrw4/P8+Oy0tERERElYvmwrR+/freyIOIiIiIKjnNhel//vMf6bZPPfWU1vBEREREVElpLkznz5+PixcvIjMzE6GhoQCAa9euISAgANWr//UrTUVRWJgSERERkTTN/SrMnj0bbdq0QWJiIq5cuYIrV64gMTERbdu2xSuvvIKkpCQkJSXh+PHj3siXiIiIiCoozYXptGnT8Pbbb6Np06auaU2bNsX8+fPx0ksveTQ5IiIiIqo8NH+Un5ycDLvdXmS6w+HA+fPnPZJUSWw2G2w2m+t5ampeJ9QG4YRBlNwZtHJ9niIcUpW4Qch1sF/wsfS28h3sy8ZU1FOE4nS4Paq3l+hgv8C+hEoOMttRsJ1Me5PaSgEYr7cxSrSVXa+WHM0yB6dAO5n2Fok2WuLltVM/Lz2dY14sUeCx9GWsRvVzMr+NTFsAMKr3ya4ppvx2y+9Lqe02CLdHNf5W9Q33uz7ogp/E4AuwyN3X8DMb3B5LY4T69aX1+rbb1Qen0IId7HuOIvm3URMnO9j3BIeHrxuHQz6e5pGf+vfvjzNnzmDRokVo27YtAGDXrl144oknUKdOHXzzzTfastUgLi4OM2bMKDJ9+fLlCAgI8Np6iYiIiKhsMjMzMXjwYKmRnzQXphcvXsTw4cOxdu1amM1mAIDdbkefPn2wePFi1KhRo0xJL1u2DP/85z9dz7///nt07drVrU1xd0wjIiIw+9vf4RdYpcTYinCgYeohHA9uCuHBIUljso4g0b+xR4ckbZxxGEcCm6jGDPGTG5K05pVEnK8W47khSYUDda4exJmq0ar7Mk1tSMPrDMKBpplHcChAfV+evGordT6Qdyelt+VP/JhTHzL3yOuEWKVybJZ9BAf81HNMOJWiGg/Iu3M26JbzWHlZfbjPdIkhSc2KE4/VvYKPTldTjQfI3RUzK04MqXERyy5UVx82VXIIO7MiMLrBNbyXFIpclTsmZ86mqcazGgVmxDowfZvRo0OSvtzBjpm/qQ9zGlE3RDUeoO34nE2W2G6DwNR2OZi90wKbU32bTm34XrWNn8WIhc92x5h58cjOUbl+g8JU4wF5d0oX/rMFxrz/h+pr65DR96nGM8KJnqYT+NkeKXV9T+7ZSCpPWbxj6jleGZLUwDumnnAt08NDkqalolmD2t4ZkrR69epYs2YNDh8+jIMHDwIAoqOj0aRJk7Jle929996Ljh07up7XqVOnSBur1QqrtWgR4VQMpRYL+aepUIyqRUVePPmLxSkRU0s82ZgyhWbBtjLtBSTeo1z/uyIU9ZgSfysLtVffbruGr0U7YJBqL3NOFGyr1l6mKCzcXm0ZtbHVC8eTaW/UGNNzOTqvx1RUl5EpNAu2lSpMNZQBMjG1HBtA7vho2m6n3HZnSb5RBIDsHId6e6vcR+mumLlOZOWUvowD8teiAwap9iaTWTqmDBamnmPyRmFqZGHqCUaTpnuW6vGM8teh5sI0X2RkJIQQaNSoEUymModxCQoKQlBQ0A3HISIiIqLySfNbi8zMTDz22GMICAhA8+bNcfLkSQDAk08+iTlz5ng8QSIiIiKqHDQXplOmTMGePXuwceNG+Pn5uab37t0bK1eu9GhyRERERFR5aP4MfvXq1Vi5ciVuv/12KAW+w9K8eXMcO3bMo8kRERERUeWh+Y7pxYsXi/3lfUZGhluhSkRERESkhebCtF27dvjuu+9cz/OL0UWLFiE2NtZzmRERERFRpaL5o/xXX30Vffv2xYEDB2C32/HWW2/hwIED2Lp1K+Lj472RIxERERFVApoL0y5duiAhIQFz5sxBy5Yt8cMPP6Bt27bYtm0bWrZs6Y0cVa345U8YrYElzrcaBKa2AVbE/ynVEXVKSpZqG6tRYG43YMnaw6p9CNaqJdcNlsUg0LQp8O2us8hRybNqiF+p84HrHXqHA9/tuyDVt2Z6pnon7hbFiacaAV/vOKvaF+O1a9mq8YC84zOtLfDlppOqx+dY4knVeP5mBX0eroaVX+9CVq56X2wxrRuotrEYBFq0AL7dcUb12Jglhl7Mj4kw4PzlTNWYaWnqAwtYDQKIAM6fT5c6z8+dvqzaxs8EjLg3AL9tT0J20ZGI3WRevKgaD8g7PmhYD79v3KN+fJIPq8ezGoEuvXB+0w9SfXUqjdrK5RhbDcl/nlfN8fxpue32MwGICMGu7cdU92WtCPWBSpzXh3Z1OgWcEl2KRv6tv2qbvKFQbajXu6/q61q3tnXVV4r8YYSP45En7lPtV7hpDX/VeIpwAFeBxtX9pQZMOZ8i9zoky2LyfD+ZZi/0vWn2cJ7e6Aw/V0N/vbIMkkMEy9I4BpGUHIfnY6ZJDMKixQWJvzlaZKSnS7ctUwekjRo1wocffliWRYmIiIiIiiVdmNrtdjgcDreRl86fP4/33nsPGRkZuPfee9GlSxevJElEREREFZ90YTpq1ChYLBa8//77AIC0tDS0b98e2dnZCA8Px/z58/Hf//4X/fr181qyRERERFRxSX8JZcuWLXjwwQddzz/99FM4HA4cOXIEe/bswcSJE/Haa695JUkiIiIiqvikC9MzZ86gcePGrucbNmzAgw8+iJCQEADA8OHDsX//fs9nSERERESVgnRh6ufnh6ysv36t/uuvv6Jjx45u89M1/OqKiIiIiKgg6cK0TZs2WLp0KQBg06ZNOH/+PHr16uWaf+zYMdSuXVvTyn/55Rf0798ftWvXhqIoWL16tabliYiIiKjikC5MX375Zbz11lto1KgR+vTpgxEjRiA8PNw1f9WqVejcubOmlWdkZKB169ZYsGCBpuWIiIiIqOKR/lV+9+7dsWvXLvzwww+oVasWHnroIbf5bdq0QYcOHTStvG/fvujbt6+mZYiIiIioYtLUwX5MTAxiYmKKnffEE094JKHS2Gw22Gx/jUaQmpoKALAqAkZDySMpWK7Ps5TSpqC80U/k2si0lV2vljzNEqNb5LeRaQvkjeqk3kYUeCy9vdUL2+1vVh8pxM9U8FG9vcx6NR0bL2y3zL7Uep77SVz97vuydELi2ACA3/V2fjLtreoj+/hZjG6PahQPn0NGyVdRLfvS069BAGCU2D3555nM+WZSuf5d673ezijRXhESI3ddbyPTFgCcDpVhtjRyKJ4fpckgMTKf5pjyH4jKxfP8YEUQBm+M/OTZmN4Y+cnhhZGfHB4+z4XH48ldrwCgCG/s9TJQFAWrVq3CfffdV2KbuLg4zJgxo8j05cuXIyAgwIvZEREREVFZZGZmYvDgwUhJSUFwcHCpbcs0JKmvTJkyBRMnTnQ9T01NRUREBF7fGwCjX2CJy1kMApNaZeK1vQGqY5IDQEpqlmobq1FgVmeBaVsU1TGla9YMUo2Xn+eTjVPx9pFg1TxDg62lzgfy7pQOq3UJn54LQ67EO/KMTPWxdi2KwOiG1/De8VDkiNJzTEmRG2vXYhB4oU0W/p3gr7rdSYdPq8bzMwHv/r0axn55RXVMcgBo0qK+VI4Tm6XjjQNVVHM0m+XuVGg53ukS4xZbDAJT2trwr9+tUuf5+bNXVNv4mYA3+gVg4ppM1X2ZdemSajwg707pwqERGLP0FLJVxqHH+aPq8SxGLHy2O8bMi0d2jsQdtwZt1GNqOIeMJrk7tX4m4O37QvDk6hTVmDXrVleNZzUKTGufi1k7zKqvQQBglBjr3GoQePG2HLy6ywKbyjnUuXUd1XhA3p3SOywnsCEnEg6Vu3hNa/qrxlOEA/WuHcTJ0GgIRX3fd6t3i1Sesjw9Bj0AmA1eiOnhPM1yp7kmhkp6xzTXC3dM02T+2GlwSeJvjhbm9DTptuWqMLVarW5DouazCQVGiT/EOU5F9cUWgNSLfB4Bm0O9MJUpEgq3V1tGptAs2FamfY5UTOf1topqe5l97bZ+ieOTpVbIAMj/6DXbLtdey/GROTbCC8dby76UPc+1vI5l29Xbyx2bAjFzhfoyNvmPf7JzHMiSaK94+BzS+iIqsy/lX4Mg9RoEAEaJr7W4YjrVY9o1flTsgEF1GZlCs2BbmfYG2e9aSDIaPV9EeiWmhwtTmTc2mmNW0sLUqXg+psxXdbRQjPKvvXLx5BP0/NVARERERFQGmgvThg0b4vLly0WmX7t2DQ0bNtQUKz09HQkJCUhISAAAJCUlISEhASdPntSaFhERERGVc5o/4zhx4gQcxfy6ymaz4cyZM5pi7dy5Ez179nQ9z//+6PDhw7F48WKtqRERERFROSZdmH7zzTeu/69btw4hISGu5w6HAxs2bEBkZKSmlffo0cMr398gIiIiovJHujDN78ZJURQMHz7cbZ7ZbEZkZCTmzZvn0eSIiIiIqPKQLkydzrxfYzdo0AA7duxAWFiY15IiIiIiospH83dMk5KSvJEHEREREVVymgvTmTNnljr/5ZdfLnMyZRUS4geTn1+J8/OG2sxASIhVqq/OkBD1zuvzYl5FRN0Q1Zi5uXL9geV3v6Yof/2/JKGBFtV4+UMGhgRYpPoclOn/LX9406BAi2rfqIH+ZtV4f8XMRHitKqoxr1yuqhrP73p3aaFhofCT2PWBgep55g/XGhhgglklR9lOo/Nj+lmNqkMSpnm4s2MACAwueVCKfH7Xh7sMDAqAUaVfy8xrKXIrzh8S1GwBcPO/Yy6unFNvYzEAqAZx9TxETulDaTrDasut9/pmC4cTTpXz8tRh9Z5J/M0KcHsNnDl6WqoP2dCa6h3N5w2XakLK1QzVvlarSAwXCwD5I6YGWgxwqPQ7em9MuGo8h92O/dv3o1+TWjCa1P+EVQ2Qex2SZZUcQEMLxcN9b5YXXukj1MMh7Q65oXe18XzMQKtn++ut6+F+cNNM6gP45NO8JatWrXJ7npubi6SkJJhMJjRq1MgnhSkRERERlX+aC9Pdu3cXmZaamooRI0bg/vvv90hSRERERFT5eORebXBwMGbMmIFp06Z5IhwRERERVUIe+xJBSkoKUlIkv2NGRERERFSI5o/y//Of/7g9F0IgOTkZS5cuRd++fT2WGBERERFVLpoL0/nz57s9NxgMqF69OoYPH44pU6Z4LDEiIiIiqlx00Y/pggUL8Nprr+HcuXNo3bo13n77bXTo0MHj6yEiIiIi/bqh75ieOnUKp06duqEEVq5ciYkTJ2L69On4/fff0bp1a/Tp0wcXLly4obhEREREVL5oLkztdjumTZuGkJAQREZGIjIyEiEhIXjppZeQmyvfgWq+N954A6NGjcLIkSPRrFkzvPfeewgICMDHH3+sORYRERERlV+aP8p/8skn8fXXX2Pu3LmIjY0FAGzbtg1xcXG4fPkyFi5cKB0rJycHu3btcvtuqsFgQO/evbFt27Yi7W02G2y2v0bASU1NBZA3go5JKXkkBbMiCjyqj7ggJEaj0RJTMcgNRWG53s4i0d4ksR35bWTaAn+N6iTTRqatQZHbbi0x/SQGm8kfkEZyYBrXCEyl0XK8ZUbQ0hrTKnFOaDl/ACDXqN7OfV+W3t7fLLfdftfb+cm0lziIfhaj26Mqi/r7cb/ro/v4SYzyY5R8FfUzuT+WxiHU942m/Si5Xi3XjlHIjWhnvH5uG+FUHejLYVcZbqpAG5m2ACDZTJrRc53ZuHDkJ8/x+MhPTs+P0mS3ez6mw+7ZDXd4eMQrp4YLUREaz4yQkBCsWLGiyC/w16xZg0ceeURTl1Fnz55FnTp1sHXrVleRCwDPP/884uPjsX37drf2cXFxmDFjRpE4y5cvR0BAgJbNICIiIqKbIDMzE4MHD0ZKSgqCg4NLbav5jqnVakVkZGSR6Q0aNIDFoj5++42YMmUKJk6c6HqempqKiIgIfPBnVZj8Sh7326wI/DPyKt4/URW5EnciZGp1syIwusE1vJcUqhozV/LdkcUgMD4qFe8cDUaOs/SY9WoGqcYzwYn7Qs5idUpt2CXe5adlqX8Vw6w48Y+aF/F/56urjmvvkHzralacGB5+CUuSw1RjHjigPs651Qj8q6cBU352wiZxU6dBQ/UxxLUcb4NB/o7pE/Wv4oM/1c/LS5cyVeNZDAJT2trwr9+tqucPAKSnZau2sRqBV7oBL/0C1X156bT6sQHy7vAtfLgmxqw4j2y1Md5PH1CPZzFi4bPdMWZePLJzJA54qPp47H5mAxY+3hRjFh1Cdm7p16+xWi31dSLvjuU7D1bF+K+uqo5D75D4WpSfWcHCQdUxZuVF9f0IIKRGNdU2ViPw7ztMeGGDXfV4P3RXtGo8IO9O6e04jl/REA6V16EnOtZXjeew23FwVzyib+sOo0n9T1hogGfHELd6eAxxgHdMPak83DHN8cIdU5vEa4AWuR6+Y5qelirdVvMVO378eMyaNQuffPIJrFYrgLyP2GfPno3x48drihUWFgaj0Yjz58+7TT9//jxq1Sr6Ym+1Wl3rLChHGOAstaDJ28G5QkGOSuEDyF4s8jFzNV4pOU5FtbCQKTQLtpVpr1YUFm6rWphqfNGRiZkt9+khgLxCSqa9zDmh5XgbJN78aI1pkyg08+U4Fan22Q6ZmHnHMG9flt4+S+MLY3auUF9G5p1FfrwcB7Jk2ufIv+Bm5zqRpdLeqPGj4mw7kKVamMrvS6n9CMCqIU+bA+rFsyL51YnrqTlgUF1GptAs2Famvclklo4pwyTx9Q6tWJh6jqcLU3i4QAMAp+TX67SwCw9/9C7xFTctDBqubc2F6e7du7FhwwbUrVsXrVu3BgDs2bMHOTk5uOOOO/DAAw+42n799delxrJYLLjtttuwYcMG3HfffQAAp9OJDRs2aC5yiYiIiKh801yYhoaG4sEHH3SbFhERUeYEJk6ciOHDh6Ndu3bo0KED3nzzTWRkZGDkyJFljklERERE5Y/mwvSTTz7xaAKDBg3CxYsX8fLLL+PcuXNo06YN1q5di5o1a3p0PURERESkb5q/LNOrVy9cu3atyPTU1FT06tWrTEmMHz8ef/75J2w2G7Zv346OHTuWKQ4RERERlV+aC9ONGzciJyenyPTs7Gxs2rTJI0kRERERUeUj/VH+3r17Xf8/cOAAzp37q2sYh8OBtWvXok6dOp7NjoiIiIgqDenCtE2bNlAUBYqiFPuRvb+/P95++22PJkdERERElYd0YZqUlAQhBBo2bIjffvsN1atXd82zWCyoUaMGjEbJfu08zG53APaS+y80XB+i0e5wwi7RyVlggPpAAfnDZ1qtJhhU+qBMTbWVOj+f83qe2Vl21X4oT1/MUI1nUZxACHD2UoZUX502m3pHhxaDAGoBl65mqfa16iczBiL+GhI0KztXNc/w2qFyOSINNcNDpDqaT0sr+tWUwvKHBE1Pz1U9NmdOXlKNB1wfIjLSioMHklX7jEz7/RfVeP5WI9CuF06sXyPVn2dg6y6qbZwmAPBHZnqWao6RzRqoxgMAq1EAsKFedH3YVPpGTaur3nl9/jC11Tr01NTPrVTMJtGqMcOqqw92AeSfQxloGF1X9RwKC1MfyS7vNegSOvZoJtUH8bVr6gMq5F874XWqql47D7dQH6gAAJwOO84dOIoHm9WCQWX81gCJYWXthrzXC3+LASaTenut/SmryZQZxIF8x/Ndo3qcN1KUHdhFmg9Pc+nCtH79vBE5nF4YBYGIiIiISHN3UZ9++mmp84cNG1bmZIiIiIio8tJcmD799NNuz3Nzc5GZmQmLxYKAgAAWpkRERERUJpq7i7p69arbv/T0dBw6dAhdunTBZ5995o0ciYiIiKgS0FyYFqdx48aYM2dOkbupRERERESyPFKYAoDJZMLZs2c9FY6IiIiIKhnN3zH95ptv3J4LIZCcnIx33nkHnTt39lhiRERERFS5aC5M77vvPrfniqKgevXq6NWrF+bNm+epvIiIiIioktFcmPqyH1ObzQab7a/O6lNTUwHkdQptMpTcZa1FEX89Snx5Ib/zfJk2Mm2tpeRWkOV6O4tEe4vEet22G+rthcx6PZwjAJiv52mWyNPu4RwBwCnRAbeWmJLjCrjaybS3W9U7Eve73jm5n0Qn5bLr1ZJjXsf5Eu2u70OZ6yJXYlPyd43ELpKmJaY3rm9PvwbJrldLjk6H+oAcBdvJtLfbc1XbOOx2t0dVwsMdj5O+lYMO9j096AMAOByejemwe7bWc8perwAUIcq2hy5dyhvdJiwsrCyLl0lcXBxmzJhRZPry5csREKA+UgoRERER3VyZmZkYPHgwUlJSEBwcXGpbTYXptWvXMHXqVKxcuRJXr14FAFStWhUPP/wwXnnlFYSGhpY56WXLluGf//yn6/n333+Prl27urUp7o5pREQEurzyHUx+gSXGtigCYxul4N1jIciRePcc4G9WbWNWnBhZ+zI+OXuL6nCAly5lqsYD8u5SPNcyE6/vC1AdDrBKFfVhUy2KwOiG1/De8VCp7bZJDLVnMQg83TQNbx0KUh+SVPIWllkRGN3gGt5LCkWuSp6ZWXLDpj4Tk475iVWkhiR1SrzTtBgEnm2RgXl/BKrGPHv6smo8IO8u5Gt3WTHpB5vqcJ/pe7aox7MYsfDZ7hgzLx7ZEscyoEWsVI7z+vrj2e/VhyQNqxWqGg/Iu8P44m05eHWXRXVozvRU9WE0rUbg1R4KXtwoIDESq1yOGmLeElZFKqaW67vaLf6q8cyKE4/VuYKPzlSTGpI0JUV9aGQt1878h1qrxgPy7pReOPQrajS9XXVI0roS2+2w27F728+4NbYnjCb1W/kmI++YViqV9I6p3cN3TG25nr1jmp6WirZN60oVptIf5V+5cgWxsbE4c+YMhgwZgpiYGADAgQMHsHjxYmzYsAFbt25F1apVy5T0vffei44dO7qe16lTp0gbq9UKq9VaZHqOU4GztBfR66/ZOUKRKlTMEi/y+XKFQfWPgtof38JynIrqMhapHPNOrByhqI5Bn7de+RMxx6m+Lw3S+zFvvbkSecocv4JtZdo7NFx/MsdGrYArrr3aMlkaKq7sHIdUe4OGPGVyVBv3vkh7p6K6jNo49e7r19beUzG9cX3LFJoF28q09/S1o1ZkFtdebRmTSf2mQD6jySTVnoVpJVMOClPFC4WpUDwb0yg8+2JqkHgTmU+65cyZM2GxWHDs2DHUrFmzyLy77roLM2fOxPz58+UzLSAoKAhBQUFlWpaIiIiIyj/pt+WrV6/G66+/XqQoBYBatWph7ty5WLVqlUeTIyIiIqLKQ7owTU5ORvPmzUuc36JFC5w7d84jSRERERFR5SNdmIaFheHEiRMlzk9KSkK1atU8kRMRERERVULShWmfPn0wdepU5OTkFJlns9kwbdo0/O1vf/NockRERERUeWj68VO7du3QuHFjjBs3DtHR0RBCIDExEe+++y5sNhuWLl3qzVyJiIiIqAKTLkzr1q2Lbdu2YezYsZgyZQryuz9VFAV33nkn3nnnHURERHgtUSIiIiKq2DR1RNegQQN8//33uHr1Ko4cOQIAiIqK4ndLiYiIiOiGaesh+bqqVauiQ4cOns6lzOpWrwKzf8mjr5jgBHANdcKqwC7xtdrgAIlOm+EEcBH1q6vHzJUcczZvbPkMhIb6qXY0X/eWkke6cs/xKurVCJLaboNBvSPqvJipaFqvqmpMP7PcyE/5eTatqx7zz4vpqvHyxg5Pwy1V/aU6Hk/LkBsRBwD8A0wwqnQ8nnZwj2o8ALBbDEC/dkg/vA9ZOZ4dZUNGg8a1VNvkbXcq6jeqqdrhulGyI/O8mDYEB/upxqxaVX0koLx411C3XlWpTuQzM9XHY7caBIAM3BJWRbUz/FaN5YZlzjvPM9C80S2q57nMyDCm6wNT1Ajxl7q+qwf7ScZMQ6O6IaoxD1xOUY0HAHA6EArg4JVUwFD6a0KOxGgXTkfeSA/HL2RIdfLfNNyz/WMr7K+f9MjDnfZ7egwALfHkhxchIiIiIvIiFqZEREREpAssTImIiIhIF1iYEhEREZEusDAlIiIiIl3QRWG6YMECREZGws/PDx07dsRvv/3m65SIiIiI6CbzeWG6cuVKTJw4EdOnT8fvv/+O1q1bo0+fPrhw4YKvUyMiIiKim8jnhekbb7yBUaNGYeTIkWjWrBnee+89BAQE4OOPP/Z1akRERER0E5Wpg31PycnJwa5duzBlyhTXNIPBgN69e2Pbtm1F2ttsNthsf3WGnpqaCiCvU+j8zqaLkz+vtDbFtS+N8Xobo0TbvI7z1ZmvtzNLtJfJUet2GyDbwb7s+uV6otayL2X2jZb9CPzVeX6pbRTx16PK2zl/i9z7Pb/r7fxk2jvUByvwsxjdHtVIbff1NjJtjZJvc7XENEjE1BIPABwe3m6trysy7RXIr1d2/U5Px3Q6pNbraifRPr/z/NKI622Ewy615Xa7+oAKWrCDfZ3zcMfw3iAzgIbmmHbPxnRIDgwky2lXv7bzKUJ4YQ9JOnv2LOrUqYOtW7ciNjbWNf35559HfHw8tm/f7tY+Li4OM2bMKBJn+fLlCAgI8Hq+RERERKRNZmYmBg8ejJSUFAQHB5fa1qd3TLWaMmUKJk6c6HqempqKiIgIrE6pDXNO6UOS3h96Fquu1fbYkKRGONHb8id+zKkPh0rMY8mpqvGAvDt8o+pdxYcnq6oOpVm7mtyQpPcEnca3aXU9OiRpv8BTWJMR4bEhSY1w4g7LCWzIiVTdlycvyQ1JOjz8EpYkh0kNSZqemaPaxqIIjG2UgnePhSBHlL6f9nz7o2o8IO9O6cKxbTHm3d+RrTYkaZb6OeRnMWLhs90xZl48snPU707FDLhPtY3FIDAhOg1vHgzy6JCkTzZOxdtHglVjypyTFoPAuEYpWHAsRGpI0iyJIUktBoHnWmbi9X0BqjGbN7pFNR6Qd+3cXeU0vktXvx5lhyS9N+gMvkmrI3V9O51yMe8LOYvVKeqvlb2bVlONl7diB0Iv7Me1Gs1VhyRtGFLy63g+4bDj6tHfUDWqAxSJIUkb11KPqQXvmOpcJb1javfwHdPsXM/eMU1Pk6uDAB8XpmFhYTAajTh//rzb9PPnz6NWraLjeFutVlit1iLT7TBAkXhhtsMg9QIu0yafQyKm2rj3heUKg+oyWnKU3W6Zj/K1xNSSIyC3L2UKzYJtZdrLFDP5aeUIRbW91nHvs3Oc6svYJD82BZCd40CWRHup7S7QVrUw1fgXWyamlnNSJh4A2DRut1p7ree5zLXj0PDXVfb6lvkoX1NMlSKz2PYqyxgkCs38K0UxmqTam0zqNxq0YGGqc+WgMFW8UJgKya/0yDIKz8YzmOTLTZ/++MliseC2227Dhg0bXNOcTic2bNjg9tE+EREREVV8Pv8of+LEiRg+fDjatWuHDh064M0330RGRgZGjhzp69SIiIiI6CbyeWE6aNAgXLx4ES+//DLOnTuHNm3aYO3atahZs6avUyMiIiKim8jnhSkAjB8/HuPHj/d1GkRERETkQz7vYJ+IiIiICGBhSkREREQ6wcKUiIiIiHRBF98xvVFDb6uDwCpBJc4XDjuyjp3GsNtqS3XIXCdEfRQph8OO478nYczt9WFUifnhzlOq8QDAKBwALqNl/apwKKX393dvk+qq8YTDjrSjJzH0VrntDpDoEN/psONC4p94vF1d1T4Ea4b4qcYDAIfdjj+2H8fErg1gVOnrbF9yimo84bRDnLiAUZ3qQTGob/eq/RdU2+QN0XgNjeuEqvbv+HvNhqrxAADm6x0i1mgA5Kr0a5e0Wy6mBnVqqnc8njesaypq1whU7RO2UQ25jszzzvMU3N60uup5HhFqUY2nCAdw7Rrub18bQiUeAOxLzlBtk3e8M9CmaXXV4/1EuwjVeED+tXMSo9pHqF47IRKDfDgcdhzddQrP92ik+hoEAMcvq2+3cNphTzqNh9uEq147NavIXd9Ohx1nzwHNwkJUt9vPLNEfq0PBFQChAWapfkwlxmigiqQcHG+TFzrDNVs9W84Fyl3e0sxC/bU8H++YEhEREZEusDAlIiIiIl1gYUpEREREusDClIiIiIh0gYUpEREREekCC1MiIiIi0gWfFqZpaWmYMGEC6tevD39/f3Tq1Ak7duzwZUpERERE5CM+LUwff/xxrF+/HkuXLsW+fftw1113oXfv3jhz5owv0yIiIiIiH/BZYZqVlYWvvvoKc+fORbdu3RAVFYW4uDhERUVh4cKFvkqLiIiIiHzEZyM/2e12OBwO+Pm5Dy/g7++PzZs3F7uMzWaDzWZzPU9NTQWQN8KRcNhLXJdwOtwe1ThKiZXPeb2NU6Jt3kg36oxw/vWoMhBQadvrauO0uz2qcRpUVgpt2+2wy603v51Me5lt0Xq8Tdf3u0wbmbb+ZrlRPfxMittjqazqIxr5WYxuj2ryRnWSayPT1hvnuSIRM7+NTFvA88db5loo2E7q2nGonxNa4gGev3a8sd1Og8zIT9q2227PlWpHVJ4pHh5NytODUzk0XIeKEEK9GvGSTp06wWKxYPny5ahZsyY+++wzDB8+HFFRUTh06FCR9nFxcZgxY0aR6cuXL0dAgPowokRERER0c2VmZmLw4MFISUlBcHBwqW19WpgeO3YMjz76KH755RcYjUa0bdsWTZo0wa5du5CYmFikfXF3TCMiIvD1tkMIrBJU4nqE04HspF3wa3AbFIP63aTwEPUi1+mw48SezYhs3UV1vOYlv59WjQfk3UHqhOPYioZwqHzLol9UmGo84bQj/fhOVGnYTmrMeH+z+r5xOuy4dHg7wpp0VN3uGsFyg+067HYk7opHzG3dYTSVHnP/+RTVeMLpAE7uBurdKnW8v028pNrGBCf6+J/Euqx6qmOnf7lql2o8IO9O6cKB1TDm8yvItqtchn/uUY9nMWLhs90xZl48snPU73jdMWqIahuz4sSgW85j5eWayBWlb3eDsEDVeEDeeX47juNXifO8Tqj6+MqKcKB+yiH8GdIUQlE/3gfOZaq2McGJu/xO4ods9eM94tY6qvEAbddOcIBZKt7xhM1o2Eb9NQgATlzJUG0jnA44/vwdxvptVa+dGpKDaTsddpxL3IZaMbGqefqZ5e6Ynti7BZGtOkttd80Qq1SeROWZ3u+YpqWmomHdmlKFqc8+ygeARo0aIT4+HhkZGUhNTUV4eDgGDRqEhg0bFtvearXCai36IqMYTVAkXqAUg1GqnVGiTT6D0aTa3iHxxxKA62NNBwyqy8hsh6utQW7/GIySeSJvu9X+KKgVmcW1V1tGpsAG8nalYjBKtVcrPAq3VWuflavtvV62XagvY5P7mBoAsnMcyJJor1ZoFm6r1t4b57lMoVmwrUx7Tx9vmeKocHvVa8fDr0GA/LWT11b92vHGdhuM8sdGJh4AmEzqRT5Reaf3wtSo4TrURT+mgYGBCA8Px9WrV7Fu3ToMGDDA1ykRERER0U3m0zum69atgxACTZs2xdGjRzFp0iRER0dj5MiRvkyLiIiIiHzAp3dMU1JSMG7cOERHR2PYsGHo0qUL1q1bB7OZH70QERERVTY+vWM6cOBADBw40JcpEBEREZFO6OI7pkRERERELEyJiIiISBd8+lH+jcrvgjUzPa30dg47sjMz4UxPk+o2Kd0gMzKLHZmZmUhPS1XtqsWWma4aD8gbOScTmbAhXbUbnYx09T4ExfUcDZLbLUwSfQg6HcjMzERGehoMKv0cphlyVOMBef2YZmZmIi0tVbW7KLVjDeT13yoyM6Gkp0l1kZMjcXyccCJTZCInK121+yCRm6UaDwAEFGRmZkLkZkGodRflUN+Xwm7Ii2e3QTjURyzKzZI4LxUnMjMzkZuVrtpdlE29e1AA2s7zbInvmysiL8dsczqEon4O52SqJ+qEE5nOTORkqx/vjLRU1XiAtmvH6FDfbi2vQQCQma6+3cJphz0zEyaJayddyF3fzgJ5qnXvZJfsx1Q2HgCkSr4OEZVneu8uKu3666RM1/k+7WD/Rp0+fRoRERG+ToOIiIiIVJw6dQp169YttU25LkydTifOnj2LoKCgUt8t5I8QderUKdURB2SVh5jlIUdvxCwPOXojZnnI0Rsxy0OO3ohZHnL0Rkxv5EhE3iWEQFpaGmrXrg2DofRPRsr1R/kGg0G18i4oODjY4y9k5SFmecjRGzHLQ47eiFkecvRGzPKQozdiloccvRHTGzkSkfeEhIRIteOPn4iIiIhIF1iYEhEREZEuVIrC1Gq1Yvr06bBarZUqZnnI0Rsxy0OO3ohZHnL0RszykKM3YpaHHL0R0xs5EpF+lOsfPxERERFRxVEp7pgSERERkf6xMCUiIiIiXWBhSkRERES6UOELU7tdfXhRIiIiIvK9Cl2Y7t+/H//617+QlqY+vjqVbunSpVi1apWv0yAiIqIKrMIWpnv27EHLli1hNpsRFBTk63TKtYyMDHz66ad47bXXsGbNGl+nU645nU44HA6Px3Q6nR6N6U167wikPO1LT/H0OVTezkki0o8KWZgeOHAAsbGxePnllzF58mSvrCMzM9MrcT1lz549OHPmjEdiBQYG4tNPP0XdunXx2muv4X//+59H4nrLJ598gvnz5/s6jSIOHDiAYcOGoU+fPhgzZgy2bt3qkZgjRoxA79698cQTT2DFihUeyNQ7srKyYLPZcOrUKWRnZ3ss7o1ei6dOncKXX34JAFixYgVGjRrl8TcPnuTp3Dx9DnnjnLxy5QoOHjyII0eOICcn54bjEZF+VbjC9I8//kD37t0RGRmJuLg4AJ7/nunmzZvx3HPPYf/+/R6J9+uvv+L999/H7NmzsXHjxhuOt3r1avTr1w8LFy5Eenr6DcUSQiA3Nxfh4eGIi4uDv78/5s6di3Xr1t1wnocOHcLOnTuxefPmG46Vz2az4csvv0R8fLzHYnrCoUOH0KlTJzgcDrRv3x7btm3D008/jf/85z9ljnnw4EF06dIFFosF99xzD06ePIlp06bhySef9GDmnpGYmIh//OMfaNeuHRo1aoTY2FiPvGn88ccfMW3aNOzevbtMy+fm5uL555/H/PnzMXHiRAwePBidOnWC0Wi84dy84fDhw3jzzTeRnJzskXiePoe8cU7+8ccf6N27NwYOHIiWLVti7ty5un7jQEQ3SFQgCQkJIiAgQPTo0UPUrl1bPPXUU655drvdY+v5+OOPRZ06dcRTTz0lDhw4cEOxvvzySxESEiIefvhh0alTJ9GuXTvxxBNPlDnet99+K/z9/cWHH34ozp49e0O5CSGE0+kUQgixcuVKMXDgQBEbGysCAgJEVFSU+O6778ocd9WqVSIyMlLExMQIf39/8eijj95wvvm57ty5UwQHB4v//ve/NxTPU5xOp3jxxRfFwIEDXdNSU1PFK6+8Itq0aSP+/e9/a46ZnZ0thgwZ4naOZ2VliVtvvVUoiiIeeeQRj+TuCXv37hUhISFi3LhxYtGiReLrr78WAwYMEFarVdxzzz0iJyenTHG/+uor4e/vL2bNmiV27txZ5vyuXr0qOnbsKBRFEWPGjHFNdzgcZY7pDUeOHBHVqlUTiqKIKVOmiIsXL95QPE+fQ944J/fv3y9uueUW8dxzz4n9+/eL119/XSiKIk6ePKk5FhGVDxWmMN2xY4cwm80iLi5O2O128f7774uwsDCvFadLliwRTZs2FePGjStzcXrgwAFRr1498d5777me+/v7iylTppQpXlZWlnjooYfEiy++KIQQIiMjQxw7dkzMnj1brFq1SqSmppYp7q+//ioCAgLERx99JA4ePCiOHDkievToIWJjY8WaNWs0x1u3bp0IDQ0V77//vrDZbOL7778XiqKIhx9+WJw6dapMORaUkpIiBg4cKJ5++mkhhD4KjBEjRohu3bq5TUtNTRWvv/66aNeunfi///s/zTHvuOMOERcXJ4TIO/ZCCPH888+LBx98ULRt21a89tprN574Dbpw4YK49dZbxeTJk4tMf+edd0RgYKAYNGiQ5riHDh0SDRo0EO++++4N55iTkyN69eol2rRpI+688063Y6GHc0cIIdLT08Wjjz4qRowYIRYsWCAURRGTJk264eLU0+eQJ+NdvHhRdOvWzXUdC5H3Ju9vf/ub2Lp1q9i9ezcLVKIKqMIUpvHx8W5F6LVr1zxanB47dkycOXPGbdonn3wioqOjxZgxY8Thw4c1x1y3bp249dZbhRBCHD9+XNSvX9/tbumuXbs0xcvMzBTt2rUTTz75pLh8+bIYP3686N69u6hbt66oWbOmmDVrluYchRDi/fffF82aNROZmZmuaadPnxZdunQRUVFRYt26ddKxUlJSxBNPPCFmzJghhMjb7kaNGom///3vIjQ0VAwYMED8+eefmvJ74403xOuvv+5W1H7wwQciMDBQHD16VAjx193Umy1/vf/5z39E586dxcGDB93mX7lyRYwaNUp06tRJZGRkSMfMyMgQXbt2FUOHDhW5ublCiLxjUr9+ffHxxx+Lf/zjH6Jnz56e3Zgy+P3330WLFi3Evn37XNdefrF37do18corr4iAgACxatUqTXHXr18vmjRpIk6cOOGadiPHODs7WyQnJ4u7775b9OzZs8gbBU++qS2LzMxMsWDBArFixQohRN4nGDdSnHr6HPLGOXnp0iXx6quvur22zpw5UyiKItq0aSPq1q0r+vTpIzZt2qRhy4lI7ypMYVpQ/h+olJQUjxSnV65cEeHh4eLFF18s8nHzokWLhNlsFk8++aTYu3evprg//PCD6Nevn0hKShJ169YVTzzxhCu3LVu2iBdeeEHzHYElS5YIf39/ERwcLO6//36xZMkSIYQQEyZMED179izTHaBPP/1UNG3aVFy4cEEIIVwfve7du1dUqVJFtGrVSnz//fdSsWw2m/j888/F0aNHxeXLl8Wtt94qHnvsMSGEEJ999plQFEX069dPnD59WipeZmameOGFF0RISIjo1auXePTRR8Xly5dFVlaWGDJkiBg7dmyZPyr2pKNHj4qwsDDx6KOPirS0NCHEX+fpyZMnhaIo0vsw3+bNm4XBYBDdunUTQ4cOFYGBgeLxxx8XQgixb98+ERQUJA4ePOizolyIvDdvfn5+rueFczl+/LgICQnRfGdu1apVIiIiwlWYFjyvN27cqPlNXb5jx46Ju+++W9xxxx1i6dKlQgghpk6dKkaNGuXT/ShE3l3TglasWCEURRHPPfecuHTpkhAibz8cP35cOqanzyFPxyv4KU/+68PKlSvF5cuXRXx8vGjfvr3rDi0RVQwVsjAtqGBx+swzz5Q5zs8//ywiIyPFjBkzitw5ve2220RISIh4/vnnhc1mk46ZlJQkAgIChKIoboWzEEI89dRT4q677hJXrlzRnOv+/fvFDz/8IIT46w/2uHHjxLBhw0R2drbmeEeOHBF+fn5i2rRpbtN37twpunfvLh555BFNdznzP+JbunSpiI2Ndd3p/Oyzz0SPHj1E/fr1Nd81PXXqlPjggw9E27ZtRXR0tBg2bJi4++67xd13312kEPSVn376SVitVjFu3Di3u1zJycmidevWYuvWrZpj/vbbb+If//iHePzxx8WCBQtc0//73/+KmJgYce3aNY/kXlabNm0Sfn5+4ssvvyyxza233iomTJigKe7x48eFv7+/62srBU2YMEG8/PLLZX5Dcvz4cXH//feLFi1aiPbt24vg4GDx66+/limWN9jtdte5nF+sTZo0SZw5c0Y888wz4oEHHpC++y6E588hb52TJ06cKPKG4+677xb9+/cvUzwi0qcKX5gKkVecfvjhh0JRlCLfddNi06ZNom7dumLmzJmuO6cZGRli9OjR4tVXX9V0pyLf6tWrRWBgoHjhhRfE4cOHxb59+8Rzzz0nQkNDxb59+8qca77ExETx4osvipCQkBuKt3TpUmE2m8WLL74okpKSxNWrV8W0adPE8OHDRUpKSplizpw5U7Ro0cJVfE+ePFm8/fbbN3yH84MPPhBPP/20UBRFKIoiXnnllRuK50nffPONsFqt4oEHHhArVqwQBw4cEJMnTxbh4eFl/n5tcQX3c889J3r06FHmY+Mpp06dEjVq1BD33nuv28fu+W+Yrly5Ijp16uS6O6nFRx99JMxms5g0aZLYt2+fOHDggHj++edFaGioSExMvKG8T58+LT766CMxY8aMIl+/0AOn0+nahytWrBBms1k0bdpUmEwmsXv37jLFK+xGziFvn5MOh0NkZWWJQYMGidmzZ99wPCLSj0pRmAqR9322xYsXi0OHDt1QnE2bNonIyEgxfvx4sXz5cjF16lTRrFmzMr/Y2u128cknn4jg4GBRt25dERMTI1q3bi1+//33G8pTiLw7mo888oiIiYkRCQkJNxTL6XSK5cuXiypVqogGDRqIRo0aiWrVqpX5I1Mh8r5/aLVaRefOncUdd9whgoODxZ49e24ox4J+++03MXz4cNGvXz+fF2gF7dq1S3Tv3l3Ur19fNGrUSDRp0sQjx1uIvK9XjB07VgQHB9/wMfeUr776SlgsFjF06FDxxx9/uM176aWXRGRkpFvRKsvhcIjPP/9cVK1aVdStW1dERUWJpk2bemxf6p3T6XSd87169RLVqlXT/HWi4nj6HPLWOTlt2jRRr169Mn2/n4j0SxFC58OweJAQAoqi3HCcnTt3YuLEiThx4gSCgoKwdOlStG3b9oZinj59GidOnECVKlVQt25dhIWF3XCeWVlZ2LlzJyIjIxEREXHD8QDgxIkT2Lt3L7KystCxY0dERkbeULxt27bh3XffRUhICMaMGYPmzZt7JM9827dvR/fu3fHDDz+gW7duHo19I1JTU3HlyhWkpaUhPDzcI8fbZrNhzZo1+PzzzzFlyhS0atXKA5neOIfDgUWLFmH8+PFo1KgROnfujPDwcCQlJeH777/Hhg0bcOutt5Y5/tmzZ/Hnn39CURQ0aNAANWvW9GD2+uZwODBp0iS8+eabSEhIuOFj7ulzyBvn5BdffIH4+HisWLEC69evv6Fzh4j0p1IVpp6UlpaGa9euwc/PD9WrV/d1OuWa0+mEoigeedNQUP4bkdjYWIwZMwbDhg3zaHw9stlssNvtCAwM9HUqRWzfvh1z587FoUOHEBoaitatW+PJJ59EdHS0r1MrtxwOBxYvXozbbrsNbdq08UhMT59Dno63f/9+zJw5E3FxcYiJifFITCLSDxamVKF98MEHGD16NI4cOYJGjRr5Op1Kz+FwwGAwQFEUOJ1OGAwVbvC5m85TnwSVJ7m5uTCbzb5Og4i8gIUpVWjHjh2DzWZDs2bNfJ0Kwb2IqowFFRERlY6FKRERERHpAj9HIyIiIiJdYGFKRERERLrAwpSIiIiIdIGFKRERERHpAgtTIiIiItIFFqZEREREpAssTImIiIhIF1iYEhEREZEusDAlonJtxIgRuO+++276ehcvXozQ0FDVdg6HA3PmzEF0dDT8/f1RrVo1dOzYEYsWLfJ+kkRE5YzJ1wkQEVVkM2bMwPvvv4933nkH7dq1Q2pqKnbu3ImrV6/6OjUiIt3hHVMiqlB69OiBp556Cs8//zyqVauGWrVqIS4uzq2NoihYuHAh+vbtC39/fzRs2BBffvmla/7GjRuhKAquXbvmmpaQkABFUXDixAls3LgRI0eOREpKChRFgaIoRdaR75tvvsHYsWPx0EMPoUGDBmjdujUee+wxPPfcc642TqcT//rXv9CgQQP4+/ujdevWbvkAwJo1a9CkSRP4+/ujZ8+eWLx4sVuOcXFxaNOmjdsyb775JiIjI92mLVq0CDExMfDz80N0dDTeffdd17wTJ05AURR8/fXX6NmzJwICAtC6dWts27bNLcaWLVvQo0cPBAQEoGrVqujTp4+r0JbZFiKikrAwJaIKZ8mSJQgMDMT27dsxd+5czJw5E+vXr3drM23aNDz44IPYs2cPhgwZgocffhiJiYlS8Tt16oQ333wTwcHBSE5ORnJysluhWVCtWrXw008/4eLFiyXG+9e//oVPP/0U7733Hvbv349nnnkG//jHPxAfHw8AOHXqFB544AH0798fCQkJePzxxzF58mTJvfGXZcuW4eWXX8bs2bORmJiIV199FdOmTcOSJUvc2k2dOhXPPfccEhIS0KRJEzzyyCOw2+0A8gr0O+64A82aNcO2bduwefNm9O/fHw6HQ2pbiIhKJYiIyrHhw4eLAQMGuJ53795ddOnSxa1N+/btxQsvvOB6DkCMHj3arU3Hjh3FmDFjhBBC/PzzzwKAuHr1qmv+7t27BQCRlJQkhBDik08+ESEhIar57d+/X8TExAiDwSBatmwp/vnPf4o1a9a45mdnZ4uAgACxdetWt+Uee+wx8cgjjwghhJgyZYpo1qyZ2/wXXnjBLcfp06eL1q1bu7WZP3++qF+/vut5o0aNxPLly93azJo1S8TGxgohhEhKShIAxKJFi9zyByASExOFEEI88sgjonPnzsVuq8y2EBGVht8xJaIKp1WrVm7Pw8PDceHCBbdpsbGxRZ4nJCR4PJdmzZrhjz/+wK5du7Blyxb88ssv6N+/P0aMGIFFixbh6NGjyMzMxJ133um2XE5ODm699VYAQGJiIjp27Fhq/moyMjJw7NgxPPbYYxg1apRrut1uR0hIiFvbgvsvPDwcAHDhwgVER0cjISEBDz30ULHrkNkWIqLSsDAlogrHbDa7PVcUBU6nU3p5gyHvW05CCNe03NzcMudjMBjQvn17tG/fHhMmTMD//d//YejQoZg6dSrS09MBAN999x3q1KnjtpzVatW0joL5Fs45fz0ffvhhkSLXaDS6PS+4/xRFAQDX/vP39y8xB09tCxFVXixMiahS+vXXXzFs2DC35/l39apXrw4ASE5ORtWqVQGgyN1Ui8Xi+l6lVs2aNQOQdxezWbNmsFqtOHnyJLp3715s+5iYGHzzzTdF8i+oevXqOHfuHIQQrmKyYM41a9ZE7dq1cfz4cQwZMqRMeQN5d1M3bNiAGTNmFLtdattCRFQaFqZEVCl98cUXaNeuHbp06YJly5bht99+w0cffQQAiIqKQkREBOLi4jB79mwcPnwY8+bNc1s+MjIS6enp2LBhA1q3bo2AgAAEBAQUWc/f//53dO7cGZ06dUKtWrWQlJSEKVOmoEmTJoiOjobJZMJzzz2HZ555Bk6nE126dEFKSgq2bNmC4OBgDB8+HKNHj8a8efMwadIkPP7449i1axcWL17stp4ePXrg4sWLmDt3Lv7+979j7dq1+P777xEcHOxqM2PGDDz11FMICQnB3/72N9hsNlfXVRMnTpTab1OmTEHLli0xduxYjB49GhaLBT///DMeeughhIWFqW4LEVGpfPwdVyKiG1Lcj5+efvpptzYDBgwQw4cPdz0HIBYsWCDuvPNOYbVaRWRkpFi5cqXbMps3bxYtW7YUfn5+omvXruKLL75w+/GTEEKMHj1a3HLLLQKAmD59erH5ffDBB6Jnz56ievXqwmKxiHr16okRI0aIEydOuNo4nU7x5ptviqZNmwqz2SyqV68u+vTpI+Lj411t/ve//4moqChhtVpF165dxccff1zkB1oLFy4UERERIjAwUAwbNkzMnj3b7cdPQgixbNky0aZNG2GxWETVqlVFt27dxNdffy2E+OvHT7t373a1v3r1qgAgfv75Z9e0jRs3ik6dOgmr1SpCQ0NFnz59XHnIbAsRUUkUIQp9KYmIqIJTFAWrVq3yyYhRnrJx40b07NkTV69elRqBioioPGA/pkRERESkCyxMiYiIiEgX+FE+EREREekC75gSERERkS6wMCUiIiIiXWBhSkRERES6wMKUiIiIiHSBhSkRERER6QILUyIiIiLSBRamRERERKQLLEyJiIiISBdYmBIRERGRLrAwJSIiIiJdYGFKRERERLrAwpSIiIiIdIGFKRERERHpAgtTIiIiItIFFqZEREREpAssTImIiIhIF1iYEhEREZEusDAlIiIiIl1gYUpEREREusDClIiIiIh0gYUpEREREekCC1MiIiIi0gUWpkRERESkCyxMiYiIiEgXWJgSERERkS6wMCUiIiIiXWBhSkRERES6wMKUiIiIiHSBhSkRERER6QILUyIiIiLSBRamRERERKQLLEyJiIiISBdYmBIRERGRLrAwJSIiIiJdYGFKRERERLrAwpSIiIiIdIGFKRERERHpAgtTIiIiItIFFqZEREREpAssTImIiIhIF1iYEhEREZEusDAlIiIiIl1gYUpEREREusDClIiIiIh0gYUpEREREekCC1MiIiIi0gUWpkRERESkCyxMiYiIiEgXWJgSERERkS6wMCUiIiIiXWBhSkRERES6wMKUiIiIiHSBhSkRERER6QILUyIiIiLSBRamRERERKQLLEyJiIiISBdYmBIRERGRLrAwJSIiIiJdYGFKRERERLrAwpSIiIiIdIGFKRERERHpAgtTIiIiItIFFqZEREREpAssTImIiIhIF1iYEhEREZEusDAlIiIiIl1gYUpEREREusDClIiIiIh0gYUpEREREekCC1MiIiIi0gUWpkRERESkCyxMiYiIiEgXWJgSERERkS6wMCUiIiIiXWBhSkRERES6wMKUiIiIiHSBhSkRERER6QILUyIiIiLSBRamRERERKQLLEyJiIiISBdYmBIRERGRLrAwJSIiIiJdYGFKRERERLrAwpSIiIiIdIGFKRERERHpAgtTIiIiItIFFqZEREREpAssTImIiIhIF1iYEhEREZEusDAlIiIiIl1gYUpEREREusDClIiIiIh0gYUpEREREekCC1MiIiIi0gUWpkRERESkCyxMiYiIiEgXWJgSERERkS6wMCUiIiIiXWBhSkRERES6wMKUiIiIiHSBhSkRERER6QILUyIiIiLSBRamRERERKQLLEyJiIiISBdYmBIRERGRLrAwJSIiIiJdYGFKRERERLrAwpSIiIiIdIGFKRERERHpAgtTIiIiItIFFqZEREREpAssTImIiIhIF1iYEhEREZEusDAlIiIiIl1gYUpEREREusDClIiIiIh0gYUpEREREekCC1MiIiIi0gUWpkRERESkCyxMiYiIiEgXWJgSERERkS6wMCUiIiIiXWBhSkRERES6wMKUiIiIiHSBhSkRERER6QILUyIiIiLSBRamRERERKQLLEyJiIiISBdYmBIRERGRLrAwJSIiIiJdYGFKRERERLrAwpSIiIiIdIGFKRERERHpAgtTIiIiItIFFqZEREREpAssTImIiIhIF1iYEhEREZEusDAlIiIiIl1gYUpEREREusDClIiIiIh0gYUpEREREekCC1MiIiIi0gUWpkRERESkCyxMiYiIiEgXWJgSERERkS6wMCUiIiIiXWBhSkRERES6wMKUiIiIiHSBhSkRERER6QILUyIiIiLSBRamRERERKQLLEyJiIiISBdYmBIRERGRLrAwJSIiIiJdYGFKRERERLrAwpSIiIiIdIGFKRERERHpAgtTIiIiItIFFqZEREREpAssTImIiIhIF1iYEhEREZEusDAlIiIiIl1gYUpEREREusDClIiIiIh0gYUpEREREekCC1MiIiIi0gUWpkRERESkCyxMiYiIiEgXWJgSERERkS6wMCUiIiIiXWBhSkRERES6wMKUiIiIiHSBhSkRERER6QILUyIiIiLSBRamRERERKQLLEyJiIiISBdYmBIRERGRLrAwJSIiIiJdYGFKRERERLrAwpSIiIiIdIGFKRERERHpAgtTIiIiItIFFqZEREREpAssTImIiIhIF1iYEhEREZEusDAlIiIiIl1gYUpEREREusDClIiIiIh0gYUpEREREekCC1MiIiIi0gUWpkRERESkCyxMiYiIiEgXWJgSERERkS6wMCUiIiIiXWBhSkRERES6wMKUiIiIiHSBhSkRERER6QILUyIiIiLSBRamRERERKQLLEyJiIiISBdYmBIRERGRLrAwJSIiIiJdYGFKRERERLrAwpSIiIiIdIGFKRERERHpAgtTIiIiItIFFqZEREREpAssTImIiIhIF1iYEhEREZEusDAlIiIiIl1gYUpEREREusDClIiIiIh0gYUpEREREekCC1MiIiIi0gUWpkRERESkCyxMiYiIiEgXWJgSERERkS6wMCUiIiIiXWBhSkRERES6wMKUiIiIiHSBhSkRERER6QILUyIiIiLSBRamRERERKQLLEyJiIiISBdYmBIRERGRLrAwJSIiIiJdYGFKRERERLrAwpSIiIiIdIGFKRERERHpAgtTIiIiItIFFqZEREREpAssTImIiIhIF1iYEhEREZEusDAlIiIiIl1gYUpEREREusDClIiIiIh0gYUpEREREekCC1MiIiIi0gUWpkRERESkCyxMiYiIiEgXWJgSERERkS6wMCUiIiIiXTD5OgHSJjs7Gzk5Ob5Og4iIqFKxWCzw8/PzdRoVHgvTciQ7OxsNGjTAuXPnfJ0KERFRpVKrVi0kJSWxOPUyFqblSE5ODs6dO4cjSacQFBQMABAQuP6fgg8QovR5BZfNfw63+aJQW7j9p/CyBeeXNE8UClJkvsr2lLa90LC9N7KvPLlsaTk78/dRCbGcovR5BZ+75heYXtI54Cy8jGsd12MX+H9x8wo+L3xeOYUoZZni1+8UBba3UBu42ri3FYWmCyEKtHHfbmfhfVNk2WJyKrA9Ja2vcM5q+6hwLCFE0XmFli28PhSYX3ie67wqZj2F8yiScwnnSuH9L5NzSevNW1a9TcGgBeerLlvkWhCuxxL3hVNt/cXHKzZHiKLtSli25O0smHPpbQpfwAXzUN+fJedT8nqc6s8LLwOJZfKflzSvyAVc2noLzZNZv+sEdgKOHJw7sAQ5OTksTL2MhWk5FBwc7PXCtLTCs7hlC84vc2GqIWe9FJfeWrboH3b3WM5i9/Nf8wo+L1J03eTCtOA2lbxMoe0u9Ly4wvSvNiUt+9e6ZAvTguvLn14kLtzbFre+/OdFphXa/pK2uyyFqShme71ZmBaXR2nbo/ZctjAt7nnZly25MFVKKEyLO0ZaikrXo0qbGylMC893exEo/IJQXJtiH0ubJ1PkaShECz9XLUydJccoy/qKWTZ/N5H38cdPRERERKQLLEyJiIiISBdYmBIRERGRLrAwJSIiIiJdYGFKRERERLrAwpSIiIiIdIGFKRERERHpAgtTIiIiItIFFqZEREREpAssTImIiIhIF1iYEhEREZEusDAlIiIiIl1gYUpEREREusDClIiIiIh0gYUpEREREekCC1MiIiIi0gUWpkRERESkCyxMiYiIiEgXWJgSERERkS6wMCUiIiIiXWBhSkRERES6wMKUiIiIiHTB5OsESLvU1FQIkfd/Add/Cj5AiNLnFVw2/znc5otCbeH2n8LLFpxf0jxRKEiR+SrbU9r2QsP23si+8uSypeXszN9HJcRyitLnFXzuml9geknngLPwMq51XI9d4P/FzSv4vPB55RSilGWKX79TFNjeQm3gauPeVhSaLoQo0MZ9u52F902RZYvJqcD2lLS+wjmr7aPCsYQQRecVWrbw+lBgfuF5rvOqmPUUzqNIziWcK4X3v0zOJa03b1n1NgWDFpyvumyRa0G4HkvcF0619Rcfr9gcIYq2K2HZkrezYM6ltyl8ARfMQ31/lpxPyetxqj8vvAwklsl/XtK8IhdwaestNE9m/a4T2Ak4ckA3BwvTckQIgSpVqqBxgwhfp0JERFSpVKlSpcgbQfI8FqbliKIoSE9Px6lTpxAcHOzrdCqd1NRUREREcP/7CPe/b3H/+xb3v2/l739FUXydSoXHwrQcCg4O5guTD3H/+xb3v29x//sW9z9VdPzxExERERHpAgtTIiIiItIFFqbliNVqxfTp02G1Wn2dSqXE/e9b3P++xf3vW9z/vsX9f/Mogj8xIyIiIiId4B1TIiIiItIFFqZEREREpAssTImIiIhIF1iYEhEREZEusDDVmQULFiAyMhJ+fn7o2LEjfvvtt1Lbf/HFF4iOjoafnx9atmyJNWvW3KRMKyYt+//DDz9E165dUbVqVVStWhW9e/dWPV5UOq3nf74VK1ZAURTcd9993k2wgtO6/69du4Zx48YhPDwcVqsVTZo04WvQDdC6/9988000bdoU/v7+iIiIwDPPPIPs7OyblG3F8ssvv6B///6oXbs2FEXB6tWrVZfZuHEj2rZtC6vViqioKCxevNjreVYKgnRjxYoVwmKxiI8//ljs379fjBo1SoSGhorz588X237Lli3CaDSKuXPnigMHDoiXXnpJmM1msW/fvpucecWgdf8PHjxYLFiwQOzevVskJiaKESNGiJCQEHH69OmbnHnFoHX/50tKShJ16tQRXbt2FQMGDLg5yVZAWve/zWYT7dq1E/369RObN28WSUlJYuPGjSIhIeEmZ14xaN3/y5YtE1arVSxbtkwkJSWJdevWifDwcPHMM8/c5MwrhjVr1oipU6eKr7/+WgAQq1atKrX98ePHRUBAgJg4caI4cOCAePvtt4XRaBRr1669OQlXYCxMdaRDhw5i3LhxrucOh0PUrl1b/Otf/yq2/cCBA8Xdd9/tNq1jx47in//8p1fzrKi07v/C7Ha7CAoKEkuWLPFWihVaWfa/3W4XnTp1EosWLRLDhw9nYXoDtO7/hQsXioYNG4qcnJyblWKFpnX/jxs3TvTq1ctt2sSJE0Xnzp29mmdlIFOYPv/886J58+Zu0wYNGiT69OnjxcwqB36UrxM5OTnYtWsXevfu7ZpmMBjQu3dvbNu2rdhltm3b5tYeAPr06VNieypZWfZ/YZmZmcjNzUW1atW8lWaFVdb9P3PmTNSoUQOPPfbYzUizwirL/v/mm28QGxuLcePGoWbNmmjRogVeffVVOByOm5V2hVGW/d+pUyfs2rXL9XH/8ePHsWbNGvTr1++m5FzZ8e+v95h8nQDluXTpEhwOB2rWrOk2vWbNmjh48GCxy5w7d67Y9ufOnfNanhVVWfZ/YS+88AJq165d5MWK1JVl/2/evBkfffQREhISbkKGFVtZ9v/x48fx008/YciQIVizZg2OHj2KsWPHIjc3F9OnT78ZaVcYZdn/gwcPxqVLl9ClSxcIIWC32zF69Gi8+OKLNyPlSq+kv7+pqanIysqCv7+/jzIr/3jHlMgD5syZgxUrVmDVqlXw8/PzdToVXlpaGoYOHYoPP/wQYWFhvk6nUnI6nahRowY++OAD3HbbbRg0aBCmTp2K9957z9epVQobN27Eq6++infffRe///47vv76a3z33XeYNWuWr1MjuiG8Y6oTYWFhMBqNOH/+vNv08+fPo1atWsUuU6tWLU3tqWRl2f/5Xn/9dcyZMwc//vgjWrVq5c00Kyyt+//YsWM4ceIE+vfv75rmdDoBACaTCYcOHUKjRo28m3QFUpbzPzw8HGazGUaj0TUtJiYG586dQ05ODiwWi1dzrkjKsv+nTZuGoUOH4vHHHwcAtGzZEhkZGXjiiScwdepUGAy87+RNJf39DQ4O5t3SG8QzVycsFgtuu+02bNiwwTXN6XRiw4YNiI2NLXaZ2NhYt/YAsH79+hLbU8nKsv8BYO7cuZg1axbWrl2Ldu3a3YxUKySt+z86Ohr79u1DQkKC69+9996Lnj17IiEhARERETcz/XKvLOd/586dcfToUdcbAgA4fPgwwsPDWZRqVJb9n5mZWaT4zH+TIITwXrIEgH9/vcrXv76iv6xYsUJYrVaxePFiceDAAfHEE0+I0NBQce7cOSGEEEOHDhWTJ092td+yZYswmUzi9ddfF4mJiWL69OnsLuoGaN3/c+bMERaLRXz55ZciOTnZ9S8tLc1Xm1Cuad3/hfFX+TdG6/4/efKkCAoKEuPHjxeHDh0S3377rahRo4Z45ZVXfLUJ5ZrW/T99+nQRFBQkPvvsM3H8+HHxww8/iEaNGomBAwf6ahPKtbS0NLF7926xe/duAUC88cYbYvfu3eLPP/8UQggxefJkMXToUFf7/O6iJk2aJBITE8WCBQvYXZSHsDDVmbffflvUq1dPWCwW0aFDB/Hrr7+65nXv3l0MHz7crf3nn38umjRpIiwWi2jevLn47rvvbnLGFYuW/V+/fn0BoMi/6dOn3/zEKwit539BLExvnNb9v3XrVtGxY0dhtVpFw4YNxezZs4Xdbr/JWVccWvZ/bm6uiIuLE40aNRJ+fn4iIiJCjB07Vly9evXmJ14B/Pzzz8W+nufv8+HDh4vu3bsXWaZNmzbCYrGIhg0bik8++eSm510RKULwnj8RERER+R6/Y0pEREREusDClIiIiIh0gYUpEREREekCC1MiIiIi0gUWpkRERESkCyxMqVLZuHEjFEXBtWvXpJeJi4tDmzZtvJbTjYiMjMSbb755U9Y1dOhQvPrqq15fT48ePTBhwoQbiiFznBcvXozQ0FDX88LHecSIEbjvvvtuKI/KLjMzEw8++CCCg4M1X3eKomD16tVey83TtmzZgpYtW8JsNper8+bSpUuoUaMGTp8+7etUiACwMKUKaNu2bTAajbj77rt9nUqFsWfPHqxZswZPPfWUa1qPHj2gKAoURYGfnx+aNWuGd99914dZajNo0CAcPny4xPlvvfUWFi9e7HruiYLZUwoX1Z5UljdvJVmyZAk2bdqErVu3Ijk5GSEhIUXa3Mw3fhs3bkRkZCSAvDcecXFxHos9ceJEtGnTBklJSVi8eLHu3tDGxcVhxIgRAPLe0G7cuBFA3nCow4YNw/Tp032XHFEBLEypwvnoo4/w5JNP4pdffsHZs2d9nU6F8Pbbb+Ohhx5ClSpV3KaPGjUKycnJOHDgAAYOHIhx48bhs88+KzZGTk7OzUhVmr+/P2rUqFHi/JCQEK8Vf5XFsWPHEBMTgxYtWqBWrVpQFMXXKXnNsWPH0KtXL9StW7fcnTcjR47EsmXLcOXKFV+nQsTClCqW9PR0rFy5EmPGjMHdd9/tdserOPl3nlavXo3GjRvDz88Pffr0walTp4q0Xbp0KSIjIxESEoKHH34YaWlprnlr165Fly5dEBoailtuuQX33HMPjh07VuJ6P/jgA9SuXdttnHEAGDBgAB599FEAeX/oBgwYgJo1a6JKlSpo3749fvzxxxJjnjhxAoqiICEhwTXt2rVrUBTFdXcEAP744w/07dsXVapUQc2aNTF06FBcunSpxLgOhwNffvkl+vfvX2ReQEAAatWqhYYNGyIuLg6NGzfGN998AyDvDuP48eMxYcIEhIWFoU+fPgCA+Ph4dOjQAVarFeHh4Zg8eTLsdrtbXLvdjvHjxyMkJARhYWGYNm2a2/jfS5cuRbt27RAUFIRatWph8ODBuHDhQpH8tmzZglatWsHPzw+33347/vjjD9c8tbuOBT/KHzFiBOLj4/HWW2+57hInJSUhKioKr7/+uttyCQkJUBQFR48eLTau0+nEzJkzUbduXVitVrRp0wZr1651zS/ujmV+zBMnTmDjxo0YOXIkUlJSXLnk3/mLjIzErFmz8MgjjyAwMBB16tTBggULXHHUzpETJ06gZ8+eAICqVatCURTXXbbifPXVV2jevDmsVisiIyMxb94817wePXpg3rx5+OWXX6AoCnr06FFk+cWLF2PGjBnYs2ePa1sKXrOXLl3C/fffj4CAALdzK5/Wc7k07777rus1oGbNmvj73//ummez2fDUU0+hRo0a8PPzQ5cuXbBjxw4Af+3Ty5cv49FHH3VtQ0nbpSgK3n//fdxzzz0ICAhATEwMtm3bhqNHj6JHjx4IDAxEp06d3F4/1F4LDh48iICAACxfvtw17fPPP4e/vz8OHDiguu3NmzdH7dq1sWrVqjLtOyJPYmFKFcrnn3+O6OhoNG3aFP/4xz/w8ccfQ21ws8zMTMyePRuffvoptmzZgmvXruHhhx92a3Ps2DGsXr0a3377Lb799lvEx8djzpw5rvkZGRmYOHEidu7ciQ0bNsBgMOD+++8vUnjme+ihh3D58mX8/PPPrmlXrlzB2rVrMWTIEAB5RXa/fv2wYcMG7N69G3/729/Qv39/nDx5sqy7B9euXUOvXr1w6623YufOnVi7di3Onz+PgQMHlrjM3r17kZKSgnbt2qnG9/f3d7szumTJElgsFmzZsgXvvfcezpw5g379+qF9+/bYs2cPFi5ciI8++givvPKKW5wlS5bAZDLht99+w1tvvYU33ngDixYtcs3Pzc3FrFmzsGfPHqxevRonTpwotoCaNGkS5s2bhx07dqB69ero378/cnNzJfaUu7feeguxsbGuO8TJycmoV68eHn30UXzyySdubT/55BN069YNUVFRJcaaN28eXn/9dezduxd9+vTBvffeiyNHjkjl0qlTJ7z55psIDg525fLcc8+55r/22mto3bo1du/ejcmTJ+Ppp5/G+vXrpWJHRETgq6++AgAcOnQIycnJeOutt4ptu2vXLgwcOBAPP/ww9u3bh7i4OEybNs1VgH399dcYNWoUYmNjkZycjK+//rpIjEGDBuHZZ59F8+bNXdsyaNAg1/wZM2Zg4MCB2Lt3L/r164chQ4a47uqV5Vwuyc6dO/HUU09h5syZOHToENauXYtu3bq55j///PP46quvsGTJEvz++++IiopCnz59cOXKFURERCA5ORnBwcF48803XdtQ2nbNmjULw4YNQ0JCAqKjozF48GD885//xJQpU7Bz504IITB+/HhXe7XXgujoaLz++usYO3YsTp48idOnT2P06NH497//jWbNmkntgw4dOmDTpk2a9x2Rx/l0QFQiD+vUqZN48803hRB5Y0mHhYWJn3/+2TU/fzzk/PGkP/nkEwHAbUzqxMREAUBs375dCCHE9OnTRUBAgEhNTXW1mTRpkujYsWOJeVy8eFEAEPv27SuxzYABA8Sjjz7qev7++++L2rVrC4fDUeIyzZs3F2+//bbref369cX8+fOFEEIkJSUJAGL37t2u+VevXhUAXPtg1qxZ4q677nKLeerUKQFAHDp0qNh1rlq1ShiNRuF0Ot2md+/eXTz99NNCCCHsdrtYunSpACDeeecd1/xbb73VbZkXX3xRNG3a1C3WggULRJUqVVzb3b17dxETE+PW5oUXXhAxMTEl7pcdO3YIACItLU0I8ddxXrFihavN5cuXhb+/v1i5cqUQIu/Yh4SEuOZPnz5dtG7d2vV8+PDhYsCAAcVub74zZ84Io9HoOldycnJEWFiYWLx4cYm51q5dW8yePdttWvv27cXYsWPdci845vnu3bsFAJGUlFRs7vnq168v/va3v7lNGzRokOjbt68QQu4cKW79xRk8eLC488473aZNmjRJNGvWzPX86aefLjK+eGGF93s+AOKll15yPU9PTxcAxPfffy+EKNu5XJKvvvpKBAcHu13jBddrNpvFsmXLXNNycnJE7dq1xdy5c13TQkJC3MZKl92ubdu2CQDio48+ck377LPPhJ+fX6k5F34tEEKIu+++W3Tt2lXccccd4q677ipyzZbmmWeeET169JBuT+QtvGNKFcahQ4fw22+/4ZFHHgEAmEwmDBo0CB999FGpy5lMJrRv3971PDo6GqGhoUhMTHRNi4yMRFBQkOt5eHi420fHR44cwSOPPIKGDRsiODjY9QOL0u5uDhkyBF999RVsNhsAYNmyZXj44YdhMORdlunp6XjuuecQExOD0NBQVKlSBYmJiTd0x3TPnj34+eefUaVKFde/6OhoACjxqwdZWVmwWq3Ffj/w3XffRZUqVeDv749Ro0bhmWeewZgxY1zzb7vtNrf2iYmJiI2NdYvVuXNnpKenu/0q+Pbbb3drExsbiyNHjsDhcADIu1vXv39/1KtXD0FBQejevTuAovs7NjbW9f9q1aqhadOmbsf1RtWuXRt33303Pv74YwDA//73P9hsNjz00EPFtk9NTcXZs2fRuXNnt+mdO3f2WF4Ftzn/uSe3OV9iYmKx21HwON2oVq1auf4fGBiI4OBg13VXlnO5JHfeeSfq16+Phg0bYujQoVi2bBkyMzNdsXJzc9221Ww2o0OHDmXerwW3q2bNmgCAli1buk3Lzs5GamoqAPnXgo8//hh79+7F77//jsWLF2v6Tq+/v79rm4l8yeTrBIg85aOPPoLdbkft2rVd04QQsFqteOedd4r9RbAss9ns9lxRFLeP6fv374/69evjww8/dH13tEWLFqX+4Kd///4QQuC7775D+/btsWnTJsyfP981/7nnnsP69evx+uuvIyoqCv7+/vj73/9eYsz8glYU+OpC4Y+t09PT0b9/f/z73/8usnx4eHixccPCwpCZmYmcnBxYLBa3eUOGDMHUqVPh7++P8PBwVw75AgMDS9z+ssrIyECfPn3Qp08fLFu2DNWrV8fJkyfRp08fn/zA6vHHH8fQoUMxf/58fPLJJxg0aBACAgLKHE/mOOoxtjeUdt2V5VwuSVBQEH7//Xds3LgRP/zwA15++WXExcW5vkfqaQW3K794LG5a/rbKvhbs2bMHGRkZMBgMSE5O1rQfrly5gurVq5d5m4g8hXdMqUKw2+349NNPMW/ePCQkJLj+7dmzB7Vr1y7xl+L5y+7cudP1/NChQ7h27RpiYmKk1n358mUcOnQIL730Eu644w7ExMTg6tWrqsv5+fnhgQcewLJly/DZZ5+hadOmaNu2rWv+li1bMGLECNx///1o2bIlatWqhRMnTpQYL/+PSnJysmtawR+5AEDbtm2xf/9+REZGIioqyu1fSUVkfpc3xf2IIiQkBFFRUahTp06RorQ4+T/0KFgYbdmyBUFBQahbt65r2vbt292W+/XXX9G4cWMYjUYcPHgQly9fxpw5c9C1a1dER0cX+8On/OXyXb16FYcPH5Y+roVZLJZi7wT269cPgYGBWLhwIdauXev68VpxgoODUbt2bWzZssVt+pYtW1zfBZQ5jiXlArhvc/7z/G2WjQ1A9a5nTExMsdvRpEkTGI3GUpctvL6y3GEty7lcGpPJhN69e2Pu3LnYu3cvTpw4gZ9++gmNGjVyfU86X25uLnbs2FHq9zfLul3FkXktuHLlCkaMGIGpU6dixIgRGDJkCLKysqTX8ccff+DWW2/1SL5EN4KFKVUI3377La5evYrHHnsMLVq0cPv34IMPlvpxvtlsxpNPPont27dj165dGDFiBG6//XZ06NBBat1Vq1bFLbfcgg8++ABHjx7FTz/9hIkTJ0otO2TIEHz33Xf4+OOPXT96yte4cWN8/fXXrgJ78ODBJf6YCsj7KO7222/HnDlzkJiYiPj4eLz00ktubcaNG4crV67gkUcewY4dO3Ds2DGsW7cOI0eOLPGPaPXq1dG2bVts3rxZaptKM3bsWJw6dQpPPvkkDh48iP/+97+YPn06Jk6c6FbYnjx5EhMnTsShQ4fw2Wef4e2338bTTz8NAKhXrx4sFgvefvttHD9+HN988w1mzZpV7PpmzpyJDRs24I8//sCIESMQFhZW5s7PIyMjsX37dpw4cQKXLl1yHQuj0YgRI0ZgypQpaNy4cZGP0gubNGkS/v3vf2PlypU4dOgQJk+ejISEBNf2RUVFISIiAnFxcThy5Ai+++47t1+75+eSnp6ODRs24NKlS24fwW7ZsgVz587F4cOHsWDBAnzxxReu2DLnSP369aEoCr799ltcvHgR6enpxW7Hs88+iw0bNmDWrFk4fPgwlixZgnfeecfth1iy+zUpKQkJCQm4dOmS66staspyLpfk22+/xX/+8x8kJCTgzz//xKeffgqn04mmTZsiMDAQY8aMwaRJk7B27VocOHAAo0aNQmZmJh577DGPb1dxZF4LRo8ejYiICLz00kt444034HA4pI9FZmYmdu3ahbvuuqvMORJ5jC+/4ErkKffcc4/o169fsfO2b98uAIg9e/YU++OnkJAQ8dVXX4mGDRsKq9UqevfuLf7880/X8sX9iGH+/Pmifv36rufr168XMTExwmq1ilatWomNGzcKAGLVqlWl5u1wOER4eLgAII4dO+Y2LykpSfTs2VP4+/uLiIgI8c477xT5AU7BHz8JIcSBAwdEbGys8Pf3F23atBE//PCD2w9bhBDi8OHD4v777xehoaHC399fREdHiwkTJpT6Q4l3331X3H777W7TivsxkMz8jRs3ivbt2wuLxSJq1aolXnjhBZGbm+u23NixY8Xo0aNFcHCwqFq1qnjxxRfd8lu+fLmIjIwUVqtVxMbGim+++cbtRz35x/l///ufaN68ubBYLKJDhw5iz549rhhaf/x06NAhcfvttwt/f3+3HyIJIcSxY8cEALcfw5TE4XCIuLg4UadOHWE2m0Xr1q1dP+jJt3nzZtGyZUvh5+cnunbtKr744osi6xw9erS45ZZbBAAxffp0IUTe+TBjxgzx0EMPiYCAAFGrVi3x1ltvucWWOUdmzpwpatWqJRRFEcOHDy9xW7788kvRrFkzYTabRb169cRrr73mNl/mx0/Z2dniwQcfFKGhoQKA6wdExV0/hX9gVJZzuTibNm0S3bt3F1WrVhX+/v6iVatWrh/JCSFEVlaWePLJJ0VYWJiwWq2ic+fO4rfffis1N9ntKu4HaYVfp9ReC5YsWSICAwPF4cOHXTG2b98uzGazWLNmjer2L1++XDRt2lRuZxF5mSKESl86RBXY4sWLMWHCBI+MclORZWVloWnTpli5cqXqHcHKaNOmTbjjjjtw6tQp149ZfCEyMhITJkzQzQhVVD7cfvvteOqppzB48GBfp0LEHz8RkTp/f398+umnZe68vKKy2Wy4ePEi4uLi8NBDD/m0KCUqi0uXLuGBBx5w9WZC5GssTIlISnEj91R2n332GR577DG0adMGn376qa/TIdIsLCwMzz//vK/TIHLhR/lEREREpAv8VT4RERER6QILUyIiIiLSBRamRERERKQLLEyJiIiISBdYmBIRERGRLrAwJSIiIiJdYGFKRERERLrAwpSIiIiIdOH/ATg5bB5Y9SZGAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 800x850 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "attention_map = plot_attention_map(model, human_vocab, inv_machine_vocab, \"Tuesday 09 Oct 1993\", num = 7, n_s = 64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the generated plot you can observe the values of the attention weights for each character of the predicted output. Examine this plot and check that where the network is paying attention makes sense to you.\n",
    "\n",
    "In the date translation application, you will observe that most of the time attention helps predict the year, and hasn't much impact on predicting the day/month."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Congratulations!\n",
    "\n",
    "\n",
    "You have come to the end of this assignment \n",
    "\n",
    "<font color='blue'> **Here's what you should remember from this notebook**:\n",
    "\n",
    "- Machine translation models can be used to map from one sequence to another. They are useful not just for translating human languages (like French->English) but also for tasks like date format translation. \n",
    "- An attention mechanism allows a network to focus on the most relevant parts of the input when producing a specific part of the output. \n",
    "- A network using an attention mechanism can translate from inputs of length $T_x$ to outputs of length $T_y$, where $T_x$ and $T_y$ can be different. \n",
    "- You can visualize attention weights $\\alpha^{\\langle t,t' \\rangle}$ to see what the network is paying attention to while generating each output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations on finishing this assignment! You are now able to implement an attention model and use it to learn complex mappings from one sequence to another. "
   ]
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "nlp-sequence-models",
   "graded_item_id": "n16CQ",
   "launcher_item_id": "npjGi"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}