cap12_4_sentiment_analysis_lstm_bert.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "cap12_4_sentiment_analysis_lstm_bert.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMgPK//3v/EI/+3ZWCVHsVJ",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/viniciusrpb/cic0193_machine_learning/blob/main/cap12_4_sentiment_analysis_lstm_bert.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Capítulo 12 - Transformers\n",
        "\n",
        "\n",
        "### Tarefa: classificação de polaridade em tweets\n",
        "\n",
        "Classificar um *tweet* em alguma das três polaridades: positiva (*positive*), negativa (*negative*) e positiva (*neutral*).\n",
        "\n",
        "Fonte (TensorFlow Hub):\n",
        "\n",
        "https://www.tensorflow.org/text/tutorials/classify_text_with_bert\n",
        "\n",
        "### O *corpus*\n",
        "\n",
        "Para isso, vamos precisar utilizar as bibliotecas keras, tensorflow, numpy, sklearn, matplotlib.\n",
        "\n",
        "Fonte: [SemEval-2013 Task 2: Sentiment Analysis in Twitter](https://aclanthology.org/S13-2052/)\n",
        "\n",
        "O *corpus* de tweets empregado nesse estudo de caso podem ser acessados na pasta \"corpus_tweets\" deste repositório. Segue a especificação de cada arquivo:\n",
        "\n",
        "*   ```twitter-2013train-A.txt```: conjunto de tweets para treinamento\n",
        "*   ```twitter-2013dev-A.txt```: conjunto de tweets para validação\n",
        "*   ```twitter-2013test-A.txt```: conjunto de tweets para testes"
      ],
      "metadata": {
        "id": "IipByK9A-0GD"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AuV9oScUNg6G",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2f92a3ef-32e3-48d4-d310-b762cab89126"
      },
      "source": [
        "!pip install -U keras\n",
        "!pip install -U tensorflow\n",
        "!pip install transformers\n",
        "!pip install tensorflow_text\n",
        "!pip install -q -U \"tensorflow-text==2.8.*\""
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: keras in /usr/local/lib/python3.7/dist-packages (2.8.0)\n",
            "Requirement already satisfied: tensorflow in /usr/local/lib/python3.7/dist-packages (2.8.0)\n",
            "Requirement already satisfied: gast>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (0.5.3)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (1.1.0)\n",
            "Collecting tf-estimator-nightly==2.8.0.dev2021122109\n",
            "  Downloading tf_estimator_nightly-2.8.0.dev2021122109-py2.py3-none-any.whl (462 kB)\n",
            "\u001b[K     |████████████████████████████████| 462 kB 5.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: flatbuffers>=1.12 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (2.0)\n",
            "Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (1.21.5)\n",
            "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (3.10.0.2)\n",
            "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (0.24.0)\n",
            "Requirement already satisfied: absl-py>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (1.0.0)\n",
            "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (1.6.3)\n",
            "Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (3.1.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (1.1.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from tensorflow) (57.4.0)\n",
            "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (1.44.0)\n",
            "Requirement already satisfied: libclang>=9.0.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (13.0.0)\n",
            "Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (1.14.0)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (3.3.0)\n",
            "Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (3.17.3)\n",
            "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (0.2.0)\n",
            "Requirement already satisfied: keras<2.9,>=2.8.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (2.8.0)\n",
            "Requirement already satisfied: tensorboard<2.9,>=2.8 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (2.8.0)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow) (1.15.0)\n",
            "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.7/dist-packages (from astunparse>=1.6.0->tensorflow) (0.37.1)\n",
            "Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py>=2.9.0->tensorflow) (1.5.2)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow) (2.23.0)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow) (1.8.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow) (3.3.6)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow) (0.6.1)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow) (0.4.6)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow) (1.35.0)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow) (1.0.1)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow) (4.2.4)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow) (0.2.8)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow) (4.8)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.9,>=2.8->tensorflow) (1.3.1)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<2.9,>=2.8->tensorflow) (4.11.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.9,>=2.8->tensorflow) (3.7.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow) (0.4.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow) (2021.10.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.9,>=2.8->tensorflow) (3.2.0)\n",
            "Installing collected packages: tf-estimator-nightly\n",
            "Successfully installed tf-estimator-nightly-2.8.0.dev2021122109\n",
            "Collecting transformers\n",
            "  Downloading transformers-4.18.0-py3-none-any.whl (4.0 MB)\n",
            "\u001b[K     |████████████████████████████████| 4.0 MB 4.7 MB/s \n",
            "\u001b[?25hCollecting sacremoses\n",
            "  Downloading sacremoses-0.0.49-py3-none-any.whl (895 kB)\n",
            "\u001b[K     |████████████████████████████████| 895 kB 28.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers) (4.11.3)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers) (21.3)\n",
            "Collecting tokenizers!=0.11.3,<0.13,>=0.11.1\n",
            "  Downloading tokenizers-0.11.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 6.5 MB 35.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.6.0)\n",
            "Collecting pyyaml>=5.1\n",
            "  Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)\n",
            "\u001b[K     |████████████████████████████████| 596 kB 12.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.21.5)\n",
            "Collecting huggingface-hub<1.0,>=0.1.0\n",
            "  Downloading huggingface_hub-0.5.1-py3-none-any.whl (77 kB)\n",
            "\u001b[K     |████████████████████████████████| 77 kB 2.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.63.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<1.0,>=0.1.0->transformers) (3.10.0.2)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->transformers) (3.0.7)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers) (3.7.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2021.10.8)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.15.0)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.1.0)\n",
            "Installing collected packages: pyyaml, tokenizers, sacremoses, huggingface-hub, transformers\n",
            "  Attempting uninstall: pyyaml\n",
            "    Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "Successfully installed huggingface-hub-0.5.1 pyyaml-6.0 sacremoses-0.0.49 tokenizers-0.11.6 transformers-4.18.0\n",
            "Collecting tensorflow_text\n",
            "  Downloading tensorflow_text-2.8.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.9 MB)\n",
            "\u001b[K     |████████████████████████████████| 4.9 MB 5.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: tensorflow<2.9,>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow_text) (2.8.0)\n",
            "Requirement already satisfied: tensorflow-hub>=0.8.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow_text) (0.12.0)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (3.3.0)\n",
            "Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (1.21.5)\n",
            "Requirement already satisfied: flatbuffers>=1.12 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (2.0)\n",
            "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (0.2.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (1.1.0)\n",
            "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (0.24.0)\n",
            "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (1.6.3)\n",
            "Requirement already satisfied: tf-estimator-nightly==2.8.0.dev2021122109 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (2.8.0.dev2021122109)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (1.1.2)\n",
            "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (3.10.0.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (57.4.0)\n",
            "Requirement already satisfied: tensorboard<2.9,>=2.8 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (2.8.0)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (1.15.0)\n",
            "Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (1.14.0)\n",
            "Requirement already satisfied: libclang>=9.0.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (13.0.0)\n",
            "Requirement already satisfied: absl-py>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (1.0.0)\n",
            "Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (3.1.0)\n",
            "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (1.44.0)\n",
            "Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (3.17.3)\n",
            "Requirement already satisfied: keras<2.9,>=2.8.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (2.8.0)\n",
            "Requirement already satisfied: gast>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9,>=2.8.0->tensorflow_text) (0.5.3)\n",
            "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.7/dist-packages (from astunparse>=1.6.0->tensorflow<2.9,>=2.8.0->tensorflow_text) (0.37.1)\n",
            "Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py>=2.9.0->tensorflow<2.9,>=2.8.0->tensorflow_text) (1.5.2)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (0.6.1)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (1.8.1)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (0.4.6)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (3.3.6)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (1.35.0)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (1.0.1)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (2.23.0)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (4.2.4)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (0.2.8)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (4.8)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (1.3.1)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (4.11.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (3.7.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (0.4.8)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (2021.10.8)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (1.24.3)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.9,>=2.8->tensorflow<2.9,>=2.8.0->tensorflow_text) (3.2.0)\n",
            "Installing collected packages: tensorflow-text\n",
            "Successfully installed tensorflow-text-2.8.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Importando as classes necessárias a partir das bibliotecas"
      ],
      "metadata": {
        "id": "WWVshYwOMSFz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.model_selection import train_test_split\n",
        "from keras.models import Model,Sequential\n",
        "from keras.layers import Dense,Activation,Input,LSTM,Bidirectional,Dropout,SpatialDropout1D\n",
        "from tensorflow.keras.optimizers import SGD,Adam\n",
        "from sklearn.metrics import classification_report\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import transformers\n",
        "print(transformers.__version__)\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "import tensorflow_text as text\n",
        "from keras.layers.embeddings import Embedding"
      ],
      "metadata": {
        "id": "GETnhj__U_Iy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "84d58b53-a5ea-4fb0-8a23-a5ab2708beb3"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "4.18.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Carregamento do corpus para Pandas DataFrames\n",
        "\n",
        "\n",
        "\n",
        "**PS.:** Mais informações sobre a divisão de um conjunto de dados em treinamento, validação e teste, você encontra nesse [link](https://machinelearningmastery.com/difference-test-validation-datasets/).\n"
      ],
      "metadata": {
        "id": "fp0s4KlqiQUq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "path_train = 'twitter-2013train-A.txt'\n",
        "path_valid = 'twitter-2013dev-A.txt'\n",
        "path_test = 'twitter-2013test-A.txt'"
      ],
      "metadata": {
        "id": "MY8M1KY0TpiW"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_train = pd.read_csv(path_train, names=[\"id\",\"polarity\",\"text\"],encoding='utf8', sep='\\t')\n",
        "df_valid = pd.read_csv(path_valid, names=[\"id\",\"polarity\",\"text\"],encoding='utf8', sep='\\t')\n",
        "df_test = pd.read_csv(path_test, names=[\"id\",\"polarity\",\"text\"],encoding='utf8', sep='\\t')"
      ],
      "metadata": {
        "id": "2MJ0FVjpkEHG"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_train"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "uq2wnaK6fRVh",
        "outputId": "70f4c82f-c277-4132-aae7-90af179ca5f0"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                      id  polarity  \\\n",
              "0     264183816548130816  positive   \n",
              "1     263405084770172928  negative   \n",
              "2     262163168678248449  negative   \n",
              "3     264249301910310912  negative   \n",
              "4     262682041215234048   neutral   \n",
              "...                  ...       ...   \n",
              "9679  103158179306807296  positive   \n",
              "9680  103157324096618497  positive   \n",
              "9681  100259220338905089   neutral   \n",
              "9682  104230318525001729  positive   \n",
              "9683  100461938533863424  negative   \n",
              "\n",
              "                                                   text  \n",
              "0     Gas by my house hit $3.39!!!! I\\u2019m going t...  \n",
              "1     Theo Walcott is still shit\\u002c watch Rafa an...  \n",
              "2     its not that I\\u2019m a GSP fan\\u002c i just h...  \n",
              "3     Iranian general says Israel\\u2019s Iron Dome c...  \n",
              "4     Tehran\\u002c Mon Amour: Obama Tried to Establi...  \n",
              "...                                                 ...  \n",
              "9679  RT @MNFootNg It's monday and Monday Night Foot...  \n",
              "9680  All I know is the road for that Lomardi start ...  \n",
              "9681  All Blue and White fam, we r meeting at Golden...  \n",
              "9682  @DariusButler28   Have a great game agaist Tam...  \n",
              "9683  I'm pisseeedddd that I missed Kid Cudi's show ...  \n",
              "\n",
              "[9684 rows x 3 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f2d7ec92-eb96-4cd4-aa4b-05684dce30c2\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>polarity</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>264183816548130816</td>\n",
              "      <td>positive</td>\n",
              "      <td>Gas by my house hit $3.39!!!! I\\u2019m going t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>263405084770172928</td>\n",
              "      <td>negative</td>\n",
              "      <td>Theo Walcott is still shit\\u002c watch Rafa an...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>262163168678248449</td>\n",
              "      <td>negative</td>\n",
              "      <td>its not that I\\u2019m a GSP fan\\u002c i just h...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>264249301910310912</td>\n",
              "      <td>negative</td>\n",
              "      <td>Iranian general says Israel\\u2019s Iron Dome c...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>262682041215234048</td>\n",
              "      <td>neutral</td>\n",
              "      <td>Tehran\\u002c Mon Amour: Obama Tried to Establi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9679</th>\n",
              "      <td>103158179306807296</td>\n",
              "      <td>positive</td>\n",
              "      <td>RT @MNFootNg It's monday and Monday Night Foot...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9680</th>\n",
              "      <td>103157324096618497</td>\n",
              "      <td>positive</td>\n",
              "      <td>All I know is the road for that Lomardi start ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9681</th>\n",
              "      <td>100259220338905089</td>\n",
              "      <td>neutral</td>\n",
              "      <td>All Blue and White fam, we r meeting at Golden...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9682</th>\n",
              "      <td>104230318525001729</td>\n",
              "      <td>positive</td>\n",
              "      <td>@DariusButler28   Have a great game agaist Tam...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9683</th>\n",
              "      <td>100461938533863424</td>\n",
              "      <td>negative</td>\n",
              "      <td>I'm pisseeedddd that I missed Kid Cudi's show ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>9684 rows × 3 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f2d7ec92-eb96-4cd4-aa4b-05684dce30c2')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-f2d7ec92-eb96-4cd4-aa4b-05684dce30c2 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-f2d7ec92-eb96-4cd4-aa4b-05684dce30c2');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Informações sobre cada parte do *corpus* de tweets:"
      ],
      "metadata": {
        "id": "oXUO88MRwdcY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(f\"O conjunto de treinamento possui {df_train.shape[0]} tweets\")\n",
        "print(f\"O conjunto de validação possui {df_valid.shape[0]} tweets\")\n",
        "print(f\"O conjunto de testes possui {df_test.shape[0]} tweets\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Le_z-K3-wivo",
        "outputId": "7aaf7b71-e86e-45e2-9d43-7c57c93186c1"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "O conjunto de treinamento possui 9684 tweets\n",
            "O conjunto de validação possui 1654 tweets\n",
            "O conjunto de testes possui 3547 tweets\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Pré-processamento dos Tweets\n",
        "\n",
        "Remove o atributo **id**, pois não será necessário o uso desse atributo no processo de classificação.\n",
        "\n"
      ],
      "metadata": {
        "id": "cycRNw7tfZLZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_train.drop(labels=['id'],axis=1)\n",
        "df_valid.drop(labels=['id'],axis=1)\n",
        "df_test.drop(labels=['id'],axis=1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "W4ZbUgNheyGK",
        "outputId": "d920ca29-4a49-4b74-e514-eef41e1b7e9f"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      polarity                                               text\n",
              "0     positive  @jjuueellzz down in the Atlantic city, ventnor...\n",
              "1     positive  Musical awareness: Great Big Beautiful Tomorro...\n",
              "2      neutral  On Radio786 100.4fm 7:10 Fri Oct 19 Labour ana...\n",
              "3     negative  Kapan sih lo ngebuktiin,jan ngomong doang Susa...\n",
              "4      neutral  Excuse the connectivity of this live stream, f...\n",
              "...        ...                                                ...\n",
              "3542  negative  Khaleda Zia's present India visit may have a b...\n",
              "3543   neutral  FYI, golf fans: @jameslepp will join Moj on We...\n",
              "3544  negative  @__Aniko you think mr.Calle let practice with ...\n",
              "3545  positive  Don't hide under your desk! It's just a salsa ...\n",
              "3546   neutral  Saturday flashmob on the music of The Runaways...\n",
              "\n",
              "[3547 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-114b93c9-6a58-418f-9161-35deb8f7d7fd\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>polarity</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>positive</td>\n",
              "      <td>@jjuueellzz down in the Atlantic city, ventnor...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>positive</td>\n",
              "      <td>Musical awareness: Great Big Beautiful Tomorro...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>neutral</td>\n",
              "      <td>On Radio786 100.4fm 7:10 Fri Oct 19 Labour ana...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>negative</td>\n",
              "      <td>Kapan sih lo ngebuktiin,jan ngomong doang Susa...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>neutral</td>\n",
              "      <td>Excuse the connectivity of this live stream, f...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3542</th>\n",
              "      <td>negative</td>\n",
              "      <td>Khaleda Zia's present India visit may have a b...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3543</th>\n",
              "      <td>neutral</td>\n",
              "      <td>FYI, golf fans: @jameslepp will join Moj on We...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3544</th>\n",
              "      <td>negative</td>\n",
              "      <td>@__Aniko you think mr.Calle let practice with ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3545</th>\n",
              "      <td>positive</td>\n",
              "      <td>Don't hide under your desk! It's just a salsa ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3546</th>\n",
              "      <td>neutral</td>\n",
              "      <td>Saturday flashmob on the music of The Runaways...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3547 rows × 2 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-114b93c9-6a58-418f-9161-35deb8f7d7fd')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-114b93c9-6a58-418f-9161-35deb8f7d7fd button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-114b93c9-6a58-418f-9161-35deb8f7d7fd');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Além disso, por questões do uso dos métodos da classe Sequential no Keras-Tensorflow, convertemos o atributo de classe para o tipo categórico e para cada instância, mapeamos seus valores para números inteiros. Lembre-se que os **modelos de aprendizado de máquina processam apenas números inteiros ou reais**."
      ],
      "metadata": {
        "id": "Plz1Pm15ztNO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_train['polarity'] = pd.Categorical(df_train['polarity'])\n",
        "df_valid['polarity'] = pd.Categorical(df_valid['polarity'])\n",
        "df_test['polarity'] = pd.Categorical(df_test['polarity'])\n",
        "\n",
        "y_train = df_train['polarity'].cat.codes\n",
        "y_valid = df_valid['polarity'].cat.codes\n",
        "y_test = df_test['polarity'].cat.codes"
      ],
      "metadata": {
        "id": "4MbE7PZqztrV"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Vamos ver como e repare que o mapeamento do atributo categórico \"polarity\" para valores categóricos inteiros ficou como:\n",
        "\n",
        "*   polaridade \"negativa\" *(negative)*: valor $0$\n",
        "*   polaridade \"neutra\" *(neutral)*: valor $1$\n",
        "*   polaridade \"positiva\" *(positive)*: valor $2$"
      ],
      "metadata": {
        "id": "soW04sEVto_b"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for i in range(0,5):\n",
        "    v = df_train['polarity'][i]\n",
        "    print(f'Valor original da classe {v} Transformação inteira: {y_train[i]}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4LxkFq1ttpRZ",
        "outputId": "ea68c038-0ac1-4fc2-e021-d6ee4be1b0be"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Valor original da classe positive Transformação inteira: 2\n",
            "Valor original da classe negative Transformação inteira: 0\n",
            "Valor original da classe negative Transformação inteira: 0\n",
            "Valor original da classe negative Transformação inteira: 0\n",
            "Valor original da classe neutral Transformação inteira: 1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "A seguir, transformamos os valores da classe \"polaridade\", nossa classe de interesse, para one-hot-encoding. Nessa representação, como temos três classes, utilizaremos três bits, em que para cada instância, apenas um deles estará ativado (com valor $1$) com base na presença do valor classe.\n",
        "\n",
        "Veja como fica a representação one-hot encoding para cada classe:\n",
        "\n",
        "*   polaridade \"negativa\" *(negative)*: valor $0$, one hot encoding: $[1,0,0]$\n",
        "*   polaridade \"neutra\" *(neutral)*: valor $1$, one hot encoding: $[0,1,0]$\n",
        "*   polaridade \"positiva\" *(positive)*: valor $2$, one hot encoding: $[0,0,1]$\n",
        "\n",
        "Utilizaremos a função *to_categorical* do Keras para esse propósito (se bem que poderíamos facilmente implementar uma por nossa conta):"
      ],
      "metadata": {
        "id": "qRN-0SW50IG0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from keras.utils.np_utils import to_categorical\n",
        "\n",
        "num_classes = 3\n",
        "\n",
        "y_train_enc = to_categorical(y_train,3)\n",
        "y_valid_enc = to_categorical(y_valid,3)\n",
        "y_test_enc = to_categorical(y_test,3)"
      ],
      "metadata": {
        "id": "bZqJdMXW0INV"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Apenas verificando como ficou o pré-processamento do atributo de classe:"
      ],
      "metadata": {
        "id": "ONGb3W5ky3Sv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for i in range(0,5):\n",
        "    v = df_train['polarity'][i]\n",
        "    print(f\"Tweet {i}: valor original da classe '{v}'; mapeamento para inteiro: {y_train[i]}; representação one-hot-encoding [{int(y_train_enc[i][0])},{int(y_train_enc[i][1])},{int(y_train_enc[i][2])}]\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4qWlGYzJxb-s",
        "outputId": "055dcdc5-2be8-4dd3-8a24-bc536eeb31bb"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tweet 0: valor original da classe 'positive'; mapeamento para inteiro: 2; representação one-hot-encoding [0,0,1]\n",
            "Tweet 1: valor original da classe 'negative'; mapeamento para inteiro: 0; representação one-hot-encoding [1,0,0]\n",
            "Tweet 2: valor original da classe 'negative'; mapeamento para inteiro: 0; representação one-hot-encoding [1,0,0]\n",
            "Tweet 3: valor original da classe 'negative'; mapeamento para inteiro: 0; representação one-hot-encoding [1,0,0]\n",
            "Tweet 4: valor original da classe 'neutral'; mapeamento para inteiro: 1; representação one-hot-encoding [0,1,0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Extração de características dos tweets\n",
        "\n",
        " Bidirectional Encoder Representations from Transformers (BERT). The BERT architecture is composed of several Transformer encoders stacked together. Further, each Transformer encoder is composed of two sub-layers: a feed-forward layer and a self-attention layer.\n",
        "\n",
        " Primeiramente, fazemos os downloads do modelo BERT de referência:\n"
      ],
      "metadata": {
        "id": "Q_mzac_gfstB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "bert_model_name = 'bert_en_uncased_L-12_H-768_A-12'\n",
        "tfhub_handle_encoder = 'https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/3'\n",
        "tfhub_handle_preprocess = 'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3'\n",
        "\n",
        "print(f'BERT model selected           : {tfhub_handle_encoder}')\n",
        "print(f'Preprocess model auto-selected: {tfhub_handle_preprocess}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aDstcaHEZtrR",
        "outputId": "4cf6c9e9-1362-4f57-deb2-3d5ded2b5035"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "BERT model selected           : https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/3\n",
            "Preprocess model auto-selected: https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "bert_preprocess = hub.KerasLayer(tfhub_handle_preprocess)\n",
        "bert_encoder = hub.KerasLayer(tfhub_handle_encoder)"
      ],
      "metadata": {
        "id": "hsvvNT4EPHmM"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Colocamos as instâncias (juntamente com seus vetores BERT) nas respectivas matrizes $X$ (para cada tipo de dados), em que cada linha de $X$ está associada a um *tweet*. Repare que as classes (labels, rótulos etc) já foram separadas anteriormente e estão armazenadas nas variáveis ```y```. Por exemplo, a instância de treinamento $\\mathbf{x_i}$, denotada por ```X_train[i]``` é da classe ```y_train[i]``` $\\in \\{0,1,2\\}$."
      ],
      "metadata": {
        "id": "GGw9jEA2gCTn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X_train = df_train['text'].values\n",
        "X_valid = df_valid['text'].values\n",
        "X_test = df_test['text'].values"
      ],
      "metadata": {
        "id": "kXNL8-WsT6wl"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X_train[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "PssUon-vRx-t",
        "outputId": "1aa29338-5e3d-4c02-a95e-dfb0830a7539"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Gas by my house hit $3.39!!!! I\\\\u2019m going to Chapel Hill on Sat. :)'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sample = [X_train[0]]\n",
        "text_preprocessed = bert_preprocess(sample)\n",
        "\n",
        "print(f'Keys       : {list(text_preprocessed.keys())}')\n",
        "print(f'Shape      : {text_preprocessed[\"input_word_ids\"].shape}')\n",
        "print(f'Word Ids   : {text_preprocessed[\"input_word_ids\"][0, :12]}')\n",
        "print(f'Input Mask : {text_preprocessed[\"input_mask\"][0, :12]}')\n",
        "print(f'Type Ids   : {text_preprocessed[\"input_type_ids\"][0, :12]}')\n",
        "\n",
        "bert_model = hub.KerasLayer(tfhub_handle_encoder)\n",
        "\n",
        "bert_results = bert_model(text_preprocessed)\n",
        "\n",
        "print(f'Loaded BERT: {tfhub_handle_encoder}')\n",
        "print(f'Pooled Outputs Shape:{bert_results[\"pooled_output\"].shape}')\n",
        "print(f'Pooled Outputs Values:{bert_results[\"pooled_output\"][0, :12]}')\n",
        "print(f'Sequence Outputs Shape:{bert_results[\"sequence_output\"].shape}')\n",
        "print(f'Sequence Outputs Values:{bert_results[\"sequence_output\"][0, :12]}')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ml-OM3S4b1kQ",
        "outputId": "74a2dba2-dba2-4638-ea45-870b0a9261d8"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Keys       : ['input_type_ids', 'input_mask', 'input_word_ids']\n",
            "Shape      : (1, 128)\n",
            "Word Ids   : [ 101 3806 2011 2026 2160 2718 1002 1017 1012 4464  999  999]\n",
            "Input Mask : [1 1 1 1 1 1 1 1 1 1 1 1]\n",
            "Type Ids   : [0 0 0 0 0 0 0 0 0 0 0 0]\n",
            "Loaded BERT: https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/3\n",
            "Pooled Outputs Shape:(1, 768)\n",
            "Pooled Outputs Values:[-0.8057022  -0.44450736 -0.55610144  0.6081395   0.03992073 -0.09596545\n",
            "  0.69608736  0.33400092 -0.37008688 -0.9999634   0.11522834  0.7892676 ]\n",
            "Sequence Outputs Shape:(1, 128, 768)\n",
            "Sequence Outputs Values:[[-0.15877226  0.1460731   0.43972343 ... -0.31484985  0.3212449\n",
            "   0.31777155]\n",
            " [ 0.65560526 -0.18834352  1.0243313  ... -0.44978327  0.01051322\n",
            "   0.29176   ]\n",
            " [ 0.13443717 -0.35418344  1.087825   ... -0.7229347  -0.14338884\n",
            "   0.23410942]\n",
            " ...\n",
            " [-0.07321975  0.24194658  0.11324732 ... -0.40200403  0.5916259\n",
            "  -0.3293868 ]\n",
            " [ 0.25970036  0.30408114  1.3485484  ...  0.17821038 -0.11198954\n",
            "  -0.10162394]\n",
            " [ 0.46989918  0.29996172  1.5466263  ... -0.04923994 -0.27422243\n",
            "   0.04177291]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "vocab_size = 128"
      ],
      "metadata": {
        "id": "n5Qal1wGGk8j"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Modelo de Classificação: BERT Input + LSTM + Sigmoid Function\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "TSV_ATefiiUB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "num_neurons = 64\n",
        "num_embeddings = 128"
      ],
      "metadata": {
        "id": "YgW5-c3z1Zrg"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "text_input = Input(shape=(), dtype=tf.string, name='text')\n",
        "preprocessing_layer = hub.KerasLayer(tfhub_handle_preprocess, name='preprocessing')\n",
        "encoder_inputs = preprocessing_layer(text_input)\n",
        "encoder = hub.KerasLayer(tfhub_handle_encoder, trainable=True, name='BERT_encoder')\n",
        "outputs = encoder(encoder_inputs)\n",
        "net = outputs['pooled_output']\n",
        "model = Model(text_input, net)"
      ],
      "metadata": {
        "id": "LvWB8hO2jGCf"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "inputs = model.inputs[:2]\n",
        "bert_out = model.output\n",
        "newReshape = tf.keras.layers.Reshape((768,1), name='newReshape')(bert_out)\n",
        "#bert_out.shape"
      ],
      "metadata": {
        "id": "uab6wXbS6-8n"
      },
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "lstm = LSTM(256)(newReshape)\n",
        "newout = Dense(768,activation='relu')(lstm)\n",
        "preds = Dense(num_classes, activation='sigmoid')(newout)\n",
        "model = Model(inputs, preds)"
      ],
      "metadata": {
        "id": "HOWc-Pzu69-X"
      },
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "O ajuste do modelo para o treinamento consiste em utilizarmos um otimizador da classe SGD (*Stochastic Gradient Descent*), em que colocamos aleatoriamente um valor da taxa de aprendizado ```learning_rate=0.001```.\n",
        "\n",
        "Outra alteração se trata da função *loss* (que calcula o erro da modelo). Como temos um problema de classificação multiclasse, temos que empregar a função ```categorical_crossentropy```."
      ],
      "metadata": {
        "id": "peU5AQYB0Rc6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#sgd = SGD(learning_rate=3e-5)\n",
        "\n",
        "adam = Adam(learning_rate=0.00001,decay = 1e-6)\n",
        "\n",
        "model.compile(loss=\"categorical_crossentropy\",optimizer=adam,metrics=['accuracy'])\n",
        "history_fine = model.fit(X_train,y_train_enc,validation_data=(X_valid,y_valid_enc),epochs=10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T95n0SaLzpj1",
        "outputId": "a8198bcb-7a74-403a-c84b-6da05564b6be"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "303/303 [==============================] - 606s 2s/step - loss: 0.9471 - accuracy: 0.6301 - val_loss: 0.7768 - val_accuracy: 0.6524\n",
            "Epoch 2/10\n",
            "303/303 [==============================] - 581s 2s/step - loss: 0.6377 - accuracy: 0.7244 - val_loss: 0.7282 - val_accuracy: 0.6524\n",
            "Epoch 3/10\n",
            "303/303 [==============================] - 581s 2s/step - loss: 0.5219 - accuracy: 0.7605 - val_loss: 0.7426 - val_accuracy: 0.6499\n",
            "Epoch 4/10\n",
            "303/303 [==============================] - 581s 2s/step - loss: 0.4304 - accuracy: 0.8052 - val_loss: 0.7921 - val_accuracy: 0.7128\n",
            "Epoch 5/10\n",
            "303/303 [==============================] - 581s 2s/step - loss: 0.3522 - accuracy: 0.8869 - val_loss: 0.8244 - val_accuracy: 0.7316\n",
            "Epoch 6/10\n",
            "303/303 [==============================] - 581s 2s/step - loss: 0.2883 - accuracy: 0.9153 - val_loss: 0.8650 - val_accuracy: 0.7364\n",
            "Epoch 7/10\n",
            "303/303 [==============================] - 580s 2s/step - loss: 0.2215 - accuracy: 0.9327 - val_loss: 0.8485 - val_accuracy: 0.7455\n",
            "Epoch 8/10\n",
            "303/303 [==============================] - 581s 2s/step - loss: 0.1753 - accuracy: 0.9475 - val_loss: 0.9708 - val_accuracy: 0.7261\n",
            "Epoch 9/10\n",
            "303/303 [==============================] - 581s 2s/step - loss: 0.1485 - accuracy: 0.9576 - val_loss: 0.9023 - val_accuracy: 0.7527\n",
            "Epoch 10/10\n",
            "303/303 [==============================] - 580s 2s/step - loss: 0.1199 - accuracy: 0.9673 - val_loss: 1.0028 - val_accuracy: 0.7497\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Vamos analisar o comportamento da função *loss* no decorrer das épocas:"
      ],
      "metadata": {
        "id": "SVYG8Cut07Zi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "loss = history_fine.history['loss']\n",
        "val_loss = history_fine.history['val_loss']\n",
        "\n",
        "\n",
        "plt.plot(history_fine.history['loss'])\n",
        "plt.plot(history_fine.history['val_loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['training set','validation set'], loc='upper left')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "sUDT-dn41VLH",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "945c414d-b0ab-422e-8ff1-573947de6051"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deVxVdf748debXVaVTQUFVFLAXdyX3DLbNCtbrWyzsbKa6etMzVbTd5rp961ps9I022ayxiyXytJcc881UyRBRQUFARcQBVk+vz/OFXFBUbkc4L6fjwcPuOeee+6bq5z3+Szn/RFjDEoppVyXm90BKKWUspcmAqWUcnGaCJRSysVpIlBKKReniUAppVycJgKllHJxmgiUqiIR+UhE/l7FfdNEZMiVHkepmqCJQCmlXJwmAqWUcnGaCFS94uiSmSAiW0SkQESmiUi4iHwnIvkislBEGlXYf7iIbBORIyKyVETiKjzXWUQ2Ol73X8DnrPe6UUQ2O167SkQ6XGbMj4hIqogcEpG5ItLMsV1E5HUROSgieSLyi4i0czx3vYgkOWLLEJH/uawPTCk0Eaj66VbgGuAq4CbgO+CPQCjW//knAUTkKuAz4GnHc/OAr0XES0S8gNnAv4HGwBeO4+J4bWfgA+BRIBh4D5grIt6XEqiIDAL+CdwONAX2AJ87nh4K9Hf8HkGOfXIdz00DHjXGBADtgMWX8r5KVaSJQNVHE40xWcaYDGA5sNYYs8kYUwjMAjo79rsD+NYY84Mxphh4FWgA9AZ6Ap7AG8aYYmPMTGBdhfcYC7xnjFlrjCk1xnwMFDledynuAT4wxmw0xhQBzwG9RCQaKAYCgLaAGGO2G2MOOF5XDMSLSKAx5rAxZuMlvq9S5TQRqPooq8LPJ87z2N/xczOsK3AAjDFlwD4gwvFchjmzKuOeCj9HAc84uoWOiMgRoLnjdZfi7BiOYV31RxhjFgNvA+8AB0VkiogEOna9Fbge2CMiy0Sk1yW+r1LlNBEoV7Yf64QOWH3yWCfzDOAAEOHYdkqLCj/vA14yxjSs8OVrjPnsCmPww+pqygAwxrxljOkKxGN1EU1wbF9njBkBhGF1Yc24xPdVqpwmAuXKZgA3iMhgEfEEnsHq3lkFrAZKgCdFxFNEbgG6V3jtVOA3ItLDMajrJyI3iEjAJcbwGfCAiHRyjC/8A6srK01EujmO7wkUAIVAmWMM4x4RCXJ0aeUBZVfwOSgXp4lAuSxjzK/AaGAikIM1sHyTMeakMeYkcAswBjiENZ7wVYXXrgceweq6OQykOva91BgWAn8BvsRqhbQC7nQ8HYiVcA5jdR/lAq84nrsXSBORPOA3WGMNSl0W0YVplFLKtWmLQCmlXJwmAqWUcnGaCJRSysVpIlBKKRfnYXcAlyokJMRER0fbHYZSStUpGzZsyDHGhJ7vuTqXCKKjo1m/fr3dYSilVJ0iInsqe067hpRSysVpIlBKKReniUAppVxcnRsjOJ/i4mLS09MpLCy0OxR1Fh8fHyIjI/H09LQ7FKVUJepFIkhPTycgIIDo6GjOLBap7GSMITc3l/T0dGJiYuwORylViXrRNVRYWEhwcLAmgVpGRAgODtaWmlK1nNMSgYh84FhrdWslz4uIvOVYq3WLiHS5wve7kpcrJ9F/F6VqP2e2CD4Chl3g+euAWMfXWGCSE2NRSqm66+Rx+OGvcGSvUw7vtERgjPkRq457ZUYAnxjLGqChiDR1VjzOdOTIEd59993Leu3111/PkSNHLrjPX//6VxYuXHhZx78Ss2fPJikpqcbfVylVwZ7VMLkPrHwTUhY45S3sHCOIwFru75R0x7ZziMhYEVkvIuuzs7NrJLhLcaFEUFJScsHXzps3j4YNG15wnxdffJEhQ4ZcdnyXSxOBUjY6eRy+fw4+vA7KSuH+r6Hbw055qzoxWGyMmWKMSTTGJIaGnrdUhq2effZZdu7cSadOnZgwYQJLly6lX79+DB8+nPj4eABuvvlmunbtSkJCAlOmTCl/bXR0NDk5OaSlpREXF8cjjzxCQkICQ4cO5cSJEwCMGTOGmTNnlu///PPP06VLF9q3b09ycjIA2dnZXHPNNSQkJPDwww8TFRVFTk7OGXGWlpYyZswY2rVrR/v27Xn99dcB2LlzJ8OGDaNr167069eP5ORkVq1axdy5c5kwYQKdOnVi586dTv8clVIOe1ZZrYA171on/3GrIKa/097OzumjGVgLhZ8S6dh2Rf729TaS9udd6WHOEN8skOdvSqj0+ZdffpmtW7eyefNmAJYuXcrGjRvZunVr+bTJDz74gMaNG3PixAm6devGrbfeSnBw8BnHSUlJ4bPPPmPq1KncfvvtfPnll4wePfqc9wsJCWHjxo28++67vPrqq7z//vv87W9/Y9CgQTz33HN8//33TJs27ZzXbd68mYyMDLZutcbvT3VJjR07lsmTJxMbG8vatWt57LHHWLx4McOHD+fGG2/ktttuu7wPTil1aU4WwKL/hbWToWELuP8biOnn9Le1MxHMBZ4Qkc+BHsBRY8wBG+OpVt27dz9j7vxbb73FrFmzANi3bx8pKSnnJIKYmBg6deoEQNeuXUlLSzvvsW+55Zbyfb76ylpGd8WKFeXHHzZsGI0aNTrndS1btmTXrl2MHz+eG264gaFDh3Ls2DFWrVrFqFGjyvcrKiq6zN9aKXXZ9qyC2Y/B4d3QfSwMfh68/WvkrZ2WCETkM2AAECIi6cDzgCeAMWYyMA+4HmvR7+PAA9Xxvhe6cq9Jfn5+5T8vXbqUhQsXsnr1anx9fRkwYMB559Z7e3uX/+zu7l7eNVTZfu7u7hcdg6ioUaNG/Pzzz8yfP5/JkyczY8YM3njjDRo2bFjemlFK1bCTBbDoRVj7HjSKqrFWQEVOSwTGmLsu8rwBHnfW+9ekgIAA8vPzK33+6NGjNGrUCF9fX5KTk1mzZk21x9CnTx9mzJjBH/7wBxYsWMDhw4fP2ScnJwcvLy9uvfVW2rRpw+jRowkMDCQmJoYvvviCUaNGYYxhy5YtdOzY8aK/l1JOdWgXFBdCeLzdkThP2kqY87ijFfAoDHkevPwu/rpqVicGi2u74OBg+vTpQ7t27ZgwYcI5zw8bNoySkhLi4uJ49tln6dmzZ7XH8Pzzz7NgwQLatWvHF198QZMmTQgICDhjn4yMDAYMGECnTp0YPXo0//znPwH49NNPmTZtGh07diQhIYE5c+YAcOedd/LKK6/QuXNnHSxWNStpDkzqC5N6wcfDIWUhGGN3VNXnZAHM+z18dD1gYMy3cP3/2ZIEAMTUsQ83MTHRnL0wzfbt24mLi7MpotqhqKgId3d3PDw8WL16NePGjas13T3676OqrKwMlv4DfnwFIhKhzXWw7n3IPwBh8dDrCWh/G3h4X/xYtVXaCkcrIA16/AYG/7VGEoCIbDDGJJ7vuXpRdE7B3r17uf322ykrK8PLy4upU6faHZJSl6YwD74aCzu+g06j4YZ/gacP9H4Stn4JqybCnMes/vQej0LiA9Dg3EkRtVbRMVj0N/hpCjSKgTHzILqP3VEBmgjqjdjYWDZt2mR3GEpdnpxU+PxuyE2F6/7PmjVzqk6Vhxd0ugs63gk7F1kJYdHf4MdXoct90HOcNcham+1ebrUCjuyp0VZAVblUIiguLcPTXYdFlKpVUn6AmQ+BmzvcN7vyG6dEoPUQ6+vAFlj9Nqybal1hJ9xsdRtFXFHtyupXdAwWvmDFWctaARW5zFnxYF4hOzLzKSktszsUpRRYg78rXodPR0HD5jB2adXvnm3aAW6ZAk/9DL0egx0LYOpA+OhG2DHfGmuw2+4fYVJva4yj52PW3cG1MAmACyWCgAaelBpDTsFJu0NRSp08Dl8+ZF0tx4+AhxZcXvdOUCQM/Tv8bpv1/dAumH47vNsTNn5iTT+taUXH4Ntn4OObrFbOA/Ng2D/By7fmY6kil0kEDTzdCWrgSe6xIkprw9WCUq7qyF744FrY+pXVVz7qoyvvL/cJgt7jrRbCLVOtcYW54+GN9tYMpOMXKoRcjXb/aE15XTfNagX8ZiVE9a6Z974CLpMIAMICvCktM+Qes79V4O9v3Tq+f//+Smv5DBgwgLOnyp7tjTfe4Pjx4+WPq1LWurqlpaUxffr0Gn1PVUelrYApA6ypk3f/F/o9c3pQuDq4e0KH2+HR5XDfHKsLafHf4fUEmDcBDu2uvveqqOgYfPM7RyvAEx74rta3AipyqUTQwMuDQB9Pco4VUVpWO+6faNasWXll0ctxdiKoSlnr6qaJQF2UMfDTVPhkBDRoDI8shquudd77iUDLATD6S6tvPmEkrP8QJnaBGfdB+oUvsC7JrmVWK2D9B9DzcfjNCojqVX3HrwEulQjAahWUlBkOFVRfYbVnn32Wd955p/zxCy+8wKuvvsqxY8cYPHhwecnoU3fsVpSWlka7du0AOHHiBHfeeSdxcXGMHDnyjFpD48aNIzExkYSEBJ5//nnAKmS3f/9+Bg4cyMCBA4HTZa0BXnvtNdq1a0e7du144403yt+vsnLXFX3xxRe0a9eOjh070r+/NYBXWlrKhAkT6NatGx06dOC9994r//2XL19Op06dyktbK1WupMjqppn3P9BqMDyyCEJia+79wxPg5nfh6V+gz1Owcym8Pxg+GAbJ317+wHJRPnzzW/hkuNUKePB7GPaPOtMKqKj+TR/97lnI/KXSp32B2OJSyozBeLkjVKFZ2qQ9XPdypU/fcccdPP300zz+uFU6acaMGcyfPx8fHx9mzZpFYGAgOTk59OzZk+HDh1e6ju+kSZPw9fVl+/btbNmyhS5dTk+Fe+mll2jcuDGlpaUMHjyYLVu28OSTT/Laa6+xZMkSQkJCzjjWhg0b+PDDD1m7di3GGHr06MHVV19No0aNqlTu+sUXX2T+/PlERESUdzVNmzaNoKAg1q1bR1FREX369GHo0KG8/PLLvPrqq3zzzTcX/yyVa8nPhP/eC+k/Wd1AA/9kDaDaIbApDHnBimPTf2D1u9a9C8Gtodfj0PEu8GxQtWPtWgpzxsPRfda01YF/qpMJ4BSXaxEAeHm4YQyUlFZP91Dnzp05ePAg+/fv5+eff6ZRo0Y0b94cYwx//OMf6dChA0OGDCEjI4OsrKxKj/Pjjz+Wn5A7dOhAhw4dyp+bMWMGXbp0oXPnzmzbtu2iK4etWLGCkSNH4ufnh7+/P7fccgvLly8Hqlbuuk+fPowZM4apU6dSWloKwIIFC/jkk0/o1KkTPXr0IDc3l5SUlEv6rJQLSd9gjQdkbYXbPrQGhu1KAhV5B1g3oT25CW77ALz8rSv71xNg6ctQkFP5a4vy4eunrS4uDy94cD5c+1KdTgJQH1sEF7hyP8UdyMo+xsmSMtqEB+DmduWDVaNGjWLmzJlkZmZyxx13AFYxt+zsbDZs2ICnpyfR0dHnLT99Mbt37+bVV19l3bp1NGrUiDFjxlzWcU6pSrnryZMns3btWr799lu6du3Khg0bMMYwceJErr32zL7dpUuXXnYsqp7aPN06YfqHW1NDm7S3O6JzuXtAu1sh4RbYs9K6Y3npP617GzrdbV3pB7c6vf/OJVYX19F067lBf656C6KWc8kWAUB4gDfFpWUcPl49M4juuOMOPv/8c2bOnFm+yMvRo0cJCwvD09OTJUuWsGfPngseo3///uWDrlu3bmXLli0A5OXl4efnR1BQEFlZWXz33Xflr6msVHS/fv2YPXs2x48fp6CggFmzZtGvX9VrnO/cuZMePXrw4osvEhoayr59+7j22muZNGkSxcXFAOzYsYOCggItV61OKy2x1tmdPQ6ad7duEquNSaAiEYjua81ievwna9bRpv/AxK7w+T1WN9DXT8O/b7aK3Z1qBdSTJAD1sUVQRX7eHvh6eZCdX0QjPy/crnAKW0JCAvn5+URERNC0aVMA7rnnHm666Sbat29PYmIibdu2veAxxo0bxwMPPEBcXBxxcXF07doVgI4dO9K5c2fatm1L8+bN6dPn9N2JY8eOZdiwYTRr1owlS5aUb+/SpQtjxoyhe/fuADz88MN07ty50lXPzjZhwgRSUlIwxjB48GA6duxIhw4dSEtLo0uXLhhjCA0NZfbs2XTo0AF3d3c6duzImDFj+O1vf3spH52qL44fgi/GwO5lVj2doX+3pnPWJaFtYPhEGPhnqyzEuvch+RtArPsUBv6pXiWAU1y6DHVeYTFpOQVENmpAY786XNa2ltMy1C4gaxt8dpdVLvrG16HzuWtt10knC2D719Ysp4iudkdzRbQMdSUCvD1o4OXOwfwiGvl6VTqbRyl1AUlzYNY4axB2zDxo3s3uiKqPl59V9bSec9kxAgARISzAh5MlZRw5Xmx3OErVLWVl1l27M+6DsDhrPKA+JQEXUm9aBMaYy7qiD/TxwMfTahU09PXUVkE1q2tdj6qKKltERtVJ9aJF4OPjQ25u7mWddKxWgTdFJaUcPaGtgupkjCE3NxcfHz1B1Cs5qfD+EEhZANe9AiPe1iRQx9WLFkFkZCTp6elkZ2df1uuNgUP5hRzKgLAAn2qtgeXqfHx8iIyMtDsMVV1SFsLMBy++iIyqU+pFIvD09CQmJuaKjrFjUwZP/3czk0d3ZVi7JtUUmVL1hDGw8g1Y+Derds+d02v/8pCqyupFIqgON3ZoyhsLdzBxcQrXJoTrWIGqOcbAwSTrztWsbeAXAoERENjs9Hf/MPvKM5w8DnOfsBaQTxgJI96pVevtqiunicDBw92Nxwa25vczt7Dk14MMahtud0iqPss7ALuWWHet7loKxxw1qPzD4cRhKD3rjndxh4CmjuRQIUFU/DmgSfXfwHVkr3V3beYvVq2gvr+r3vUDVK2giaCCkZ0jeGtRCm8tSmVgmzBtFajqU3TMqmezc4mVALKTre2+IVbd/FYDre9BkVYL4Xgu5GVA3v4K3x0/Z2211uUtObtGlFiJ5JxkUTFpNLPKJFRF2gpramhpsVV+wZnrByhbaSKowNPdjXEDWvGnWVtZkZpDv9hQu0NSdVVpCezfZJ30dy6xyjCXlYCHj7V0Yad7rBN/eDtwO2vynojVPeQXAk07nv/4xkDhkTMTRMXvuanWsolFeee+1jek8lbFqe+bP4Xvn4VGMXDXZzW7foCqcZoIznJb10jeXpzKxEWpmghU1RljLZy+c7HV1bN7ORQdBcQ6mfd6wrrqb96zeqZaikCDRtZXeELl+xXmWWUfzm5V5O23aunvW2N1RZ1P7LVw61RrPWBVr2kiOIu3hzuP9m/JC18nsWZXLj1bBtsdkqqtCnJh91JHd88yOLrX2h7UAhJGQMuBEHM1+Nn4f8gn0PoKbVP5PiePn5UsMqwE0+X+2rF+gHI6TQTncWf3Fry9ZCcTF6doIlCnFRfC3tWOAd4lcGALYMA7CGL6Qd+nrJN/45Z1a0DVy9equ1+x9r5yKZoIzsPH02oVvDRvOxv2HKZrVCO7Q1J2KCuzBmZP9fPvXQ0lhdb6tM27WyWJWw2Epp2sRU6UqqP0f28l7unZgknLrFbBRw90tzscVVOOpp+e2bNrGRx3LFsYGgeJD1oDvFF9wNvfziiVqlaaCCrh6+XBQ31jeGX+r2xJP0KHyIZ2h6Sq26kB3owNsO8nq8sn17EGs384tB5infhbDrAWPleqntJEcAH39YrivWU7eXtxKlPuO+96DqouOX7IOumnr4eM9dbPp2bMePpZ0zoTH7D6+cPi6lY/v1JXQBPBBQT4ePJAnxjeXJTC9gN5xDUNtDskVVUlRdbdsOmOE37GeuvqHwCxTvRtb4TIRIhIhNC22s+vXJZT/+eLyDDgTcAdeN8Y8/JZz7cAPgYaOvZ51hgzz5kxXaoH+8QwbcVu3l6Syjt3d7E7HHU+Fbt4Tl3tZ/5yukxDQFNrmcEu91kn/WadrNW0lFKAExOBiLgD7wDXAOnAOhGZa4xJqrDbn4EZxphJIhIPzAOinRXT5Qjy9eS+XlFMWraT1IP5tA7TE4jtLtbF06wz9BxnnfwjEiEowt54larlnNki6A6kGmN2AYjI58AIoGIiMMCp/pYgYL8T47lsD/WN4cOVabyzZCev39HJ7nBci3bxKOV0zvyLiQD2VXicDvQ4a58XgAUiMh7wA4Y4MZ7LFuzvzeieLZi2YjdPDY4lOkRL8DqFdvEoZQu7L53uAj4yxvxLRHoB/xaRdsaYsoo7ichYYCxAixYtbAgTHunfkk9W7+Hdpan8322VFAJTl6a02KpwuXeNdvEoZSNnJoIMoHmFx5GObRU9BAwDMMasFhEfIAQ4WHEnY8wUYApAYmKiLauhhwX4cFf3FvxnzR7GD4qleWNfO8Ko+0qKrBu2kubAr99CoaMwm3bxKGUbZ/6lrQNiRSQGKwHcCdx91j57gcHARyISB/gAl7fwcA149OqWTF+7l8nLdvLSyPZ2h1N3FJ+A1IWOk//3cDLfqmjZ5nqIG27V6dEuHqVs47REYIwpEZEngPlYU0M/MMZsE5EXgfXGmLnAM8BUEfkt1sDxGGOMc674S0usSopXcJNQ06AG3JYYyRfr0xk/KJYmQdVQTri+KjoGqT9YJ/8dC6C4ABo0hoSbIf5ma9FzDy+7o1RKAeKs866zJCYmmvXr11/6Czd+AstegbibrK/m3S+rxO6+Q8cZ+OpSRveM4oXhF6gD74oK86yVs5JmWy2AkkLwC7U+7/gRENVXu3uUsomIbDDGnLdEguv8VQY1h/B4WDcV1rxj1ZJpe4N1koruV+W1Xps39mVk5wg++2kvjw1sRViAi7cKThyGX7+zrvx3LrZm+AQ0tWb2xI+AFr20pr1StZzrtAhOKcxzdFnMhZQfrC4Ln4aO/uqboNWgi64glZZTwKB/LeXhfi354/Vxlx9LXVWQaw30Js2xCrWVlUBgpHXijx8Bkd3OXX5RKWUrbRFU5BMI7W61vopPWFex27+2Tmw/Twcvf4i9xhrEjL3mvIOY0SF+DO/YjP+s2cNvrm5FYz8X6Os+dtD6nJLmWFM+TSk0ioZej1sn/2ZdtEibUnWU6yWCijwbWN1DbW+w5rTv/tE62SV/A9tmgbu31UKIHw5XDQPfxuUvfWJQa+b8vJ9pK3Yx4dq2Nv4STpS333Hynwt7VgIGgltD36etk3+TDnryV6oecL2uoaooK4V9a62T4PavrUW+xd2a5hg33JrvHhDO459uZNmObFb+YRBBvlUbY6j1juyD7XOtK/99a61toXGnu320PLNSddKFuoY0EVyMMbB/kyMpzIXcVECgRU+yIq7hliXBjBrSm6eHXFVzMVW3Q7usq/6kObB/o7WtSXvrxB83AkLr8O+mlAI0EVQfYyA72Tppbv8asn4BYBstad3/Lrw7jISQWHtiq6qSIjiWBfmZsHuZ9btkbrGea9bFceU/3FqAXSlVb2gicJZDuziw5gsOrJlBF7dUa1toW6v7KO4m66q6JrpRjLGmcR47CMcyHd8dJ/uK2/IzofDIma+N7O648r8JGkU5P1allC00ETjZmA9/4sC+XXw95DBeO761BlZNmTWrJu4mKzFEJF76lMqSk1BwEPKzrBP7GSf5U9scX6cqdFbk4WPdLxHQBPzDwL+J43G49b1JewhsVi2fgVKqdtPpo042flAst07K5uOS7jwyZhwU5MCv86zuozWTYdVE6yartjeebikUZJ//hF5x24lD539D32DHST3M6ooqP8mHOU764dbP3oE6sKuUuihtEVSTe95fw6+Zx1jxh4H4eFa4k7bwqFVrZ/scSFkIJSfOfwB379NX6qe+zriSDzt9gq/iXdBKKXWKtghqwPhBsdw5ZQ2f/7SXMX1iTj/hEwQdRllfJ4/DzkVweM+ZXTT+4dZ+evWulLKBJoJq0rNlMN2jGzN52S7u6tECb4/z1Nfx8rW6hpRSqhbRgjDVaPzg1mTmFTJzQ7rdoSilVJVpIqhGfVuH0Kl5QyYt3UlxadnFX6CUUrWAJoJqJCI8Obg16YdPMGvT2atyKqVU7aSJoJoNbBNGu4hA3l2SSom2CpRSdYAmgmomIjwxMJa03ON8s+WA3eEopdRFaSJwgqHx4bQJD+DtJamUldWt+zSUUq5HE4ETuLkJTwxqTerBY3y3NdPucJRS6oI0ETjJ9e2b0jLUj4mLU7RVoJSq1TQROIm7m/DEwNYkZ+azcHuW3eEopVSlNBE40fCOzYgK9mXi4lTqWk0npZTr0ETgRB7ubjw2oBW/ZBxl6Y5su8NRSqnz0kTgZCM7RxLRsAETF6Voq0ApVStpInAyLw83fjOgFRv3HmHVzly7w1FKqXNoIqgBo7pGEh7ozVuLUuwORSmlzqGJoAb4eLrzaP9WrN19iJ92V7LqmFJK2UQTQQ25q3sLQvy9mLhYWwVKqdpFE0ENaeDlziP9WrI8JYdNew/bHY5SSpXTRFCDRveMoqGvJ//8LpmCohK7w1FKKUATQY3y8/bgj9fHsT7tECPeWUlKVr7dISmllCaCmnZ7YnP+81APjhw/yfC3VzJnsy5go5SylyYCG/RuHcI34/vRLiKQpz7fzF9mb6WopNTusJRSLkoTgU2aBPkw/ZGejO3fkn+v2cOoyavZd+i43WEppVyQJgIbebq78cfr45g8uiu7swu4ceIKFidrpVKlVM1yaiIQkWEi8quIpIrIs5Xsc7uIJInINhGZ7sx4aqth7Zrw9fi+RDRswIMfrefV+b9SqmsYKKVqiNMSgYi4A+8A1wHxwF0iEn/WPrHAc0AfY0wC8LSz4qntokP8+Oqx3tzZrTlvL0nl3mlryc4vsjsspZQLcGaLoDuQaozZZYw5CXwOjDhrn0eAd4wxhwGMMQedGE+t5+Ppzsu3duCV2zqwYc9hbnhruZakUEo5nTMTQQSwr8LjdMe2iq4CrhKRlSKyRkSGne9AIjJWRNaLyPrs7Ppf139UYnNmP94HXy937pq6hik/7tQS1kopp7F7sNgDiAUGAHcBU0Wk4dk7GWOmGGMSjTGJoaGhNRyiPeKaBjJ3fF+Gxofzj3nJPPrvDRw9UWx3WEqpesiZiSADaF7hcaRjW0XpwFxjTLExZjewAysxKCDQx5N37+nCX1BD7wkAABgfSURBVG6MZ3HyQYa/vYJt+4/aHZZSqp6pUiIQkadEJFAs00Rko4gMvcjL1gGxIhIjIl7AncDcs/aZjdUaQERCsLqKdl3Sb1DPiQgP9Y3hv4/2pKi4jJHvruK/6/ZqV5FSqtpUtUXwoDEmDxgKNALuBV6+0AuMMSXAE8B8YDswwxizTUReFJHhjt3mA7kikgQsASYYY3QZr/PoGtWYb57sS/foxvzhy1+YMHMLJ07q3chKqSsnVbmyFJEtxpgOIvImsNQYM0tENhljOjs/xDMlJiaa9evX1/Tb1hqlZYY3F6UwcXEKbcIDePeeLrQM9bc7LKVULSciG4wxied7rqotgg0isgC4HpgvIgFAWXUFqKrO3U343TVX8eGYbmTlFTL87ZXM++WA3WEppeqwqiaCh4BngW7GmOOAJ/CA06JSFzWgTRjfPNmP1mH+PPbpRv73mySKSzU3K6UuXVUTQS/gV2PMEREZDfwZ0OkrNoto2IAZj/ZiTO9opq3YzZ1T1nDg6Am7w1JK1TFVTQSTgOMi0hF4BtgJfOK0qFSVeXm48cLwBN6+uzPJB/K44a0VLE+p/zfdKaWqT1UTQYmxRpVHAG8bY94BApwXlrpUN3Zoxpwn+hLi78V9H/zEmwtTKNPCdUqpKqhqIsgXkeewpo1+KyJuWOMEqhZpHebP7Mf7cHOnCF5fuIMxH63jUMFJu8NSStVyVU0EdwBFWPcTZGLdJfyK06JSl83Xy4PXbu/IP0a2Z83OXG58azmb9h62OyylVC1WpUTgOPl/CgSJyI1AoTFGxwhqKRHh7h4t+HJcb9zchNvfW81HK3fr3chKqfOqaomJ24GfgFHA7cBaEbnNmYGpK9c+Mohvx/ejf2woL3ydxPjPNnGsqMTusJRStYxHFff7E9Y9BAcBRCQUWAjMdFZgqnoE+Xoy9b5E3vtxF6/MTybpQB6T7ulKmyY61q+UslR1jMDtrEVjci/htcpmbm7CuAGtmP5IT/JOlDDinRV8tTHd7rCUUrVEVU/m34vIfBEZIyJjgG+Bec4LSzlDz5bBzHuyLx0jG/K7GT/zuxmbyT2my2Eq5eqqVHQOQERuBfo4Hi43xsxyWlQX4OpF56pDSWkZby5KYdLSnfh5ezDh2jbc1b0F7m5id2hKKSe5UNG5KieC2kITQfVJPZjPX2ZvY/WuXDpEBvG/I9rRsfk5C8QppeqBy64+KiL5IpJ3nq98EclzTriqprQOC2D6Iz14885OHDhayM3vruRPs37hyHG9CU0pV3LBWUPGGJ1aUs+JCCM6RTCobRiv/5DCx6vT+G5rJs9e15bbukTipt1FStV7OvNHARDg48lfb4rn6yf6EhPix+9nbmHUe6tJ2q8NP6XqO00E6gzxzQL54tFe/N9tHdidU8CNE5fzt6+3kV9YbHdoSikn0USgzuHmJtye2JzFz1zNXd1b8NGqNAb9axlzNmdomQql6iFNBKpSDX29eGlke2Y/1oemQT489flm7p66lpSsfLtDU0pVI00E6qI6Nm/IrMf68Peb25F0II/r3lzOP7/bToHWLVKqXtBEoKrE3U0Y3TOKxc9czcjOEby3bBfXvLaM7345oN1FStVxmgjUJQn29+aVUR2Z+ZteBDbwZNynG7n/w3XszimwOzSl1GXSRKAuS2J0Y74Z35e/3hjPxj2Hufb1H3nthx0UFpfaHZpS6hJpIlCXzcPdjQf7xrD4mau5rn0T3lqUwjWvL2NxcpbdoSmlLoEmAnXFwgJ9ePPOzkx/pAfeHu48+NF6HvlkPfsOHbc7NKVUFWgiUNWmd6sQ5j3Zjz8Ma8uKlByueX0Z7yxJpahEu4uUqs00Eahq5eXhxrgBrVj4zNUMbBPGK/N/5bo3lrMiJcfu0JRSldBEoJwiomEDJo3uykcPdKPUGEZPW8vj0zeSebTQ7tCUUmfRRKCcakCbMOY/3Z/fDrmKhUlZDP7XUqb+uIvi0jK7Q1NKOWgiUE7n4+nOU0Ni+eG3V9OjZTAvzdvOjW+tYO2uXLtDU0qhiUDVoBbBvky7P5Ep93blWFEJd0xZw+/+u5mDedpdpJSdLrgwjVLVTUQYmtCEfrGhvL0khSk/7uLbXw5wf+9ofnN1Kxr7edkdolIuR9csVrbak1vAmwtTmLU5Az8vDx7sG8PD/WII9PG0OzSl6hVdvF7VeilZ+by+cAfzfskkqIEnj17dkjG9o/H10karUtVBE4GqM7ZmHOW1H3awOPkgIf5ePDagNXf3aIGPp7vdoSlVp10oETh1sFhEhonIryKSKiLPXmC/W0XEiMh5g1Suo11EEB+M6caX43pxVXgAL36TxMBXlzJ97V6dcqqUkzgtEYiIO/AOcB0QD9wlIvHn2S8AeApY66xYVN3TNaox0x/pyfSHe9AkyIc/zvqFwf9axlcb0yktq1utWKVqO2e2CLoDqcaYXcaYk8DnwIjz7Pe/wP8DdA6hOkfv1iF8Na43H4xJxN/bg9/N+Jlr3/iReb8coEwTglLVwpmJIALYV+FxumNbORHpAjQ3xnx7oQOJyFgRWS8i67Ozs6s/UlWriQiD2obzzfi+vHtPFwAe+3QjN729giXJB3WFNKWukG03lImIG/Aa8MzF9jXGTDHGJBpjEkNDQ50fnKqV3NyE69s3Zf7T/Xnt9o7kF5bwwEfruG3yalbt1KJ2Sl0uZyaCDKB5hceRjm2nBADtgKUikgb0BObqgLG6GHc34ZYukSx65mr+MbI9GYdPcPfUtdzz/ho27j1sd3hK1TlOmz4qIh7ADmAwVgJYB9xtjNlWyf5Lgf8xxlxwbqhOH1VnKywuZfravby7NJWcYycZ1DaMZ4ZeRUKzILtDU6rWsGX6qDGmBHgCmA9sB2YYY7aJyIsiMtxZ76tcj4+nOw/2jWHZhIFMuLYN69MOccNbK3j8042kHsy3Ozylaj29oUzVO0dPFDNt+S6mrdjNieJSbu4cwdODr6JFsK/doSllG72zWLmkQwUnmbxsJx+vSqO0zHB7t+aMH9SapkEN7A5NqRqniUC5tIN5hby9JJXPftqLiDC6RxSPDWxFiL+33aEpVWM0ESgF7Dt0nImLU5i5IR0fT3ce6BPN2H6tCPLVSqeq/tNEoFQFu7KP8cbCFL7esh9/bw/G9mvJA31j8PfWSqeq/tJEoNR5JGfm8a8FO/ghKYvGfl6M7tGCu3tE0STIx+7QlKp2mgiUuoCf9x1h4uIUFiUfxE2EYQlNuL93NN2iGyEidoenVLXQRKBUFezNPc5/1u7hv+v2cfREMW2bBHB/72hGdGqmC+SoOk8TgVKX4MTJUub+nMFHq/aw/UAegT4e3NGtOaN7RhEV7Gd3eEpdFk0ESl0GYwzr9xzm41VpfL81k1JjGNgmjPt6RdE/NhQ3N+02UnXHhRKBtneVqoSI0C26Md2iG5OVV8j0tXuZ/tNexny4jpgQP+7tGcVtiZEE+uj0U1W3aYtAqUtwsqSM77Ye4JPVe9iw5zC+Xu6M7BzBfb2iadMkwO7wlKqUdg0p5QRbM47y8ao05vy8n5MlZfRqGcz9vaMYEheOh7ttS30odV6aCJRyokMFJ5mxfh//Xr2HjCMnaBrkw+ieUdzZrTnBWsZC1RKaCJSqAaVlhkXbs/hk9R5WpObg5e7GjR2bcn+vaDo2b2h3eMrF6WCxUjXA3U0YmtCEoQlNSD2Yz79X72HmhnS+2phBx+YNGdM7iuvbN8Xbw93uUJU6g7YIlHKi/MJivtqYwcer09iVXUCwnxd3dW/BPT1baDlsVaO0a0gpmxljWJmay0er0liUnIWbCEPjw7m/dzQ9YhprKQvldNo1pJTNRIS+sSH0jQ1h36HTpSy+25pJm/AA7usdxcjOEVrKQtlCWwRK2aSwuJS5m/fz0ao0kg7kEeDjwe2JzRnTO5rmjXVZTVW9tGtIqVrMGMPGvYf5eNUe5v1ygDJjGBrfhIf6xZAYpRVQVfXQRKBUHZF5tJBPVqfx6dq9HD1RTIfIIB7qG8P17ZviqTepqSugiUCpOub4yRK+3JjBhyt2syungCaBPtzfO5q7u7fQpTXVZdFEoFQdVVZmWLrjINNW7GZlai4NPN25rWskD/SJpmWov93hqTpEE4FS9UDS/jw+WLmbuZv3U1xWxqA2YTzUL4ZeLYN1HEFdlCYCpeqRg/mF/GfNXj5ds4fcgpPENQ3kob4x3NRR71pWldNEoFQ9VFhcypzNGUxbsZsdWccIDfDm3p5R3NOjhRa7U+fQRKBUPWaMYXlKDtNW7GbZjmy8Pdy4pUsED/aJITZc10hQFr2zWKl6TETof1Uo/a8KJSUrnw9WpvHVxnQ++2kf/a8K5aG+MfSPDdFxBFUpbREoVQ8dKjjJ9LV7+Hj1HrLzi4gN8+ehvjHc3DkCH08dR3BF2jWklIsqKinlm58PMG3FbpIO5NHYz4vRPVowulcUYQE+doenapAmAqVcnDGGNbsOMW3FbhYlZ+Hp5sZNHZvxUN8Y4psF2h2eqgE6RqCUixMRerUKplerYHbnFPDhyt18sT6dLzem06tlMA/1jWFQ2zDc3HQcwRVpi0ApF3X0eDGfrdvLx6vSOHC0kJYhfjzQJ5pbu0ZqOex6SLuGlFKVKi4t47utmUxbvouf048S1MCTUV0jua59Ezo1b4S7thLqBU0ESqmLMsawYc9hpq3YzQ9JWZSUGYL9vBjUNoxr4sPpGxuiLYU6TMcIlFIXJSIkRjcmMboxeYXFLP01m4VJWXy/LZMvNqTj7eFG39YhXBMfzqC4MJ11VI84NRGIyDDgTcAdeN8Y8/JZz/8OeBgoAbKBB40xe5wZk1Lq4gJ9PBnesRnDOzajuLSMn3Yf4oekLH5IymJR8kFEoFPzhgyJC+ea+HBiw/z1hrU6zGldQyLiDuwArgHSgXXAXcaYpAr7DATWGmOOi8g4YIAx5o4LHVe7hpSyjzGG5Mx8FiZlsXB7Fj+nHwUgKti3PCkkRjXCQxfRqXVsGSMQkV7AC8aYax2PnwMwxvyzkv07A28bY/pc6LiaCJSqPTKPFrIo2WoprErN5WRpGUENPMvHFfpfFYq/t/ZA1wZ2jRFEAPsqPE4Helxg/4eA7873hIiMBcYCtGjRorriU0pdoSZBPtzTI4p7ekRRUFTC8pRsFiRlsTj5ILM2ZeDl7kbPVsFcEx/OkLgwmgY1sDtkdR7ObBHcBgwzxjzseHwv0MMY88R59h0NPAFcbYwputBxtUWgVO1XUlrGxr1H+CEpkx+SskjLPQ5A+4gghsSFMyQ+jPimgTquUIPsahFkAM0rPI50bDuDiAwB/kQVkoBSqm7wcHeje0xjusc05o/Xx7Ezu4AfHOMKbyzawesLdxDRsAFD4sK4Jr4J3WMa4+Wh4wp2cWaLwANrsHgwVgJYB9xtjNlWYZ/OwEyslkNKVY6rLQKl6rbs/CKWJB/kh+1ZLE/JprC4jABvD65uE8o18eEMaBNGUANPu8Osd2y7oUxErgfewJo++oEx5iUReRFYb4yZKyILgfbAAcdL9hpjhl/omJoIlKo/TpwsZWVqjmNaahY5x07i4Sb0aNmYIXHh9GkdQssQP52FVA30zmKlVK1XVmbYnH7E6kJKyiLl4DEAvDzcaB3qT9umAbRtEkDbJoG0bRJAaIC3jjFcAk0ESqk6Jy2ngE37DpN8IJ/kzHySM/PIyjs9jNjYz4s24QHlCaJNk0CuCvfXMhiV0BITSqk6JzrEj+gQP+h8etvhgpMkZ+bza2aeIznk8/lP+zhRXAqACEQ19qVtk0DaNAkgrqmVIFo09tXieRegiUApVWc08vMqX1fhlLIyw77Dx9l+IJ9fM/P5NSuP5AP5LEjKpMzR4dHA052rwv3LE4TVigiksZ+XTb9J7aJdQ0qpeunEyVJSDjq6lQ6cThC5BSfL9wkL8LYSg2PsoU2TAFqH+dfLdZ21a0gp5XIaeLnTIbIhHSIbnrE9O7+I5Mw8fs3Mt1oRWXl8vHoPJ0vKAHB3E2JC/BzJwUoQcc0CaRbkU28HpzURKKVcSmiAN6EBofSLDS3fVlJaRlrucX51DEonZ+bzc/oRvtlyoHyfhr6exDcNJL5pIAkRgcQ3DaJVaP2Y2qqJQCnl8jzc3Wgd5k/rMH9u6NC0fPuxohJHyyGPbfvzSDqQx7/X7KHI0Xrw8nCjbZMAKzk0CyS+WSBtmwTiV8cK7dWtaJVSqgb5e3vQNaoRXaMalW8rKS1jd05BeWJI2p/H/G2ZfL7OqrEpAjHBfsQ1cySHplaCqM0L+WgiUEqpS+Dh7kZseACx4QHc3DkCsNZpyMwrZFvG6eSwJf0I31boWgoN8D6j5ZDQLIioxr641YJprZoIlFLqCokITYMa0DSoAUPiw8u3Hz1RzHZHYjjVglj54y5KHPNafb3ciWt6uuWQ0CyI2PCan7Wk00eVUqoGFZWUkpJ1rLzlkORIEMeKSgBr1lLrUP/ylkO8I0k09L2yex50+qhSStUS3h7utIsIol1EUPm2UzfFndFy2JnDV5tOV+6PaNiA3w9rw4hOEdUekyYCpZSymZubEBXsR1SwH9e1Pz1rKedY0ekZS/vzCA3wdsr7ayJQSqlaKsTfm36xZ97z4Ax1/04IpZRSV0QTgVJKuThNBEop5eI0ESillIvTRKCUUi5OE4FSSrk4TQRKKeXiNBEopZSLq3O1hkQkG9hzmS8PAXKqMZy6Tj+PM+nncZp+FmeqD59HlDHmvHem1blEcCVEZH1lRZdckX4eZ9LP4zT9LM5U3z8P7RpSSikXp4lAKaVcnKslgil2B1DL6OdxJv08TtPP4kz1+vNwqTECpZRS53K1FoFSSqmzaCJQSikX5zKJQESGicivIpIqIs/aHY9dRKS5iCwRkSQR2SYiT9kdU20gIu4isklEvrE7FruJSEMRmSkiySKyXUR62R2TXUTkt46/k60i8pmI+NgdkzO4RCIQEXfgHeA6IB64S0Ti7Y3KNiXAM8aYeKAn8LgLfxYVPQVstzuIWuJN4HtjTFugIy76uYhIBPAkkGiMaQe4A3faG5VzuEQiALoDqcaYXcaYk8DnwAibY7KFMeaAMWaj4+d8rD/y6l8Nuw4RkUjgBuB9u2Oxm4gEAf2BaQDGmJPGmCP2RmUrD6CBiHgAvsB+m+NxCldJBBHAvgqP03Hxkx+AiEQDnYG19kZiuzeA3wNldgdSC8QA2cCHjq6y90XEz+6g7GCMyQBeBfYCB4CjxpgF9kblHK6SCNRZRMQf+BJ42hiTZ3c8dhGRG4GDxpgNdsdSS3gAXYBJxpjOQAHgkmNqItIIq+cgBmgG+InIaHujcg5XSQQZQPMKjyMd21ySiHhiJYFPjTFf2R2PzfoAw0UkDavLcJCI/MfekGyVDqQbY061EmdiJQZXNATYbYzJNsYUA18BvW2OySlcJRGsA2JFJEZEvLAGfObaHJMtRESw+n+3G2NeszseuxljnjPGRBpjorH+Xyw2xtTLq76qMMZkAvtEpI1j02AgycaQ7LQX6Ckivo6/m8HU04FzD7sDqAnGmBIReQKYjzXy/4ExZpvNYdmlD3Av8IuIbHZs+6MxZp6NManaZTzwqeOiaRfwgM3x2MIYs1ZEZgIbsWbbbaKelprQEhNKKeXiXKVrSCmlVCU0ESillIvTRKCUUi5OE4FSSrk4TQRKKeXiNBEoVYNEZIBWOFW1jSYCpZRycZoIlDoPERktIj+JyGYRec+xXsExEXndUZ9+kYiEOvbtJCJrRGSLiMxy1KhBRFqLyEIR+VlENopIK8fh/SvU+//UcdeqUrbRRKDUWUQkDrgD6GOM6QSUAvcAfsB6Y0wCsAx43vGST4A/GGM6AL9U2P4p8I4xpiNWjZoDju2dgaex1sZoiXW3t1K2cYkSE0pdosFAV2Cd42K9AXAQq0z1fx37/Af4ylG/v6ExZplj+8fAFyISAEQYY2YBGGMKARzH+8kYk+54vBmIBlY4/9dS6vw0ESh1LgE+NsY8d8ZGkb+ctd/l1mcpqvBzKfp3qGymXUNKnWsRcJuIhAGISGMRicL6e7nNsc/dwApjzFHgsIj0c2y/F1jmWP0tXURudhzDW0R8a/S3UKqK9EpEqbMYY5JE5M/AAhFxA4qBx7EWaenueO4g1jgCwP3AZMeJvmK1znuB90TkRccxRtXgr6FUlWn1UaWqSESOGWP87Y5DqeqmXUNKKeXitEWglFIuTlsESinl4jQRKKWUi9NEoJRSLk4TgVJKuThNBEop5eL+P8H04yqF1k9PAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "O comportamento acima das curvas de loss mostra que o modelo aprende bem os padrões dos dados de treinamento, ganhando mais confiança nas suas classificações. Já a curva para os dados de validação (desconhecidos para o treinamento do modelo) mostram que a *loss* se estabiliza após algumas épocas.\n",
        "\n",
        "**O que fazer para melhorar?**\n",
        "\n",
        "Dá para fazer muita coisa...\n",
        "\n",
        "*   Otimização dos hiperparâmetros: altere a quantidade de camadas intermediárias, a quantidade de neurônios em cada camada intermediária, a quantidade de épocas de treinamento, o algoritmo de otimização (veja [aqui](https://keras.io/api/optimizers/) outros otimizadores);\n",
        "*   Etapas de pré-processamento dos *tweets*: remoção de URLs, @'s, etc...\n",
        "*   A caracterização dos *tweets* (por exemplo, o [BERT](https://keras.io/examples/nlp/text_extraction_with_bert/)!)...\n",
        "\n",
        "\n",
        "### Avaliação de performance utilizando os dados de teste\n",
        "\n",
        "Agora fazemos a classificação das instâncias do conjunto de teste utilizando o método *predict*. Observe que uma matriz é gerada na saída, em que cada linha representa um *tweet* e as colunas representam:\n",
        "\n",
        "\n",
        "*   Coluna $0$: probabilidade estimada do modelo em classificar o $i$-ésimo *tweet* como sendo da classe \"0\" (polaridade negativa)\n",
        "*   Coluna $1$: probabilidade estimada do modelo em classificar o $i$-ésimo *tweet* como sendo da classe \"1\" (polaridade neutra)\n",
        "*   Coluna $2$: probabilidade estimada do modelo em classificar o $i$-ésimo *tweet* como sendo da classe \"2\" (polaridade positiva)\n",
        "\n"
      ],
      "metadata": {
        "id": "uWuQ0prhiiY4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "y = model.predict(X_test)\n",
        "y"
      ],
      "metadata": {
        "id": "V1mox-in1dt-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2267bfcb-f742-44a1-9dc5-11fe56f92aa7"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.2264621 , 0.8613842 , 0.40541023],\n",
              "       [0.34821957, 0.4485632 , 0.70630455],\n",
              "       [0.03780433, 0.99058   , 0.27396843],\n",
              "       ...,\n",
              "       [0.22388306, 0.93936676, 0.25324726],\n",
              "       [0.05695976, 0.20156829, 0.9876643 ],\n",
              "       [0.0455583 , 0.98867524, 0.27305946]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "De maneira genérica, podemos calcular argmax das probabilidades, tomando-se as colunas como referência. Suponha que todas as probabilidades estejam na variável ```y_prob```, em que cada linha se trata de uma instância e teremos três colunas:\n",
        "\n",
        "*   ```y_prob[i][0]```: estimativa da probabilidade de classificar a $i$-ésima instância  de ```X_test``` como sendo da classe \"0\"\n",
        "*   ```y_prob[i][1]```: estimativa da probabilidade de classificar a $i$-ésima instância de ```X_test``` como sendo da classe \"1\"\n",
        "*   ```y_prob[i][2]```: estimativa da probabilidade de classificar a $i$-ésima instância de ```X_test``` como sendo da classe \"2\"\n",
        "\n",
        "Assim, com o uso de $argmax$, para cada instância $i$ podemos obter o índice $j$ da coluna associada à classe com a maior probabilidade:"
      ],
      "metadata": {
        "id": "Vlq2q3Fuk185"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "y_prob = model.predict(X_test)\n",
        "y_pred = np.argmax(y_prob,axis=1)"
      ],
      "metadata": {
        "id": "ScVRiIVWk2px"
      },
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Vamos imprimir os resultados da classificação de *tweets* para os dados de teste:"
      ],
      "metadata": {
        "id": "_QjL-SHZrl7h"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(classification_report(y_test,y_pred))"
      ],
      "metadata": {
        "id": "lSqjqALvrmDw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4417fa85-e57f-40d7-e8b8-a51c3c4c822c"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.74      0.72      0.73       559\n",
            "           1       0.74      0.82      0.78      1513\n",
            "           2       0.83      0.75      0.79      1475\n",
            "\n",
            "    accuracy                           0.77      3547\n",
            "   macro avg       0.77      0.76      0.76      3547\n",
            "weighted avg       0.78      0.77      0.77      3547\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Outra possibilidade é o uso do método *evaluate*:\n",
        "\n"
      ],
      "metadata": {
        "id": "PvAl8nUD2CQ9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(model.evaluate(X_test,y_test_enc))"
      ],
      "metadata": {
        "id": "6RMsY7GK2E1r",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c7ac924f-2838-4520-cb94-1d490cb18384"
      },
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "111/111 [==============================] - 77s 696ms/step - loss: 0.9379 - accuracy: 0.7745\n",
            "[0.9379117488861084, 0.7744572758674622]\n"
          ]
        }
      ]
    }
  ]
}