DistillBERT_2_class_Bayesian_Optimisation_Tuning.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1ysSqAbGQ0Kb"
      },
      "source": [
        "# Hyperparameter tuning of DistillBERT with Classification Head\n",
        "\n",
        "Link to Colab Notebook: https://colab.research.google.com/drive/1M3twE8OhurhJ8h5IQLE-yNliqdEXHxu0?authuser=3#scrollTo=1ysSqAbGQ0Kb\n",
        "\n",
        "This notebook aims to improve on the limitation of the paper:\n",
        "Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019). How to Fine-Tune BERT for Text Classification? Computation and Language (Cs.CL). https://doi.org/10.48550/arXiv.1905.05583\n",
        "\n",
        "The authors did not conduct hyperparameter tuning for their neural networks. For our project, we attempt to choose values for dropout rate and LEARNING_RATE variables used in 'DistillBERT_finetuning_2_target.ipynb' notebook, since learning rate affects convergence while dropout rate affects generalisation ability and train time of model. We used a subset of the training data, and utilized Bayesian Optimisation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bE2E6VdPx308",
        "outputId": "2e146478-a12e-4adc-a697-762a5fb9c329"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting bayesian-optimization\n",
            "  Downloading bayesian_optimization-1.4.3-py3-none-any.whl (18 kB)\n",
            "Requirement already satisfied: numpy>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from bayesian-optimization) (1.25.2)\n",
            "Requirement already satisfied: scipy>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from bayesian-optimization) (1.11.4)\n",
            "Requirement already satisfied: scikit-learn>=0.18.0 in /usr/local/lib/python3.10/dist-packages (from bayesian-optimization) (1.2.2)\n",
            "Collecting colorama>=0.4.6 (from bayesian-optimization)\n",
            "  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
            "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.18.0->bayesian-optimization) (1.3.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.18.0->bayesian-optimization) (3.4.0)\n",
            "Installing collected packages: colorama, bayesian-optimization\n",
            "Successfully installed bayesian-optimization-1.4.3 colorama-0.4.6\n"
          ]
        }
      ],
      "source": [
        "!pip install bayesian-optimization"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gZ3WdQcfsALW",
        "outputId": "8b4d66d4-6918-402f-e1a0-c16ace5466a4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ],
      "source": [
        "# Import libraries\n",
        "from google.colab import drive\n",
        "import numpy as np\n",
        "import datetime\n",
        "import pandas as pd\n",
        "import torch\n",
        "import transformers\n",
        "from torch.utils.data import Dataset, DataLoader, SequentialSampler\n",
        "from transformers import DistilBertModel, DistilBertTokenizer, AdamW\n",
        "import re\n",
        "\n",
        "# Setting up the device for GPU usage\n",
        "from torch import cuda\n",
        "device = 'cuda' if cuda.is_available() else 'cpu'\n",
        "\n",
        "# Import data and extract mini-set (500 rows) for hyperparametertuning. train_split was created with shuffling so no need shuffle again.\n",
        "drive.mount('/content/drive')\n",
        "train_split = pd.read_csv('/content/drive/MyDrive/train_data.csv')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vS8gzZDmtIRs",
        "outputId": "3329fe27-4302-4f12-8d2a-8a0451cd368c"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "<ipython-input-3-3807951b1270>:12: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  mini_set['Sentiment'] = mini_set['Sentiment'].apply(good_bad)\n",
            "<ipython-input-3-3807951b1270>:13: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  val_set['Sentiment'] = val_set['Sentiment'].apply(good_bad)\n"
          ]
        }
      ],
      "source": [
        "# Extracting subset of train data\n",
        "mini_set = train_split[0:500]\n",
        "val_set = train_split[500:551]\n",
        "\n",
        "# Convert to binary classification\n",
        "def good_bad(row):\n",
        "  if row < 5:\n",
        "    return 0\n",
        "  else:\n",
        "    return 1\n",
        "\n",
        "mini_set['Sentiment'] = mini_set['Sentiment'].apply(good_bad)\n",
        "val_set['Sentiment'] = val_set['Sentiment'].apply(good_bad)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3izVj0sKt-GG",
        "outputId": "307e1eee-f899-420d-e94e-b2cfffc2a02f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "|   iter    |  target   | dropou... | lr_exp... |\n",
            "-------------------------------------------------\n",
            "| \u001b[0m1        \u001b[0m | \u001b[0m84.31    \u001b[0m | \u001b[0m3.085    \u001b[0m | \u001b[0m5.881    \u001b[0m |\n",
            "| \u001b[0m2        \u001b[0m | \u001b[0m58.82    \u001b[0m | \u001b[0m1.001    \u001b[0m | \u001b[0m4.209    \u001b[0m |\n",
            "| \u001b[0m3        \u001b[0m | \u001b[0m43.14    \u001b[0m | \u001b[0m1.734    \u001b[0m | \u001b[0m3.369    \u001b[0m |\n",
            "| \u001b[0m4        \u001b[0m | \u001b[0m78.43    \u001b[0m | \u001b[0m1.931    \u001b[0m | \u001b[0m4.382    \u001b[0m |\n",
            "| \u001b[95m5        \u001b[0m | \u001b[95m86.27    \u001b[0m | \u001b[95m2.984    \u001b[0m | \u001b[95m5.155    \u001b[0m |\n",
            "| \u001b[0m6        \u001b[0m | \u001b[0m82.35    \u001b[0m | \u001b[0m3.096    \u001b[0m | \u001b[0m5.741    \u001b[0m |\n",
            "| \u001b[0m7        \u001b[0m | \u001b[0m78.43    \u001b[0m | \u001b[0m2.022    \u001b[0m | \u001b[0m6.512    \u001b[0m |\n",
            "| \u001b[0m8        \u001b[0m | \u001b[0m84.31    \u001b[0m | \u001b[0m1.137    \u001b[0m | \u001b[0m5.682    \u001b[0m |\n",
            "| \u001b[0m9        \u001b[0m | \u001b[0m84.31    \u001b[0m | \u001b[0m3.087    \u001b[0m | \u001b[0m5.235    \u001b[0m |\n",
            "| \u001b[0m10       \u001b[0m | \u001b[0m56.86    \u001b[0m | \u001b[0m1.702    \u001b[0m | \u001b[0m3.792    \u001b[0m |\n",
            "| \u001b[0m11       \u001b[0m | \u001b[0m78.43    \u001b[0m | \u001b[0m5.004    \u001b[0m | \u001b[0m6.873    \u001b[0m |\n",
            "| \u001b[0m12       \u001b[0m | \u001b[0m84.31    \u001b[0m | \u001b[0m2.567    \u001b[0m | \u001b[0m5.769    \u001b[0m |\n",
            "| \u001b[0m13       \u001b[0m | \u001b[0m64.71    \u001b[0m | \u001b[0m5.382    \u001b[0m | \u001b[0m6.578    \u001b[0m |\n",
            "| \u001b[0m14       \u001b[0m | \u001b[0m56.86    \u001b[0m | \u001b[0m1.425    \u001b[0m | \u001b[0m3.156    \u001b[0m |\n",
            "| \u001b[0m15       \u001b[0m | \u001b[0m84.31    \u001b[0m | \u001b[0m1.849    \u001b[0m | \u001b[0m6.513    \u001b[0m |\n",
            "| \u001b[0m16       \u001b[0m | \u001b[0m80.39    \u001b[0m | \u001b[0m1.031    \u001b[0m | \u001b[0m6.605    \u001b[0m |\n",
            "| \u001b[0m17       \u001b[0m | \u001b[0m86.27    \u001b[0m | \u001b[0m1.946    \u001b[0m | \u001b[0m5.284    \u001b[0m |\n",
            "| \u001b[0m18       \u001b[0m | \u001b[0m70.59    \u001b[0m | \u001b[0m3.847    \u001b[0m | \u001b[0m6.729    \u001b[0m |\n",
            "| \u001b[0m19       \u001b[0m | \u001b[0m78.43    \u001b[0m | \u001b[0m3.224    \u001b[0m | \u001b[0m4.382    \u001b[0m |\n",
            "| \u001b[0m20       \u001b[0m | \u001b[0m84.31    \u001b[0m | \u001b[0m1.668    \u001b[0m | \u001b[0m5.919    \u001b[0m |\n",
            "| \u001b[0m21       \u001b[0m | \u001b[0m43.14    \u001b[0m | \u001b[0m6.0      \u001b[0m | \u001b[0m3.0      \u001b[0m |\n",
            "| \u001b[0m22       \u001b[0m | \u001b[0m56.86    \u001b[0m | \u001b[0m4.463    \u001b[0m | \u001b[0m4.807    \u001b[0m |\n",
            "| \u001b[0m23       \u001b[0m | \u001b[0m56.86    \u001b[0m | \u001b[0m3.829    \u001b[0m | \u001b[0m3.0      \u001b[0m |\n",
            "| \u001b[0m24       \u001b[0m | \u001b[0m58.82    \u001b[0m | \u001b[0m1.554    \u001b[0m | \u001b[0m7.0      \u001b[0m |\n",
            "| \u001b[0m25       \u001b[0m | \u001b[0m70.59    \u001b[0m | \u001b[0m1.742    \u001b[0m | \u001b[0m4.627    \u001b[0m |\n",
            "| \u001b[95m26       \u001b[0m | \u001b[95m88.24    \u001b[0m | \u001b[95m1.334    \u001b[0m | \u001b[95m5.137    \u001b[0m |\n",
            "| \u001b[0m27       \u001b[0m | \u001b[0m76.47    \u001b[0m | \u001b[0m2.531    \u001b[0m | \u001b[0m4.632    \u001b[0m |\n",
            "| \u001b[0m28       \u001b[0m | \u001b[0m88.24    \u001b[0m | \u001b[0m2.997    \u001b[0m | \u001b[0m5.157    \u001b[0m |\n",
            "| \u001b[0m29       \u001b[0m | \u001b[0m84.31    \u001b[0m | \u001b[0m1.339    \u001b[0m | \u001b[0m5.143    \u001b[0m |\n",
            "| \u001b[0m30       \u001b[0m | \u001b[0m86.27    \u001b[0m | \u001b[0m1.312    \u001b[0m | \u001b[0m5.078    \u001b[0m |\n",
            "=================================================\n"
          ]
        }
      ],
      "source": [
        "from bayes_opt import BayesianOptimization\n",
        "import warnings\n",
        "\n",
        "# Ignore warnings in run logs\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "\n",
        "# Model and custom dataset classes similar to main notebook\n",
        "# Define Custom Dataset\n",
        "class CustomDataset(Dataset):\n",
        "    ''' Custom dataset class defined to create '''\n",
        "\n",
        "    def __init__(self, dataframe, tokenizer, max_len):\n",
        "        self.tokenizer = tokenizer\n",
        "        self.data = dataframe\n",
        "        self.Content = dataframe.Text.to_numpy()\n",
        "        self.targets = dataframe.Sentiment.to_numpy()\n",
        "        self.max_len = max_len\n",
        "\n",
        "    # __len__ and __getitem__ methods to create map-style dataset to be interfaced by torch DataLoader method\n",
        "    def __len__(self):\n",
        "        return len(self.Content)\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        # Data preprocessing code to remove trailing whitespace, html tags and urls\n",
        "        Content = re.sub(r'<[^>]+>', '', self.Content[index])\n",
        "        Content = re.sub(r'https://\\S+|www\\.\\S+', '', Content)\n",
        "        Content = re.sub(r'br\\s', '', Content)\n",
        "        Content = \" \".join(Content.split())\n",
        "\n",
        "        rating = self.targets[index]\n",
        "\n",
        "        # Tokenisation of text\n",
        "        inputs = self.tokenizer.encode_plus(\n",
        "            Content,\n",
        "            None,\n",
        "            add_special_tokens=True,\n",
        "            max_length=self.max_len,\n",
        "            truncation=True,\n",
        "            pad_to_max_length=True,\n",
        "        )\n",
        "        ids = inputs['input_ids']\n",
        "        mask = inputs['attention_mask']\n",
        "\n",
        "        return {\n",
        "            'ids': torch.tensor(ids, dtype=torch.long),\n",
        "            'mask': torch.tensor(mask, dtype=torch.long),\n",
        "            'targets': torch.tensor(rating, dtype=torch.int)\n",
        "        }\n",
        "\n",
        "class DistillBERTClass(torch.nn.Module):\n",
        "    def __init__(self, dropout_val):\n",
        "        super(DistillBERTClass, self).__init__()\n",
        "        self.l1 = DistilBertModel.from_pretrained(\"distilbert-base-uncased\")\n",
        "        self.pre_classifier = torch.nn.Linear(768, 768)\n",
        "        self.dropout = torch.nn.Dropout(dropout_val)\n",
        "        self.classifier = torch.nn.Linear(768, 2)\n",
        "\n",
        "    # Note: DistilBERT outputs a tuple where the first element at index 0\n",
        "    # represents the hidden-state at the output of the model's last layer.\n",
        "    # It is a tensor of shape (batch_size, sequence_length, hidden_size=768)\n",
        "    def forward(self, input_ids, attention_mask):\n",
        "        output_1 = self.l1(input_ids=input_ids, attention_mask=attention_mask)\n",
        "        hidden_state = output_1[0]\n",
        "        pooler = hidden_state[:, 0]\n",
        "        pooler = self.pre_classifier(pooler)\n",
        "        pooler = torch.nn.ReLU()(pooler)\n",
        "        pooler = self.dropout(pooler)\n",
        "        output = self.classifier(pooler)\n",
        "        return output\n",
        "\n",
        "# Function to calcuate the accuracy of the model\n",
        "def calcuate_accu(big_idx, targets):\n",
        "    n_correct = (big_idx==targets).sum().item()\n",
        "    return n_correct\n",
        "\n",
        "# Training Parameters\n",
        "MAX_LEN = 512\n",
        "EPOCHS = 5\n",
        "tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')\n",
        "\n",
        "# Create validation set (fixed for all experiments)\n",
        "test_params = {'batch_size': 1,\n",
        "                'shuffle': False,\n",
        "                'sampler': SequentialSampler(val_set),\n",
        "                'num_workers': 0\n",
        "                }\n",
        "val_data = CustomDataset(val_set, tokenizer, MAX_LEN)\n",
        "testing_loader = DataLoader(val_data, **test_params)\n",
        "\n",
        "\n",
        "def train(lr_exponent_val, dropout_val):\n",
        "    # Ensure train_batch and lr_exponent_val are discrete\n",
        "    # Transformations to convert parameter inputs to actual model parameters\n",
        "    dropout = 0.1*int(dropout_val)\n",
        "    lr_exponent_val = int(lr_exponent_val)\n",
        "    lr = 1*10**-(lr_exponent_val)\n",
        "\n",
        "    # Create Dataset and Dataloader\n",
        "    paramtune_set = CustomDataset(mini_set, tokenizer, MAX_LEN)\n",
        "    train_params = {'batch_size': 4,\n",
        "                    'shuffle': True,\n",
        "                    'num_workers': 0\n",
        "                    }\n",
        "    paramtune_loader = DataLoader(paramtune_set, **train_params)\n",
        "\n",
        "    # Initialize model\n",
        "    model = DistillBERTClass(dropout)\n",
        "    model.to(device)\n",
        "\n",
        "    # Creating the loss function and optimizer\n",
        "    loss_function = torch.nn.CrossEntropyLoss()\n",
        "    optimizer = torch.optim.AdamW(params=model.parameters(), lr=lr)\n",
        "\n",
        "    # Training loop over mini_set\n",
        "    for epoch in range(EPOCHS):\n",
        "        model.train()\n",
        "        for _,data in enumerate(paramtune_loader, 0):\n",
        "            ids = data['ids'].to(device, dtype = torch.long)\n",
        "            mask = data['mask'].to(device, dtype = torch.long)\n",
        "            targets = data['targets'].to(device, dtype = torch.long)\n",
        "\n",
        "            outputs = model(ids, mask)\n",
        "            loss = loss_function(outputs, targets)\n",
        "\n",
        "            optimizer.zero_grad()\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "\n",
        "    # Evaluating model accuracy over test set\n",
        "    model.eval()\n",
        "    n_correct,nb_val_examples = 0,0\n",
        "    with torch.no_grad():\n",
        "        for _, data in enumerate(testing_loader, 0):\n",
        "            ids = data['ids'].to(device, dtype = torch.long)\n",
        "            mask = data['mask'].to(device, dtype = torch.long)\n",
        "            targets = data['targets'].to(device, dtype = torch.long)\n",
        "            outputs = model(ids, mask)\n",
        "            big_val, big_idx = torch.max(outputs.data, dim=1)\n",
        "            n_correct += calcuate_accu(big_idx, targets)\n",
        "            nb_val_examples+=targets.size(0)\n",
        "\n",
        "    run_accu = (n_correct*100)/nb_val_examples\n",
        "\n",
        "    return run_accu\n",
        "\n",
        "# Parameters to tune (learning rate and train batch size)\n",
        "pbounds = {\n",
        "    'dropout_val': (1,6),\n",
        "    'lr_exponent_val': (3, 7),\n",
        "    }\n",
        "\n",
        "optimizer = BayesianOptimization(\n",
        "    f=train,\n",
        "    pbounds=pbounds,\n",
        "    verbose=2,\n",
        "    random_state=1,\n",
        ")\n",
        "\n",
        "# Bayesian Optimisation Algorithm. init_points parameter initiates 15 random points to explore during search. Helps by diversifying exploration space, increasing chances of finding global maxima.\n",
        "# n_iter specifies number of iterations of bayesian optimisation to run. Total iterations would be sum of n_iter and init_points.\n",
        "optimizer.maximize(init_points=15, n_iter=15)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uBBKrZhaWzk_",
        "outputId": "717a6c4a-41cd-4ecc-cbe9-a3247d537d0b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'target': 88.23529411764706, 'params': {'dropout_val': 1.33403703170643, 'lr_exponent_val': 5.137147476858718}}\n"
          ]
        }
      ],
      "source": [
        "print(optimizer.max)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D1deGVrKTbxt"
      },
      "source": [
        "Note: Parameter numbers in table above are not the final parameters. Decimals are truncated and value is transformed to appropriate form (i.e. dropout_val of 1.334 means p=0.1 for dropout layer was used; lr_exponent_val = 5.137 means lr=1e-5 for AdamW optimiser was used. (See first 3 lines of `train` function for transformations).\n",
        "\n",
        "The cell above prints the optimal hyperparameters. The dropout rate of 0.1 aligns with related works. Hence, learning rate of 1e-5 and p=0.1 will be used for model training."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}