ex4-task2-squad_as_sent_classification_unsolved.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/textual-data-analysis-course/blob/main/ex4-task2-squad_as_sent_classification_unsolved.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "08ba83f3-24d6-41e2-85a2-7c5cc314fc6e",
      "metadata": {
        "id": "08ba83f3-24d6-41e2-85a2-7c5cc314fc6e"
      },
      "source": [
        "# QA as sentence classification\n",
        "\n",
        "## Task 1:\n",
        "\n",
        "* Familiarize yourself with the whole notebook and what is going on there\n",
        "* At the very bottom there is a place where you can simply run a trained model\n",
        "* Print some interesting correct and incorrect classifications (e.g. cases where the model failed to answer \"yes\" - false negatives, or cases where it overpredicted a \"yes\" - false positives)\n",
        "* Do these make sense? Would you do better?\n",
        "\n",
        "## Task 2:\n",
        "\n",
        "* Try to see if you can run the training yourself on the dataset\n",
        "* AND add precision and recall to the reported metrics so you can better see what happens to the positive class\n",
        "* Make sure you save the models etc\n",
        "\n",
        "## Task 3:\n",
        "\n",
        "* You will notice that the basic model is heavily skewed towards predicting the (more common) negative class\n",
        "* The correct way to address this is by introducing class weights into training\n",
        "* Try, for example, to train a model with a 0.9 weight on the positive class and 0.1 on the negative, does anything change? And what if you turn the weights around, and give the positive class a 0.1 weight, will the model ever predict any?\n",
        "* Google will help you with how to add class weights into the training process (it involves overriding the compute_loss() method of the trainer)\n",
        "\n",
        "## Task 4:\n",
        "\n",
        "* Those of you who want, once you make it to the point you can train your own model, you can do it on Finnish\n",
        "* The Finnish QA data is pointed to below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "180f00f0-6065-45e6-bc51-a3c249ced122",
      "metadata": {
        "id": "180f00f0-6065-45e6-bc51-a3c249ced122"
      },
      "outputs": [],
      "source": [
        "!pip3 install evaluate transformers datasets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "de99d32a-14a3-4d94-9443-999f4ea19f24",
      "metadata": {
        "id": "de99d32a-14a3-4d94-9443-999f4ea19f24"
      },
      "outputs": [],
      "source": [
        "import transformers\n",
        "import datasets\n",
        "import evaluate"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "03288d5e-b428-46d2-9011-b211b96ada56",
      "metadata": {
        "id": "03288d5e-b428-46d2-9011-b211b96ada56"
      },
      "source": [
        "# SQuAD v2\n",
        "\n",
        "* First, we will load and explore the SQuAD v2 dataset\n",
        "* It is also quite important to read through the SQuAD v2 paper to understand the finer points of the data\n",
        "* `squad_v2` is the original English dataset\n",
        "* `TurkuNLP/squad_v2_fi` is a machine-translated version produced by TurkuNLP"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "30a4569e-6754-4cf3-87a0-3a1eee4f845e",
      "metadata": {
        "colab": {
          "referenced_widgets": [
            "96120576fcfc4543bbaa2144d8e05a7a"
          ]
        },
        "id": "30a4569e-6754-4cf3-87a0-3a1eee4f845e",
        "outputId": "56d7fa8c-c6f2-440b-fdef-e8f0da197c7d"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Found cached dataset squad_v2 (/users/ginter/.cache/huggingface/datasets/squad_v2/squad_v2/2.0.0/09187c73c1b837c95d9a249cd97c2c3f1cebada06efe667b4427714b27639b1d)\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "96120576fcfc4543bbaa2144d8e05a7a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/2 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "#squad_dataset_full=datasets.load_dataset(\"TurkuNLP/squad_v2_fi\")\n",
        "squad_dataset_full=datasets.load_dataset(\"squad_v2\")\n",
        "squad_dataset_full=squad_dataset_full.shuffle()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c7933312-dde0-4dba-868e-24b3956e2fb4",
      "metadata": {
        "id": "c7933312-dde0-4dba-868e-24b3956e2fb4",
        "outputId": "b045d2ed-9dbe-4fca-b5b2-797efe57dfad"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'id': '57321f24e17f3d14004226b0', 'title': 'Party_leaders_of_the_United_States_House_of_Representatives', 'context': 'In brief, there is disagreement among historical analysts as to the exact time period when the minority leadership emerged officially as a party position. Nonetheless, it seems safe to conclude that the position emerged during the latter part of the 19th century, a period of strong party organization and professional politicians. This era was \"marked by strong partisan attachments, resilient patronage-based party organizations, and...high levels of party voting in Congress.\" Plainly, these were conditions conducive to the establishment of a more highly differentiated House leadership structure.', 'question': 'What party characteristics emerged in the house in late 19th century?', 'answers': {'text': ['strong party organization and professional politicians'], 'answer_start': [276]}}\n",
            "\n",
            "{'id': '572fb50004bcaa1900d76c1f', 'title': 'Armenia', 'context': \"Gorbachev's inability to alleviate any of Armenia's problems created disillusionment among the Armenians and fed a growing hunger for independence. In May 1990, the New Armenian Army (NAA) was established, serving as a defence force separate from the Soviet Red Army. Clashes soon broke out between the NAA and Soviet Internal Security Forces (MVD) troops based in Yerevan when Armenians decided to commemorate the establishment of the 1918 First Republic of Armenia. The violence resulted in the deaths of five Armenians killed in a shootout with the MVD at the railway station. Witnesses there claimed that the MVD used excessive force and that they had instigated the fighting.\", 'question': 'What does NAA stand for?', 'answers': {'text': ['New Armenian Army'], 'answer_start': [165]}}\n",
            "\n",
            "{'id': '572707be5951b619008f84ef', 'title': 'Crimean_War', 'context': 'For months each side had been building forward rifle pits and defensive positions, which resulted in many skirmishes. Artillery fire aiming to gain superiority over the enemy guns.:450–462 September saw the final assault. On 5 September, another French bombardment (the sixth) was followed by an assault by the French Army on 8 September resulting in the capture of the Malakoff by the French, and following their failure to retake it, the collapse of the Russian defences. Meanwhile, the British captured the Great Redan, just south of the city of Sevastopol. The Russians retreated to the north, blowing up their magazines and the city fell on 9 September 1855 after a 337-day-long siege.:106', 'question': 'What was being built that caused unpremeditated fighting on each side?', 'answers': {'text': ['forward rifle pits and defensive positions'], 'answer_start': [39]}}\n",
            "\n",
            "{'id': '5a46f2ae5fd40d001a27dd01', 'title': 'Rajasthan', 'context': 'Rajasthani cooking was influenced by both the war-like lifestyles of its inhabitants and the availability of ingredients in this arid region. Food that could last for several days and could be eaten without heating was preferred. The scarcity of water and fresh green vegetables have all had their effect on the cooking. It is known for its snacks like Bikaneri Bhujia. Other famous dishes include bajre ki roti (millet bread) and lashun ki chutney (hot garlic paste), mawa kachori Mirchi Bada, Pyaaj Kachori and ghevar from Jodhpur, Alwar ka Mawa(Milk Cake), malpauas from Pushkar and rassgollas from Bikaner. Originating from the Marwar region of the state is the concept Marwari Bhojnalaya, or vegetarian restaurants, today found in many parts of India, which offer vegetarian food of the Marwari people. 4 Dal-Bati-Churma is very popular in Rajasthan. The traditional way to serve it is to first coarsely mash the Baati then pour pure Ghee on top of it. It is served with the daal (lentils) and spicy garlic chutney. Also served with Besan (gram flour) ki kadi. It is commonly served at all festivities, including religious occasions, wedding ceremonies, and birthday parties in Rajasthan. \"Dal-Baati-Churma\", is a combination of three different food items — Daal (lentils), Baati and Churma (Sweet). It is a typical Rajasthani dish.', 'question': 'How is Pyaaj Kachori served?', 'answers': {'text': [], 'answer_start': []}}\n",
            "\n",
            "{'id': '5a555ca9134fea001a0e1a56', 'title': 'Computer_security', 'context': 'Today, computer security comprises mainly \"preventive\" measures, like firewalls or an exit procedure. A firewall can be defined as a way of filtering network data between a host or a network and another network, such as the Internet, and can be implemented as software running on the machine, hooking into the network stack (or, in the case of most UNIX-based operating systems such as Linux, built into the operating system kernel) to provide real time filtering and blocking. Another implementation is a so-called physical firewall which consists of a separate machine filtering network traffic. Firewalls are common amongst machines that are permanently connected to the Internet.', 'question': 'Which part of a Linux computer is the firewall built into?', 'answers': {'text': [], 'answer_start': []}}\n",
            "\n"
          ]
        }
      ],
      "source": [
        "for item in squad_dataset_full[\"train\"].select(range(5)):\n",
        "    print(item)\n",
        "    print()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "865f6078-8159-4ad3-86d1-c14635a19f37",
      "metadata": {
        "id": "865f6078-8159-4ad3-86d1-c14635a19f37"
      },
      "source": [
        "# QA as a classification task\n",
        "\n",
        "* As a warm-up problem, let us cast QA as a classification problem\n",
        "* The task is: does the string at hand contain the answer to the question?\n",
        "* This is not how you would do QA but it is a good start, and an interesting problem in its own right\n",
        "* Maybe we can give ourselves some slack, and go sentence-by-sentence\n",
        "* So, we need to split the context into sentences, and then assemble a yes/no classification data\n",
        "* But how do we split the text into sentences?\n",
        "* Udpipe is an old, but trusty library to do this\n",
        "* Trained models: https://ufal.mff.cuni.cz/udpipe/1/models#universal_dependencies_25_models\n",
        "* Go familiarize yourself with that tool and models!\n",
        "* For Finnish, make sure you use the TDT model (Made in Turku), which is the best\n",
        "* For English, the EWT (English Web Treebank) model is a good choice"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "557eeba8-e8ca-4785-b298-bc1427c76e9f",
      "metadata": {
        "id": "557eeba8-e8ca-4785-b298-bc1427c76e9f",
        "outputId": "322e39b0-de18-4b58-fdff-5b3a04f2a5d2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: ufal.udpipe in /users/ginter/.local/lib/python3.9/site-packages (1.2.0.3)\n"
          ]
        }
      ],
      "source": [
        "!pip3 install ufal.udpipe"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2418d68d-1f81-49f4-8fc6-28e52a1a3fdc",
      "metadata": {
        "id": "2418d68d-1f81-49f4-8fc6-28e52a1a3fdc",
        "outputId": "5c827cd6-9a78-4f01-a63a-8879f792a5f1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2023-01-31 11:32:18--  https://lindat.mff.cuni.cz/repository/xmlui/bitstream/handle/11234/1-3131/english-ewt-ud-2.5-191206.udpipe?sequence=17&isAllowed=y\n",
            "Resolving lindat.mff.cuni.cz (lindat.mff.cuni.cz)... 195.113.20.140\n",
            "Connecting to lindat.mff.cuni.cz (lindat.mff.cuni.cz)|195.113.20.140|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 16309608 (16M) [application/octet-stream]\n",
            "Saving to: 'english-ewt.udpipe'\n",
            "\n",
            "english-ewt.udpipe  100%[===================>]  15.55M  30.4MB/s    in 0.5s    \n",
            "\n",
            "2023-01-31 11:32:19 (30.4 MB/s) - 'english-ewt.udpipe' saved [16309608/16309608]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget -O english-ewt.udpipe 'https://lindat.mff.cuni.cz/repository/xmlui/bitstream/handle/11234/1-3131/english-ewt-ud-2.5-191206.udpipe?sequence=17&isAllowed=y'"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2ca8dc89-999d-4df0-80e0-79cb0115f015",
      "metadata": {
        "id": "2ca8dc89-999d-4df0-80e0-79cb0115f015"
      },
      "source": [
        "# Running UDPipe\n",
        "\n",
        "* This is surprisingly easy, once you figure it out\n",
        "* The documentation (from which I figured it out) is here:\n",
        "  * https://ufal.mff.cuni.cz/udpipe/1/users-manual#run_udpipe_tokenizer\n",
        "  * https://ufal.mff.cuni.cz/udpipe/1/api-reference#pipeline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e475b270-d2e2-4347-865c-d1fc87929bc8",
      "metadata": {
        "id": "e475b270-d2e2-4347-865c-d1fc87929bc8"
      },
      "outputs": [],
      "source": [
        "import ufal.udpipe\n",
        "udpipemodel=ufal.udpipe.Model.load(\"english-ewt.udpipe\")\n",
        "tokenizer=ufal.udpipe.Pipeline(udpipemodel,\"tokenizer=ranges\",\"none\",\"none\",\"conllu\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9e6f55a8-7e62-4081-bae3-deb88db4c8c8",
      "metadata": {
        "id": "9e6f55a8-7e62-4081-bae3-deb88db4c8c8",
        "outputId": "c29f3027-5ea0-4505-b904-55a0fe05756f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "# newdoc\n",
            "# newpar\n",
            "# sent_id = 1\n",
            "# text = I have a dog.\n",
            "1\tI\t_\t_\t_\t_\t_\t_\t_\tTokenRange=0:1\n",
            "2\thave\t_\t_\t_\t_\t_\t_\t_\tTokenRange=2:6\n",
            "3\ta\t_\t_\t_\t_\t_\t_\t_\tTokenRange=7:8\n",
            "4\tdog\t_\t_\t_\t_\t_\t_\t_\tSpaceAfter=No|TokenRange=9:12\n",
            "5\t.\t_\t_\t_\t_\t_\t_\t_\tTokenRange=12:13\n",
            "\n",
            "# sent_id = 2\n",
            "# text = The dog is cute.\n",
            "1\tThe\t_\t_\t_\t_\t_\t_\t_\tTokenRange=14:17\n",
            "2\tdog\t_\t_\t_\t_\t_\t_\t_\tTokenRange=18:21\n",
            "3\tis\t_\t_\t_\t_\t_\t_\t_\tTokenRange=22:24\n",
            "4\tcute\t_\t_\t_\t_\t_\t_\t_\tSpaceAfter=No|TokenRange=25:29\n",
            "5\t.\t_\t_\t_\t_\t_\t_\t_\tTokenRange=29:30\n",
            "\n",
            "# sent_id = 3\n",
            "# text = And brown at that.\n",
            "1\tAnd\t_\t_\t_\t_\t_\t_\t_\tTokenRange=31:34\n",
            "2\tbrown\t_\t_\t_\t_\t_\t_\t_\tTokenRange=35:40\n",
            "3\tat\t_\t_\t_\t_\t_\t_\t_\tTokenRange=41:43\n",
            "4\tthat\t_\t_\t_\t_\t_\t_\t_\tSpaceAfter=No|TokenRange=44:48\n",
            "5\t.\t_\t_\t_\t_\t_\t_\t_\tSpacesAfter=\\n|TokenRange=48:49\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(tokenizer.process(\"I have a dog. The dog is cute. And brown at that.\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "11cf8162-c521-43de-a5d2-e41b9856905a",
      "metadata": {
        "id": "11cf8162-c521-43de-a5d2-e41b9856905a"
      },
      "source": [
        "# Find the sentence with the answer\n",
        "* Let's take a shortcut and simply use the `text=` field to find sentences which have the answer\n",
        "* Of course this might lead to spurious hits where the string is repeated in several sentences, but acts as the correct answer only in one\n",
        "* The right solution would be to use the `TokenRange` data to find the correct sentence that has the right answer\n",
        "* I leave that to you as an extra exercise if you wish\n",
        "* All you need is to parse the output of UDPipe, takes an extra for-loop, really"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "678d1651-c46a-4cd1-8ada-68dadeb09068",
      "metadata": {
        "id": "678d1651-c46a-4cd1-8ada-68dadeb09068"
      },
      "outputs": [],
      "source": [
        "import tqdm\n",
        "def get_sentences(parsed):\n",
        "    # gather the text= lines with the sentences\n",
        "    sents=[line.replace(\"# text = \",\"\") for line in parsed.split(\"\\n\") if line.startswith(\"# text = \")]\n",
        "    return sents\n",
        "\n",
        "def qa_to_binary(example,tokenizer):\n",
        "    context=example[\"context\"]\n",
        "    question=example[\"question\"]\n",
        "    sentences=get_sentences(tokenizer.process(context))\n",
        "    result=[]\n",
        "    #compare every sentence with every answer, when you find an answer, you have an example\n",
        "    for sent in sentences:\n",
        "        for answer in example[\"answers\"][\"text\"]:\n",
        "            if answer in sent:\n",
        "                result.append({\"question\":question,\"context\":sent,\"label\":1,\"answer\":answer})\n",
        "                break\n",
        "        else: #we found no answers, so this sentence is then 0\n",
        "            result.append({\"question\":question,\"context\":sent,\"label\":0,\"answer\":None})\n",
        "    return result\n",
        "\n",
        "train_processed=[]\n",
        "for item in tqdm.tqdm(squad_dataset_full[\"train\"].select(range(50000))):\n",
        "    train_processed.extend(qa_to_binary(item,tokenizer))\n",
        "test_processed=[]\n",
        "for item in tqdm.tqdm(squad_dataset_full[\"validation\"].select(range(2000))):\n",
        "    test_processed.extend(qa_to_binary(item,tokenizer))\n",
        "\n",
        "                      \n",
        "\n",
        "        "
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5f60f800-4de1-44d0-897b-c33fa9906c6f",
      "metadata": {
        "id": "5f60f800-4de1-44d0-897b-c33fa9906c6f"
      },
      "source": [
        "# Save the data\n",
        "\n",
        "* Now the usual trick of saving the data in a nice, processed form as jsonl\n",
        "* And then load as a HF Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "32d3267d-962c-4073-8ac0-466cae821f4d",
      "metadata": {
        "id": "32d3267d-962c-4073-8ac0-466cae821f4d"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "with open(\"squad_2_binarized_train.json\",\"wt\") as f:\n",
        "    json.dump(train_processed,f,ensure_ascii=False,indent=2)\n",
        "with open(\"squad_2_binarized_test.json\",\"wt\") as f:\n",
        "    json.dump(test_processed,f,ensure_ascii=False,indent=2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "a09276bc-3444-4e97-aa2c-15df6979ae2a",
      "metadata": {
        "colab": {
          "referenced_widgets": [
            "ece561ff96d84573ba4afc505ecd67f4",
            "0056bc7acd664ef59a87497e5f379fdb",
            "f2c86f462c1346a4860d7ddb38ae1ca8",
            "",
            "c837323f063e4a4aa07020e9ef536db9"
          ]
        },
        "id": "a09276bc-3444-4e97-aa2c-15df6979ae2a",
        "outputId": "c7a9d17c-059e-4a8d-b5a1-0b28a4358a6e"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Using custom data configuration default-5ff78b1b6d565ae6\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading and preparing dataset json/default (download: 74.41 MiB, generated: 58.22 MiB, post-processed: Unknown size, total: 132.62 MiB) to /users/ginter/.cache/huggingface/datasets/json/default-5ff78b1b6d565ae6/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab...\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ece561ff96d84573ba4afc505ecd67f4",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data files:   0%|          | 0/2 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "  "
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "0056bc7acd664ef59a87497e5f379fdb",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Extracting data files #1:   0%|          | 0/1 [00:00<?, ?obj/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "f2c86f462c1346a4860d7ddb38ae1ca8",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Extracting data files #0:   0%|          | 0/1 [00:00<?, ?obj/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating train split:   0%|          | 0/259266 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating test split:   0%|          | 0/10703 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Dataset json downloaded and prepared to /users/ginter/.cache/huggingface/datasets/json/default-5ff78b1b6d565ae6/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab. Subsequent calls will reuse this data.\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c837323f063e4a4aa07020e9ef536db9",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/2 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "#here you have a version you can download:\n",
        "dset=datasets.load_dataset(\"json\",data_files={\"train\":\"http://dl.turkunlp.org/TKO_8964_2023/squad_2_binarized_train.json\",\"test\":\"http://dl.turkunlp.org/TKO_8964_2023/squad_2_binarized_test.json\"})\n",
        "#dset=datasets.load_dataset(\"json\",data_files={\"train\":\"squad_2_binarized_train.json\",\"test\":\"squad_2_binarized_test.json\"},download_mode=\"force_redownload\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f3d6cf37-abac-4ca8-92a7-e77fd507e3dd",
      "metadata": {
        "id": "f3d6cf37-abac-4ca8-92a7-e77fd507e3dd"
      },
      "outputs": [],
      "source": [
        "dset=dset.shuffle()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f19e1760-3ee4-4235-b8a1-70d1ac70e9a0",
      "metadata": {
        "id": "f19e1760-3ee4-4235-b8a1-70d1ac70e9a0"
      },
      "source": [
        "# Train the classifier\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6e58c148-5e9e-456d-928f-8acc94ebe323",
      "metadata": {
        "id": "6e58c148-5e9e-456d-928f-8acc94ebe323"
      },
      "outputs": [],
      "source": [
        "MODEL = 'bert-base-cased'\n",
        "tokenizer = transformers.AutoTokenizer.from_pretrained(MODEL)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7e6d8bd2-863b-4bb7-9df9-9a90b9acaa3b",
      "metadata": {
        "id": "7e6d8bd2-863b-4bb7-9df9-9a90b9acaa3b"
      },
      "source": [
        "* The tokenizer now needs a pair of texts (the question and the context)\n",
        "* For BERT, it will actually then do the right thing with token type IDs\n",
        "* For Finnish you can grab the Finnish BERT model from huggingface (FinBERT)\n",
        "* The two texts are passed as `text` and `text_pair` (for some confusing naming)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "35ac7cd3-77e6-4e14-b631-2966f75b62ba",
      "metadata": {
        "id": "35ac7cd3-77e6-4e14-b631-2966f75b62ba",
        "outputId": "888d968f-20e8-4b34-c8b4-cee46f7a0a49"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Loading cached processed dataset at shuffled_binarized_squad2_dataset/train/cache-35d9c69123ad365c.arrow\n",
            "Loading cached processed dataset at shuffled_binarized_squad2_dataset/test/cache-94d56838e5efe70a.arrow\n"
          ]
        }
      ],
      "source": [
        "def tokenize(example):\n",
        "    return tokenizer(text=example[\"question\"],text_pair=example[\"context\"], truncation=True)\n",
        "\n",
        "dataset = dset.map(tokenize)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "079909a4-ea61-4f84-97e8-d271fad8ef74",
      "metadata": {
        "id": "079909a4-ea61-4f84-97e8-d271fad8ef74"
      },
      "outputs": [],
      "source": [
        "trainer_args = transformers.TrainingArguments(\n",
        "    output_dir='checkpoints',\n",
        "    evaluation_strategy='steps',\n",
        "    logging_strategy='steps',\n",
        "    load_best_model_at_end=True,\n",
        "    eval_steps=250,\n",
        "    logging_steps=250,\n",
        "    learning_rate=0.000015,\n",
        "    per_device_train_batch_size=32,\n",
        "    per_device_eval_batch_size=32,\n",
        "    max_steps=5000,\n",
        "    save_steps=250,\n",
        "    save_total_limit=6 #I will have an early stopping with patience 5, so max 6 checkpoints sounds reasonable\n",
        ")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1711a9c2-7f7d-4583-b7d9-53265f73b6cb",
      "metadata": {
        "id": "1711a9c2-7f7d-4583-b7d9-53265f73b6cb"
      },
      "source": [
        "# How to evaluate?\n",
        "\n",
        "* The data is imbalanced\n",
        "* The positive class is much rarer than the negative class\n",
        "* High accuracy does not tell the whole story\n",
        "* It will be good to report also precision and recall, so we can gauge what happens to the class"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fc0bf723-bc29-4ec5-8a96-30fb44faf94f",
      "metadata": {
        "colab": {
          "referenced_widgets": [
            "10e19c8d52cf4953b9aae556851769b9",
            "20e57ae95a484b59b9a92e10db35be70"
          ]
        },
        "id": "fc0bf723-bc29-4ec5-8a96-30fb44faf94f",
        "outputId": "f25ef2d9-b017-4613-8f93-6622f3843b99"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "10e19c8d52cf4953b9aae556851769b9",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading builder script:   0%|          | 0.00/7.55k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "20e57ae95a484b59b9a92e10db35be70",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading builder script:   0%|          | 0.00/7.36k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "accuracy = evaluate.load('accuracy')\n",
        "\n",
        "\n",
        "def compute_accuracy(outputs_and_labels):\n",
        "    outputs, labels = outputs_and_labels\n",
        "    predictions = outputs.argmax(axis=-1) # pick index of \"winning\" label\n",
        "    acc=accuracy.compute(predictions=predictions, references=labels)\n",
        "    return acc"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "26e5c19e-245e-4aa6-9485-1f46e3c7ead1",
      "metadata": {
        "id": "26e5c19e-245e-4aa6-9485-1f46e3c7ead1"
      },
      "source": [
        "* Let's also add EarlyStopping"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fb80b16d-5fe1-4119-88bd-6a24873de874",
      "metadata": {
        "id": "fb80b16d-5fe1-4119-88bd-6a24873de874"
      },
      "outputs": [],
      "source": [
        "estopping_cback=transformers.EarlyStoppingCallback(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e7a2e347-175d-41ce-b1db-e3aadb04e635",
      "metadata": {
        "id": "e7a2e347-175d-41ce-b1db-e3aadb04e635"
      },
      "source": [
        "# Class imbalance\n",
        "\n",
        "* One way to deal with imbalanced data is to give the minority class higher weight during training\n",
        "* I.e. the loss is weighted by class-dependent weights\n",
        "* A way to implement this can be copied straight from the HF documentation\n",
        "* You can invent some weights, or maybe you can use the formula/code from e.g. here https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ecec50b5-f1c7-4449-bbce-943a9e5d7b0f",
      "metadata": {
        "id": "ecec50b5-f1c7-4449-bbce-943a9e5d7b0f"
      },
      "outputs": [],
      "source": [
        "# A little housekeeping\n",
        "# to help with various memory leaks\n",
        "\n",
        "import gc\n",
        "try:\n",
        "    del trainer\n",
        "except:\n",
        "    pass\n",
        "try:\n",
        "    del model\n",
        "except:\n",
        "    pass\n",
        "gc.collect()\n",
        "torch.cuda.empty_cache()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "805c5133-9c05-41e9-a73f-90f97a034d25",
      "metadata": {
        "id": "805c5133-9c05-41e9-a73f-90f97a034d25"
      },
      "outputs": [],
      "source": [
        "#This is something I needed on CSC's puhti\n",
        "#why, I have no idea\n",
        "try:\n",
        "    import mlflow\n",
        "    mlflow.end_run()\n",
        "    mlflow.start_run()\n",
        "except:\n",
        "    pass\n",
        "\n",
        "model = transformers.AutoModelForSequenceClassification.from_pretrained(\n",
        "    MODEL,\n",
        "    num_labels=2\n",
        " )\n",
        "\n",
        "#So, somewhere here would come in a code which does\n",
        "#the class weightinh\n",
        "\n",
        "trainer = transformers.Trainer(\n",
        "    model=model,\n",
        "    args=trainer_args,\n",
        "    train_dataset=dataset['train'],\n",
        "    eval_dataset=dataset['test'],\n",
        "    compute_metrics=compute_accuracy,\n",
        "    tokenizer=tokenizer,\n",
        "    callbacks=[estopping_cback], \n",
        ")\n",
        "\n",
        "trainer.train()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "514b26ef-9de9-4c3b-9684-f5cbe454670c",
      "metadata": {
        "id": "514b26ef-9de9-4c3b-9684-f5cbe454670c"
      },
      "outputs": [],
      "source": [
        "trainer.save_model(\"english-binarized.model\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Trained models\n",
        "\n",
        "* I left the trained models for you here, in case you want them\n",
        "* http://dl.turkunlp.org/TKO_8964_2023/"
      ],
      "metadata": {
        "id": "evEfKK43KCKo"
      },
      "id": "evEfKK43KCKo"
    },
    {
      "cell_type": "markdown",
      "id": "1d8d2a1e-1143-48fa-b996-e5401512a25c",
      "metadata": {
        "id": "1d8d2a1e-1143-48fa-b996-e5401512a25c"
      },
      "source": [
        "# Test the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "b654c239-9add-4228-b459-8e4446dda795",
      "metadata": {
        "id": "b654c239-9add-4228-b459-8e4446dda795"
      },
      "outputs": [],
      "source": [
        "#try also the -weighted model from the url above\n",
        "model=transformers.AutoModelForSequenceClassification.from_pretrained(\"english-binarized.model\")\n",
        "tokenizer=transformers.AutoTokenizer.from_pretrained(\"english-binarized.model\",truncation=True,padding=True)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dset=datasets.load_dataset(\"json\",data_files={\"train\":\"http://dl.turkunlp.org/TKO_8964_2023/squad_2_binarized_train.json\",\"test\":\"http://dl.turkunlp.org/TKO_8964_2023/squad_2_binarized_test.json\"})"
      ],
      "metadata": {
        "id": "LmVLqfl1KXma"
      },
      "id": "LmVLqfl1KXma",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "02f9cd75-93a4-48e7-a686-3460ef06390c",
      "metadata": {
        "id": "02f9cd75-93a4-48e7-a686-3460ef06390c"
      },
      "outputs": [],
      "source": [
        "pipe=transformers.TextClassificationPipeline(model=model,tokenizer=tokenizer,device=0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "9fadaf07-5a45-4443-968b-0b08c5a68c72",
      "metadata": {
        "id": "9fadaf07-5a45-4443-968b-0b08c5a68c72"
      },
      "outputs": [],
      "source": [
        "# Hmm, this is not the right way to go about it, perhaps, but the KeyPairDataset\n",
        "# I could not get working, even though that would be the right way to do it\n",
        "# {\"text\":[listoftexts],\"text_pair\":[listoftexts]} should have worked according to docs,\n",
        "# but did not in reality :D\n",
        "res=pipe([{\"text\":e[\"question\"],\"text_pair\":e[\"context\"]} for e in dset[\"test\"].select(range(100))])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d58c99a4-3778-4b7e-8ca7-80d597168493",
      "metadata": {
        "id": "d58c99a4-3778-4b7e-8ca7-80d597168493"
      },
      "outputs": [],
      "source": [
        "# Let's print out some interesting misclassifications\n",
        "\n",
        "for prediction,datapoint in zip(res,dset[\"test\"].select(range(100))):\n",
        "    if prediction[\"label\"]==\"LABEL_0\" and datapoint[\"label\"]==1:\n",
        "        print(datapoint[\"question\"])\n",
        "        print(datapoint[\"context\"])\n",
        "        print()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.13"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "accelerator": "GPU",
    "gpuClass": "standard",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {}
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}