UmojaHack2021BioChallenge.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "indaba_challenge",
      "language": "python",
      "name": "indaba_challenge"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.9"
    },
    "colab": {
      "name": "UmojaHack2021BioChallenge.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/NaderJS/hacknotebooks/blob/master/UmojaHack2021BioChallenge.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iaSrJlia0b3_"
      },
      "source": [
        "# DeepChain Antibody Classification Challenge\n",
        "### Evaluating neutralising antibodies for the next influenza pandemic using the DeepChain™ platform\n",
        "Authors : Mohamed Jedidi (meds.jedidi@instadeep.com), Marcin Skwark (m.skwark@instadeep.com), Nicolas Lopez Carranza  (n.lopezcarranza@instadeep.com). Any Issues running this notebook feeel free to email us. \n",
        "\n",
        "Welcome to the challenge. In this notebook we will create a neural network that is able to estimate the binding energy between an antibody and Influenza HA protein\n",
        "\n",
        "\n",
        "\n",
        "<img src=\"influenza-infection2.png\" width=\"600\">\n",
        "\n",
        "## How we will build a scorer for Antibody Binding?\n",
        "The antibody sequence is 221 amino acids long and will be difficult to learn from the entire sequence. Hence we will explore in DeepChain which are the regions on the antibody that are in contact with the receptor binding domain of the virus. These regions are called [CDR regions](https://en.wikipedia.org/wiki/Complementarity-determining_region) and DeepChains shows them in green\n",
        "\n",
        "<img src=\"deepchain.png\" width=\"800\">\n",
        "\n",
        "\n",
        "We already extracted these 3 regions for you but feel free to use the entire sequence as input for your model. \n",
        "\n",
        "Enough said. Lets get our hands dirty and build a simple biLSTM model!\n",
        "\n",
        "## Binding scorer example"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XzE-S1fN0b4E"
      },
      "source": [
        "Download relevant Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mxe5BJdh6hUX",
        "outputId": "84c987d6-217d-4307-f66f-d1fee356c4f6"
      },
      "source": [
        "from google.colab  import drive\n",
        "\n",
        "drive.mount(\"/content/gdrive\")\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/gdrive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7jBb0HyREBK4"
      },
      "source": [
        "!cp \"/content/gdrive/MyDrive/UHDeepChain.zip\" ."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RL0TvtZWELKT",
        "outputId": "c1722ff9-2d8e-4f5c-83a7-6fce73059633"
      },
      "source": [
        "!unzip  -P lpdsv \"UHDeepChain.zip\""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Archive:  UHDeepChain.zip\n",
            "   creating: DeepChain/\n",
            "  inflating: DeepChain/SampleSubmission.csv  \n",
            "  inflating: DeepChain/CDR2_ProtBert_embeddings.csv  \n",
            "  inflating: DeepChain/CDR3_ProtBert_embeddings.csv  \n",
            "  inflating: DeepChain/train.csv     \n",
            "  inflating: DeepChain/UmojaHack2021BioChallenge.ipynb  \n",
            "  inflating: DeepChain/influenza-infection2.png  \n",
            "  inflating: DeepChain/deepchain.png  \n",
            "   creating: DeepChain/.ipynb_checkpoints/\n",
            "  inflating: DeepChain/.ipynb_checkpoints/DataCheck-checkpoint.ipynb  \n",
            "  inflating: DeepChain/CDR1_ProtBert_embeddings.csv  \n",
            "  inflating: DeepChain/test.csv      \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2-_7YB_V0b4G",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f243c248-e919-4220-c7cc-6c5871c4b0e4"
      },
      "source": [
        "import pandas as  pd \n",
        "import numpy as np \n",
        "import tensorflow as tf\n",
        "import os \n",
        "import urllib\n",
        "import datetime\n",
        "\n",
        "from tqdm  import tqdm\n",
        "import random\n",
        "import torch\n",
        "from torch import optim\n",
        "from torch import nn\n",
        "from torch.nn import functional as F\n",
        "if torch.cuda.is_available():  \n",
        "  dev = \"cuda:0\" \n",
        "  print(\"gpu up\")\n",
        "else:  \n",
        "  dev = \"cpu\"  \n",
        "device = torch.device(dev)\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"3\" # change it to \"0\" if yo have only one gpu or the gpu numbe  that you would like to use \n",
        "tf.get_logger().setLevel('ERROR')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "gpu up\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xTE6jEZpDnSV"
      },
      "source": [
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed(42)\n",
        "random.seed(42)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2NOMvnIg0b4M"
      },
      "source": [
        "data = pd.read_csv(\"DeepChain/train.csv\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "T5Xh6YVk0b4N",
        "outputId": "7f41609a-f903-47ca-db9b-ccfa74c899d4"
      },
      "source": [
        "# display the first 5 rows from the train file \n",
        "data.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ID</th>\n",
              "      <th>sequence</th>\n",
              "      <th>CDR1</th>\n",
              "      <th>CDR2</th>\n",
              "      <th>CDR3</th>\n",
              "      <th>binding</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>train_seq_0</td>\n",
              "      <td>QVQLKEHGPGLVNPSQSLSVTCSVSGFLLISNGVHWVRQPPGKGLE...</td>\n",
              "      <td>LLISN</td>\n",
              "      <td>WAGGNTN</td>\n",
              "      <td>YDYDNFTY</td>\n",
              "      <td>-11.36922</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>train_seq_1</td>\n",
              "      <td>QVQLKEYGPGLVAPSQSLSITCTVSGFLLISNGVHWVRQPPGKGLE...</td>\n",
              "      <td>LLISN</td>\n",
              "      <td>WAGGMTA</td>\n",
              "      <td>YCYDVFYY</td>\n",
              "      <td>-14.23689</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>train_seq_2</td>\n",
              "      <td>QVQLKESGGGLVAPSQSLSITCTVSGFLLTSAGVHWVRQPPGKGLH...</td>\n",
              "      <td>LLTSA</td>\n",
              "      <td>WAGGYVN</td>\n",
              "      <td>YDYDHFYY</td>\n",
              "      <td>-10.90296</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>train_seq_3</td>\n",
              "      <td>QVQLKESGPGLVAPSSSLSDTCEVSGFLLQSHGVHWVRQPPGKGLE...</td>\n",
              "      <td>LLQSH</td>\n",
              "      <td>WAGGNTN</td>\n",
              "      <td>HDADCFYY</td>\n",
              "      <td>-9.19866</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>train_seq_4</td>\n",
              "      <td>QCQLKESGPGMVAPSQSLSITCTVSGFLLTSNGVHWVRQPPGKGLE...</td>\n",
              "      <td>LLTSN</td>\n",
              "      <td>WAGGHTN</td>\n",
              "      <td>YDYDRFYY</td>\n",
              "      <td>-11.25864</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            ID  ...   binding\n",
              "0  train_seq_0  ... -11.36922\n",
              "1  train_seq_1  ... -14.23689\n",
              "2  train_seq_2  ... -10.90296\n",
              "3  train_seq_3  ...  -9.19866\n",
              "4  train_seq_4  ... -11.25864\n",
              "\n",
              "[5 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l1zreRNI0b4O"
      },
      "source": [
        "Split the train file to 90% train and 10% validation "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yeHecYCf0b4P",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4e69488c-2422-49bb-8998-9aa3ab1a5e8d"
      },
      "source": [
        "len(voc_set)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "23"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 66
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rJQJn5KW0b4Q"
      },
      "source": [
        "# create dictionary to map Amino acids to integer\n",
        "voc_set=set(['P', 'V', 'I', 'K', 'N', 'B', 'F', 'Y', 'E', 'W', 'R', 'D', 'X', 'S', 'C', 'U', 'Q', 'A', 'M', 'H', 'L', 'G', 'T'])\n",
        "voc_set_map={ k:v for k , v in zip(voc_set,range(1,len(voc_set)+1))}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 759
        },
        "id": "E9gtRIuZitFD",
        "outputId": "879bed71-a4a5-46fa-ac56-de0c2f33bc5c"
      },
      "source": [
        "train "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ID</th>\n",
              "      <th>sequence</th>\n",
              "      <th>CDR1</th>\n",
              "      <th>CDR2</th>\n",
              "      <th>CDR3</th>\n",
              "      <th>binding</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>3996</th>\n",
              "      <td>[3996]</td>\n",
              "      <td>QSQLRESGPGLVAPSQSLSITCTVSGFLLVSNGVHWVSQPPGGGCE...</td>\n",
              "      <td>[19, 19, 23, 22, 6]</td>\n",
              "      <td>[10, 8, 15, 15, 21, 11, 6, 13, 14, 13, 14, 23,...</td>\n",
              "      <td>[13, 14, 13, 14, 23, 15, 13, 13]</td>\n",
              "      <td>-12.72924</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25846</th>\n",
              "      <td>[25846]</td>\n",
              "      <td>QVQLKEEGPGLVAPSQSLSITCTVSGFLLISNGVHWVRQPPGKGLE...</td>\n",
              "      <td>[19, 19, 16, 22, 6]</td>\n",
              "      <td>[10, 8, 15, 15, 14, 11, 6]</td>\n",
              "      <td>[13, 14, 13, 14, 3, 15, 13, 13]</td>\n",
              "      <td>-8.93703</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31731</th>\n",
              "      <td>[31731]</td>\n",
              "      <td>QVLLKESGPGLVAPSQSLSITCTVSGFLLQSNGVHWVRQGPGYGLE...</td>\n",
              "      <td>[19, 19, 20, 22, 6]</td>\n",
              "      <td>[10, 8, 15, 15, 6, 11, 6]</td>\n",
              "      <td>[13, 14, 13, 14, 9, 9, 13, 13]</td>\n",
              "      <td>-11.29854</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>448</th>\n",
              "      <td>[448]</td>\n",
              "      <td>QCALKESGPGLVAPSQSLSITCTVSGFLLSSNGVHWVRQPPGKGLE...</td>\n",
              "      <td>[19, 19, 22, 22, 6]</td>\n",
              "      <td>[10, 8, 15, 15, 6, 11, 8]</td>\n",
              "      <td>[13, 3, 13, 14, 23, 13, 13, 13]</td>\n",
              "      <td>-11.81610</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25903</th>\n",
              "      <td>[25903]</td>\n",
              "      <td>QVQLKESGPGLVAPNQSLSITCLVSGFLLIKNGVHWVAQPPGKGLS...</td>\n",
              "      <td>[19, 19, 16, 18, 6]</td>\n",
              "      <td>[10, 8, 15, 15, 6, 11, 22]</td>\n",
              "      <td>[13, 14, 13, 14, 23, 9, 13, 13]</td>\n",
              "      <td>-12.67680</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29375</th>\n",
              "      <td>[29375]</td>\n",
              "      <td>TVQLHECGPGLVAPSQSLSITCTVSGFLLASNGVHWVRQPPGKGLE...</td>\n",
              "      <td>[19, 19, 8, 22, 6]</td>\n",
              "      <td>[10, 8, 15, 15, 6, 3, 6]</td>\n",
              "      <td>[13, 14, 13, 14, 23, 8, 13, 13]</td>\n",
              "      <td>-9.71109</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17950</th>\n",
              "      <td>[17950]</td>\n",
              "      <td>QVQLKESGIGLVAPSKSLSITCTVSGFLLISNGVHWVRQPPGKGLI...</td>\n",
              "      <td>[19, 19, 16, 22, 6]</td>\n",
              "      <td>[10, 8, 15, 15, 2, 11, 6]</td>\n",
              "      <td>[13, 14, 13, 14, 22, 3, 13, 13]</td>\n",
              "      <td>-11.02665</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26074</th>\n",
              "      <td>[26074]</td>\n",
              "      <td>QVQLIESGPGCVAPGQSLSITCTVSGFLLISAGVHWVRQPPGKGLE...</td>\n",
              "      <td>[19, 19, 16, 22, 8]</td>\n",
              "      <td>[10, 8, 15, 15, 2, 11, 6]</td>\n",
              "      <td>[19, 14, 14, 14, 23, 9, 13, 13]</td>\n",
              "      <td>-11.08821</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37200</th>\n",
              "      <td>[37200]</td>\n",
              "      <td>QVQLHELGPGLVAPSQSLSITCTVSGFLLIGNGVHWVRQPPGKGLE...</td>\n",
              "      <td>[19, 19, 16, 15, 6]</td>\n",
              "      <td>[10, 8, 15, 15, 6, 11, 22]</td>\n",
              "      <td>[13, 14, 13, 14, 17, 9, 13, 13]</td>\n",
              "      <td>-12.13245</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32971</th>\n",
              "      <td>[32971]</td>\n",
              "      <td>IVQLKESGPGLVAPSQSLSITCTVSGFLLMSNGVHWVRQPPGKGLE...</td>\n",
              "      <td>[19, 19, 2, 22, 6]</td>\n",
              "      <td>[10, 8, 15, 15, 14, 11, 6]</td>\n",
              "      <td>[13, 14, 13, 14, 21, 9, 13, 13]</td>\n",
              "      <td>-11.96487</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>32147 rows × 6 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "            ID  ...   binding\n",
              "3996    [3996]  ... -12.72924\n",
              "25846  [25846]  ...  -8.93703\n",
              "31731  [31731]  ... -11.29854\n",
              "448      [448]  ... -11.81610\n",
              "25903  [25903]  ... -12.67680\n",
              "...        ...  ...       ...\n",
              "29375  [29375]  ...  -9.71109\n",
              "17950  [17950]  ... -11.02665\n",
              "26074  [26074]  ... -11.08821\n",
              "37200  [37200]  ... -12.13245\n",
              "32971  [32971]  ... -11.96487\n",
              "\n",
              "[32147 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oEAPkP5O0b4R",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "814c8a47-e927-4bd3-8f78-a354120ecf95"
      },
      "source": [
        "train,val=train_test_split(data,test_size=0.2,random_state=1994)\n",
        "test = pd.read_csv(\"DeepChain/test.csv\")\n",
        "\n",
        "# process sequences \n",
        "# first step : map Amino acids to integer \n",
        "# second step : convert the sequence of integer  to a list \n",
        "\n",
        "train.CDR1=train.CDR1.apply(lambda x: [voc_set_map[e] for e in x ])\n",
        "train.CDR2=train.CDR2.apply(lambda x: [voc_set_map[e] for e in x ])\n",
        "train.CDR3=train.CDR3.apply(lambda x: [voc_set_map[e] for e in x ])\n",
        "train.sequence=train.sequence.apply(lambda x: [voc_set_map[e] for e in x ])\n",
        "cdr1 = np.array(np.array(train[[\"CDR1\"]].values).tolist()).reshape(len(train),-1)\n",
        "cdr2 = np.array(np.array(train[[\"CDR2\"]].values).tolist()).reshape(len(train),-1)\n",
        "cdr3 = np.array(np.array(train[[\"CDR3\"]].values).tolist()).reshape(len(train),-1)\n",
        "sequence = np.array(np.array(train[[\"sequence\"]].values).tolist()).reshape(len(train),-1)\n",
        "train[\"CDR\"]= np.concatenate((cdr1,cdr2,cdr3,sequence),axis=1).tolist()\n",
        "\n",
        "val.CDR1=val.CDR1.apply(lambda x: [voc_set_map[e] for e in x ])\n",
        "val.CDR2=val.CDR2.apply(lambda x: [voc_set_map[e] for e in x ])\n",
        "val.CDR3=val.CDR3.apply(lambda x: [voc_set_map[e] for e in x ])\n",
        "val.sequence=val.sequence.apply(lambda x: [voc_set_map[e] for e in x ])\n",
        "cdr1 = np.array(np.array(val[[\"CDR1\"]].values).tolist()).reshape(len(val),-1)\n",
        "cdr2 = np.array(np.array(val[[\"CDR2\"]].values).tolist()).reshape(len(val),-1)\n",
        "cdr3 = np.array(np.array(val[[\"CDR3\"]].values).tolist()).reshape(len(val),-1)\n",
        "sequence = np.array(np.array(val[[\"sequence\"]].values).tolist()).reshape(len(val),-1)\n",
        "val[\"CDR\"]= np.concatenate((cdr1,cdr2,cdr3,sequence),axis=1).tolist()\n",
        "\n",
        "\n",
        "test.CDR1=test.CDR1.apply(lambda x: [voc_set_map[e] for e in x ])\n",
        "test.CDR2=test.CDR2.apply(lambda x: [voc_set_map[e] for e in x ])\n",
        "test.CDR3=test.CDR3.apply(lambda x: [voc_set_map[e] for e in x ])\n",
        "test.sequence=test.sequence.apply(lambda x: [voc_set_map[e] for e in x ])\n",
        "cdr1 = np.array(np.array(test[[\"CDR1\"]].values).tolist()).reshape(len(test),-1)\n",
        "cdr2 = np.array(np.array(test[[\"CDR2\"]].values).tolist()).reshape(len(test),-1)\n",
        "cdr3 = np.array(np.array(test[[\"CDR3\"]].values).tolist()).reshape(len(test),-1)\n",
        "sequence = np.array(np.array(test[[\"sequence\"]].values).tolist()).reshape(len(test),-1)\n",
        "test[\"CDR\"]= np.concatenate((cdr1,cdr2,cdr3,sequence),axis=1).tolist()\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/pandas/core/generic.py:5170: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  self[name] = value\n",
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:16: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  app.launch_new_instance()\n",
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:26: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yDZaqf5RawnX"
      },
      "source": [
        "class AntiBodyTransformer(nn.Module):\n",
        "  def __init__(self):\n",
        "    super(AntiBodyTransformer,self).__init__()\n",
        "    self.cdr1EmbeddingLayer = nn.Embedding(25,48)\n",
        "    self.cdr2EmbeddingLayer = nn.Embedding(25,48)\n",
        "    self.cdr3EmbeddingLayer = nn.Embedding(25,48)\n",
        "    self.sequenceEmbeddingLayer = nn.Embedding(25,48)\n",
        "    \n",
        "    # self.cdr1EmbeddingLayer.weight.requires_grad = False\n",
        "    # self.cdr2EmbeddingLayer.weight.requires_grad = False\n",
        "    # self.cdr3EmbeddingLayer.weight.requires_grad = False\n",
        "    self.transEncoderLayer = nn.TransformerEncoderLayer(48,8)\n",
        "    self.batchnorm1 = nn.BatchNorm1d(241)\n",
        "    self.linear1= nn.Linear(48,1)\n",
        "    self.drop1 = nn.Dropout(0.2)\n",
        "    self.drop2 = nn.Dropout(0.2)\n",
        "    self.linear2= nn.Linear(241,1)\n",
        "\n",
        "  def forward(self,x):\n",
        "    cdr1EmbeddingVectors = self.cdr1EmbeddingLayer(x[:,:5].long())\n",
        "    cdr2EmbeddingVectors = self.cdr2EmbeddingLayer(x[:,5:12].long())\n",
        "    cdr3EmbeddingVectors = self.cdr3EmbeddingLayer(x[:,12:20].long())\n",
        "    sequenceEmbeddingVectors = self.sequenceEmbeddingLayer(x[:,20:].long())\n",
        "    \n",
        "    x= torch.cat([cdr1EmbeddingVectors,cdr2EmbeddingVectors,cdr3EmbeddingVectors,sequenceEmbeddingVectors],axis=1)\n",
        "    # print(x.size())\n",
        "    # return x\n",
        "    atten= self.transEncoderLayer(x)\n",
        "    atten= self.drop1(atten)\n",
        "    out = F.leaky_relu(self.linear1(atten))\n",
        "    \n",
        "    out = out.view(-1,241)\n",
        "    out=  self.batchnorm1(out)\n",
        "    # out = self.batchnorm1(out)\n",
        "    out = self.drop2(out)\n",
        "\n",
        "    out = self.linear2(out)\n",
        "    return out\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0pSKLzFLnIuv"
      },
      "source": [
        "\n",
        "batch_size = 512\n",
        "\n",
        "model = AntiBodyTransformer().to(device)\n",
        "optimizer = optim.Adam(model.parameters(),lr = 0.01,weight_decay=0.0001)\n",
        "npData = np.array(train[[\"CDR\",\"binding\"]].values)\n",
        "valnp = np.array(val[[\"CDR\",\"binding\"]].values)\n",
        "# target = npData[:,1]\n",
        "def ceil(a,b):\n",
        "    return -(-a//b)\n",
        "\n",
        "n_samples = len(npData)\n",
        "better_batch_size = ceil(n_samples, ceil(n_samples, batch_size))\n",
        "\n",
        "for e in range(510):\n",
        "  model.train()\n",
        "  trainerrs = []\n",
        "  for i in range(ceil(n_samples, better_batch_size)):\n",
        "    batchx = np.array(npData[i*better_batch_size : (i+1)*better_batch_size,0].tolist())\n",
        "    batchy = np.array(npData[i*better_batch_size : (i+1)*better_batch_size,1:].tolist())\n",
        "    batchy = torch.Tensor(batchy.astype(\"float32\")).to(device)\n",
        "    batchx = torch.Tensor(batchx.astype(\"float32\")).to(device)\n",
        "    model.zero_grad()\n",
        "    pred = model(batchx)\n",
        "    # break\n",
        "    err = F.mse_loss(pred,batchy)\n",
        "    err.backward()\n",
        "    trainerrs.append(np.sqrt(err.item()))\n",
        "    optimizer.step()\n",
        "  # break\n",
        "  model.eval()\n",
        "  valtrues = torch.Tensor([])\n",
        "  valpreds = torch.Tensor([])\n",
        "  for i in range(ceil(len(val), better_batch_size)):\n",
        "    validTensorx = torch.Tensor(np.array(valnp[i*better_batch_size : (i+1)*better_batch_size,0].tolist()).astype(\"float32\")).to(device)\n",
        "    validTensory = torch.Tensor(np.array(valnp[i*better_batch_size : (i+1)*better_batch_size,1:].tolist()).astype(\"float32\"))\n",
        "    valpred = model(validTensorx).cpu().detach()\n",
        "    valtrues = torch.cat([valtrues,validTensory])\n",
        "    valpreds = torch.cat([valpreds,valpred])\n",
        "\n",
        "  err = F.mse_loss(valpreds,valtrues)\n",
        "  if np.sqrt(err.item()) < bestvalid :\n",
        "    print(\"new best\")\n",
        "    torch.save(model.state_dict(),\"./model.tar\")\n",
        "    bestvalid =np.sqrt(err.item())\n",
        "\n",
        "  print(\"train \",np.mean(trainerrs), \" valid\",np.sqrt(err.item()))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0tAQ29jGH2XA",
        "outputId": "ce12365a-f94c-4b0b-8005-81903aec279e"
      },
      "source": [
        "checkpoint = torch.load(\"./model.tar\")\n",
        "model.load_state_dict(checkpoint)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<All keys matched successfully>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KFT9vR2jJUi0"
      },
      "source": [
        "nptest = np.array(test[[\"CDR\"]].values)\n",
        "valpreds = torch.Tensor([])\n",
        "for i in range(ceil(len(nptest), better_batch_size)):\n",
        "  validTensorx = torch.Tensor(np.array(nptest[i*better_batch_size : (i+1)*better_batch_size,0].tolist()).astype(\"float32\")).to(device)\n",
        "  # validTensory = torch.Tensor(np.array(val[i*better_batch_size : (i+1)*better_batch_size,1:].tolist()).astype(\"float32\"))\n",
        "  valpred = model(validTensorx).cpu().detach()\n",
        "  valpreds = torch.cat([valpreds,valpred])\n",
        "\n",
        "\n",
        "sub=test[[\"ID\"]].copy()\n",
        "sub[\"binding\"]=valpreds.numpy()\n",
        "sub.to_csv(\"sub.csv\",index=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4otTj5FN0b4Z"
      },
      "source": [
        "test_pred=model.predict(X_test,verbose=True)\n",
        "sub=test[[\"ID\"]].copy()\n",
        "sub[\"binding\"]=test_pred\n",
        "sub.to_csv(\"sub.csv\",index=False)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}