workshop_2_deliverable.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/CriticalDatathon/workshops/blob/main/workshop_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Workshop 2: Clinical Variables Selection & Feature Engineering"
      ],
      "metadata": {
        "id": "M65WOuF6u62h"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 🎯 Workshop Goals\n",
        "The goal of this workshop is to provide participants with a deep understanding of data preprocessing in the data science workflow. Specifically, by the end of this workshop, participants should be able to:\n",
        "\n",
        "1. **Understand the importance of Data Preprocessing Techniques**: Understanding thes significance of data preprocessing in the data science workflow. This includes to be able to apply common techniques such as cleaning, normalization, transformation, and reduction of data. This also includes handling missing data, outliers, skewed data, and data with different scales.\n",
        "\n",
        "3. **Familiarize with Data Pre-processing Techniques**: such as feature scaling, dimensionality reduction, and feature engineering.\n",
        "\n",
        "4. **Apply data pre-processing techniques**: This involved the practical application of data preprocessing techniques to real-world datasets and to be able to evaluate the impact of different preprocessing techniques on machine learning model performance.\n",
        "\n",
        "Be aware of the potential biases that can be introduced in data preprocessing, and how to identify and mitigate them.\n",
        "Throughout the workshop, participants will engage in hands-on activities, case studies, and real-world examples. They will work in groups to apply the concepts learned to real datasets, and engage in discussions to share their experiences and insights. By the end of the workshop, participants should have gained a solid understanding of data preprocessing techniques and their importance in the data science workflow, and be able to apply these techniques to improve the performance of machine learning models."
      ],
      "metadata": {
        "id": "p-DviQlnd7vT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# ✏️ Expected Deliverables\n",
        "\n",
        "1. A report or dashboard summarizing the results of EDA, including visualizations and statistical summaries of the data distribution and correlations.\n",
        "\n",
        "2. A set of code scripts or pipelines that automate the data preprocessing process, making it easier and more efficient to apply these techniques to future datasets.\n",
        "\n",
        "3. A cleaned dataset that has undergone preprocessing techniques such as removal of duplicates, handling missing data, and dealing with outliers. The cleaned dataset should be ready to be fed into machine learning models."
      ],
      "metadata": {
        "id": "lHt_KgdnGilV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# ❗ Highlighted Pitfall(s)\n",
        "1. Make sure to apply preprocessing steps only to the training data and avoid using information from the validation set to prevent data leakage.\n",
        "\n",
        "2. Check for potential biases that could be introduced or amplified by preprocessing techniques, and evaluate the impact of these techniques on different subgroups of the data.\n",
        "\n",
        "3. Carefully evaluate the appropriateness of different preprocessing techniques for a given dataset and ensure that the techniques are applied correctly to avoid incorrect preprocessing that could lead to poor model performance or incorrect conclusions about the data."
      ],
      "metadata": {
        "id": "EnR2K_EKGrEV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **1.  Setup environment**\n",
        "---"
      ],
      "metadata": {
        "id": "EffSajZhq7lo"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "otcVcnAOQEUH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fc5e0525-5551-42ab-fa3c-8ad4a0263751"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: ipykernel in /usr/local/lib/python3.10/dist-packages (6.23.1)\n",
            "Requirement already satisfied: comm>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from ipykernel) (0.1.3)\n",
            "Requirement already satisfied: debugpy>=1.6.5 in /usr/local/lib/python3.10/dist-packages (from ipykernel) (1.6.6)\n",
            "Requirement already satisfied: ipython>=7.23.1 in /usr/local/lib/python3.10/dist-packages (from ipykernel) (7.34.0)\n",
            "Requirement already satisfied: jupyter-client>=6.1.12 in /usr/local/lib/python3.10/dist-packages (from ipykernel) (6.1.12)\n",
            "Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /usr/local/lib/python3.10/dist-packages (from ipykernel) (5.3.0)\n",
            "Requirement already satisfied: matplotlib-inline>=0.1 in /usr/local/lib/python3.10/dist-packages (from ipykernel) (0.1.6)\n",
            "Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.10/dist-packages (from ipykernel) (1.5.6)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from ipykernel) (23.1)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from ipykernel) (5.9.5)\n",
            "Requirement already satisfied: pyzmq>=20 in /usr/local/lib/python3.10/dist-packages (from ipykernel) (23.2.1)\n",
            "Requirement already satisfied: tornado>=6.1 in /usr/local/lib/python3.10/dist-packages (from ipykernel) (6.3.1)\n",
            "Requirement already satisfied: traitlets>=5.4.0 in /usr/local/lib/python3.10/dist-packages (from ipykernel) (5.7.1)\n",
            "Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel) (67.7.2)\n",
            "Requirement already satisfied: jedi>=0.16 in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel) (0.18.2)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel) (4.4.2)\n",
            "Requirement already satisfied: pickleshare in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel) (0.7.5)\n",
            "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel) (3.0.38)\n",
            "Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel) (2.14.0)\n",
            "Requirement already satisfied: backcall in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel) (0.2.0)\n",
            "Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/dist-packages (from ipython>=7.23.1->ipykernel) (4.8.0)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.10/dist-packages (from jupyter-client>=6.1.12->ipykernel) (2.8.2)\n",
            "Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.10/dist-packages (from jupyter-core!=5.0.*,>=4.12->ipykernel) (3.3.0)\n",
            "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /usr/local/lib/python3.10/dist-packages (from jedi>=0.16->ipython>=7.23.1->ipykernel) (0.8.3)\n",
            "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.10/dist-packages (from pexpect>4.3->ipython>=7.23.1->ipykernel) (0.7.0)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=7.23.1->ipykernel) (0.2.6)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.1->jupyter-client>=6.1.12->ipykernel) (1.16.0)\n"
          ]
        }
      ],
      "source": [
        "#@markdown ## Import dependencies\n",
        "!pip install --upgrade ipykernel\n",
        "# Data reading in Dataframe format and data preprocessing\n",
        "import pandas as pd\n",
        "\n",
        "# Data Visualization\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "# For missing values\n",
        "import missingno as msno\n",
        "\n",
        "# Linear algebra operations\n",
        "import numpy as np\n",
        "\n",
        "# Preprocessing\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn import preprocessing\n",
        "\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "from sklearn.linear_model import LinearRegression\n",
        "\n",
        "from sklearn.feature_selection import SelectKBest\n",
        "from sklearn.feature_selection import f_classif, chi2, mutual_info_classif, mutual_info_regression\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown ##Play the cell to connect your Google Drive to Colab\n",
        "\n",
        "\n",
        "# mount user's Google Drive to Google Colab.\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "id": "Frc9RcmKR3Ea",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ede425e2-fcf9-4ac8-a3fe-9f6828bea196"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Set your path to the dataset \n",
        "\n",
        "Hint: `%cd /content/drive/MyDrive/Data_dir`"
      ],
      "metadata": {
        "id": "RFFYYJxYGyDb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Add you path to the dataset here:\n",
        "\n",
        "!ls"
      ],
      "metadata": {
        "id": "pYsx3v330G4u"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        " ## **2. Data Analysis**\n",
        "---\n",
        "\n",
        "\n",
        "Once we understand the problem, we have formed a multidisciplinary team, formulated the research question,  hypothesis, we have to start working with the data. What data do we have? How do they look? What distributions do they have?\n",
        "\n",
        "Let's remember that in data science, data is the most important thing, and whether or not we can solve a problem depends on the data quality. At the same time, understanding the data also helps us to have a clearer vision of what we are facing and in case the data is not very good, we can at least fix it.\n",
        "\n",
        "Specifically, understanding the problem and the data are the essential phases in a data science project. An error in this phase is much more critical than an error in the modeling and evaluation phases. We must bear in mind that machine learning is not a magical tool that solves any type of problem, but rather a mathematical/statistical tool that learns from what we teach it, therefore if the data has biases, the model will also have them."
      ],
      "metadata": {
        "id": "wsZkeFBG08pU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Read the dataset\n",
        "\n",
        "Let's asume that you already have the dataset of [workshop 1](https://github.com/CriticalDatathon/workshops/blob/main/solutions/workshop_1.py):"
      ],
      "metadata": {
        "id": "Hzzgjobkzz1u"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Read the dataset and use the function null_values to see which columns has missing data."
      ],
      "metadata": {
        "id": "ewfSG4dLHFxK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown ### ✏️ Read the dataset and use the function null_values to see which columns has missing data.\n",
        "\n",
        "df_path = \"/content/drive/MyDrive/datasets/mit-critical-datathon-2023/cleaned_pulseOx_data.csv\" #@param {type:\"string\"}\n"
      ],
      "metadata": {
        "id": "8F5j91Ii3WpP"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def null_values(df):\n",
        "  \"\"\" \n",
        "  This function checks if there are null values in the dataframe:\n",
        "  In case of not having null values, print: 'There aren't null values in the dataframe'\n",
        "  In case of having null values, print the columns with these values\n",
        "  \"\"\"\n",
        "  \n",
        "  nulos = df.isnull().sum().any()\n",
        "  if nulos == False:\n",
        "    print(\"There aren't null values in the dataframe\")\n",
        "  else:\n",
        "    print('Null values: ')\n",
        "    print(df.isnull().sum()[df.isnull().sum()  > 0])"
      ],
      "metadata": {
        "id": "T5NLdL3K1cib"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Read file here:\n",
        "df = pd.read_csv(df_path)\n",
        "print(f'The dataset has {df.shape[0]} rows and {df.shape[1]} columns')\n",
        "\n",
        "# Print first 5 rows\n",
        "df"
      ],
      "metadata": {
        "id": "vwAy1XAHR3Gq",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 674
        },
        "outputId": "cd00d05f-a642-4747-db89-c70cffa11db1"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The dataset has 40791 rows and 140 columns\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "       level_0  index  subject_id   stay_id       SaO2_timestamp  SaO2  \\\n",
              "0            0      0    10001884  37510196  2131-01-12 21:04:00  90.0   \n",
              "1            2      2    10002013  39060235  2160-05-18 16:03:00  99.0   \n",
              "2            7      7    10002428  38875437  2156-04-19 19:41:00  96.0   \n",
              "3            8      8    10002428  38875437  2156-04-20 00:46:00  97.0   \n",
              "4            9      9    10002428  38875437  2156-04-20 03:42:00  97.0   \n",
              "...        ...    ...         ...       ...                  ...   ...   \n",
              "40786    81595  81787    19997752  34531437  2128-03-02 17:54:00  97.0   \n",
              "40787    81597  81789    19998330  33428243  2178-11-28 07:14:00  95.0   \n",
              "40788    81599  81791    19998843  30988867  2187-02-06 05:11:00  98.0   \n",
              "40789    81600  81792    19999287  35165301  2197-08-07 02:12:00  97.0   \n",
              "40790    81603  81795    19999840  38978960  2164-09-17 08:11:00  93.0   \n",
              "\n",
              "       delta_SpO2  SpO2  hidden_hypoxemia   hadm_id  ... delta_mbp    mbp  \\\n",
              "0              -4    89                 0  26184834  ...        -3   96.0   \n",
              "1               0    99                 0  23581541  ...        -3   86.5   \n",
              "2             -11    94                 0  28662225  ...        10   98.0   \n",
              "3               0   100                 0  28662225  ...        -1   68.0   \n",
              "4             -16    98                 0  28662225  ...         3   77.0   \n",
              "...           ...   ...               ...       ...  ...       ...    ...   \n",
              "40786          -9    99                 0  29452285  ...         6   62.0   \n",
              "40787         -14    99                 0  24096838  ...       -14  112.0   \n",
              "40788         -11   100                 0  24842066  ...       -11   76.0   \n",
              "40789         -12    98                 0  20175828  ...       -12   83.0   \n",
              "40790         -11    93                 0  21033226  ...       -11   71.0   \n",
              "\n",
              "       delta_resp_rate resp_rate delta_temperature temperature delta_glucose  \\\n",
              "0                 -4.0      19.5             -64.0       36.72         445.0   \n",
              "1                 -2.0      14.0              -3.0       36.90           0.0   \n",
              "2                  4.0      22.0             -56.0       37.00           2.0   \n",
              "3                 -1.0      18.0              14.0       36.94         164.0   \n",
              "4                  3.0      20.0              18.0       37.39         -12.0   \n",
              "...                ...       ...               ...         ...           ...   \n",
              "40786              6.0      17.0            -114.0       36.33           0.0   \n",
              "40787            -14.0      25.0              46.0       36.50          46.0   \n",
              "40788            -11.0      27.0             -11.0       38.90           0.0   \n",
              "40789            -12.0      16.0             108.0       36.89         -12.0   \n",
              "40790            -11.0      23.0             -11.0       36.78           0.0   \n",
              "\n",
              "       glucose  delta_heart_rhythm                     heart_rhythm  \n",
              "0        199.0                -4.0                SR (Sinus Rhythm)  \n",
              "1        155.0                -3.0                SR (Sinus Rhythm)  \n",
              "2        183.0                19.0          ST (Sinus Tachycardia)   \n",
              "3        163.0                14.0                SR (Sinus Rhythm)  \n",
              "4        163.0                18.0                SR (Sinus Rhythm)  \n",
              "...        ...                 ...                              ...  \n",
              "40786    153.0                 6.0                          V Paced  \n",
              "40787    175.0               -14.0         AF (Atrial Fibrillation)  \n",
              "40788    142.0               -11.0                SR (Sinus Rhythm)  \n",
              "40789    129.0               -12.0  1st AV (First degree AV Block)   \n",
              "40790    166.0               -11.0                SR (Sinus Rhythm)  \n",
              "\n",
              "[40791 rows x 140 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6d59a140-72c7-4a4a-afeb-5a3874f73ffc\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>level_0</th>\n",
              "      <th>index</th>\n",
              "      <th>subject_id</th>\n",
              "      <th>stay_id</th>\n",
              "      <th>SaO2_timestamp</th>\n",
              "      <th>SaO2</th>\n",
              "      <th>delta_SpO2</th>\n",
              "      <th>SpO2</th>\n",
              "      <th>hidden_hypoxemia</th>\n",
              "      <th>hadm_id</th>\n",
              "      <th>...</th>\n",
              "      <th>delta_mbp</th>\n",
              "      <th>mbp</th>\n",
              "      <th>delta_resp_rate</th>\n",
              "      <th>resp_rate</th>\n",
              "      <th>delta_temperature</th>\n",
              "      <th>temperature</th>\n",
              "      <th>delta_glucose</th>\n",
              "      <th>glucose</th>\n",
              "      <th>delta_heart_rhythm</th>\n",
              "      <th>heart_rhythm</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>10001884</td>\n",
              "      <td>37510196</td>\n",
              "      <td>2131-01-12 21:04:00</td>\n",
              "      <td>90.0</td>\n",
              "      <td>-4</td>\n",
              "      <td>89</td>\n",
              "      <td>0</td>\n",
              "      <td>26184834</td>\n",
              "      <td>...</td>\n",
              "      <td>-3</td>\n",
              "      <td>96.0</td>\n",
              "      <td>-4.0</td>\n",
              "      <td>19.5</td>\n",
              "      <td>-64.0</td>\n",
              "      <td>36.72</td>\n",
              "      <td>445.0</td>\n",
              "      <td>199.0</td>\n",
              "      <td>-4.0</td>\n",
              "      <td>SR (Sinus Rhythm)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>10002013</td>\n",
              "      <td>39060235</td>\n",
              "      <td>2160-05-18 16:03:00</td>\n",
              "      <td>99.0</td>\n",
              "      <td>0</td>\n",
              "      <td>99</td>\n",
              "      <td>0</td>\n",
              "      <td>23581541</td>\n",
              "      <td>...</td>\n",
              "      <td>-3</td>\n",
              "      <td>86.5</td>\n",
              "      <td>-2.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>-3.0</td>\n",
              "      <td>36.90</td>\n",
              "      <td>0.0</td>\n",
              "      <td>155.0</td>\n",
              "      <td>-3.0</td>\n",
              "      <td>SR (Sinus Rhythm)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>7</td>\n",
              "      <td>7</td>\n",
              "      <td>10002428</td>\n",
              "      <td>38875437</td>\n",
              "      <td>2156-04-19 19:41:00</td>\n",
              "      <td>96.0</td>\n",
              "      <td>-11</td>\n",
              "      <td>94</td>\n",
              "      <td>0</td>\n",
              "      <td>28662225</td>\n",
              "      <td>...</td>\n",
              "      <td>10</td>\n",
              "      <td>98.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>22.0</td>\n",
              "      <td>-56.0</td>\n",
              "      <td>37.00</td>\n",
              "      <td>2.0</td>\n",
              "      <td>183.0</td>\n",
              "      <td>19.0</td>\n",
              "      <td>ST (Sinus Tachycardia)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "      <td>10002428</td>\n",
              "      <td>38875437</td>\n",
              "      <td>2156-04-20 00:46:00</td>\n",
              "      <td>97.0</td>\n",
              "      <td>0</td>\n",
              "      <td>100</td>\n",
              "      <td>0</td>\n",
              "      <td>28662225</td>\n",
              "      <td>...</td>\n",
              "      <td>-1</td>\n",
              "      <td>68.0</td>\n",
              "      <td>-1.0</td>\n",
              "      <td>18.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>36.94</td>\n",
              "      <td>164.0</td>\n",
              "      <td>163.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>SR (Sinus Rhythm)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>9</td>\n",
              "      <td>9</td>\n",
              "      <td>10002428</td>\n",
              "      <td>38875437</td>\n",
              "      <td>2156-04-20 03:42:00</td>\n",
              "      <td>97.0</td>\n",
              "      <td>-16</td>\n",
              "      <td>98</td>\n",
              "      <td>0</td>\n",
              "      <td>28662225</td>\n",
              "      <td>...</td>\n",
              "      <td>3</td>\n",
              "      <td>77.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>20.0</td>\n",
              "      <td>18.0</td>\n",
              "      <td>37.39</td>\n",
              "      <td>-12.0</td>\n",
              "      <td>163.0</td>\n",
              "      <td>18.0</td>\n",
              "      <td>SR (Sinus Rhythm)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40786</th>\n",
              "      <td>81595</td>\n",
              "      <td>81787</td>\n",
              "      <td>19997752</td>\n",
              "      <td>34531437</td>\n",
              "      <td>2128-03-02 17:54:00</td>\n",
              "      <td>97.0</td>\n",
              "      <td>-9</td>\n",
              "      <td>99</td>\n",
              "      <td>0</td>\n",
              "      <td>29452285</td>\n",
              "      <td>...</td>\n",
              "      <td>6</td>\n",
              "      <td>62.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>17.0</td>\n",
              "      <td>-114.0</td>\n",
              "      <td>36.33</td>\n",
              "      <td>0.0</td>\n",
              "      <td>153.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>V Paced</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40787</th>\n",
              "      <td>81597</td>\n",
              "      <td>81789</td>\n",
              "      <td>19998330</td>\n",
              "      <td>33428243</td>\n",
              "      <td>2178-11-28 07:14:00</td>\n",
              "      <td>95.0</td>\n",
              "      <td>-14</td>\n",
              "      <td>99</td>\n",
              "      <td>0</td>\n",
              "      <td>24096838</td>\n",
              "      <td>...</td>\n",
              "      <td>-14</td>\n",
              "      <td>112.0</td>\n",
              "      <td>-14.0</td>\n",
              "      <td>25.0</td>\n",
              "      <td>46.0</td>\n",
              "      <td>36.50</td>\n",
              "      <td>46.0</td>\n",
              "      <td>175.0</td>\n",
              "      <td>-14.0</td>\n",
              "      <td>AF (Atrial Fibrillation)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40788</th>\n",
              "      <td>81599</td>\n",
              "      <td>81791</td>\n",
              "      <td>19998843</td>\n",
              "      <td>30988867</td>\n",
              "      <td>2187-02-06 05:11:00</td>\n",
              "      <td>98.0</td>\n",
              "      <td>-11</td>\n",
              "      <td>100</td>\n",
              "      <td>0</td>\n",
              "      <td>24842066</td>\n",
              "      <td>...</td>\n",
              "      <td>-11</td>\n",
              "      <td>76.0</td>\n",
              "      <td>-11.0</td>\n",
              "      <td>27.0</td>\n",
              "      <td>-11.0</td>\n",
              "      <td>38.90</td>\n",
              "      <td>0.0</td>\n",
              "      <td>142.0</td>\n",
              "      <td>-11.0</td>\n",
              "      <td>SR (Sinus Rhythm)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40789</th>\n",
              "      <td>81600</td>\n",
              "      <td>81792</td>\n",
              "      <td>19999287</td>\n",
              "      <td>35165301</td>\n",
              "      <td>2197-08-07 02:12:00</td>\n",
              "      <td>97.0</td>\n",
              "      <td>-12</td>\n",
              "      <td>98</td>\n",
              "      <td>0</td>\n",
              "      <td>20175828</td>\n",
              "      <td>...</td>\n",
              "      <td>-12</td>\n",
              "      <td>83.0</td>\n",
              "      <td>-12.0</td>\n",
              "      <td>16.0</td>\n",
              "      <td>108.0</td>\n",
              "      <td>36.89</td>\n",
              "      <td>-12.0</td>\n",
              "      <td>129.0</td>\n",
              "      <td>-12.0</td>\n",
              "      <td>1st AV (First degree AV Block)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40790</th>\n",
              "      <td>81603</td>\n",
              "      <td>81795</td>\n",
              "      <td>19999840</td>\n",
              "      <td>38978960</td>\n",
              "      <td>2164-09-17 08:11:00</td>\n",
              "      <td>93.0</td>\n",
              "      <td>-11</td>\n",
              "      <td>93</td>\n",
              "      <td>0</td>\n",
              "      <td>21033226</td>\n",
              "      <td>...</td>\n",
              "      <td>-11</td>\n",
              "      <td>71.0</td>\n",
              "      <td>-11.0</td>\n",
              "      <td>23.0</td>\n",
              "      <td>-11.0</td>\n",
              "      <td>36.78</td>\n",
              "      <td>0.0</td>\n",
              "      <td>166.0</td>\n",
              "      <td>-11.0</td>\n",
              "      <td>SR (Sinus Rhythm)</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>40791 rows × 140 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6d59a140-72c7-4a4a-afeb-5a3874f73ffc')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-6d59a140-72c7-4a4a-afeb-5a3874f73ffc button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-6d59a140-72c7-4a4a-afeb-5a3874f73ffc');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df['diff_SpO2_SaO2'] = df['SpO2'] - df['SaO2']\n",
        "\n",
        "# df.diff_SpO2_SaO2.describe()\n",
        "\n",
        "df['diff_bins'] = np.nan\n",
        "\n",
        "# if df['diff_SpO2_SaO2'] <= -10:\n",
        "#   df['diff_bins'] = 0\n",
        "\n",
        "df.loc[df['diff_SpO2_SaO2'] < -10, ['diff_bins']] = 0\n",
        "\n",
        "df.loc[(df['diff_SpO2_SaO2'] > -10) & \n",
        "      (df['diff_SpO2_SaO2'] <= -8), ['diff_bins']] = 1\n",
        "\n",
        "df.loc[(df['diff_SpO2_SaO2'] > -8) & \n",
        "      (df['diff_SpO2_SaO2'] <= -6), ['diff_bins']] = 2\n",
        "\n",
        "df.loc[(df['diff_SpO2_SaO2'] > -6) & \n",
        "      (df['diff_SpO2_SaO2'] <= -4), ['diff_bins']] = 3\n",
        "\n",
        "df.loc[(df['diff_SpO2_SaO2'] > -4) & \n",
        "      (df['diff_SpO2_SaO2'] <= -2), ['diff_bins']] = 4\n",
        "\n",
        "df.loc[(df['diff_SpO2_SaO2'] > -2) & \n",
        "      (df['diff_SpO2_SaO2'] <= 0), ['diff_bins']] = 5\n",
        "\n",
        "df.loc[(df['diff_SpO2_SaO2'] > 0) & \n",
        "      (df['diff_SpO2_SaO2'] <= 2), ['diff_bins']] = 6\n",
        "\n",
        "df.loc[(df['diff_SpO2_SaO2'] > 2) & \n",
        "      (df['diff_SpO2_SaO2'] <= 4), ['diff_bins']] = 7\n",
        "\n",
        "df.loc[(df['diff_SpO2_SaO2'] > 4) & \n",
        "      (df['diff_SpO2_SaO2'] <= 6), ['diff_bins']] = 8\n",
        "\n",
        "df.loc[(df['diff_SpO2_SaO2'] > 6) & \n",
        "      (df['diff_SpO2_SaO2'] <= 8), ['diff_bins']] = 9\n",
        "\n",
        "df.loc[(df['diff_SpO2_SaO2'] > 8) & \n",
        "      (df['diff_SpO2_SaO2'] <= 10), ['diff_bins']] = 10\n",
        "\n",
        "df.loc[df['diff_SpO2_SaO2'] > 10, ['diff_bins']] = 11\n",
        "\n",
        "df['diff_bins']"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "guKzX8NuYqV0",
        "outputId": "356ed190-ab63-4752-94be-1582b6c9e7d7"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0        5.0\n",
              "1        5.0\n",
              "2        4.0\n",
              "3        7.0\n",
              "4        6.0\n",
              "        ... \n",
              "40786    6.0\n",
              "40787    7.0\n",
              "40788    6.0\n",
              "40789    6.0\n",
              "40790    5.0\n",
              "Name: diff_bins, Length: 40791, dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df['diff_bins'].value_counts().sort_index().plot(kind='bar', title='Bins for SpO2 - SaO2')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 158
        },
        "id": "4AapeZ5-XHuY",
        "outputId": "7ab95f8a-6ea9-4cd4-f8fb-0f1683e7e97d"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "error",
          "ename": "TypeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m/tmp/ipykernel_10423/3689423203.py\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'diff_bins'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue_counts\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mby\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'race_group'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msort_index\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mplot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkind\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'bar'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtitle\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'Bins for SpO2 - SaO2'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m: IndexOpsMixin.value_counts() got an unexpected keyword argument 'by'"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.diff_SpO2_SaO2.hist()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 445
        },
        "id": "nEqg6Y5nZtw2",
        "outputId": "d8f259fd-329e-4735-9eb1-5ceaa2cbeafd"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<Axes: >"
            ]
          },
          "metadata": {},
          "execution_count": 12
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjkAAAGdCAYAAADwjmIIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAkxUlEQVR4nO3df3DU9Z3H8VcSkg1RQgBLYkrAtFhB+SkpIVYtSkhApleUYYpyHnopnlxyY0iLGo9CkM7EwypSjIdOK3g3cCDOHHcCA9mGA7QsIpFUQOFqDw853NCKsPzQzZL93h+dfGW74UcwZDfvPB8zGdzv95PN5/vJF326+/0mCY7jOAIAADAmMdYTAAAAuBqIHAAAYBKRAwAATCJyAACASUQOAAAwicgBAAAmETkAAMAkIgcAAJjULdYTiKVwOKyjR4+qR48eSkhIiPV0AADAZXAcR6dOnVJ2drYSEy/8ek2XjpyjR48qJycn1tMAAABX4JNPPlG/fv0uuL9LR06PHj0k/XmR0tPTYzyb9hUKhVRbW6uioiIlJyfHejpdAmve8VjzjseadzzWPFogEFBOTo773/EL6dKR0/IWVXp6usnISUtLU3p6On8pOghr3vFY847Hmnc81vzCLnWpCRceAwAAk4gcAABgEpEDAABMInIAAIBJRA4AADCJyAEAACYROQAAwCQiBwAAmETkAAAAk4gcAABgEpEDAABMInIAAIBJRA4AADCJyAEAACZ1i/UEAHQONzy5IdZTiOJJcrRotDSkarOCzQlR+z9+ZlIMZgUgXvBKDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJrUpcqqrq/Xd735XPXr0UN++fTV58mQdPHgwYsyXX36p0tJS9enTR9dee62mTJmixsbGiDGHDx/WpEmTlJaWpr59+2rOnDk6d+5cxJitW7fq1ltvlcfj0cCBA7VixYqo+dTU1OiGG25Qamqq8vPztWvXrrYcDgAAMKxNkbNt2zaVlpZq586d8nq9CoVCKioq0pkzZ9wxs2fP1ptvvqm1a9dq27ZtOnr0qO677z53f3NzsyZNmqSmpibt2LFDr732mlasWKF58+a5Yw4dOqRJkybprrvuUkNDg8rLy/XjH/9YmzdvdsesWbNGFRUVmj9/vt577z0NHz5cxcXFOnbs2NdZDwAAYES3tgzetGlTxOMVK1aob9++qq+v15133qmTJ0/q17/+tVatWqW7775bkrR8+XINHjxYO3fu1JgxY1RbW6sPPvhAv/nNb5SZmakRI0Zo4cKFeuKJJ1RVVaWUlBQtW7ZMubm5eu655yRJgwcP1ttvv63FixeruLhYkvT8889r5syZevjhhyVJy5Yt04YNG/Tqq6/qySef/NoLAwAAOrc2Rc5fOnnypCSpd+/ekqT6+nqFQiEVFha6YwYNGqT+/fvL5/NpzJgx8vl8Gjp0qDIzM90xxcXFmjVrlvbv36+RI0fK5/NFPEfLmPLycklSU1OT6uvrVVlZ6e5PTExUYWGhfD7fBecbDAYVDAbdx4FAQJIUCoUUCoWucBXiU8vxWDuueGZ9zT1JTqynEMWT6ET8+Zesfi9iyfp5Ho9Y82iXuxZXHDnhcFjl5eX63ve+pyFDhkiS/H6/UlJSlJGRETE2MzNTfr/fHXN+4LTsb9l3sTGBQEBffPGFPv/8czU3N7c65sCBAxecc3V1tRYsWBC1vba2VmlpaZdx1J2P1+uN9RS6HKtrvmh0rGdwYQvzwq1u37hxYwfPpOuwep7HM9b8K2fPnr2scVccOaWlpdq3b5/efvvtK32KDldZWamKigr3cSAQUE5OjoqKipSenh7DmbW/UCgkr9er8ePHKzk5OdbT6RKsr/mQqs2XHtTBPImOFuaF9bPdiQqGE6L276sqjsGsbLN+nscj1jxayzsxl3JFkVNWVqb169dr+/bt6tevn7s9KytLTU1NOnHiRMSrOY2NjcrKynLH/OVdUC13X50/5i/vyGpsbFR6erq6d++upKQkJSUltTqm5Tla4/F45PF4orYnJyebPXEsH1u8srrmweboiIgXwXBCq/Oz+H2IF1bP83jGmn/lctehTXdXOY6jsrIy/fu//7u2bNmi3NzciP2jRo1ScnKy6urq3G0HDx7U4cOHVVBQIEkqKCjQ3r17I+6C8nq9Sk9P18033+yOOf85Wsa0PEdKSopGjRoVMSYcDquurs4dAwAAurY2vZJTWlqqVatW6T/+4z/Uo0cP9xqanj17qnv37urZs6dKSkpUUVGh3r17Kz09Xf/wD/+ggoICjRkzRpJUVFSkm2++WQ8++KAWLVokv9+vuXPnqrS01H2V5dFHH9WLL76oxx9/XH/7t3+rLVu26PXXX9eGDRvcuVRUVGjGjBnKy8vT6NGj9cILL+jMmTPu3VYAAKBra1Pk/PM//7MkaezYsRHbly9froceekiStHjxYiUmJmrKlCkKBoMqLi7WSy+95I5NSkrS+vXrNWvWLBUUFOiaa67RjBkz9PTTT7tjcnNztWHDBs2ePVtLlixRv3799Ktf/cq9fVySfvSjH+mPf/yj5s2bJ7/frxEjRmjTpk1RFyMDAICuqU2R4ziXvoU0NTVVNTU1qqmpueCYAQMGXPKuh7Fjx2rPnj0XHVNWVqaysrJLzgkAAHQ9/O4qAABgEpEDAABMInIAAIBJRA4AADCJyAEAACYROQAAwCQiBwAAmETkAAAAk4gcAABgEpEDAABMInIAAIBJRA4AADCJyAEAACYROQAAwCQiBwAAmETkAAAAk4gcAABgEpEDAABMInIAAIBJRA4AADCJyAEAACYROQAAwCQiBwAAmETkAAAAk4gcAABgUrdYTwAArpYbntwQ6ym02cfPTIr1FAAzeCUHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGBSmyNn+/bt+sEPfqDs7GwlJCRo3bp1EfsfeughJSQkRHxMmDAhYszx48c1ffp0paenKyMjQyUlJTp9+nTEmPfff1933HGHUlNTlZOTo0WLFkXNZe3atRo0aJBSU1M1dOhQbdy4sa2HAwAAjGpz5Jw5c0bDhw9XTU3NBcdMmDBBn376qfvxb//2bxH7p0+frv3798vr9Wr9+vXavn27HnnkEXd/IBBQUVGRBgwYoPr6ej377LOqqqrSK6+84o7ZsWOH7r//fpWUlGjPnj2aPHmyJk+erH379rX1kAAAgEHd2voJEydO1MSJEy86xuPxKCsrq9V9H374oTZt2qR3331XeXl5kqSlS5fqnnvu0S9+8QtlZ2dr5cqVampq0quvvqqUlBTdcsstamho0PPPP+/G0JIlSzRhwgTNmTNHkrRw4UJ5vV69+OKLWrZsWVsPCwAAGNPmyLkcW7duVd++fdWrVy/dfffd+vnPf64+ffpIknw+nzIyMtzAkaTCwkIlJibqnXfe0b333iufz6c777xTKSkp7pji4mL90z/9kz7//HP16tVLPp9PFRUVEV+3uLg46u2z8wWDQQWDQfdxIBCQJIVCIYVCofY49LjRcjzWjiueWV9zT5IT6ylE8SQ6EX9aEO/nj/XzPB6x5tEudy3aPXImTJig++67T7m5ufrDH/6gp556ShMnTpTP51NSUpL8fr/69u0bOYlu3dS7d2/5/X5Jkt/vV25ubsSYzMxMd1+vXr3k9/vdbeePaXmO1lRXV2vBggVR22tra5WWlnZFxxvvvF5vrKfQ5Vhd80WjYz2DC1uYF471FNpNZ7m20Op5Hs9Y86+cPXv2ssa1e+RMmzbN/eehQ4dq2LBh+va3v62tW7dq3Lhx7f3l2qSysjLi1Z9AIKCcnBwVFRUpPT09hjNrf6FQSF6vV+PHj1dycnKsp9MlWF/zIVWbYz2FKJ5ERwvzwvrZ7kQFwwmxnk672FdVHOspXJT18zwesebRWt6JuZSr8nbV+b71rW/puuuu00cffaRx48YpKytLx44dixhz7tw5HT9+3L2OJysrS42NjRFjWh5fasyFrgWS/nytkMfjidqenJxs9sSxfGzxyuqaB5vjNyKC4YS4nl9bdJZzx+p5Hs9Y869c7jpc9Z+Tc+TIEX322We6/vrrJUkFBQU6ceKE6uvr3TFbtmxROBxWfn6+O2b79u0R77l5vV7ddNNN6tWrlzumrq4u4mt5vV4VFBRc7UMCAACdQJsj5/Tp02poaFBDQ4Mk6dChQ2poaNDhw4d1+vRpzZkzRzt37tTHH3+suro6/fCHP9TAgQNVXPznl2AHDx6sCRMmaObMmdq1a5d++9vfqqysTNOmTVN2drYk6YEHHlBKSopKSkq0f/9+rVmzRkuWLIl4q+mxxx7Tpk2b9Nxzz+nAgQOqqqrS7t27VVZW1g7LAgAAOrs2R87u3bs1cuRIjRw5UpJUUVGhkSNHat68eUpKStL777+vv/qrv9J3vvMdlZSUaNSoUXrrrbci3iZauXKlBg0apHHjxumee+7R7bffHvEzcHr27Kna2lodOnRIo0aN0k9+8hPNmzcv4mfp3HbbbVq1apVeeeUVDR8+XG+88YbWrVunIUOGfJ31AAAARrT5mpyxY8fKcS58u+bmzZe+OLF3795atWrVRccMGzZMb7311kXHTJ06VVOnTr3k1wMAAF0Pv7sKAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgEltjpzt27frBz/4gbKzs5WQkKB169ZF7HccR/PmzdP111+v7t27q7CwUL///e8jxhw/flzTp09Xenq6MjIyVFJSotOnT0eMef/993XHHXcoNTVVOTk5WrRoUdRc1q5dq0GDBik1NVVDhw7Vxo0b23o4AADAqDZHzpkzZzR8+HDV1NS0un/RokX65S9/qWXLlumdd97RNddco+LiYn355ZfumOnTp2v//v3yer1av369tm/frkceecTdHwgEVFRUpAEDBqi+vl7PPvusqqqq9Morr7hjduzYofvvv18lJSXas2ePJk+erMmTJ2vfvn1tPSQAAGBQt7Z+wsSJEzVx4sRW9zmOoxdeeEFz587VD3/4Q0nSv/zLvygzM1Pr1q3TtGnT9OGHH2rTpk169913lZeXJ0launSp7rnnHv3iF79Qdna2Vq5cqaamJr366qtKSUnRLbfcooaGBj3//PNuDC1ZskQTJkzQnDlzJEkLFy6U1+vViy++qGXLll3RYgAAADva9ZqcQ4cOye/3q7Cw0N3Ws2dP5efny+fzSZJ8Pp8yMjLcwJGkwsJCJSYm6p133nHH3HnnnUpJSXHHFBcX6+DBg/r888/dMed/nZYxLV8HAAB0bW1+Jedi/H6/JCkzMzNie2ZmprvP7/erb9++kZPo1k29e/eOGJObmxv1HC37evXqJb/ff9Gv05pgMKhgMOg+DgQCkqRQKKRQKHTZx9kZtByPteOKZ9bX3JPkxHoKUTyJTsSfFsT7+WP9PI9HrHm0y12Ldo2ceFddXa0FCxZEba+trVVaWloMZnT1eb3eWE+hy7G65otGx3oGF7YwLxzrKbSbznIDhdXzPJ6x5l85e/bsZY1r18jJysqSJDU2Nur66693tzc2NmrEiBHumGPHjkV83rlz53T8+HH387OystTY2BgxpuXxpca07G9NZWWlKioq3MeBQEA5OTkqKipSenp6Ww417oVCIXm9Xo0fP17Jycmxnk6XYH3Nh1RtjvUUongSHS3MC+tnuxMVDCfEejrtYl9VcayncFHWz/N4xJpHa3kn5lLaNXJyc3OVlZWluro6N2oCgYDeeecdzZo1S5JUUFCgEydOqL6+XqNGjZIkbdmyReFwWPn5+e6Yf/zHf1QoFHK/oV6vVzfddJN69erljqmrq1N5ebn79b1erwoKCi44P4/HI4/HE7U9OTnZ7Ilj+djildU1DzbHb0QEwwlxPb+26CznjtXzPJ6x5l+53HVo84XHp0+fVkNDgxoaGiT9+WLjhoYGHT58WAkJCSovL9fPf/5z/ed//qf27t2rv/mbv1F2drYmT54sSRo8eLAmTJigmTNnateuXfrtb3+rsrIyTZs2TdnZ2ZKkBx54QCkpKSopKdH+/fu1Zs0aLVmyJOJVmMcee0ybNm3Sc889pwMHDqiqqkq7d+9WWVlZWw8JAAAY1OZXcnbv3q277rrLfdwSHjNmzNCKFSv0+OOP68yZM3rkkUd04sQJ3X777dq0aZNSU1Pdz1m5cqXKyso0btw4JSYmasqUKfrlL3/p7u/Zs6dqa2tVWlqqUaNG6brrrtO8efMifpbObbfdplWrVmnu3Ll66qmndOONN2rdunUaMmTIFS0EAACwpc2RM3bsWDnOhe9kSEhI0NNPP62nn376gmN69+6tVatWXfTrDBs2TG+99dZFx0ydOlVTp069+IQBAECXxO+uAgAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAExq98ipqqpSQkJCxMegQYPc/V9++aVKS0vVp08fXXvttZoyZYoaGxsjnuPw4cOaNGmS0tLS1LdvX82ZM0fnzp2LGLN161bdeuut8ng8GjhwoFasWNHehwIAADqxq/JKzi233KJPP/3U/Xj77bfdfbNnz9abb76ptWvXatu2bTp69Kjuu+8+d39zc7MmTZqkpqYm7dixQ6+99ppWrFihefPmuWMOHTqkSZMm6a677lJDQ4PKy8v14x//WJs3b74ahwMAADqhblflSbt1U1ZWVtT2kydP6te//rVWrVqlu+++W5K0fPlyDR48WDt37tSYMWNUW1urDz74QL/5zW+UmZmpESNGaOHChXriiSdUVVWllJQULVu2TLm5uXruueckSYMHD9bbb7+txYsXq7i4+GocEgAA6GSuSuT8/ve/V3Z2tlJTU1VQUKDq6mr1799f9fX1CoVCKiwsdMcOGjRI/fv3l8/n05gxY+Tz+TR06FBlZma6Y4qLizVr1izt379fI0eOlM/ni3iOljHl5eUXnVcwGFQwGHQfBwIBSVIoFFIoFGqHI48fLcdj7bjimfU19yQ5sZ5CFE+iE/GnBfF+/lg/z+MRax7tctei3SMnPz9fK1as0E033aRPP/1UCxYs0B133KF9+/bJ7/crJSVFGRkZEZ+TmZkpv98vSfL7/RGB07K/Zd/FxgQCAX3xxRfq3r17q3Orrq7WggULorbX1tYqLS3tio433nm93lhPocuxuuaLRsd6Bhe2MC8c6ym0m40bN8Z6CpfF6nkez1jzr5w9e/ayxrV75EycONH952HDhik/P18DBgzQ66+/fsH46CiVlZWqqKhwHwcCAeXk5KioqEjp6ekxnFn7C4VC8nq9Gj9+vJKTk2M9nS7B+poPqYq/a948iY4W5oX1s92JCoYTYj2ddrGvKr7fcrd+nscj1jxayzsxl3JV3q46X0ZGhr7zne/oo48+0vjx49XU1KQTJ05EvJrT2NjoXsOTlZWlXbt2RTxHy91X54/5yzuyGhsblZ6eftGQ8ng88ng8UduTk5PNnjiWjy1eWV3zYHP8RkQwnBDX82uLznLuWD3P4xlr/pXLXYer/nNyTp8+rT/84Q+6/vrrNWrUKCUnJ6uurs7df/DgQR0+fFgFBQWSpIKCAu3du1fHjh1zx3i9XqWnp+vmm292x5z/HC1jWp4DAACg3SPnpz/9qbZt26aPP/5YO3bs0L333qukpCTdf//96tmzp0pKSlRRUaH/+q//Un19vR5++GEVFBRozJgxkqSioiLdfPPNevDBB/W73/1Omzdv1ty5c1VaWuq+CvPoo4/qf/7nf/T444/rwIEDeumll/T6669r9uzZ7X04AACgk2r3t6uOHDmi+++/X5999pm+8Y1v6Pbbb9fOnTv1jW98Q5K0ePFiJSYmasqUKQoGgyouLtZLL73kfn5SUpLWr1+vWbNmqaCgQNdcc41mzJihp59+2h2Tm5urDRs2aPbs2VqyZIn69eunX/3qV9w+DgAAXO0eOatXr77o/tTUVNXU1KimpuaCYwYMGHDJOwzGjh2rPXv2XNEcAQCAffzuKgAAYBKRAwAATCJyAACASUQOAAAwicgBAAAmETkAAMAkIgcAAJhE5AAAAJOIHAAAYBKRAwAATCJyAACASUQOAAAwicgBAAAmETkAAMAkIgcAAJhE5AAAAJOIHAAAYBKRAwAATCJyAACASUQOAAAwicgBAAAmETkAAMAkIgcAAJhE5AAAAJOIHAAAYBKRAwAATCJyAACASUQOAAAwicgBAAAmETkAAMAkIgcAAJhE5AAAAJOIHAAAYBKRAwAATCJyAACASUQOAAAwicgBAAAmETkAAMAkIgcAAJhE5AAAAJOIHAAAYBKRAwAATCJyAACASUQOAAAwicgBAAAmETkAAMAkIgcAAJhE5AAAAJOIHAAAYBKRAwAATCJyAACASUQOAAAwicgBAAAmETkAAMAkIgcAAJhE5AAAAJOIHAAAYBKRAwAATCJyAACASUQOAAAwicgBAAAmETkAAMAkIgcAAJhE5AAAAJOIHAAAYBKRAwAATCJyAACASUQOAAAwqVusJwB0NTc8uSHWUwCALqHTv5JTU1OjG264QampqcrPz9euXbtiPSUAABAHOnXkrFmzRhUVFZo/f77ee+89DR8+XMXFxTp27FispwYAAGKsU79d9fzzz2vmzJl6+OGHJUnLli3Thg0b9Oqrr+rJJ5+M8ezQEeLprR9PkqNFo6UhVZsVbE6I9XTQScXTOd2a1s7zj5+ZFONZAa3rtJHT1NSk+vp6VVZWutsSExNVWFgon8/X6ucEg0EFg0H38cmTJyVJx48fVygUuroT7mChUEhnz57VZ599puTk5Mv6nPzquqs8q/YXTydwt7Cjs2fD6hZKVHOYyOkIrHnHa23NB/709RjPyjZPoqO5I8Nt+ve5dadOnZIkOY5z0XHx9N+INvnTn/6k5uZmZWZmRmzPzMzUgQMHWv2c6upqLViwIGp7bm7uVZkjup4HYj2BLog173isecdjzVt36tQp9ezZ84L7O23kXInKykpVVFS4j8PhsI4fP64+ffooIcHW/wUGAgHl5OTok08+UXp6eqyn0yWw5h2PNe94rHnHY82jOY6jU6dOKTs7+6LjOm3kXHfddUpKSlJjY2PE9sbGRmVlZbX6OR6PRx6PJ2JbRkbG1ZpiXEhPT+cvRQdjzTsea97xWPOOx5pHutgrOC067d1VKSkpGjVqlOrqvrqOJBwOq66uTgUFBTGcGQAAiAed9pUcSaqoqNCMGTOUl5en0aNH64UXXtCZM2fcu60AAEDX1akj50c/+pH++Mc/at68efL7/RoxYoQ2bdoUdTFyV+TxeDR//vyot+dw9bDmHY8173isecdjza9cgnOp+68AAAA6oU57TQ4AAMDFEDkAAMAkIgcAAJhE5AAAAJOIHMOCwaBGjBihhIQENTQ0ROx7//33dccddyg1NVU5OTlatGhRbCZpwMcff6ySkhLl5uaqe/fu+va3v6358+erqakpYhxr3r5qamp0ww03KDU1Vfn5+dq1a1esp2RGdXW1vvvd76pHjx7q27evJk+erIMHD0aM+fLLL1VaWqo+ffro2muv1ZQpU6J+OCuu3DPPPKOEhASVl5e721jztiNyDHv88cdb/ZHXgUBARUVFGjBggOrr6/Xss8+qqqpKr7zySgxm2fkdOHBA4XBYL7/8svbv36/Fixdr2bJleuqpp9wxrHn7WrNmjSoqKjR//ny99957Gj58uIqLi3Xs2LFYT82Ebdu2qbS0VDt37pTX61UoFFJRUZHOnDnjjpk9e7befPNNrV27Vtu2bdPRo0d13333xXDWdrz77rt6+eWXNWzYsIjtrPkVcGDSxo0bnUGDBjn79+93JDl79uxx97300ktOr169nGAw6G574oknnJtuuikGM7Vp0aJFTm5urvuYNW9fo0ePdkpLS93Hzc3NTnZ2tlNdXR3DWdl17NgxR5Kzbds2x3Ec58SJE05ycrKzdu1ad8yHH37oSHJ8Pl+spmnCqVOnnBtvvNHxer3O97//feexxx5zHIc1v1K8kmNQY2OjZs6cqX/9139VWlpa1H6fz6c777xTKSkp7rbi4mIdPHhQn3/+eUdO1ayTJ0+qd+/e7mPWvP00NTWpvr5ehYWF7rbExEQVFhbK5/PFcGZ2nTx5UpLcc7q+vl6hUCjiezBo0CD179+f78HXVFpaqkmTJkWsrcSaXykixxjHcfTQQw/p0UcfVV5eXqtj/H5/1E+Fbnns9/uv+hyt++ijj7R06VL93d/9nbuNNW8/f/rTn9Tc3NzqerKW7S8cDqu8vFzf+973NGTIEEl/PmdTUlKifsEx34OvZ/Xq1XrvvfdUXV0dtY81vzJETifx5JNPKiEh4aIfBw4c0NKlS3Xq1ClVVlbGesqd3uWu+fn+7//+TxMmTNDUqVM1c+bMGM0caD+lpaXat2+fVq9eHeupmPbJJ5/oscce08qVK5Wamhrr6ZjRqX93VVfyk5/8RA899NBFx3zrW9/Sli1b5PP5on7HSV5enqZPn67XXntNWVlZUVfktzzOyspq13l3Zpe75i2OHj2qu+66S7fddlvUBcWsefu57rrrlJSU1Op6spbtq6ysTOvXr9f27dvVr18/d3tWVpaampp04sSJiFcW+B5cufr6eh07dky33nqru625uVnbt2/Xiy++qM2bN7PmVyLWFwWhff3v//6vs3fvXvdj8+bNjiTnjTfecD755BPHcb66CLapqcn9vMrKSi6C/RqOHDni3Hjjjc60adOcc+fORe1nzdvX6NGjnbKyMvdxc3Oz881vfpMLj9tJOBx2SktLnezsbOe///u/o/a3XAT7xhtvuNsOHDjARbBfQyAQiPh39969e528vDznr//6r529e/ey5leIyDHu0KFDUXdXnThxwsnMzHQefPBBZ9++fc7q1audtLQ05+WXX47dRDuxI0eOOAMHDnTGjRvnHDlyxPn000/djxaseftavXq14/F4nBUrVjgffPCB88gjjzgZGRmO3++P9dRMmDVrltOzZ09n69atEefz2bNn3TGPPvqo079/f2fLli3O7t27nYKCAqegoCCGs7bn/LurHIc1vxJEjnGtRY7jOM7vfvc75/bbb3c8Ho/zzW9+03nmmWdiM0EDli9f7khq9eN8rHn7Wrp0qdO/f38nJSXFGT16tLNz585YT8mMC53Py5cvd8d88cUXzt///d87vXr1ctLS0px77703Iuzx9f1l5LDmbZfgOI7T4e+RAQAAXGXcXQUAAEwicgAAgElEDgAAMInIAQAAJhE5AADAJCIHAACYROQAAACTiBwAAGASkQMAAEwicgAAgElEDgAAMInIAQAAJv0/e4ZmFK794HgAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        " ## **3. Understand your Data**\n",
        "---\n"
      ],
      "metadata": {
        "id": "sKiLCcTe_1s2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Understand the variables:\n",
        "\n",
        "The first step is to understand what variables your dataset has and how these variables are distributed. The columns and data types are:\n",
        "\n",
        "Discuss with your team about the problem and what variables are needed \n",
        "\n",
        "\n",
        "| name                           | type    | description                                                                                                |\n",
        "|:-------------------------------|:--------|:-----------------------------------------------------------------------------------------------------------|\n",
        "| subject_id                     | int64   | Unique identifier for each patient                                                                         |\n",
        "| stay_id                        | int64   | Unique identifier for each hospital stay                                                                   |\n",
        "| SaO2_timestamp                 | object  | Timestamp for SaO2 measurement                                                                             |\n",
        "| SaO2                           | float64 | Arterial oxygen saturation                                                                                 |\n",
        "| delta_SpO2                     | int64   | Time offset (in minutes) in the measurement of peripheral oxygen saturation                                |\n",
        "| SpO2                           | int64   | Peripheral oxygen saturation                                                                               |\n",
        "| hidden_hypoxemia               | int64   | Indicates if the patient had hypoxemia without clinical signs                                              |\n",
        "| hadm_id                        | int64   | Unique identifier for each hospital admission                                                              |\n",
        "| gender                         | object  | Gender of the patient                                                                                      |\n",
        "| sex_female                     | int64   | Indicates if the patient is female                                                                         |\n",
        "| anchor_age                     | int64   | Age of the patient at the time of admission                                                                |\n",
        "| race                           | object  | Race of the patient                                                                                        |\n",
        "| race_group                     | object  | Grouping of race into broader categories                                                                   |\n",
        "| language                       | object  | Primary language spoken by the patient                                                                     |\n",
        "| insurance                      | object  | Type of insurance of the patient                                                                           |\n",
        "| weight                         | float64 | Weight of the patient in kilograms                                                                         |\n",
        "| height                         | float64 | Height of the patient in centimeters                                                                       |\n",
        "| BMI                            | float64 | Body Mass Index of the patient                                                                             |\n",
        "| anchor_year_group              | object  | Grouping of admission year into broader categories                                                         |\n",
        "| first_hosp_stay                | bool    | Indicates if this is the first hospital stay for the patient                                               |\n",
        "| first_icu_stay                 | bool    | Indicates if this is the first ICU stay for the patient                                                    |\n",
        "| icustay_seq                    | int64   | Sequence number of ICU stay for the patient                                                                |\n",
        "| admittime                      | object  | Timestamp for hospital admission                                                                           |\n",
        "| dischtime                      | object  | Timestamp for hospital discharge                                                                           |\n",
        "| icu_intime                     | object  | Timestamp for ICU admission                                                                                |\n",
        "| icu_outtime                    | object  | Timestamp for ICU discharge                                                                                |\n",
        "| los_hospital                   | int64   | Length of hospital stay in days                                                                            |\n",
        "| los_icu                        | float64 | Length of ICU stay in days                                                                                 |\n",
        "| CCI                            | int64   | Charlson Comorbidity Index                                                                                 |\n",
        "| SOFA_admission                 | int64   | Sequential Organ Failure Assessment (SOFA) score at admission                                              |\n",
        "| mortality_in                   | int64   | Indicates if the patient died during the hospital stay                                                     |\n",
        "| delta_vent_start               | float64 | Time since ventilation started (in minutes) at the time of the measurement                                 |\n",
        "| ventilation_status             | object  | Indicates if the patient was on mechanical ventilation                                                     |\n",
        "| invasive_vent                  | int64   | Indicates if the patient was on invasive mechanical ventilation                                            |\n",
        "| delta_FiO2                     | float64 | Time offset (in minutes) in the measurement of inspired oxygen (FiO2)                                      |\n",
        "| FiO2                           | float64 | Fraction of inspired oxygen                                                                                |\n",
        "| delta_rrt                      | float64 | Time since renal replacement therapy (in minutes) at the time of the measurement                           |\n",
        "| rrt                            | int64   | Indicates if the patient was on renal replacement therapy                                                  |\n",
        "| delta_vp_start                 | float64 | Time since vasopressor therapy started (in minutes) at the time of the measurement                         |\n",
        "| norepinephrine_equivalent_dose | float64 | Dose of norepinephrine equivalent to other vasopressors (in mcg/kg/min)                                    |\n",
        "| delta_sofa_coag                | float64 | Time offset (in minutes) in the measurement of SOFA score for coagulation from the previous measurement    |\n",
        "| sofa_coag                      | float64 | SOFA score for coagulation                                                                                 |\n",
        "| delta_sofa_liver               | float64 | Time offset (in minutes) in the measurement of SOFA score for liver from the previous measurement          |\n",
        "| sofa_liver                     | float64 | SOFA score for liver                                                                                       |\n",
        "| delta_sofa_cv                  | int64   | Time offset (in minutes) in the measurement of SOFA score for cardiovascular from the previous measurement |\n",
        "| sofa_cv                        | int64   | Cardiovascular component of Sequential Organ Failure Assessment (SOFA) score                               |\n",
        "| delta_sofa_cns                 | float64 | Time offset (in minutes) in the measurement of central nervous system component of SOFA                    |\n",
        "| sofa_cns                       | float64 | Central nervous system component of SOFA score                                                             |\n",
        "| delta_sofa_renal               | float64 | Time difference between the SaO2 measurement and the timestamp of the SaO2 measurement                     |\n",
        "| sofa_renal                     | float64 | Renal component of SOFA score                                                                              |\n",
        "| delta_sofa_resp                | float64 | Time offset (in minutes) in the measurement of respiratory component of SOFA                               |\n",
        "| sofa_resp                      | float64 | Respiratory component of SOFA score                                                                        |\n",
        "| delta_hemoglobin               | float64 | Time offset (in minutes) in the measurement of hemoglobin level                                            |\n",
        "| hemoglobin                     | float64 | Hemoglobin level                                                                                           |\n",
        "| delta_hematocrit               | float64 | Time offset (in minutes) in the measurement of Change in hematocrit level                                  |\n",
        "| hematocrit                     | float64 | Hematocrit level                                                                                           |\n",
        "| delta_mch                      | float64 | Time offset (in minutes) in the measurement of mean corpuscular hemoglobin                                 |\n",
        "| mch                            | float64 | Mean corpuscular hemoglobin                                                                                |\n",
        "| delta_mchc                     | float64 | Time offset (in minutes) in the measurement of mean corpuscular hemoglobin concentration                   |\n",
        "| mchc                           | float64 | Mean corpuscular hemoglobin concentration                                                                  |\n",
        "| delta_mcv                      | float64 | Time offset (in minutes) in the measurement of mean corpuscular volume                                     |\n",
        "| mcv                            | float64 | Mean corpuscular volume                                                                                    |\n",
        "| delta_platelet                 | float64 | Time offset (in minutes) in the measurement of platelet count                                              |\n",
        "| platelet                       | float64 | Platelet count                                                                                             |\n",
        "| delta_rbc                      | float64 | Time offset (in minutes) in the measurement of red blood cell count                                        |\n",
        "| rbc                            | float64 | Red blood cell count                                                                                       |\n",
        "| delta_rdw                      | float64 | Time offset (in minutes) in the measurement of Change in red cell distribution width                       |\n",
        "| rdw                            | float64 | Red cell distribution width                                                                                |\n",
        "| delta_wbc                      | float64 | Time offset (in minutes) in the measurement of white blood cell count                                      |\n",
        "| wbc                            | float64 | White blood cell count                                                                                     |\n",
        "| delta_d_dimer                  | float64 | Time offset (in minutes) in the measurement of Change in D-dimer                                           |\n",
        "| d_dimer                        | float64 | D-dimer level                                                                                              |\n",
        "| delta_fibrinogen               | float64 | Time offset (in minutes) in the measurement of fibrinogen level                                            |\n",
        "| fibrinogen                     | float64 | Fibrinogen level                                                                                           |\n",
        "| delta_thrombin                 | float64 | Time offset (in minutes) in the measurement of thrombin time                                               |\n",
        "| thrombin                       | float64 | Thrombin time                                                                                              |\n",
        "| delta_inr                      | float64 | Time offset (in minutes) in the measurement of Change in international normalized ratio (INR)              |\n",
        "| inr                            | float64 | International normalized ratio (INR)                                                                       |\n",
        "| delta_pt                       | float64 | Time offset (in minutes) in the measurement of prothrombin time (PT)                                       |\n",
        "| pt                             | float64 | Prothrombin time (PT)                                                                                      |\n",
        "| delta_ptt                      | float64 | Time offset (in minutes) in the measurement of partial thromboplastin time (PTT)                           |\n",
        "| ptt                            | float64 | Partial thromboplastin time (PTT)                                                                          |\n",
        "| delta_alt                      | float64 | Time offset (in minutes) in the measurement of alanine transaminase (ALT) level                            |\n",
        "| alt                            | float64 | Alanine transaminase (ALT) level                                                                           |\n",
        "| delta_alp                      | float64 | Time offset (in minutes) in the measurement of hhange in alkaline phosphatase (ALP) level                  |\n",
        "| alp                            | float64 | Alkaline phosphatase (ALP) level                                                                           |\n",
        "| delta_ast                      | float64 | Time offset (in minutes) in the measurement of aspartate transaminase (AST) level                          |\n",
        "| ast                            | float64 | Aspartate transaminase (AST) level                                                                         |\n",
        "| delta_bilirubin_total          | float64 | Time offset (in minutes) in the measurement of total bilirubin level                                       |\n",
        "| bilirubin_total                | float64 | Total bilirubin level                                                                                      |\n",
        "| delta_bilirubin_direct         | float64 | Time offset (in minutes) in the measurement of  direct bilirubin level                                     |\n",
        "| bilirubin_direct               | float64 | Direct bilirubin level                                                                                     |\n",
        "| delta_bilirubin_indirect       | float64 | Time offset (in minutes) in the measurement of indirect bilirubin level                                    |\n",
        "| bilirubin_indirect             | float64 | Indirect bilirubin level                                                                                   |\n",
        "| delta_ck_cpk                   | float64 | Time offset (in minutes) in the measurement of  creatine kinase (CPK) level                                |\n",
        "| ck_cpk                         | float64 | Creatine kinase (CPK) level                                                                                |\n",
        "| delta_ck_mb                    | float64 | Time offset (in minutes) in the measurement of  creatine kinase MB (CK-MB) level                           |\n",
        "| ck_mb                          | float64 | Creatine kinase MB (CK-MB) level                                                                           |\n",
        "| delta_ggt                      | float64 | Time offset (in minutes) in the measurement of  gamma-glutamyl transferase (GGT) level                     |\n",
        "| ggt                            | float64 | Gamma-glutamyl transferase (GGT) level                                                                     |\n",
        "| delta_ld_ldh                   | float64 | Time offset (in minutes) in the measurement of  lactate dehydrogenase (LDH) level                          |\n",
        "| ld_ldh                         | float64 | Lactate dehydrogenase (LDH) level                                                                          |\n",
        "| delta_albumin                  | float64 | Time offset (in minutes) in the measurement of albumin level                                               |\n",
        "| albumin                        | float64 | Albumin level                                                                                              |\n",
        "| delta_aniongap                 | float64 | Time offset (in minutes) in the measurement of anion gap                                                   |\n",
        "| aniongap                       | float64 | Anion gap                                                                                                  |\n",
        "| delta_bicarbonate              | float64 | Time offset (in minutes) in the measurement of bicarbonate level                                           |\n",
        "| bicarbonate                    | float64 | Bicarbonate level                                                                                          |\n",
        "| delta_bun                      | float64 | Time offset (in minutes) in the measurement of blood urea nitrogen (BUN) level                             |\n",
        "| bun                            | float64 | Blood urea nitrogen (BUN) level                                                                            |\n",
        "| delta_calcium                  | float64 | Time offset (in minutes) in the measurement of calcium level                                               |\n",
        "| calcium                        | float64 | Calcium level                                                                                              |\n",
        "| delta_chloride                 | float64 | Time offset (in minutes) in the measurement of chloride level                                              |\n",
        "| chloride                       | float64 | Chloride level                                                                                             |\n",
        "| delta_creatinine               | float64 | Time offset (in minutes) in the measurement of creatinine level                                            |\n",
        "| creatinine                     | float64 | Creatinine level                                                                                           |\n",
        "| delta_glucose_lab              | float64 | Time offset (in minutes) in the measurement of glucose level from laboratory                               |\n",
        "| glucose_lab                    | float64 | Glucose level from laboratory measurement                                                                  |\n",
        "| delta_sodium                   | float64 | Time offset (in minutes) in the measurement of sodium level                                                |\n",
        "| sodium                         | float64 | Sodium level                                                                                               |\n",
        "| delta_potassium                | float64 | Time offset (in minutes) in the measurement of potassium level                                             |\n",
        "| potassium                      | float64 | Potassium level                                                                                            |\n",
        "| delta_ph                       | float64 | Time offset (in minutes) in the measurement of pH level                                                    |\n",
        "| ph                             | float64 | pH level                                                                                                   |\n",
        "| delta_lactate                  | float64 | Time offset (in minutes) in the measurement of lactate level                                               |\n",
        "| lactate                        | float64 | Lactate level                                                                                              |\n",
        "| delta_heart_rate               | int64   | Time offset (in minutes) in the measurement of  heart rate                                                 |\n",
        "| heart_rate                     | float64 | Heart rate                                                                                                 |\n",
        "| delta_mbp                      | int64   | Time offset (in minutes) in the measurement of mean blood pressure (MBP)                                   |\n",
        "| mbp                            | float64 | Mean blood pressure (MBP)                                                                                  |\n",
        "| delta_resp_rate                | float64 | Time offset (in minutes) in the measurement of respiratory rate                                            |\n",
        "| resp_rate                      | float64 | Respiratory rate                                                                                           |\n",
        "| delta_temperature              | float64 | Time offset (in minutes) in the measurement of body temperature                                            |\n",
        "| temperature                    | float64 | Body temperature                                                                                           |\n",
        "| delta_glucose                  | float64 | Time offset (in minutes) in the measurement of glucose level                                               |\n",
        "| glucose                        | float64 | Glucose level                                                                                              |\n",
        "| delta_heart_rhythm             | float64 | Time offset (in minutes) in the measurement of heart rhythm                                                |\n",
        "| heart_rhythm                   | object  | Heart rhythm                                                                                               |"
      ],
      "metadata": {
        "id": "zqwtcefTVxdh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Select you variables\n",
        "**Task:** To simplify the task we will use a reduced group of variables for the following analysis. Ideally you should decide with your groups which variables you consider to be really important.\n",
        "\n",
        "The variables should be selected:\n",
        "1. Always thinking about the question to solve and the hypothesis.\n",
        "2. Using the clinical experience of the members of the group and if possible supported by literature.\n",
        "3. Using mathematical and statistical methods."
      ],
      "metadata": {
        "id": "ZEzfCAqiVRx8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Demographic Vars:\n",
        "demo_vars = ['race_group', \n",
        "             'gender', \n",
        "             'anchor_year_group', \n",
        "             'language']\n",
        "\n",
        "# Patient Info Vars:\n",
        "patient_info_vars = ['subject_id', \n",
        "                     'stay_id', \n",
        "                     'hadm_id', \n",
        "                     'insurance', \n",
        "                     'first_hosp_stay', \n",
        "                     'first_icu_stay', \n",
        "                     'icustay_seq', \n",
        "                     'admittime', \n",
        "                     'icu_intime', \n",
        "                     'CCI', \n",
        "                     'ventilation_status', \n",
        "                     'invasive_vent']\n",
        "\n",
        "# Temporal Clinical Vars:\n",
        "temporal_clinical_vars = ['SpO2', \n",
        "                          'SaO2', \n",
        "                          'delta_SpO2', \n",
        "                          'delta_vent_start', \n",
        "                          'delta_FiO2', \n",
        "                          'FiO2', \n",
        "                          'delta_rrt', \n",
        "                          'rrt', \n",
        "                          'delta_vp_start', \n",
        "                          'norepinephrine_equivalent_dose', \n",
        "                          'delta_hemoglobin', \n",
        "                          'hemoglobin', \n",
        "                          'delta_hematocrit', \n",
        "                          'hematocrit', \n",
        "                          'delta_mch', \n",
        "                          'mch', \n",
        "                          'd_dimer', \n",
        "                          'alt', \n",
        "                          'ck_cpk', \n",
        "                          'creatinine', \n",
        "                          'resp_rate', \n",
        "                          'temperature', \n",
        "                          'glucose', \n",
        "                          'heart_rate', \n",
        "                          'heart_rhythm']\n",
        "\n",
        "# SOFA Vars:\n",
        "sofa_vars = ['delta_sofa_coag', \n",
        "             'delta_sofa_liver', \n",
        "             'delta_sofa_cv', \n",
        "             'delta_sofa_cns', \n",
        "             'delta_sofa_renal', \n",
        "             'delta_sofa_resp']\n",
        "\n",
        "# Static Clinical Vars:\n",
        "static_clinical_vars = ['height', \n",
        "                        'weight']\n",
        "\n",
        "# Dependent Vars:\n",
        "dependent_vars = ['hidden_hypoxemia']"
      ],
      "metadata": {
        "id": "CjpEUOOthwFz"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Select demographic variables:\n",
        "Due to the challenge we need some demographic information select the relevant columns:"
      ],
      "metadata": {
        "id": "npybroq6Ju2a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Patient main information (include identifiers)\n",
        "patient_info = ['variable1', 'variable2', ... ]\n",
        "\n",
        "# Demographic variables\n",
        "demographic_variables = ['variable1', 'variable2', ... ]"
      ],
      "metadata": {
        "id": "ZIUc7QwLJuDT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The clinical variables selected were selected based on the variables that are most related with SaO2 and Spo2 and a correlation method was used to filter those variables. \n",
        "You can use other methods and criteria!"
      ],
      "metadata": {
        "id": "gZxXW_09KCOC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Select clinical variables\n",
        "1. Manual variable selection: Manually select with you team the variables that are clinically important. Don't worry if there are many, later we will use other methods to filter the variables"
      ],
      "metadata": {
        "id": "Ucr8SD4lW0Eh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# # Static clinical variables\n",
        "# static_clinical_variables = ['variable1', 'variable2', ... ]\n",
        "\n",
        "# # Temporal clinical variables (E.g. Sofa related variables)\n",
        "# sofa_variables = ['variable1', 'variable2', ... ]\n",
        "# temporal_clinical_variables = ['variable1', 'variable2', ... 'variable3'] + sofa_variables\n",
        "\n",
        "# outcomes = ['variable1', 'variable2', ... ]\n",
        "\n",
        "# treatment = ['variable1', 'variable2', ... ]"
      ],
      "metadata": {
        "id": "GLYKUYWtKjNh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Variable filtering\n",
        "Let's now employ a few techniques to filter the variables. You should choose a technique to do it. \n",
        "\n",
        "Hint: To choose the variables most connected with the columns SpO2 and SaO2, we utilised the correlation as an example. You are not required to employ the identical column(s). Select the columns in accordance with your hypothesis."
      ],
      "metadata": {
        "id": "lhKfPsmRW8TA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_most_correlated(df, variable, n):\n",
        "  \"\"\"\n",
        "  Function to get the most direct and inverse correlated variables to a specific variable\n",
        "  Inputs:\n",
        "  df: A pandas dataframe with all the variables to calculate correlation\n",
        "  variable: S string with the name of the variable with respect to which we want to calculate the correlation (Eg. 'SaO2')\n",
        "  n: A integer with the number of variables that we want to get as most directly and inverse correlated\n",
        "  Output:\n",
        "  A list with top n most correlated words (inversely and directly)\n",
        "  \"\"\"\n",
        "\n",
        "  # Calculate the correlation matrix\n",
        "  corr_matrix = df.corr()\n",
        "\n",
        "  # Print the correlation matrix\n",
        "  correlation = corr_matrix[variable].sort_values(ascending=False)\n",
        "  positive_correlated = correlation[:n]\n",
        "  inversely_correlated = correlation[-n:]\n",
        "  print('#'*40, f' {variable} ' , '#'*40)\n",
        "  print(f'The {n} most correlated variables to variable {variable} are: ')\n",
        "  print(positive_correlated)\n",
        "  print(f'The {n} most inversely correlated variables to variable {variable} are: ')\n",
        "  print(inversely_correlated)\n",
        "\n",
        "  return list(positive_correlated.index) + list(inversely_correlated.index)"
      ],
      "metadata": {
        "id": "bqsjG9cFIVTS"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(f'The number of initial temporal clinical variables is {len(temporal_clinical_vars)}')\n",
        "\n",
        "# Select the number of related columns:\n",
        "n1 =  8 # Eg. 8\n",
        "n2 =  10 # Eg. 10\n",
        "ref_col1 = 'SaO2' # Eg. 'SaO2'\n",
        "ref_col2 = 'SpO2' # Eg.\n",
        "\n",
        "# Get variables correlated to 'ref_col'\n",
        "columns_1 = get_most_correlated(df[temporal_clinical_vars], variable=ref_col1, n=n1)\n",
        "\n",
        "# Get variables correlated to 'ref_col_2'\n",
        "columns_2 = get_most_correlated(df[temporal_clinical_vars], variable=ref_col2, n=n2)\n",
        "\n",
        "# Merge the resulting columns in a single list:\n",
        "resulting_temporal_clinical_variables = columns_1 + columns_2\n",
        "\n",
        "print(f'The number of resulting variables is {len(resulting_temporal_clinical_variables)}')"
      ],
      "metadata": {
        "id": "rxZJnP7bIVWT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f2e9ab16-d676-4d24-929d-c27d3b0d9b43"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The number of initial temporal clinical variables is 25\n",
            "########################################  SaO2  ########################################\n",
            "The 8 most correlated variables to variable SaO2 are: \n",
            "SaO2                1.000000\n",
            "SpO2                0.475884\n",
            "delta_FiO2          0.030304\n",
            "delta_vent_start    0.027616\n",
            "glucose             0.003696\n",
            "delta_rrt           0.002084\n",
            "mch                 0.000943\n",
            "hemoglobin         -0.005187\n",
            "Name: SaO2, dtype: float64\n",
            "The 8 most inversely correlated variables to variable SaO2 are: \n",
            "ck_cpk                           -0.032298\n",
            "rrt                              -0.041036\n",
            "hematocrit                       -0.041337\n",
            "creatinine                       -0.057613\n",
            "heart_rate                       -0.100198\n",
            "resp_rate                        -0.120549\n",
            "norepinephrine_equivalent_dose   -0.125088\n",
            "FiO2                             -0.174808\n",
            "Name: SaO2, dtype: float64\n",
            "########################################  SpO2  ########################################\n",
            "The 10 most correlated variables to variable SpO2 are: \n",
            "SpO2                1.000000\n",
            "SaO2                0.475884\n",
            "delta_rrt           0.037291\n",
            "delta_vent_start    0.023304\n",
            "delta_FiO2          0.016410\n",
            "glucose             0.003107\n",
            "mch                -0.003349\n",
            "rrt                -0.006946\n",
            "delta_hematocrit   -0.010028\n",
            "delta_SpO2         -0.013347\n",
            "Name: SpO2, dtype: float64\n",
            "The 10 most inversely correlated variables to variable SpO2 are: \n",
            "hemoglobin                       -0.035574\n",
            "temperature                      -0.036033\n",
            "d_dimer                          -0.040607\n",
            "hematocrit                       -0.068189\n",
            "alt                              -0.077710\n",
            "creatinine                       -0.082644\n",
            "heart_rate                       -0.133826\n",
            "resp_rate                        -0.153639\n",
            "norepinephrine_equivalent_dose   -0.189864\n",
            "FiO2                             -0.224278\n",
            "Name: SpO2, dtype: float64\n",
            "The number of resulting variables is 36\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Optional: add another feature selection process to improve the model at a later stage\n",
        "from typing import Any, List, Union\n",
        "\n",
        "array_like = Union[pd.DataFrame, np.ndarray, List, Any]\n",
        "\n",
        "outcome_var = \"mortality_in\"\n",
        "K = 7\n",
        "method = f_classif  # f_classif, chi2, mutual_info_classif, mutual_info_regression, etc\n",
        "\n",
        "def feat_select(features: List[str], df: pd.DataFrame, outcome_var: str, K: int, method: Any) -> array_like:  \n",
        "  X = df[features]\n",
        "  y = df[outcome_var]\n",
        "  return SelectKBest(method, k=K).fit(X, y).get_feature_names_out()\n",
        "\n",
        "# resulting_temporal_clinical_variables = feat_select(resulting_temporal_clinical_variables, df, outcome_var, K, method)"
      ],
      "metadata": {
        "id": "Nlys9bNkJb2h"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the resulting dataframe\n",
        "variables = list(set(patient_info_vars + demo_vars + static_clinical_vars + resulting_temporal_clinical_variables + outcomes + treatment))\n",
        "\n",
        "df = df[variables]\n",
        "df.head()"
      ],
      "metadata": {
        "id": "NMQJzYj-DPk6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Understand how variables are distributed:"
      ],
      "metadata": {
        "id": "-MHV8nXH_pZ7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To see a description of some statistics of each numeric variable, you can use `df.describe()`:\n"
      ],
      "metadata": {
        "id": "36D4dSeJK2zU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.describe()"
      ],
      "metadata": {
        "id": "gyYJzvip-4-2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The problem with `data.describe()` is that it groups using all the data of each variable, this means that if in our group there are populations that are underestimated or overestimated, the description will not take that into account. An alternative to solve that problem is using the function `groupby()`. With group by we can generate groups of populations using variables. An exmple is:"
      ],
      "metadata": {
        "id": "q8hv2yMShisk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Group the mean of oxygen saturation of arterial blood (SaO2) by gender\n",
        "avg_sao2_by_gender = df.groupby('gender')['SaO2'].mean()\n",
        "avg_sao2_by_gender"
      ],
      "metadata": {
        "id": "KmRNIklehdSj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can also generate groups using more than one variable and more than one grouping methods"
      ],
      "metadata": {
        "id": "cUtoT_Z2yoN9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Obtain count, mean, standard deviation, min value and max value for (SaO2) and (SpO2) grouping by gender and race group:\n",
        "avg_sao2_by_ethnicity = df.groupby(['gender', 'race_group']).agg({\n",
        "    'SaO2': ['count', 'mean', 'std', 'min', 'max'], \n",
        "    'SpO2': ['count', 'mean', 'std', 'min', 'max']\n",
        "})\n",
        "avg_sao2_by_ethnicity"
      ],
      "metadata": {
        "id": "ZmqCwMY6hdVj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Alternatively, there are also libraries that allow us to do this process automatically. Below is an example using tableone library. \n",
        "\n",
        "Install using the command `!pip install tableone`"
      ],
      "metadata": {
        "id": "O-tm_ARoz-8s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# As a first step let's do a preprocessing to some variables like:\n",
        "df['language'] = df['language'].replace({'ENGLISH': 'Proficient', '?': 'Limited Proficiency'})"
      ],
      "metadata": {
        "id": "pPzYMKLP9LJa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tableone\n",
        "from tableone import TableOne\n",
        "\n",
        "groupby = ['race_group']\n",
        "\n",
        "categorical_variables = ['mortality_in','gender', 'language', 'invasive_vent']\n",
        "numerical_variables = ['anchor_age', 'los_icu', 'los_hospital', 'CCI', 'SOFA_admission']\n",
        "columns = categorical_variables + numerical_variables\n",
        "\n",
        "labels ={'anchor_age': 'age',\n",
        "         'SOFA_admission': 'SOFA'} \n",
        "\n",
        "\n",
        "mytable = TableOne(df, columns=columns, categorical=categorical_variables, groupby=groupby, nonnormal=numerical_variables, rename=labels, pval=False)\n",
        "mytable"
      ],
      "metadata": {
        "id": "foYkDJqQocnL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Use tableone\n",
        "\n",
        "Now let's create a new tableone using all the variables to to measure differences between ethnic groups!\n",
        "\n",
        "Hint: Use the function `get_categorical_numerical_variables` to get all the numeric and categorical variables. Then use those variables to generate a tableone of the whole dataset\n"
      ],
      "metadata": {
        "id": "MWKI47ieLIIL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_categorical_numerical_variables(df, limit, ignore, verbose=True):\n",
        "  \"\"\"\n",
        "  File to get numerical and categorical variables\n",
        "  Inputs:\n",
        "  df: A pandas dataframe with all the variables\n",
        "  limit: A integer with the maximum number of unique items in a variable to be categorical\n",
        "  ignore: Python list with columns to ignore\n",
        "\n",
        "  Outputs:\n",
        "  categorical_columns, numerical_columns: List with the names of categorical and numerical variables\n",
        "\n",
        "  \"\"\"\n",
        "  \n",
        "  categorical_columns = []\n",
        "  numerical_columns = []\n",
        "\n",
        "  for column in df.columns:\n",
        "    if ignore:\n",
        "      if column in ignore:\n",
        "        continue\n",
        "    unique_values = len(pd.unique(df[column]))\n",
        "    if unique_values <= limit:\n",
        "      if verbose:\n",
        "        print(f'The column {column} has {unique_values}, so is categorical')\n",
        "      categorical_columns.append(column)\n",
        "    else:\n",
        "      if verbose:\n",
        "        print(f'The column {column} has {unique_values}, so is numerical')\n",
        "      numerical_columns.append(column)\n",
        "\n",
        "  return categorical_columns, numerical_columns"
      ],
      "metadata": {
        "id": "ouIrWL4AiEM2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Select the columns for the tableone\n",
        "groupby = #\n",
        "ignore = ['column1', 'column2', ...] + groupby\n",
        "limit = # Max. Number of unique values to be categorical (E.g. 15)\n",
        "categorical_columns, numerical_columns = get_categorical_numerical_variables(df, limit=limit, ignore=ignore)"
      ],
      "metadata": {
        "id": "ilDdtS_tex3t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "Total_columns = \n",
        "\n",
        "TableOne(df, columns=Total_columns, categorical=categorical_columns, nonnormal=numerical_columns, groupby=groupby)"
      ],
      "metadata": {
        "id": "JXtGKC6jfcct"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### visualize your data\n",
        "\n",
        "While statistics are important, it is also very important and key to visualize the data. This will help to share the information and results in a visual and intuitive way, and identify anomalous and data patterns that could be useful when choosing the model."
      ],
      "metadata": {
        "id": "y-8mdEVQLSKZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Bar charts and histograms\n",
        "Bar charts and histograms can be used to see how our data is distributed and how many instances there are of each class, this way we can identify imbalances in the classes and avoid overfitting in the models\n",
        "\n",
        "To create a histogram with seaborn, you can use the code:\n",
        "\n",
        "\n",
        "```\n",
        "sns.histplot(data=df, x=\"column_name\", kde=False) #optional kwarg: hue=\"column_2\"\n",
        "plt.title('Title')\n",
        "plt.show()\n",
        "```\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "0L5teBSObmED"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Plot the distribution by race group"
      ],
      "metadata": {
        "id": "Op0QYpYxMvp-"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "nIarH7T8bmbP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Plot the distribution by gender"
      ],
      "metadata": {
        "id": "WP4GUREQM_0g"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "21beiKNxcJGf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Plot the distribution by age and race group"
      ],
      "metadata": {
        "id": "47abe3cjNCiu"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "gKDp62Uldi6l"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Box plots and violin plots\n",
        "**Outilers ❗❗❗**: While in some cases there are no null values, could be outliers values. These outliers are sometimes indicators of biases in data collection, or social/cultural biases. However, in some cases the outliers are erroneous values during data collection and storage. Identifying values such as very high SaO2 (99999) or negative is an indicator in many cases of missing values and should be taken as such.\n",
        "\n",
        "Bar and violin charts are one of the best ways to identify outliers, because they show us around which values most of our data is clustered."
      ],
      "metadata": {
        "id": "oRt_xYq87hLi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def plot_groupby(df, group1, group2, variable, barplot=True, boxplot=True, violinplot=False):\n",
        "\n",
        "  \"\"\"\n",
        "  This function creates two plots that compare the average values of a variable for two groups in a pandas dataframe.\n",
        "  The function allows the user to choose between a bar plot, box plot, and violin plot for each group.\n",
        "  The first plot compares the averages for the first group, and the second plot compares the averages for the second group.\n",
        "  \n",
        "  Inputs:\n",
        "  df: pandas dataframe\n",
        "  group1: string, name of the first categorical variable to group by\n",
        "  group2: string, name of the second categorical variable to group by\n",
        "  variable: string, name of the numerical variable to plot\n",
        "  barplot: boolean, optional, default True. Whether to plot bar plots or not\n",
        "  boxplot: boolean, optional, default True. Whether to plot box plots or not\n",
        "  violinplot: boolean, optional, default False. Whether to plot violin plots or not\n",
        "  \n",
        "  Outputs: \n",
        "  None\n",
        "  \"\"\"\n",
        "\n",
        "  avg_by_group1 = df.groupby(group1)[variable].mean()\n",
        "  avg_by_group2 = df.groupby(group2)[variable].mean()\n",
        "\n",
        "  fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 6))\n",
        "  \n",
        "  if barplot:\n",
        "    # Bar plot\n",
        "    print('Bar Plot: ')\n",
        "    ax1.bar(avg_by_group1.index, avg_by_group1.values)\n",
        "    ax1.set_xlabel(group1)\n",
        "    ax1.set_ylabel(f'Average {variable}')\n",
        "    ax2.bar(avg_by_group2.index, avg_by_group2.values)\n",
        "    ax2.set_xlabel(group2)\n",
        "    ax2.set_ylabel(f'Average {variable}')\n",
        "    plt.show()\n",
        "\n",
        "  if boxplot:\n",
        "    # Create box plots to visualize the distribution of SaO2 values by gender and ethnicity\n",
        "    print('Box Plot: ')\n",
        "    sns.boxplot(x=group1, y=variable, hue=group2, data=df)\n",
        "    plt.show()\n",
        "\n",
        "  if violinplot:\n",
        "    print('Violin Plot: ')\n",
        "    sns.violinplot(x=group1, y=variable, hue=group2, data=df)\n",
        "    plt.show()\n"
      ],
      "metadata": {
        "id": "IOWP5b6IZSyW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Plot the distribution of SaO2 by gender and race group"
      ],
      "metadata": {
        "id": "Mhi4eVrrhzJ3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Use plot_groupby():\n"
      ],
      "metadata": {
        "id": "kd8Y6R_P9qag"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Plot the distribution of SpO2 by gender and race group"
      ],
      "metadata": {
        "id": "5VbxyRjYh6UN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Use plot_groupby():\n"
      ],
      "metadata": {
        "id": "mcbHq8-0jIMB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Plot the distribution of any Time Offset by gender and race group (Use violin plot)\n",
        "\n",
        "Hint: You can use the next line of code to get all the variables related to a time offset"
      ],
      "metadata": {
        "id": "_IApC0RKfjcO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for column in df.columns:\n",
        "  if 'delta' in column:\n",
        "    print(column)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OSwa5fEdhQT_",
        "outputId": "681a42e3-6463-4946-e371-5d097527902d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "delta_SpO2\n",
            "delta_vent_start\n",
            "delta_ld_ldh\n",
            "delta_sofa_liver\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Use plot_groupby():\n"
      ],
      "metadata": {
        "id": "QCGZGTHfeuiC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4. Preprocessing\n",
        "---\n",
        "We know that there are some outliers so let's assume that those values al missing values and then analyze the amount of missing values for preprocessing"
      ],
      "metadata": {
        "id": "OQxBAUV7SCcV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Missing values\n",
        "\n",
        "Let's see which columns have variables with missing data"
      ],
      "metadata": {
        "id": "Th51UTXwSRYk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "msno.matrix(df)"
      ],
      "metadata": {
        "id": "cW7ysOZGeunN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ detect columns with null values\n",
        "Use the function `null_values` to see the columns with missing data and the number of rows missing.\n",
        "\n",
        "if the number of values with missing data is very high compared to the size of the dataset (E.g. 85%), in some cases it is better just to remove them"
      ],
      "metadata": {
        "id": "OcfF2HSqOM3f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "### List of columns with mussing values ###\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DNVsJ8FRSJ7N",
        "outputId": "0ca1ccbf-f3d8-4778-cf81-ee974506a932"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Null values: \n",
            "norepinephrine_equivalent_dose    9972\n",
            "ventilation_status                2342\n",
            "rdw                                 13\n",
            "FiO2                              2720\n",
            "lactate                             78\n",
            "bun                                 14\n",
            "glucose_lab                         14\n",
            "ph                                  18\n",
            "delta_vent_start                  2342\n",
            "mchc                                12\n",
            "chloride                            14\n",
            "BMI                               3853\n",
            "pt                                  14\n",
            "delta_ld_ldh                      2256\n",
            "sofa_resp                          519\n",
            "albumin                            676\n",
            "aniongap                            14\n",
            "mcv                                 12\n",
            "delta_sofa_liver                  5654\n",
            "ld_ldh                            2256\n",
            "dtype: int64\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Train-Test Split\n",
        "Dividing your dataset into training and test is important, since what we want is for the model to learn, not memorize.\n",
        "We want to test the behavior of the model on unknown data. For that reason, the dataset should always be divided into training and test. \n",
        "\n",
        "The way to divide it may vary... You can use 60% (training) / 40% (testing), or 70% (training) / 30% (testing), or even 50/50. The most important part is that you **make sure that in the test set you have a well-distributed data set that covers all the possibilities.**"
      ],
      "metadata": {
        "id": "LwRUZnXdsMB2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def train_test_split_with_id(df, test_size, id_column, stratify=None, random_state=None):\n",
        "  \"\"\"\n",
        "  Splits a dataset into train and test sets while ensuring that no rows with the same ID value appear in both sets.\n",
        "\n",
        "  Parameters:\n",
        "  df (pandas.DataFrame): The dataset to be split.\n",
        "  test_size (float): The proportion of the data to be used for the test set.\n",
        "  id_column (str): The name of the column containing the IDs used to identify patients. (A patient in train should not be also in test)\n",
        "  stratify (array-like): The values used for stratification. If None, stratification is not performed (optional).\n",
        "  random_state (int): The random seed used for the split (optional).\n",
        "\n",
        "  Returns:\n",
        "  tuple: A tuple of two pandas.DataFrames representing the train and test sets.\n",
        "  \"\"\"\n",
        "  if stratify:\n",
        "    id_groups = df.groupby(id_column)\n",
        "    ids = []\n",
        "    ys = []\n",
        "    for _, group in id_groups:\n",
        "        ids.append(group[id_column].iloc[0])\n",
        "        ys.append(group[stratify].iloc[0])\n",
        "    ids = np.array(ids)\n",
        "    ys = np.array(ys)\n",
        "    \n",
        "    train_ids, test_ids, train_ys, test_ys = train_test_split(ids, ys, test_size=test_size, stratify=ys, random_state=random_state)\n",
        "\n",
        "  else:\n",
        "    unique_ids = df[id_column].unique()\n",
        "    train_ids, test_ids = train_test_split(unique_ids, test_size=test_size, random_state=random_state)\n",
        "  \n",
        "  train = df[df[id_column].isin(train_ids)]\n",
        "  test = df[df[id_column].isin(test_ids)]\n",
        "\n",
        "  print(f'The train shape is {train.shape}')\n",
        "  print(f'The test shape is {test.shape}')\n",
        "  \n",
        "  return train, test\n",
        "  "
      ],
      "metadata": {
        "id": "BVzkNkfFsOhZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Split your data\n",
        "Use the provided function to split your data into train and test. We have to split the dataset now so we can avoid data leakage during data imputation and data normalization"
      ],
      "metadata": {
        "id": "RcbYiAi0PFP0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "test_size = \n",
        "id_column = # Avoid same patient in train and test!!!\n",
        "stratify = \n",
        "\n",
        "train, test = train_test_split_with_id('...')"
      ],
      "metadata": {
        "id": "L60Ly3JNxAZP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Data Imputation\n",
        "\n",
        "There are various imputation methods of different complexity, ranging from imputing using the most common value (mode) (usually used for categorical variables) or the average or median value depending on the distribution of the data (usually used for numerical variables), to imputation methods using machine learning algorithms to predict the missing value using the other variables as a reference. While any method of data imputation can be valid, you have to take care to avoid data leakage or adding bias when doing this process."
      ],
      "metadata": {
        "id": "TGfy5tf4SndZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "As we have the columns with missing data, let's divide the columns into two sub groups:\n",
        "1. Categorical columns with missing\n",
        "2. Numerical columns with missing"
      ],
      "metadata": {
        "id": "Vsi0qe3cQzmV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Get numerical, categorical columns and columns with missing data\n",
        "\n",
        "Hint: You can use the function `get_categorical_numerical_variables` to get the list of categorical and mumerical columns. \n",
        "\n",
        "Hint 2: You can use the function `null_values` as part of the logic to get the columns with missing data"
      ],
      "metadata": {
        "id": "CPJYmaBJQUIN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Get numerical and categorical columns:\n",
        "categorical_cols, numerical_cols = "
      ],
      "metadata": {
        "id": "qExWrf1JxPoU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the columns with missing data\n",
        "columns_missing = "
      ],
      "metadata": {
        "id": "0RLwYEoKynP_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def apply_data_imputation(df, impute_cols, test_df=None, groupby_cols=[], method='new_category_numeric', model=LinearRegression()):\n",
        "\n",
        "  \"\"\"\n",
        "  Apply data imputation on missing values of columns by groupby the given columns.\n",
        "\n",
        "  Important!!! \n",
        "  If you apply other technique than `new_category` on categorical variables\n",
        "  or `new_category_numeric` on numerical variables. You should do it \n",
        "  ONLY AFTER train-test split otherwise you could be introducing a data leakage in the test set.\n",
        "\n",
        "  Inputs:\n",
        "  df: Pandas DataFrame with the input data to apply data imputation.\n",
        "  impute_cols: list with the column names to apply data imputation.\n",
        "  test_df (Optional): Pandas DataFrame with the test set to apply data imputation using train as reference.\n",
        "  groupby_cols: list with the column names to groupby the data if method='median', 'mean' or 'mode' or features of the model if method='model' (E.g.: groupby_cols=['race_group', 'gender']).\n",
        "  method: str with the method to apply data imputation. Available options are 'median', 'mean', 'mode', 'new_category', 'new_category_numeric' and 'model'.\n",
        "  model: sklearn model used to predict the missing value if method == 'model'\n",
        "\n",
        "  Output:\n",
        "  Pandas DataFrame with the data after applying data imputation on missing values of columns.\n",
        "  \"\"\"\n",
        "  imputer = {}\n",
        "\n",
        "  for col in impute_cols:\n",
        "      \n",
        "    # Group the data and calculate the method (E.g. Median) for each group\n",
        "    if method == 'median':\n",
        "      # Impute using median (for numerical values)\n",
        "      imputation_values = df.groupby(groupby_cols)[col].median()\n",
        "\n",
        "    elif method == 'mean':\n",
        "      # Impute using mean (for numerical values)\n",
        "      imputation_values = df.groupby(groupby_cols)[col].mean()\n",
        "\n",
        "    elif method == 'mode':\n",
        "      # Impute using mode (for categorical values)\n",
        "      imputation_values = df.groupby(groupby_cols)[col].agg(pd.Series.mode)\n",
        "\n",
        "    elif method == 'new_category':\n",
        "      # Add a new category 'None' in the dataset (for categorical values)\n",
        "      new_category_val = 'None'\n",
        "      imputer[col] = new_category_val\n",
        "      df[col] = df[col].fillna(new_category_val)\n",
        "      if test_df is not None:\n",
        "        test_df[col] = test_df[col].fillna(new_category_val)\n",
        "      \n",
        "\n",
        "    elif method == 'new_category_numeric':\n",
        "      # Add an anomalous data in the dataset (for numerical values)\n",
        "      new_category_val = 0\n",
        "      # If there's not 0, replace nan with 0\n",
        "      if not((df[col] == 0).any()):\n",
        "        new_category_val = 0\n",
        "      # If there's not positive numbers, replace nan with 1\n",
        "      elif not((df[col] >= 0).any()):\n",
        "        new_category_val = 1\n",
        "      # If there's not negative numbers, replace nan with 1\n",
        "      elif not((df[col] <= 0).any()):\n",
        "        new_category_val = -1\n",
        "      else:      \n",
        "        # Replace with min value - 100\n",
        "        new_category_val = df[col].min() - 100\n",
        "      \n",
        "      imputer[col] = new_category_val\n",
        "      df[col] = df[col].fillna(new_category_val)\n",
        "      if test_df is not None:\n",
        "        test_df[col] = test_df[col].fillna(new_category_val)\n",
        "\n",
        "\n",
        "    elif method == 'model':\n",
        "      # If the variable is categorical, convert to numeric:\n",
        "      if df[col].dtype == 'object':\n",
        "        encoder = LabelEncoder()\n",
        "        df[col] = encoder.fit_transform(df[col])\n",
        "        if test_df is not None:\n",
        "          test_df[col] = encoder.fit_transform(test_df[col])\n",
        "\n",
        "      # Create a linear regression model to impute missing values\n",
        "      model = model\n",
        "\n",
        "      # Get the data with complete column\n",
        "      complete_data = df.dropna(subset=[col])\n",
        "\n",
        "      # Encode categorical columns if needed in x data\n",
        "      le = {}\n",
        "      for col_group in groupby_cols:\n",
        "        if df[col_group].dtype == 'object':\n",
        "          le[col_group] = LabelEncoder()\n",
        "          complete_data[col_group] = le[col_group].fit_transform(complete_data[col_group])\n",
        "\n",
        "      # Fit the model on the complete data\n",
        "      X = complete_data[groupby_cols]\n",
        "      # Replace any remaining NaNs with the column mean\n",
        "      X = X.fillna(X.mean())\n",
        "      y = complete_data[col]\n",
        "      model.fit(X, y)\n",
        "\n",
        "      if df[col].dtype == 'object':\n",
        "        imputer[col] = (model, encoder, le)\n",
        "      else:\n",
        "        imputer[col] = (model, le)\n",
        "\n",
        "      ### Impute column using the model to predict the value:\n",
        "      missing_data = df[df[col].isna()]\n",
        "      if test_df is not None:\n",
        "          missing_data_test = test_df[test_df[col].isna()]\n",
        "\n",
        "      # Encode categorical columns if needed in x data\n",
        "      for col_group in groupby_cols:\n",
        "        if missing_data[col_group].dtype == 'object':\n",
        "          missing_data[col_group] = le[col_group].transform(missing_data[col_group])\n",
        "          if test_df is not None:\n",
        "            missing_data_test[col_group] = le[col_group].transform(missing_data_test[col_group])\n",
        "\n",
        "      for index, row in missing_data.iterrows():\n",
        "          values = row[groupby_cols].values.reshape(1, -1)\n",
        "          imputed_value = model.predict(values)\n",
        "          df.at[index, col] = imputed_value[0]\n",
        "\n",
        "      if test_df is not None:\n",
        "          for index, row in missing_data_test.iterrows():\n",
        "            values = row[groupby_cols].values.reshape(1, -1)\n",
        "            imputed_value = model.predict(values)\n",
        "            test_df.at[index, col] = imputed_value[0]\n",
        "          \n",
        "    if method in ['median', 'mean', 'mode']:\n",
        "      imputer[col] = imputation_values\n",
        "      # Fill missing values with the method of the corresponding group\n",
        "      df[col] = df.apply(lambda x: imputation_values[tuple(x[groupby_cols])] if pd.isna(x[col]) else x[col], axis=1)\n",
        "      if test_df is not None:\n",
        "        test_df[col] = test_df.apply(lambda x: imputation_values[tuple(x[groupby_cols])] if pd.isna(x[col]) else x[col], axis=1)\n",
        "  \n",
        "  if test_df is not None:\n",
        "    return df, test_df, imputer\n",
        "  else:\n",
        "    return df, imputer\n"
      ],
      "metadata": {
        "id": "dqZXomYXPa3G"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Apply data imputation to categorical variables\n",
        "\n",
        "1. Get from the categorical variables the list of categorical columns with missing data\n",
        "2. Use the function `apply_data_imputation` to impute the variables of train and test data. You can also use other methods of pandas or sklearn. But think about the possible bias that those methods could be adding to the dataset."
      ],
      "metadata": {
        "id": "nTFDHd8RKbqI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Get from columns with missing data the categorical columns\n",
        "categorical_cols_missing = "
      ],
      "metadata": {
        "id": "NCNfYwpNKXRv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Important❗❗❗** If you apply other technique than `new_category` on categorical variables. You should do it after train-test split otherwise you could be introducing a data leakage in the test set"
      ],
      "metadata": {
        "id": "90vB5xhDzJvC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "method =  # 'mode', 'new_category'\n",
        "groupby_cols =  # E.g. ['race_group', 'gender'] or None if is new_category\n",
        "\n",
        "train, test, imputer = apply_data_imputation(train, impute_cols=categorical_cols_missing, method=method, groupby_cols=groupby_cols, test_df=test)"
      ],
      "metadata": {
        "id": "KSws7IRAScbC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Apply data imputation to Numerical variables\n",
        "\n",
        "1. Get from the numerical variables the list of numerical columns with missing data\n",
        "2. Use the function `apply_data_imputation` to impute the variables of train and test data. You can also use other methods of pandas or sklearn. But think about the possible bias that those methods could be adding to the dataset."
      ],
      "metadata": {
        "id": "OHwYtppJLNfd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Get from columns with missing data the numerical columns\n",
        "numerical_cols_missing = "
      ],
      "metadata": {
        "id": "7TavY8buLUIP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Important ❗❗❗** If you apply other technique than `new_category_numeric` on numerical variables. You should do it after train-test split otherwise you could be introducing a data leakage in the test set"
      ],
      "metadata": {
        "id": "ZQHc5wKdzsb1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "method = # 'median', 'mean', 'new_category_numeric', 'model'\n",
        "groupby_cols =  # E.g. ['race_group', 'gender'] or None if is new_category_numeric\n",
        "\n",
        "train, test, imputer = apply_data_imputation(train, impute_cols=numerical_cols_missing, method=method, groupby_cols=groupby_cols, test_df=test)"
      ],
      "metadata": {
        "id": "tuTNR8qtLV48"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Let's check null values again\n",
        "\n",
        "Print the coluns with missing values (if any) in train and test set. \n",
        "\n",
        "There should be no columns with missing data, if so check the previous tasks"
      ],
      "metadata": {
        "id": "Xtkuor2lfY-H"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "### List of columns with mussing values ###\n",
        "print('#'*20, ' Null train: ', '#'*20)\n",
        "null_values(train)\n",
        "### List of columns with mussing values ###\n",
        "print('#'*20, ' Null test: ', '#'*20)\n",
        "null_values(test)"
      ],
      "metadata": {
        "id": "J1C2rHdzDVIO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Encoding\n",
        "\n",
        "Since the machine learning model in the background models and finds patterns in our data. It only supports numeric values. For this reason, categorical variables must be coded to numeric values.\n",
        "\n",
        "Categorical variables can be of 3 types:\n",
        "- Binary variables: Binary variables can be represented with two values, 1 and 0. Examples are whether or not the variable belongs to a group.\n",
        "- Ordinal variables: Ordinal variables are a type of variables that have a specific order and can be represented with numeric variables through a label encoder. An example is High, Medium, and Low which can be represented as 3, 2, 1.\n",
        "- Nominal variables: Nominal variables are categorical variables that do not have a defined order, for these variables it is not recommended to use a label encoder, it is better to use one hot encoder in these cases."
      ],
      "metadata": {
        "id": "ojBX1nM1nE3W"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Which type of categorical variables we have?\n",
        "\n",
        "From the list of categorical variables, provide:\n",
        "\n",
        "* List of binary variables\n",
        "* List of ordinal variables\n",
        "* List of Nominal variables\n",
        "\n",
        "Not necessarily must have variables of each type.\n",
        "\n",
        "Hint: you can use `pd.unique(df[column])` to see the unique values in a specific column. You can use this information to make your decisions"
      ],
      "metadata": {
        "id": "Uqlk_-k_TRr8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(f'Categorical: ')\n",
        "print(categorical_cols)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oJ3BioApnFMK",
        "outputId": "dbbdaa1f-aee6-4f5e-faf2-ea77c08f1082"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Categorical: \n",
            "['invasive_vent', 'language', 'ventilation_status', 'insurance', 'race_group', 'gender', 'rrt', 'mortality_in', 'sofa_resp', 'hidden_hypoxemia']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Unique values in categorical columns"
      ],
      "metadata": {
        "id": "4AVAghb6nQmK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "* The binary variables are: `var1, var2, ...`\n",
        "* The ordinal variables are: `var1, var2, ...`\n",
        "* The Nominal variables are: `var1, var2, ...`\n"
      ],
      "metadata": {
        "id": "zOdxJcAwg-pg"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Encode binary variables\n",
        "\n",
        "Machine learning models only understand numbers, so binary variables must be encoded as 1s and 0s. Replace the values in those columns with 1s and 0s\n",
        "\n",
        "Hint: you can use `df.replace()` to do that"
      ],
      "metadata": {
        "id": "TiGW44npS-Xv"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "pbUYvdTtrAAj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Encode ordinal variables\n",
        "\n",
        "You can use the function `label_encoder ` to encode the ordinal variables as numeric variables."
      ],
      "metadata": {
        "id": "A271BKQxVWKc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Hint**: Some variables are already numerical, but have 'None' due to the imputation. You can use `df.replace` to convert that 'None' to numerical"
      ],
      "metadata": {
        "id": "Zo2r5Vq3h-xi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def label_encoder(df, cols, df_test=None):\n",
        "  \"\"\"\n",
        "  This function applies the LabelEncoder from Scikit-learn to encode categorical variables in a Pandas DataFrame.\n",
        "\n",
        "  Parameters:\n",
        "\n",
        "  df: Pandas DataFrame. The DataFrame to be encoded.\n",
        "  cols: list. List of column names to be encoded.\n",
        "  df_test: Pandas DataFrame (optional). The DataFrame to be encoded using the same encoders used on the training set.\n",
        "  Returns:\n",
        "\n",
        "  If df_test is None, the function returns a tuple containing the encoded DataFrame and a dictionary with the LabelEncoders used for each column. Otherwise, it returns a tuple with the encoded training DataFrame, the encoded test DataFrame and a dictionary with the LabelEncoders used for each column.\n",
        "  \"\"\"\n",
        "  encoders = {}\n",
        "  for col in cols:\n",
        "    encoders[col] = LabelEncoder()\n",
        "    df[col] = encoders[col].fit_transform(df[col])\n",
        "    if df_test is not None:\n",
        "      df_test[col] = encoders[col].transform(df_test[col])\n",
        "  \n",
        "  if df_test is not None:\n",
        "    return df, df_test, encoders\n",
        "  else:\n",
        "    return df, encoders\n",
        "\n",
        "def label_encoder_test(df, encoders):\n",
        "  \"\"\"\n",
        "  This function applies the LabelEncoder from Scikit-learn to encode categorical variables in a Pandas DataFrame.\n",
        "\n",
        "  Parameters:\n",
        "\n",
        "  df: Pandas DataFrame. The DataFrame to be encoded.\n",
        "  encoders: Python dictionary. Dictionary with the encoders trained on the train set.\n",
        "\n",
        "  \"\"\"\n",
        "  encoders = {}\n",
        "  for col in encoders.keys():\n",
        "    df[col] = encoders[col].transform(df[col])\n",
        "\n",
        "  return df\n"
      ],
      "metadata": {
        "id": "ZnENxfbLXGST"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "### This is not the case, but just to show how to use the function:\n",
        "# train, test, encoders = label_encoder(train, ['insurance'], df_test=test)"
      ],
      "metadata": {
        "id": "Fa28bOmRZycF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Encode nominal variables\n",
        "\n",
        "You can use the function `column_to_one_hot` to encode the nominal variables to a [one-hot representation](https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/). "
      ],
      "metadata": {
        "id": "SQvRcqdRVXJV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def column_to_one_hot(train_df, column, test_df=None):\n",
        "  # Generate a one-hot representation of the values in the column\n",
        "  train_one_hot = pd.get_dummies(train_df[column])\n",
        "  # add the one-hot encoded columns to the DataFrame\n",
        "  train_df = pd.concat([train_df, train_one_hot], axis=1)\n",
        "  # drop the original column\n",
        "  train_df = train_df.drop(column, axis=1)\n",
        "\n",
        "  if test_df is not None:\n",
        "    test_one_hot = pd.get_dummies(test_df[column])\n",
        "  \n",
        "    # Add missing columns in test data\n",
        "    missing_cols = set(train_one_hot.columns) - set(test_one_hot.columns)\n",
        "    for c in missing_cols:\n",
        "      test_one_hot[c] = 0\n",
        "    \n",
        "    # Ensure the order of column in the test set is in the same order than in train set\n",
        "    test_one_hot = test_one_hot[train_one_hot.columns]\n",
        "    test_df = pd.concat([test_df, test_one_hot], axis=1)\n",
        "    test_df = test_df.drop(column, axis=1)\n",
        "    return train_df, test_df\n",
        "  else:\n",
        "    return train_df"
      ],
      "metadata": {
        "id": "61UE_CRi-cfb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# E.g. Convert 'insurance' column into one-hot encoding:\n",
        "# train, test = column_to_one_hot(train, 'insurance', test)"
      ],
      "metadata": {
        "id": "j-Kl9LE_95Ve"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Feature Engineering\n",
        "\n",
        "Once we have cleaned the dataset we can engineer the features that can provide more information to the model to generate the predictions.\n",
        "\n",
        "We'll print the correlation to SpO2 as a reference to see the most correlated variables before and after generating the new features."
      ],
      "metadata": {
        "id": "2lt2Bd_Keu1U"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Get variables correlated to SpO2\n",
        "columns_sao2 = get_most_correlated(train, variable='SpO2', n=5)"
      ],
      "metadata": {
        "id": "XohbidtbS6qe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Length of ICU stay in days for dead\n",
        "train['los_icu_dead'] = train[train.mortality_in == 1].los_icu\n",
        "test['los_icu_dead'] = test[test.mortality_in == 1].los_icu\n",
        "\n",
        "# Length of ICU stay in days for survivors\n",
        "train['los_icu_surv'] = train[train.mortality_in == 0].los_icu\n",
        "test['los_icu_surv'] = test[test.mortality_in == 1].los_icu"
      ],
      "metadata": {
        "id": "y7gi7_6Kes54"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Get variables correlated to SpO2\n",
        "columns_sao2 = get_most_correlated(train, variable='SpO2', n=5)"
      ],
      "metadata": {
        "id": "ZmrafVfees8d"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Surprise! we can see how before the correlation `los_icu` and `mortality_in` were not there, but after the feature engineering the new feature `los_icu_dead` is there! \n",
        "\n",
        "That shows the importance of being able to do good feature engineering."
      ],
      "metadata": {
        "id": "Gd5hOGIVTtTy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Clean the new added columns:\n",
        "method = 'new_category_numeric'\n",
        "impute_cols = ['los_icu_dead', 'los_icu_surv']\n",
        "train, test, _ = apply_data_imputation(train, impute_cols=impute_cols, method=method, test_df=test)"
      ],
      "metadata": {
        "id": "kN3sRd5TCiZ0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Create your own features (Optional)\n",
        "\n",
        "Use the dataset to generate new features"
      ],
      "metadata": {
        "id": "XvY4nJDzW42S"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "4Yw58kNsiiR6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Normalization:\n",
        "Data normalization is an important step in machine learning because it helps to ensure that features are on similar scales, which can improve the performance of many machine learning algorithms.\n",
        "\n",
        "When features are not on similar scales, some algorithms may be more heavily influenced by certain features than others, which can lead to suboptimal performance. Additionally, some algorithms (such as those based on distance calculations) can be sensitive to differences in scale between features, which can lead to incorrect results.\n",
        "\n",
        "By normalizing the data, we can ensure that each feature contributes equally to the model, regardless of its scale. This can lead to better accuracy and more robust models."
      ],
      "metadata": {
        "id": "1kXGwnxLjWJF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def normalize_data(train_data, test_data=None, scaler=None, ignore_cols=[]):\n",
        "  # Filter out columns to ignore\n",
        "  train_data_filtered = train_data.drop(ignore_cols, axis=1)\n",
        "  if test_data is not None:\n",
        "    test_data_filtered = test_data.drop(ignore_cols, axis=1)\n",
        "  \n",
        "  if not scaler:\n",
        "    scaler = MinMaxScaler()\n",
        "    scaler.fit(train_data_filtered)\n",
        "  \n",
        "  normalized_train = scaler.transform(train_data_filtered)\n",
        "  \n",
        "  if test_data is not None:\n",
        "    normalized_test = scaler.transform(test_data_filtered)\n",
        "    normalized_train_df = pd.DataFrame(normalized_train, columns=train_data_filtered.columns)\n",
        "    normalized_test_df = pd.DataFrame(normalized_test, columns=test_data_filtered.columns)\n",
        "    return normalized_train_df, normalized_test_df, scaler\n",
        "  else:\n",
        "    normalized_train_df = pd.DataFrame(normalized_train, columns=train_data_filtered.columns)\n",
        "    return normalized_train_df, scaler\n"
      ],
      "metadata": {
        "id": "S2UttMiAjWU0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Normalize your data\n",
        "\n",
        "Use the function to normalize the data. Remember to exclude information that does not contribute to the final model, such as identifiers.\n",
        "\n",
        "Feel free to include other data [normalization methods](https://scikit-learn.org/stable/modules/preprocessing.html)."
      ],
      "metadata": {
        "id": "DMtjK8r-XDmp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ignore_cols = #\n",
        "normalized_train, normalized_test, scaler = "
      ],
      "metadata": {
        "id": "qA8jd0uqmhAt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Class Imbalance\n",
        "In order to allow you to play with other methods for data balancing such as modifying the weights of the model. The dataset will be delivered unbalanced. However it is important that you play around with different methods such as undersampling or oversampling to balance the training data."
      ],
      "metadata": {
        "id": "qeAp94vzQ7_Y"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Save the file!\n",
        "Finally let's save the dataset as a csv file! 😀"
      ],
      "metadata": {
        "id": "hKqDpYhNpLcC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "normalized_train"
      ],
      "metadata": {
        "id": "WY9spVmopc1T"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###### ✏️ Save your train and test dataset as a csv file:\n",
        "\n",
        "Hint: use `df.to_csv('path/data_clean.csv', index=False)`"
      ],
      "metadata": {
        "id": "GzWHbaIbXx6d"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown ###  ✏️Save your train and test dataset as a csv file:\n",
        "path_to_save_train = \"path/train.csv\" #@param {type:\"string\"}\n",
        "path_to_save_test = \"path/test.csv\" #@param {type:\"string\"}\n"
      ],
      "metadata": {
        "id": "BD4fUTkT5dUh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import pandas as pd\n",
        "\n",
        "# Function to create the directory if it doesn't exist\n",
        "def create_directory_if_not_exists(path):\n",
        "    directory = os.path.dirname(path)\n",
        "    os.makedirs(directory, exist_ok=True)\n",
        "\n",
        "# Save function to be used for both train and test datasets\n",
        "def save_dataset(dataset, path_to_save):\n",
        "    create_directory_if_not_exists(path_to_save)\n",
        "    dataset.to_csv(path_to_save, index=False)"
      ],
      "metadata": {
        "id": "y8ElZGp-7sDs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Export files:\n"
      ],
      "metadata": {
        "id": "zXD403Pto7WX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Thank you very much! \n",
        "\n",
        "From the MIT Critical Data team we wish you good luck 😀!!\n",
        "\n",
        "Your mentors:\n",
        "\n",
        "- David Restrepo: davidres@mit.edu\n",
        "- Sebastian Cajas: ulsordonez@unicauca.edu.co \n",
        "- Adrien Carrel: a.carrel@hotmail.fr\n",
        "- Jack Gallifant: jack.gallifant@nhs.net"
      ],
      "metadata": {
        "id": "L2Lov1eLUtrw"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "m37ESXcY6GPG"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}