MLflow Live Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6505a7ad",
   "metadata": {},
   "source": [
    "## What is MLFlow and its Components\n",
    "\n",
    "MLFLow is an open source platform to manage the ML lifecycle, including experimentation, reproducibility, deployment, and a central model registry. MLFLow currently offers four components:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5745f450",
   "metadata": {},
   "source": [
    "<img src=\"mlflow.png\">\n",
    "\n",
    "\n",
    "https://www.mlflow.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f44a8b9",
   "metadata": {},
   "source": [
    "_First thing first_\n",
    "### Create Conda environment"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90eb0ce0",
   "metadata": {},
   "source": [
    "##### run below commands in terminal but make sure conda is installed or use anaconda prompt which you will get as part of anaconda installation\n",
    "\n",
    "1. `conda create -n envname python=3.9 ipykernel` \n",
    "it will create a conda env named envname and install python version 3.9 and a ipykernel inside this environment\n",
    "\n",
    "2. Activate the environment\n",
    "`conda activate envname`\n",
    "\n",
    "3. add newly created environment to the notebook as kernel\n",
    "`python -m ipykernel install --user --name=envname` \n",
    "\n",
    "4. install notebook inside the environment\n",
    "`pip install notebook`\n",
    "\n",
    "5. Now install all required dependencies to run this notebook\n",
    "\n",
    "* `pip install pandas`\n",
    "* `pip install numpy`\n",
    "* `pip install scikit-learn`\n",
    "* `pip install imblearn`\n",
    "* `pip install matplotlib`\n",
    "* `pip install mlflow`\n",
    "\n",
    "Now open the notebook using below command: (from the anaconda prompt inside conda environment)\n",
    "\n",
    "`jupyter notebook`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cf5a6b9",
   "metadata": {},
   "source": [
    "#### Make sure python is used from your newly created environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9fd81db9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-06T15:46:06.997302Z",
     "start_time": "2023-02-06T15:46:06.989295Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c:\\users\\furqan ali\\desktop\\python\\env\\scripts\\python.exe\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "print(sys.executable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3caf80a9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-06T15:46:09.732801Z",
     "start_time": "2023-02-06T15:46:09.662284Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python 3.8.6\n"
     ]
    }
   ],
   "source": [
    "!python --version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae5ff2b8",
   "metadata": {},
   "source": [
    "### Create functions for all the steps involved in complete model training lifecycle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88a1ea93",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "2f0efd11",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:21:24.301935Z",
     "start_time": "2023-02-07T03:21:24.294801Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "ed9e53b1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:21:24.538724Z",
     "start_time": "2023-02-07T03:21:24.529722Z"
    }
   },
   "outputs": [],
   "source": [
    "def load_data(path):\n",
    "    data = pd.read_csv(path)\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "5611082a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:21:24.933759Z",
     "start_time": "2023-02-07T03:21:24.731508Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>job</th>\n",
       "      <th>marital</th>\n",
       "      <th>education</th>\n",
       "      <th>default</th>\n",
       "      <th>housing</th>\n",
       "      <th>loan</th>\n",
       "      <th>contact</th>\n",
       "      <th>month</th>\n",
       "      <th>day_of_week</th>\n",
       "      <th>...</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>previous</th>\n",
       "      <th>poutcome</th>\n",
       "      <th>emp_var_rate</th>\n",
       "      <th>cons_price_idx</th>\n",
       "      <th>cons_conf_idx</th>\n",
       "      <th>euribor3m</th>\n",
       "      <th>nr_employed</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>44</td>\n",
       "      <td>blue-collar</td>\n",
       "      <td>married</td>\n",
       "      <td>basic.4y</td>\n",
       "      <td>unknown</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>aug</td>\n",
       "      <td>thu</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.4</td>\n",
       "      <td>93.444</td>\n",
       "      <td>-36.1</td>\n",
       "      <td>4.963</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>53</td>\n",
       "      <td>technician</td>\n",
       "      <td>married</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>nov</td>\n",
       "      <td>fri</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>-0.1</td>\n",
       "      <td>93.200</td>\n",
       "      <td>-42.0</td>\n",
       "      <td>4.021</td>\n",
       "      <td>5195.8</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>28</td>\n",
       "      <td>management</td>\n",
       "      <td>single</td>\n",
       "      <td>university.degree</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>jun</td>\n",
       "      <td>thu</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>success</td>\n",
       "      <td>-1.7</td>\n",
       "      <td>94.055</td>\n",
       "      <td>-39.8</td>\n",
       "      <td>0.729</td>\n",
       "      <td>4991.6</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>39</td>\n",
       "      <td>services</td>\n",
       "      <td>married</td>\n",
       "      <td>high.school</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>apr</td>\n",
       "      <td>fri</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>93.075</td>\n",
       "      <td>-47.1</td>\n",
       "      <td>1.405</td>\n",
       "      <td>5099.1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>55</td>\n",
       "      <td>retired</td>\n",
       "      <td>married</td>\n",
       "      <td>basic.4y</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>aug</td>\n",
       "      <td>fri</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>success</td>\n",
       "      <td>-2.9</td>\n",
       "      <td>92.201</td>\n",
       "      <td>-31.4</td>\n",
       "      <td>0.869</td>\n",
       "      <td>5076.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   age          job  marital          education  default housing loan  \\\n",
       "0   44  blue-collar  married           basic.4y  unknown     yes   no   \n",
       "1   53   technician  married            unknown       no      no   no   \n",
       "2   28   management   single  university.degree       no     yes   no   \n",
       "3   39     services  married        high.school       no      no   no   \n",
       "4   55      retired  married           basic.4y       no     yes   no   \n",
       "\n",
       "    contact month day_of_week  ...  campaign  pdays  previous     poutcome  \\\n",
       "0  cellular   aug         thu  ...         1    999         0  nonexistent   \n",
       "1  cellular   nov         fri  ...         1    999         0  nonexistent   \n",
       "2  cellular   jun         thu  ...         3      6         2      success   \n",
       "3  cellular   apr         fri  ...         2    999         0  nonexistent   \n",
       "4  cellular   aug         fri  ...         1      3         1      success   \n",
       "\n",
       "  emp_var_rate  cons_price_idx  cons_conf_idx  euribor3m  nr_employed  y  \n",
       "0          1.4          93.444          -36.1      4.963       5228.1  0  \n",
       "1         -0.1          93.200          -42.0      4.021       5195.8  0  \n",
       "2         -1.7          94.055          -39.8      0.729       4991.6  1  \n",
       "3         -1.8          93.075          -47.1      1.405       5099.1  0  \n",
       "4         -2.9          92.201          -31.4      0.869       5076.2  1  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = load_data('./banking.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "bcdca67e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:21:24.948758Z",
     "start_time": "2023-02-07T03:21:24.935760Z"
    }
   },
   "outputs": [],
   "source": [
    "def data_cleaning(data):\n",
    "    print(\"na values available in data \\n\")\n",
    "    print(data.isna().sum())\n",
    "    data = data.dropna()\n",
    "    print(\"after droping na values \\n\")\n",
    "    print(data.isna().sum())\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "0d378c83",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:21:25.058311Z",
     "start_time": "2023-02-07T03:21:25.052304Z"
    }
   },
   "outputs": [],
   "source": [
    "def preprocessing(data):\n",
    "    data['education']=np.where(data['education'] =='basic.9y', 'Basic', data['education'])\n",
    "    data['education']=np.where(data['education'] =='basic.6y', 'Basic', data['education'])\n",
    "    data['education']=np.where(data['education'] =='basic.4y', 'Basic', data['education'])\n",
    "    \n",
    "    cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']\n",
    "    for var in cat_vars:\n",
    "        cat_list='var'+'_'+var\n",
    "        cat_list = pd.get_dummies(data[var], prefix=var)\n",
    "        data1=data.join(cat_list)\n",
    "        data=data1\n",
    "\n",
    "    # Removing categorical columns bcz we already created their dummy variable on top.  \n",
    "    cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']\n",
    "    data_vars=data.columns.values.tolist()\n",
    "    to_keep=[i for i in data_vars if i not in cat_vars]\n",
    "    \n",
    "    final_data=data[to_keep]\n",
    "    \n",
    "    \n",
    "    final_data.columns = final_data.columns.str.replace('.','_')\n",
    "    final_data.columns = final_data.columns.str.replace(' ','_')\n",
    "    return final_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "e6f78530",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:21:25.276960Z",
     "start_time": "2023-02-07T03:21:25.268599Z"
    }
   },
   "outputs": [],
   "source": [
    "def train_test_split(final_data):\n",
    "    from sklearn.model_selection import train_test_split\n",
    "    X = final_data.loc[:, final_data.columns != 'y']\n",
    "    y = final_data.loc[:, final_data.columns == 'y']\n",
    "    \n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,stratify = y, random_state=47)\n",
    "    return X_train, X_test, y_train, y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "3eb6095a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:21:25.449207Z",
     "start_time": "2023-02-07T03:21:25.439064Z"
    }
   },
   "outputs": [],
   "source": [
    "def over_sampling_target_class(X_train, y_train):\n",
    "    ### Over-sampling using SMOTE \n",
    "    from imblearn.over_sampling import SMOTE\n",
    "    os = SMOTE(random_state=0)\n",
    "\n",
    "    columns = X_train.columns\n",
    "    os_data_X,os_data_y=os.fit_resample(X_train, y_train)\n",
    "\n",
    "    os_data_X = pd.DataFrame(data=os_data_X,columns=columns )\n",
    "    os_data_y= pd.DataFrame(data=os_data_y,columns=['y'])\n",
    "    # we can Check the numbers of our data\n",
    "    print(\"length of oversampled data is \",len(os_data_X))\n",
    "    print(\"Number of no subscription in oversampled data\",len(os_data_y[os_data_y['y']==0]))\n",
    "    print(\"Number of subscription\",len(os_data_y[os_data_y['y']==1]))\n",
    "    print(\"Proportion of no subscription data in oversampled data is \",len(os_data_y[os_data_y['y']==0])/len(os_data_X))\n",
    "    print(\"Proportion of subscription data in oversampled data is \",len(os_data_y[os_data_y['y']==1])/len(os_data_X))\n",
    "    \n",
    "    X_train = os_data_X\n",
    "    y_train = os_data_y['y']\n",
    " \n",
    "    return X_train, y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "3c85667e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:22:48.257273Z",
     "start_time": "2023-02-07T03:22:48.247697Z"
    }
   },
   "outputs": [],
   "source": [
    "def training_basic_classifier(X_train,y_train):\n",
    "    from sklearn.ensemble import RandomForestClassifier\n",
    "    model = RandomForestClassifier(n_estimators=101)\n",
    "    model.fit(X_train, y_train)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "ce3c96bc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:22:48.552993Z",
     "start_time": "2023-02-07T03:22:48.535847Z"
    }
   },
   "outputs": [],
   "source": [
    "def predict_on_test_data(model,X_test):\n",
    "    y_pred = model.predict(X_test)\n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "1a475cb2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:22:49.383231Z",
     "start_time": "2023-02-07T03:22:49.376212Z"
    }
   },
   "outputs": [],
   "source": [
    "def predict_prob_on_test_data(model,X_test):\n",
    "    y_pred = model.predict_proba(X_test)\n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "07ccc67d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:22:53.060353Z",
     "start_time": "2023-02-07T03:22:53.050773Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_metrics(y_true, y_pred, y_pred_prob):\n",
    "    from sklearn.metrics import accuracy_score,precision_score,recall_score,log_loss\n",
    "    acc = accuracy_score(y_true, y_pred)\n",
    "    prec = precision_score(y_true, y_pred)\n",
    "    recall = recall_score(y_true, y_pred)\n",
    "    entropy = log_loss(y_true, y_pred_prob)\n",
    "    return {'accuracy': round(acc, 2), 'precision': round(prec, 2), 'recall': round(recall, 2), 'entropy': round(entropy, 2)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "ca1aeebe",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:22:57.483241Z",
     "start_time": "2023-02-07T03:22:57.472231Z"
    }
   },
   "outputs": [],
   "source": [
    "def create_roc_auc_plot(clf, X_data, y_data):\n",
    "    import matplotlib.pyplot as plt\n",
    "    from sklearn import metrics\n",
    "    metrics.plot_roc_curve(clf, X_data, y_data) \n",
    "    plt.savefig('roc_auc_curve.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "db16c987",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:22:57.940624Z",
     "start_time": "2023-02-07T03:22:57.935625Z"
    }
   },
   "outputs": [],
   "source": [
    "def create_confusion_matrix_plot(clf, X_test, y_test):\n",
    "    import matplotlib.pyplot as plt\n",
    "    from sklearn.metrics import plot_confusion_matrix\n",
    "    plot_confusion_matrix(clf, X_test, y_test)\n",
    "    plt.savefig('confusion_matrix.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "3aec48cc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:23:53.863339Z",
     "start_time": "2023-02-07T03:23:53.856323Z"
    }
   },
   "outputs": [],
   "source": [
    "def hyper_parameter_tuning(X_train, y_train):\n",
    "    # define random parameters grid\n",
    "    n_estimators = [5,21,51,101] # number of trees in the random forest\n",
    "    max_features = ['auto', 'sqrt'] # number of features in consideration at every split\n",
    "    max_depth = [int(x) for x in np.linspace(10, 120, num = 12)] # maximum number of levels allowed in each decision tree\n",
    "    min_samples_split = [2, 6, 10] # minimum sample number to split a node\n",
    "    min_samples_leaf = [1, 3, 4] # minimum sample number that can be stored in a leaf node\n",
    "    bootstrap = [True, False] # method used to sample data points\n",
    "\n",
    "    random_grid = {'n_estimators': n_estimators,\n",
    "                    'max_features': max_features,\n",
    "                    'max_depth': max_depth,\n",
    "                    'min_samples_split': min_samples_split,\n",
    "                    'min_samples_leaf': min_samples_leaf,\n",
    "                    'bootstrap': bootstrap\n",
    "                  }\n",
    "    \n",
    "    from sklearn.model_selection import RandomizedSearchCV\n",
    "    from sklearn.ensemble import RandomForestClassifier\n",
    "    classifier = RandomForestClassifier()\n",
    "    model_tuning = RandomizedSearchCV(estimator = classifier, param_distributions = random_grid,\n",
    "                   n_iter = 100, cv = 5, verbose=2, random_state=35, n_jobs = -1)\n",
    "    model_tuning.fit(X_train, y_train)\n",
    "\n",
    "    print ('Random grid: ', random_grid, '\\n')\n",
    "    # print the best parameters\n",
    "    print ('Best Parameters: ', model_tuning.best_params_, ' \\n')\n",
    "\n",
    "    best_params = model_tuning.best_params_\n",
    "    \n",
    "    n_estimators = best_params['n_estimators']\n",
    "    min_samples_split = best_params['min_samples_split']\n",
    "    min_samples_leaf = best_params['min_samples_leaf']\n",
    "    max_features = best_params['max_features']\n",
    "    max_depth = best_params['max_depth']\n",
    "    bootstrap = best_params['bootstrap']\n",
    "    \n",
    "    model_tuned = RandomForestClassifier(n_estimators = n_estimators, min_samples_split = min_samples_split,\n",
    "                                         min_samples_leaf= min_samples_leaf, max_features = max_features,\n",
    "                                         max_depth= max_depth, bootstrap=bootstrap) \n",
    "    model_tuned.fit( X_train, y_train)\n",
    "    return model_tuned,best_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8aeb7b1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "8acce9f7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:23:56.192487Z",
     "start_time": "2023-02-07T03:23:56.076230Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data = load_data('./banking.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "376d0587",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:23:57.144234Z",
     "start_time": "2023-02-07T03:23:57.041498Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "na values available in data \n",
      "\n",
      "age               0\n",
      "job               0\n",
      "marital           0\n",
      "education         0\n",
      "default           0\n",
      "housing           0\n",
      "loan              0\n",
      "contact           0\n",
      "month             0\n",
      "day_of_week       0\n",
      "duration          0\n",
      "campaign          0\n",
      "pdays             0\n",
      "previous          0\n",
      "poutcome          0\n",
      "emp_var_rate      0\n",
      "cons_price_idx    0\n",
      "cons_conf_idx     0\n",
      "euribor3m         0\n",
      "nr_employed       0\n",
      "y                 0\n",
      "dtype: int64\n",
      "after droping na values \n",
      "\n",
      "age               0\n",
      "job               0\n",
      "marital           0\n",
      "education         0\n",
      "default           0\n",
      "housing           0\n",
      "loan              0\n",
      "contact           0\n",
      "month             0\n",
      "day_of_week       0\n",
      "duration          0\n",
      "campaign          0\n",
      "pdays             0\n",
      "previous          0\n",
      "poutcome          0\n",
      "emp_var_rate      0\n",
      "cons_price_idx    0\n",
      "cons_conf_idx     0\n",
      "euribor3m         0\n",
      "nr_employed       0\n",
      "y                 0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "cleaned_data = data_cleaning(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "e8e4a40b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:23:57.923863Z",
     "start_time": "2023-02-07T03:23:57.806348Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Furqan Ali\\AppData\\Local\\Temp\\ipykernel_26224\\4218850270.py:21: FutureWarning: The default value of regex will change from True to False in a future version. In addition, single character regular expressions will *not* be treated as literal strings when regex=True.\n",
      "  final_data.columns = final_data.columns.str.replace('.','_')\n"
     ]
    }
   ],
   "source": [
    "final_data = preprocessing(cleaned_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "73be0d6c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:23:57.955863Z",
     "start_time": "2023-02-07T03:23:57.925854Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>duration</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>previous</th>\n",
       "      <th>emp_var_rate</th>\n",
       "      <th>cons_price_idx</th>\n",
       "      <th>cons_conf_idx</th>\n",
       "      <th>euribor3m</th>\n",
       "      <th>nr_employed</th>\n",
       "      <th>...</th>\n",
       "      <th>month_oct</th>\n",
       "      <th>month_sep</th>\n",
       "      <th>day_of_week_fri</th>\n",
       "      <th>day_of_week_mon</th>\n",
       "      <th>day_of_week_thu</th>\n",
       "      <th>day_of_week_tue</th>\n",
       "      <th>day_of_week_wed</th>\n",
       "      <th>poutcome_failure</th>\n",
       "      <th>poutcome_nonexistent</th>\n",
       "      <th>poutcome_success</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>44</td>\n",
       "      <td>210</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>93.444</td>\n",
       "      <td>-36.1</td>\n",
       "      <td>4.963</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>53</td>\n",
       "      <td>138</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.1</td>\n",
       "      <td>93.200</td>\n",
       "      <td>-42.0</td>\n",
       "      <td>4.021</td>\n",
       "      <td>5195.8</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>28</td>\n",
       "      <td>339</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.7</td>\n",
       "      <td>94.055</td>\n",
       "      <td>-39.8</td>\n",
       "      <td>0.729</td>\n",
       "      <td>4991.6</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>39</td>\n",
       "      <td>185</td>\n",
       "      <td>2</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>93.075</td>\n",
       "      <td>-47.1</td>\n",
       "      <td>1.405</td>\n",
       "      <td>5099.1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>55</td>\n",
       "      <td>137</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>-2.9</td>\n",
       "      <td>92.201</td>\n",
       "      <td>-31.4</td>\n",
       "      <td>0.869</td>\n",
       "      <td>5076.2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41183</th>\n",
       "      <td>59</td>\n",
       "      <td>222</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>94.465</td>\n",
       "      <td>-41.8</td>\n",
       "      <td>4.866</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41184</th>\n",
       "      <td>31</td>\n",
       "      <td>196</td>\n",
       "      <td>2</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.860</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41185</th>\n",
       "      <td>42</td>\n",
       "      <td>62</td>\n",
       "      <td>3</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41186</th>\n",
       "      <td>48</td>\n",
       "      <td>200</td>\n",
       "      <td>2</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>-3.4</td>\n",
       "      <td>92.431</td>\n",
       "      <td>-26.9</td>\n",
       "      <td>0.742</td>\n",
       "      <td>5017.5</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41187</th>\n",
       "      <td>25</td>\n",
       "      <td>112</td>\n",
       "      <td>4</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.859</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>41188 rows × 62 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       age  duration  campaign  pdays  previous  emp_var_rate  cons_price_idx  \\\n",
       "0       44       210         1    999         0           1.4          93.444   \n",
       "1       53       138         1    999         0          -0.1          93.200   \n",
       "2       28       339         3      6         2          -1.7          94.055   \n",
       "3       39       185         2    999         0          -1.8          93.075   \n",
       "4       55       137         1      3         1          -2.9          92.201   \n",
       "...    ...       ...       ...    ...       ...           ...             ...   \n",
       "41183   59       222         1    999         0           1.4          94.465   \n",
       "41184   31       196         2    999         0           1.1          93.994   \n",
       "41185   42        62         3    999         0           1.1          93.994   \n",
       "41186   48       200         2    999         0          -3.4          92.431   \n",
       "41187   25       112         4    999         0           1.1          93.994   \n",
       "\n",
       "       cons_conf_idx  euribor3m  nr_employed  ...  month_oct  month_sep  \\\n",
       "0              -36.1      4.963       5228.1  ...          0          0   \n",
       "1              -42.0      4.021       5195.8  ...          0          0   \n",
       "2              -39.8      0.729       4991.6  ...          0          0   \n",
       "3              -47.1      1.405       5099.1  ...          0          0   \n",
       "4              -31.4      0.869       5076.2  ...          0          0   \n",
       "...              ...        ...          ...  ...        ...        ...   \n",
       "41183          -41.8      4.866       5228.1  ...          0          0   \n",
       "41184          -36.4      4.860       5191.0  ...          0          0   \n",
       "41185          -36.4      4.857       5191.0  ...          0          0   \n",
       "41186          -26.9      0.742       5017.5  ...          1          0   \n",
       "41187          -36.4      4.859       5191.0  ...          0          0   \n",
       "\n",
       "       day_of_week_fri  day_of_week_mon  day_of_week_thu  day_of_week_tue  \\\n",
       "0                    0                0                1                0   \n",
       "1                    1                0                0                0   \n",
       "2                    0                0                1                0   \n",
       "3                    1                0                0                0   \n",
       "4                    1                0                0                0   \n",
       "...                ...              ...              ...              ...   \n",
       "41183                0                0                1                0   \n",
       "41184                0                0                1                0   \n",
       "41185                0                0                0                0   \n",
       "41186                0                0                0                1   \n",
       "41187                1                0                0                0   \n",
       "\n",
       "       day_of_week_wed  poutcome_failure  poutcome_nonexistent  \\\n",
       "0                    0                 0                     1   \n",
       "1                    0                 0                     1   \n",
       "2                    0                 0                     0   \n",
       "3                    0                 0                     1   \n",
       "4                    0                 0                     0   \n",
       "...                ...               ...                   ...   \n",
       "41183                0                 0                     1   \n",
       "41184                0                 0                     1   \n",
       "41185                1                 0                     1   \n",
       "41186                0                 0                     1   \n",
       "41187                0                 0                     1   \n",
       "\n",
       "       poutcome_success  \n",
       "0                     0  \n",
       "1                     0  \n",
       "2                     1  \n",
       "3                     0  \n",
       "4                     1  \n",
       "...                 ...  \n",
       "41183                 0  \n",
       "41184                 0  \n",
       "41185                 0  \n",
       "41186                 0  \n",
       "41187                 0  \n",
       "\n",
       "[41188 rows x 62 columns]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "7bbb6b98",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:24:02.643777Z",
     "start_time": "2023-02-07T03:23:59.707385Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(final_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "8da133cc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:24:06.820245Z",
     "start_time": "2023-02-07T03:24:02.645779Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "length of oversampled data is  51166\n",
      "Number of no subscription in oversampled data 25583\n",
      "Number of subscription 25583\n",
      "Proportion of no subscription data in oversampled data is  0.5\n",
      "Proportion of subscription data in oversampled data is  0.5\n"
     ]
    }
   ],
   "source": [
    "X_train, y_train = over_sampling_target_class(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "3c0e3da5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:24:18.684560Z",
     "start_time": "2023-02-07T03:24:13.810196Z"
    }
   },
   "outputs": [],
   "source": [
    "model = training_basic_classifier(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "2b485e52",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:24:18.889242Z",
     "start_time": "2023-02-07T03:24:18.685548Z"
    }
   },
   "outputs": [],
   "source": [
    "y_pred = predict_on_test_data(model,X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "55d29fda",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:24:22.493867Z",
     "start_time": "2023-02-07T03:24:22.484445Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, ..., 0, 0, 0], dtype=int64)"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "19e4fd5d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:24:30.365239Z",
     "start_time": "2023-02-07T03:24:30.124467Z"
    }
   },
   "outputs": [],
   "source": [
    "y_pred_prob = predict_prob_on_test_data(model,X_test) #model.predict_proba(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "5f7765b7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:24:32.249603Z",
     "start_time": "2023-02-07T03:24:32.229376Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.97029703, 0.02970297],\n",
       "       [0.97029703, 0.02970297],\n",
       "       [0.93069307, 0.06930693],\n",
       "       ...,\n",
       "       [1.        , 0.        ],\n",
       "       [0.69306931, 0.30693069],\n",
       "       [1.        , 0.        ]])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "ceef3c86",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:24:35.103496Z",
     "start_time": "2023-02-07T03:24:35.076960Z"
    }
   },
   "outputs": [],
   "source": [
    "run_metrics = get_metrics(y_test, y_pred, y_pred_prob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "2bdeda22",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:24:36.884446Z",
     "start_time": "2023-02-07T03:24:36.867428Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'accuracy': 0.91, 'precision': 0.64, 'recall': 0.52, 'entropy': 0.2}\n"
     ]
    }
   ],
   "source": [
    "print(run_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "2ff3c7fb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:24:43.177612Z",
     "start_time": "2023-02-07T03:24:41.915842Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\furqan ali\\desktop\\python\\env\\lib\\site-packages\\sklearn\\utils\\deprecation.py:87: FutureWarning: Function plot_roc_curve is deprecated; Function :func:`plot_roc_curve` is deprecated in 1.0 and will be removed in 1.2. Use one of the class methods: :meth:`sklearn.metric.RocCurveDisplay.from_predictions` or :meth:`sklearn.metric.RocCurveDisplay.from_estimator`.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1TUlEQVR4nO3deXxU5dXA8d/JRhJIQFYRxIAGWQIESUGgWCh1qQpaqyIulaovrXurYrVatWittpa2opZiRXAFsbXFpVht3UElSAgBlM0gQZYkQEhIQrbz/nFv4iQkmRuSyWQy5/shn9z9njsT5sx9nuc+j6gqxhhjwldEsAMwxhgTXJYIjDEmzFkiMMaYMGeJwBhjwpwlAmOMCXNRwQ6gqbp3765JSUnBDsMYY0LK6tWr81S1R33rQi4RJCUlkZ6eHuwwjDEmpIjI9obWWdGQMcaEOUsExhgT5iwRGGNMmLNEYIwxYc4SgTHGhLmAJQIRWSAie0Ukq4H1IiKPisgWEckUkVMCFYsxxpiGBfKOYCFwViPrvw8kuz8zgb8EMBZjjDENCNhzBKr6vogkNbLJecAz6vSD/bGIdBGR3qq6K1AxGROOVJXKKqXC/amsVCqqqqhSUBT3H+rOa828M01D69zl1Frus53PdJOP4bPOb4zO7nXO+821+43T3e6IGBs7Rz3HwDfmWvE3cPw68zXxNnL8yYN7MeL4Li30l/GNYD5Q1gfY4TOf4y47IhGIyEycuwb69evXKsGZ8KOqlFcqhysqOVxR5fyUV1JaXsXhikrK3Q/QyiqlotL9UK2qcj5gj2Le+XCufbyabTzO1xyjUimvnq90jlk9X1llY460Fz0TY9tdIvBMVecD8wHS0tLsrzpEqSol5ZXsLy5n/6EyDh2ucL6VqlKlUKVKlfuNqMpdVlmlja6vqHQ/sCsqOVxeRan7u2ZZRZU77/PhXvHNh3vdbQM9TlNUhBAZITW/oyMjvpmPFKIjnPnICCEqUoiKiKjZNjY6otZ83fXV8/Udr77jRwgggvsLQdzf38xTM++znc+21F1X5xjUmj/yGPUe32cfjjimx+M7e/vE6HOcpsRZ6zqbGGfd19HL8Rt8vaQl/wyPEMxEsBM43me+r7vMtHFVVUrh4QoKS8spLK3gYIn7u7ScA8XlHCguY19xWc0H/je/yzhcURXQ2CIEYqMj6RAVUfO7Q1QkHaIj6BAVQVx0JF3iot356vU+20Z/s8x3vw5RkURHRrgfttUf5t7mfT/4A/0f2pijEcxEsAy4QUQWA2OAAqsfaHuKDleQmXOANV85P2tzDpBXdLjRb84i0CUummM6xnBMfAx9usSRclwiXTvG0CU+hq4do+kSH0PHmCgiIiBShAj3G6qIECHOdIT77S9CvvkGW3e98035mw/vqEhrEW1MUwUsEYjIi8BEoLuI5AD3AtEAqjoPeAM4G9gCFAM/DlQspnHFZRV8tCWfbblF5BUdJq+ojLyiw+wuKGVrbhHVRcwDenRkQnJ3+h4TT2JsFImx0STERpEY5/xOiI2mS1w0iXHRREbYN19jQkUgWw1N97NegesDdX7TuL0HS/nf53t5a8MePtySV1Nk0yEqgh4JHejeqQNJ3Tty9rDejOzXhdTju9AlPibIURtjAiEkKotNy9i0p5B/rtnJu1/ksmHXQQD6dIlj+uh+nDGkF8P6dqZThygrxzYmzFgiaOdKyyv5YHMei1Zk8+GWPCIjhFEnHMPtZ53MpJN7MujYBPvgNybMWSJopzbtKeQ3r29k5bZ8yiqq6JXYgVlnnsz00f3o2tGKeIwx37BE0A4tTd/Br/6VRceYKC4fcwITBnbn2yd1J9pa1Bhj6mGJoB0pKavk7n9m8ffPchg7oBt/viSVnomxwQ7LGNPGWSJoJzJzDjBraSab9hZy8+RkbpqcbE04jTGeeEoEIhIBjACOA0qALFXdG8jAjDersvcx5z+bWLktn64dY3jmqtFMSO4R7LCMMSGk0UQgIicCvwC+B2wGcoFYYKCIFAN/BRapamD7DTD1emVNDrOWZtK9Uwd+efYgLhndj8TY6GCHZYwJMf7uCB7AGSfgJ6q1OxUQkZ7ApcAVwKLAhGfqU1WlzHt/K79b/gVjB3Tjrz8aZQnAGHPUGk0EjT0d7BYN/amlAzKN+/pACbe/nMmHW/KYMuI4HrloOB2iIoMdljEmhB11ZbGInK6qb7VkMKZh+w6VMe+9rTyzMpsIEX57wTAu+dbx9jCYMabZmtNq6CnARolpBZk5B7h6UTr5RYc5P7UPPz99IMd3jQ92WMaYdsJfZfGyhlYB3Vo+HFPXe5ty+cmz6XTr2IFXb/w2Q4/rHOyQjDHtjL87ggnA5UBRneUCjA5IRKbGnoOl3Lx4Df27d+KZq0bTI6FDsEMyxrRD/hLBx0Cxqr5Xd4WIfBGYkAw4LYNuW7qW0vJKHr90pCUBY0zA+Gs19P1G1p3W8uGYaotWZvPB5jzuPz+FAT06BTscY0w7Zr2QtUEbdx3kt//+nMmDenL5GKuPN8YEliWCNubtDXuY9teVJMZG8/CFw615qDEm4KzTuTaivLKKP/xnE/Pe28rQ4xL5y2Wj6N7J6gWMMYFniaANqKisYsbTn/LRlnymj+7HvVOGEBttTwsbY1qH56IhEbmvsXlz9BauyOajLfk8cH4Kv71gmCUBY0yrakodwWo/8+Yo7NhXzJy3NvHdQT25zCqGjTFB4DkRqOqrjc2bpispq+Qnz64mMkL49dShVjFsjAkKf11MzAW0ofWqelOLRxRGHnh9Axt3H+SpK9Os7yBjTND4qyxOb5UowtDq7ft4/pOvuGp8f747qFewwzHGhDF/TxbXGnBGROJVtTiwIbV/ew6W8vMlazmucyy3njEw2OEYY8KcpzoCERkrIhuAz935ESLyREAja6fyiw5z2d8+Ib/oMI9fdgodO1gLXmNMcHmtLP4TcCaQD6CqawHra6iJqqqUny3JYMe+YhbM+BYj+x0T7JCMMaZJrYZ21FlU2cKxtHtPffglH2zO494pQxkzwIZzMMa0DV7LJXaIyDhARSQauBnYGLiw2p9tuUU88p8v+N7gXkwffXywwzHGmBpe7wh+ClwP9AG+BlLdeePRXa9k0SEqggd/kGLPCxhj2hRPiUBV81T1MlXtpao9VPVyVc33t5+InCUiX4jIFhG5o571/UTkHRFZIyKZInL20VxEW7dx10FWbsvnxu8m0zMxNtjhGGNMLV5bDQ0QkVdFJFdE9orIv0RkgJ99IoHHge8DQ4DpIjKkzmZ3Ay+p6kjgEqBdtkRa/OlXxERGcOGovsEOxRhjjuC1aOgF4CWgN3AcsBR40c8+o4EtqrpNVcuAxcB5dbZRINGd7oxT7NSulJRV8o81O/n+sGM5pmNMsMMxxpgjeE0E8ar6rKpWuD/PAf7KOPoAvi2Nctxlvu4DLheRHOAN4Mb6DiQiM0UkXUTSc3NzPYbcNsz932YKSyu44tQTgh2KMcbUq9FEICJdRaQr8G8RuUNEkkTkBBG5HeeDu7mmAwtVtS9wNvCsiBwRk6rOV9U0VU3r0aNHC5y2dWTtLGDee1u5OK0vaUldgx2OMcbUy1/z0dU4xTfVzVx+4rNOgTsb2Xcn4NtOsq+7zNfVwFkAqrpSRGKB7sBeP3GFhDlvbSIxLpq7zqlbNWKMMW2Hv76G+jfj2KuAZBHpj5MALgEurbPNV8BkYKGIDMYpbgqtsp8GrMrex/8+38ttZwykc1x0sMMxxpgGee7oRkRScFr/1NQNqOozDW2vqhUicgPwJhAJLFDV9SIyG0hX1WXArcCTIvJznDuMGaraYLfXoWJbbhEzn0mnX9d4fjQuKdjhGGNMozwlAhG5F5iIkwjewGkS+iHQYCIAUNU3qFOXoKr3+ExvAMY3KeIQ8MS7WymvVJ69ejSJsXY3YIxp27y2GroQpwhnt6r+GBiB09zT1FFRWcXbG/dw+pBenNCtY7DDMcYYv7wmghJVrQIqRCQRpzLXOsypx6fZ+zhQXM6ZQ22wGWNMaPBaR5AuIl2AJ3FaEhUBKwMVVCj7cHMekRHChOTQaeZqjAlvnhKBql7nTs4TkeVAoqpmBi6s0PXpl/sY1qezDThjjAkZ/gavP6Wxdar6WcuHFLpKyipZm3OAq77dnFa3xhjTuvx9bf1DI+sU+G4LxhLyPvtqP+WVyqn9bdAZY0zo8PdA2aTWCqQ9+OeancTHRDK6v3UnYYwJHZ6HqjSNKywt57XMXUwdcZzVDxhjQoolghayPGs3JeWVXJRmrWqNMaHFEkELeX3dLvoeE8cp/boEOxRjjGkSryOUiYhcLiL3uPP9RGR0YEMLHQXF5Xy4OY9zhve28YiNMSHH6x3BE8BYnPEDAApxhqE0wJod+6moUr4z0B4iM8aEHq+1mmNU9RQRWQOgqvtFxMZddGXmFCACw/pY90vGmNDj9Y6g3B2MXgFEpAdQFbCoQszH2/IZ0L0jCdbTqDEmBHlNBI8CrwA9ReQ3OF1QPxiwqELIlr2FrNiazw9G1h2O2RhjQoPXvoaeF5HVOF1RC3C+qm4MaGQhYtGK7cRERTB9dL9gh2KMMUfF68A0jwKLVdUqiH3sO1TGy6tzmDriOLp16hDscIwx5qh4LRpaDdwtIltF5BERSQtkUKFi4UdfUlpRyU+/MyDYoRhjzFHzlAhUdZGqng18C/gCeFhENgc0sjautLySRSu3c8aQXpzUMyHY4RhjzFFr6pPFJwGDgBOAz1s+nNDx5vrdFJSUc+XYpGCHYowxzeL1yeLfuXcAs4EsIE1VpwQ0sjbutcxdHNc5llMHWJfTxpjQ5vWBsq3AWFXNC2QwoSRjxwEmnNSdiAjrUsIYE9r8jVA2SFU/B1YB/USkVhvJcB2hbM/BUnILDzOsrz1JbIwJff7uCG4BZlL/SGVhO0JZZk4BAMMtERhj2gF/I5TNdCe/r6qlvutEJDZgUbVx63YWECEwpLclAmNM6PPaamiFx2VhYcWWPIYcl0hcTGSwQzHGmGbzV0dwLNAHiBORkTjdSwAkAvEBjq1Nyi86zGdf7eeG7yYHOxRjjGkR/uoIzgRmAH2BOT7LC4FfBiimNu2dL3KpUvje4J7BDsUYY1qEvzqCRcAiEfmhqv69lWJq0975fC89EzqQcpzVDxhj2gd/RUOXq+pzQJKI3FJ3varOqWe3dquisooPNudy5tBj7fkBY0y74a+yuKP7uxOQUM9Po0TkLBH5QkS2iMgdDWxzsYhsEJH1IvJCE2JvdRk7DnCwtIKJJ1uxkDGm/fBXNPRX9/evm3pgd0Szx4HTgRxglYgsU9UNPtskA3cC493hL9v0J+x/NuwhOlL4dnL3YIdijDEtpil9DSWKSLSI/FdEckXkcj+7jQa2qOo2VS0DFgPn1dnm/4DHVXU/gKrubeoFtBZV5Y11uxh/Unc6x9mQlMaY9sPrcwRnqOpB4FwgG6cX0ll+9ukD7PCZz3GX+RoIDBSRj0TkYxE5q74DichMEUkXkfTc3FyPIbesrJ0HydlfwtkpvYNyfmOMCRSviaC6COkcYKmqFrTQ+aOAZGAiMB14UkS61N1IVeerapqqpvXo0aOFTt00y9fvIjJC+N6QXkE5vzHGBIrXRPCaiHwOjAL+KyI9gFI/++wEjveZ7+su85UDLFPVclX9EtiEkxjanOVZuxnTvytdO8YEOxRjjGlRXkcouwMYhzMOQTlwiCPL++taBSSLSH8RiQEuAZbV2eafOHcDiEh3nKKibV6Dby1ZOwvYmnuIs1KODXYoxhjT4rwOXh8NXA6cJiIA7wHzGttHVStE5AbgTSASWKCq60VkNpCuqsvcdWeIyAagEpilqvlHfTUB8szKbOKiIzkvtW4VhzHGhD6vA9P8BYgGnnDnr3CXXdPYTqr6BvBGnWX3+EwrTlfXRzys1lYcrqjkjXW7OXd4b2stZIxpl7wmgm+p6gif+f+JyNpABNTWfLg5j6LDFZw93FoLGWPaJ6+VxZUicmL1jIgMwCnKafeWZ+0mITaK8SfaQ2TGmPbJ6x3BLOAdEdmG0xX1CcCPAxZVG1FRWcXbG/cweVBPYqK85kxjjAktfhOB21S0AOdJ4eouIL5Q1cOBDKwt+DR7H/uLyzlzqLUWMsa0X41+zRWRa4D1wFwgA0hS1cxwSAIA/924l5ioCL5zcnAeYjPGmNbg747gZ8BQVc116wWe58hnAdqtlVvzSTvhGOJjvJagGWNM6PFX8F2mqrkAqroN6BD4kNqGguJyNu4+yKkDugU7FGOMCSh/X3X7isijDc2r6k2BCSv4VmXvQxXG9O8a7FCMMSag/CWCuj2Mrg5UIG3Nmh37iYoQRhzfJdihGGNMQHkZszgsrd1RwKDeCcRGRwY7FGOMCSh/rYaeFJGUBtZ1FJGrROSywIQWPJVVytqcAwzv2yXYoRhjTMD5Kxp6HLhHRIYBWUAuEIvTVXQisACnJVG7snHXQQpLKxidZPUDxpj2z1/RUAZwsYh0AtKA3kAJsFFVvwh8eMGxcqvTAaq1GDLGhANPDeRVtQh4N7ChtB0rt+XTv3tHju0cG+xQjDEm4KwDnTqqqpRVX+7j1AFWLGSMCQ+WCOr4al8xhYcrGGEVxcaYMNGkRCAi8YEKpK3YuOsgAEOOSwxyJMYY0zo8JQIRGecOJ/m5Oz9CRJ7ws1tI2rjrIBECA3slBDsUY4xpFV7vCP4InAnkA6jqWuC0QAUVTJv3FnFCt472IJkxJmx4LhpS1R11FrXLEcq27C3ixB6dgh2GMca0Gq+JYIeIjANURKJF5DZgYwDjCorKKiU7/xAn9uwY7FCMMabVeE0EPwWuB/oAO4FU4LoAxRQ0+YcOU16p9O0SF+xQjDGm1XgdceVkVa3Vp5CIjAc+avmQgie30Bl4rUdC2Ay7YIwxnu8I5npcFtIsERhjwlGjdwQiMhYYB/QQkVt8ViUC7a5Zzd7qRNDJupYwxoQPf0VDMUAndzvfhvUHgQsDFVSw7C4oBaBnot0RGGPCh7/eR98D3hORhaq6vZViCpqvD5TQvVMHe4bAGBNWvFYWF4vI74GhOOMRAKCq3w1IVEGy80AJfbpYsZAxJrx4rSx+Hqd7if7Ar4FsYFWAYgqa7PxD9OtmzxAYY8KL10TQTVWfAspV9T1VvQpoV3cDhysq2bm/hP7dLREYY8KL16Khcvf3LhE5B/gaaFcd9u/YV0yVwgBLBMaYMOP1juABEekM3ArcBvwN+Jm/nUTkLBH5QkS2iMgdjWz3QxFREUnzGE+L25p7CIAkSwTGmDDjdajK19zJAmAS1DxZ3CARiQQeB04HcoBVIrJMVTfU2S4BuBn4pGmht6wte4sAOLGHJQJjTHhp9I5ARCJFZLqI3CYiKe6yc0VkBfCYn2OPBrao6jZVLQMWA+fVs939wMNAadPDbzlbc4s4NjGWhNjoYIZhjDGtzl/R0FPANUA34FEReQ54BPidqo70s28fwLfr6hx3WQ0ROQU4XlVfb+xAIjJTRNJFJD03N9fPaY/O9vxikrq3+wHYjDHmCP6KhtKA4apaJSKxwG7gRFXNb+6JRSQCmAPM8Letqs4H5gOkpaVpc89dn68PlDDuxO6BOLQxxrRp/u4IylS1CkBVS4FtTUgCO4Hjfeb7usuqJQApwLsikg2cCiwLRoVxeWUVew6W2sNkxpiw5O+OYJCIZLrTApzozgugqjq8kX1XAcki0h8nAVwCXFq9UlULgJqv4CLyLnCbqqY3+Sqaac/BUqoU+hxj4xAYY8KPv0Qw+GgPrKoVInID8CZOT6ULVHW9iMwG0lV12dEeu6V9fcCppz7OBqQxxoQhf53ONaujOVV9A3ijzrJ7Gth2YnPO1RxfHygBLBEYY8KT58Hr27Pt+cWIQB9LBMaYMGSJANi8t5Djj4m37qeNMWHJcyIQkTgROTmQwQTL1txDnNSzU7DDMMaYoPCUCERkCpABLHfnU0WkzVT2Ntfeg6Uc29majhpjwpPXO4L7cLqMOACgqhk4YxOEvKoqZX9xGd06xgQ7FGOMCQqviaDcbffvKyBP+La2gpJyqhSOibdEYIwJT17HI1gvIpcCkSKSDNwErAhcWK0n/1AZAN06WSIwxoQnr3cEN+KMV3wYeAGnO+qfBSimVrX3oPMwWc8EqyMwxoQnr3cEg1T1LuCuQAYTDLsKnERglcXGmHDl9Y7gDyKyUUTurx6XoL3Y7d4RHJtoicAYE548JQJVnYQzMlku8FcRWScidwc0slbyZd4heiZ0IC7GHiYzxoQnzw+UqepuVX0U+CnOMwX19hkUarblFjHAhqc0xoQxrw+UDRaR+0RkHTAXp8VQ34BG1kq25R1iQA97qtgYE768VhYvAJYAZ6rq1wGMp1XtO1TGgeJyBnS3OwJjTPjylAhUdWygAwmG7fmHAEjqZonAGBO+Gk0EIvKSql7sFgn5PknsZYSyNm+323S0tw1RaYwJY/7uCG52f58b6ECCofoZguM62zgExpjw1WhlsarucievU9Xtvj/AdYEPL7B2HighNjqCLvHRwQ7FGGOCxmvz0dPrWfb9lgwkGDbtKSS5ZwIiEuxQjDEmaPzVEVyL881/gIhk+qxKAD4KZGCt4YvdhUxI7hHsMIwxJqj81RG8APwb+C1wh8/yQlXdF7CoWsGB4jL2Fh5mYC97hsAYE978JQJV1WwRub7uChHpGsrJYPPeIgAG9koIciTGGBNcXu4IzgVW4zQf9S1MV2BAgOIKuC1uIjjRnio2xoS5RhOBqp7r/m4Xw1L62pZbRExUBH2Osaajxpjw5rWvofEi0tGdvlxE5ohIv8CGFljbcg/Rv1tHIiOsxZAxJrx5bT76F6BYREYAtwJbgWcDFlUryNlfwvFd44MdhjHGBJ3XRFChqgqcBzymqo/jNCENWTsPlNDXioWMMcZz76OFInIncAUwQUQigJB9HPdwRSVFhyvobgPWG2OM5zuCaTgD11+lqrtxxiL4fcCiCrCi0goAOnXwmgeNMab98jpU5W7geaCziJwLlKrqMwGNLICKDruJIDZkb2qMMabFeG01dDHwKXARcDHwiYhc6GG/s0TkCxHZIiJ31LP+FhHZICKZIvJfETmhqRdwNPKKDgNY0ZAxxuC9juAu4FuquhdARHoAbwMvN7SDiEQCj+N0WJcDrBKRZaq6wWezNUCaqha7/Rr9DqcYKqCqu58+trONQ2CMMV7rCCKqk4Ar38O+o4EtqrpNVcuAxTitjmqo6juqWuzOfkwrjYO8c38JAL1tHAJjjPF8R7BcRN4EXnTnpwFv+NmnD7DDZz4HGNPI9lfjdHB3BBGZCcwE6Nev+c+xbc0tonunGDrHWR2BMcZ4HbN4lohcAHzbXTRfVV9pqSBE5HIgDfhOA+efD8wHSEtL0/q2aYrNe4usjyFjjHH5G48gGXgEOBFYB9ymqjs9HnsncLzPfF93Wd1zfA+nDuI7qnrY47GPWnllFRt3HeSyMa1SL22MMW2ev3L+BcBrwA9xeiCd24RjrwKSRaS/iMQAlwDLfDcQkZHAX4GpdeogAmbHvmJKy6sY0juxNU5njDFtnr+ioQRVfdKd/kJEPvN6YFWtEJEbgDeBSGCBqq4XkdlAuqouw3korROw1B0u8itVndrkq2iCkvJKADrF2sNkxhgD/hNBrPutvbqLzjjfeVVtNDGo6hvUqVRW1Xt8pr/X5IibqaTMSQRx0ZGtfWpjjGmT/CWCXcAcn/ndPvMKfDcQQQVSQUk5gLUYMsYYl7+BaSa1ViCtZd+hMgC6xFsiMMYY8P5AWbvx1b5iIiPEHiYzxhhX2CWCbXmH6HtMHDFRYXfpxhhTr7D7NPwy9xD9u3cMdhjGGNNmeO19VNyxiu9x5/uJyOjAhhYYuwpK6NPFioWMMaaa1zuCJ4CxwHR3vhCnZ9GQUlFZxf7icnokdAh2KMYY02Z4fapqjKqeIiJrAFR1v/u0cEipHpAmwQakMcaYGl7vCMrd8QUUasYjqApYVAGSV+Q0HbUBaYwx5hteE8GjwCtATxH5DfAh8GDAogqQb0Yms6IhY4yp5rUb6udFZDUwGad7ifNVdWNAIwuA6kTQze4IjDGmhqdEICL9gGLgVd9lqvpVoAILhK/2OYOh9T0mPsiRGGNM2+G1svh1nPoBAWKB/sAXwNAAxRUQ+UVlxMdE0qmD9TxqjDHVvBYNDfOdF5FTgOsCElEAFZSUk2gthowxppajerLY7X66sfGH26TcwsP0TLSKYmOM8eW1juAWn9kI4BTg64BEFEC5hYc5tnNssMMwxpg2xesdQYLPTwecOoPzAhVUoBwoLuOYeGsxZIwxvvzeEbgPkiWo6m2tEE9AlVUqHaLDrp89Y4xpVKOfiiISpaqVwPhWiiegyioqiYm0RGCMMb783RF8ilMfkCEiy4ClwKHqlar6jwDG1uJKyiuJi7Gxio0xxpfXBvWxQD7OGMXVzxMoEDKJ4HBFJeWVSkdLBMYYU4u/RNDTbTGUxTcJoJoGLKoAKC13+siLjbZE0JrKy8vJycmhtLQ02KEYExZiY2Pp27cv0dHen5nylwgigU7UTgDVQioRlFU4iaCDJYJWlZOTQ0JCAklJSYjU92dkjGkpqkp+fj45OTn079/f837+EsEuVZ3dvNDahtLySgDiLBG0qtLSUksCxrQSEaFbt27k5uY2aT9/TWjazf/egpJyAOtnKAgsCRjTeo7m/5u/RDD56EJpe/YXO4PSdO1oD5QZY4yvRhOBqu5rrUACrbzSrSOIsucIwk1kZCSpqamkpKQwZcoUDhw40CLHXbhwITfccEOLHCspKYlhw4aRmppKamoqK1asaJHj1pWRkcEbb7xRa9m///1v0tLSGDJkCCNHjuTWW28F4L777uORRx5psXOPGzeuZnrWrFkMHTqUWbNmMW/ePJ555plmHXvNmjVcffXVtZadf/75nHrqqbWWzZgxg5dffrnWsk6dOtVMb9q0ibPPPpvk5GROOeUULr74Yvbs2dOs2Pbt28fpp59OcnIyp59+Ovv37693u1/84hekpKSQkpLCkiVLjlh/00031Yr1scceY8GCBc2KrVrYfCqWVTh121GRVkwRbuLi4sjIyCArK4uuXbvy+OOPBzuker3zzjtkZGSQkZFR60OzMRUVFU06R91EkJWVxQ033MBzzz3Hhg0bSE9P56STTmrSMb3yTW7z588nMzOT3//+9/z0pz/lRz/6kefj1HfNDz74IDfddFPN/IEDB1i9ejUFBQVs27bN03FLS0s555xzuPbaa9m8eTOfffYZ1113XZPL2+t66KGHmDx5Mps3b2by5Mk89NBDR2zz+uuv89lnn5GRkcEnn3zCI488wsGDB2vWp6enH5FArrrqKubOndus2KqFXYG5tJ9qj5Dz61fXs+Hrg/43bIIhxyVy7xTvw2KMHTuWzMxMAD799FNuvvlmSktLiYuL4+mnn+bkk09m4cKFLFu2jOLiYrZu3coPfvADfve73wHw9NNP89vf/pYuXbowYsQIOnRwerPNzs7mqquuIi8vjx49evD000/Tr18/ZsyYQVxcHGvWrGHv3r0sWLCAZ555hpUrVzJmzBgWLlzYYKyNHTM2NpY1a9Ywfvx4rr/+eq6//npyc3OJj4/nySefZNCgQSxdupRf//rXREZG0rlzZ95++23uueceSkpK+PDDD7nzzjt5/fXXueuuuxg0aBDg3D1de+21R8Ty5JNPMn/+fMrKyjjppJN49tlniY+PP+Ic77//PuvXr+fHP/4xZWVlVFVV8fe//53k5GQ6depEUVERU6dOpaioiFGjRnHnnXeyceNGOnXqxG233cbWrVvrvZa61zxnzpya2AoLC8nMzGTEiBE1y/7xj38wZcoUevXqxeLFi/nlL3/p92/jhRdeYOzYsUyZMqVm2cSJE/3u58+//vUv3n33XQCuvPJKJk6cyMMPP1xrmw0bNnDaaacRFRVFVFQUw4cPZ/ny5Vx88cVUVlYya9YsXnjhBV555ZWafeLj40lKSuLTTz9l9OjRzYoxbO4IDlc4rYZioiwRhKvKykr++9//MnXqVAAGDRrEBx98wJo1a5g9e3atD4uMjAyWLFnCunXrWLJkCTt27GDXrl3ce++9fPTRR3z44Yds2LChZvsbb7yRK6+8kszMTC677LJa307379/PypUr+eMf/8jUqVP5+c9/zvr161m3bh0ZGRk1202aNInU1FTGjBnj95g5OTmsWLGCOXPmMHPmTObOncvq1at55JFHuO46Z6iQ2bNn8+abb7J27VqWLVtGTEwMs2fPZtq0aWRkZDBt2jSysrIYNWqU39fuggsuYNWqVaxdu5bBgwfz1FNP1XsOgHnz5nHzzTeTkZFBeno6ffv2rXWsZcuW1dylTZs2rda6hq6l7jX7Sk9PJyUlpdayF198kenTpzN9+nRefPFFv9cHeH4tCgsLa4rw6v74/k1U27NnD7179wbg2GOPrbeoacSIESxfvpzi4mLy8vJ455132LFjB+AUAU2dOrXmGL7S0tL44IMPPF1fY8LmjuCg22ooMc4GpgmWpnxzb0klJSWkpqayc+dOBg8ezOmnnw5AQUEBV155JZs3b0ZEKC8vr9ln8uTJdO7cGYAhQ4awfft28vLymDhxIj169ABg2rRpbNq0CYCVK1fyj384D9pfccUV3H777TXHmjJlCiLCsGHD6NWrF8OGOeM8DR06lOzsbFJTUwGnaKh79+41+zV2zIsuuojIyEiKiopYsWIFF110Uc26w4edsbnHjx/PjBkzuPjii7ngggua9RpmZWVx9913c+DAAYqKijjzzDMbPMfYsWP5zW9+Q05ODhdccAHJycmeztHYtfhec127du2qeU/A+eDdvHkz3/72txERoqOjycrKIiUlpd4WNU1tZZOQkFArgTeFiNR7vjPOOINVq1Yxbtw4evTowdixY4mMjOTrr79m6dKlNXcUdfXs2ZPPP//8qGLxFdA7AhE5S0S+EJEtInJHPes7iMgSd/0nIpIUqFjsyeLwVf3tc/v27ahqTR3Br371KyZNmkRWVhavvvpqraefq4t8wCkuaWpZvK/qY0VERNQ6bkRExFEft2PHjgBUVVXRpUuXmrqFjIwMNm7cCDjfzB944AF27NjBqFGjyM/PP+I4Q4cOZfXq1X7PN2PGDB577DHWrVvHvffeW/Na1XeOSy+9tOZb/9lnn83//vc/T9fU2LX4XnNdcXFxtd67l156if3799O/f3+SkpLIzs6uuSvo1q1brbL2ffv21SRfr69FU+8IevXqxa5duwAnafXs2bPe4951111kZGTw1ltvoaoMHDiQNWvWsGXLFk466SSSkpIoLi6uVYdTXazZXAFLBG731Y8D3weGANNFZEidza4G9qvqScAfgYcJkCp1K4sjrGgoXMXHx/Poo4/yhz/8gYqKCgoKCujTpw9Ao2X11caMGcN7771Hfn4+5eXlLF26tGbduHHjWLx4MQDPP/88EyZMaHa8Xo6ZmJhI//79a2JRVdauXQvA1q1bGTNmDLNnz6ZHjx7s2LGDhIQECgsLa/afNWsWDz74YM2dTVVVFfPmzTviPIWFhfTu3Zvy8nKef/75muX1nWPbtm0MGDCAm266ifPOO6+mTsafxq6lMYMHD2bLli018y+++CLLly8nOzub7OxsVq9eXfM6Tpw4kSVLllBW5jQnX7hwIZMmTQLg0ksvZcWKFbz++us1x3r//ffJysqqdb7qO4L6foYMqfsRB1OnTmXRokUALFq0iPPOO3Iol8rKyppEnZmZSWZmJmeccQbnnHMOu3fvrrmW+Pj4Wte6adOmI4rFjkYg7whGA1tUdZuqlgGLOXIwm/OARe70y8BksaePTACNHDmS4cOH8+KLL3L77bdz5513MnLkSE/fzHv37s19993H2LFjGT9+PIMHD65ZN3fuXJ5++mmGDx/Os88+y5///Odmx+r1mM8//zxPPfUUI0aMYOjQofzrX/8CnA/5YcOGkZKSwrhx4xgxYgSTJk1iw4YNpKamsmTJEoYPH86f/vQnpk+fzuDBg0lJSam3lc3999/PmDFjGD9+fE3FckPneOmll0hJSSE1NZWsrKwmtQhq6FoaM2jQIAoKCigsLCQ7O5vt27fXajbav39/OnfuzCeffMK5557LhAkTGDVqFKmpqXz00Uc1FbdxcXG89tprzJ07l+TkZIYMGcITTzxRq9jpaNxxxx289dZbJCcn8/bbb3PHHU7hSHp6Otdccw3g9Mk1YcIEhgwZwsyZM3nuueeIivJfcv/RRx/VFHU2h6gGpssgEbkQOEtVr3HnrwDGqOoNPttkudvkuPNb3W3y6hxrJjAToF+/fqO2b9/e5Hj+s343/8zYyZyLU614qBVt3Lix1gemMYHwxz/+kYSEhJoP1nCwZs0a5syZw7PPPnvEuvr+34nIalVNq+9YIdFqSFXnq2qaqqYdbXY+Y+ixPHHZKEsCxrRD1157ba36l3CQl5fH/fff3yLHCmSroZ3A8T7zfd1l9W2TIyJRQGeccQ+MMcaz2NhYrrjiimCH0apaokioWiDvCFYBySLSX0RigEuAZXW2WQZc6U5fCPxPA1VWZYLG3lJjWs/R/H8LWCJQ1QrgBuBNYCPwkqquF5HZIjLV3ewpoJuIbAFuAY5oYmpCW2xsLPn5+ZYMjGkF1eMRxMbGNmm/gFUWB0paWpqmp6cHOwzjkY1QZkzramiEssYqi8PmyWITHNHR0U0aKckY0/pCotWQMcaYwLFEYIwxYc4SgTHGhLmQqywWkVyg6Y8WO7oDeX63al/smsODXXN4aM41n6Cq9T6RG3KJoDlEJL2hWvP2yq45PNg1h4dAXbMVDRljTJizRGCMMWEu3BLB/GAHEAR2zeHBrjk8BOSaw6qOwBhjzJHC7Y7AGGNMHZYIjDEmzLXLRCAiZ4nIFyKyRUSO6NFURDqIyBJ3/ScikhSEMFuUh2u+RUQ2iEimiPxXRE4IRpwtyd81+2z3QxFREQn5poZerllELnbf6/Ui8kJrx9jSPPxt9xORd0Rkjfv3fXYw4mwpIrJARPa6IzjWt15E5FH39cgUkVOafVJVbVc/QCSwFRgAxABrgSF1trkOmOdOXwIsCXbcrXDNk4B4d/racLhmd7sE4H3gYyAt2HG3wvucDKwBjnHnewY77la45vnAte70ECA72HE385pPA04BshpYfzbwb0CAU4FPmnvO9nhHMBrYoqrbVLUMWAycV2eb84BF7vTLwGQRkVaMsaX5vWZVfUdVi93Zj3FGjAtlXt5ngPuBh4H20A+2l2v+P+BxVd0PoKp7WznGlublmhVIdKc7A1+3YnwtTlXfB/Y1ssl5wDPq+BjoIiK9m3PO9pgI+gA7fOZz3GX1bqPOADoFQLdWiS4wvFyzr6txvlGEMr/X7N4yH6+qr7dmYAHk5X0eCAwUkY9E5GMROavVogsML9d8H3C5iOQAbwA3tk5oQdPU/+9+2XgEYUZELgfSgO8EO5ZAEpEIYA4wI8ihtLYonOKhiTh3fe+LyDBVPRDMoAJsOrBQVf8gImOBZ0UkRVWrgh1YqGiPdwQ7geN95vu6y+rdRkSicG4n81slusDwcs2IyPeAu4Cpqnq4lWILFH/XnACkAO+KSDZOWeqyEK8w9vI+5wDLVLVcVb8ENuEkhlDl5ZqvBl4CUNWVQCxO52ztlaf/703RHhPBKiBZRPqLSAxOZfCyOtssA650py8E/qduLUyI8nvNIjIS+CtOEgj1cmPwc82qWqCq3VU1SVWTcOpFpqpqKI9z6uVv+584dwOISHecoqJtrRhjS/NyzV8BkwFEZDBOIsht1Shb1zLgR27roVOBAlXd1ZwDtruiIVWtEJEbgDdxWhwsUNX1IjIbSFfVZcBTOLePW3AqZS4JXsTN5/Gafw90Apa69eJfqerUoAXdTB6vuV3xeM1vAmeIyAagEpilqiF7t+vxmm8FnhSRn+NUHM8I5S92IvIiTjLv7tZ73AtEA6jqPJx6kLOBLUAx8ONmnzOEXy9jjDEtoD0WDRljjGkCSwTGGBPmLBEYY0yYs0RgjDFhzhKBMcaEOUsEYUBEKkUkw+cnqZFti1rgfAtF5Ev3XJ+5T3s29Rh/E5Eh7vQv66xb0dwY3eNUvy5ZIvKqiHTxs33q0fRsKSK9ReQ1d3qiiBS4590oIvcexfGmVvfCKSLnV79O7vxs98HBZnHfwwv9bPNuUx7Qc6/9NQ/b1dv7pog8IiLf9Xo+450lgvBQoqqpPj/ZrXDOWaqaCtyB8yBbk6jqNaq6wZ39ZZ1145ofHvDN65KC8zzJ9X62T8Vpv91UtwBP+sx/4L42aTh95DSpG2FVXaaqD7mz5+P0uFm97h5VffsoYmxLFgL19ZE0F+fvybQwSwRhSEQ6iTMmwWcisk5Ejui10/0W+77PN+YJ7vIzRGSlu+9SEenk53TvAye5+97iHitLRH7mLusoIq+LyFp3+TR3+bsikiYiDwFxbhzPu+uK3N+LReQcn5gXisiFIhIpIr8XkVXi9Nf+Ew8vy0rcjrtEZLR7jWtEZIWInOw+1TobmObGMs2NfYGIfOpuW1/vpwA/BJbXXaiqh4DVwEnu3cbHbryviMgxbiw3yTfjSCx2l80QkcdEZBwwFfi9G9OJPq/BWSKy1Oe1qfk23tT3UETucV/LLBGZL1Krp94rfP5GRrvbe31d6tVQ75uquh3oJiLHNuV4xoNg9LdtP637g/OEaYb78wrOE+WJ7rruOE8oVj9cWOT+vhW4y52OxOm7pzvOB3tHd/kvgHvqOd9C4EJ3+iLgE2AUsA7oiPOE83pgJM6H5JM++3Z2f7+LO35AdUw+21TH+ANgkTsdg9MjYxwwE7jbXd4BSAf61xNnkc/1LQXOcucTgSh3+nvA393pGcBjPvs/CFzuTnfB6denY51z9AdW+8xPBF5zp7sB2cBQIBP4jrt8NvAnd/proEP1OerG4fta+8677/FXPu/VX4DLj/I97Oqz/Flgis979KQ7fRpu//kNvS51rj0N+Fsjf7NJ1NMfP86d1Q+D/X+qvf20uy4mTL1K1CmKAEBEooEHReQ0oArnm3AvYLfPPquABe62/1TVDBH5Dk4xxEful8IYnG/S9fm9iNyN0+fL1Th9wbyizrdgROQfwAScb8p/EJGHcT4kPmjCdf0b+LOIdMApSnhfVUtE5AxguE8Zd2ecjte+rLN/nIhkuNe/EXjLZ/tFIpKM02VBdAPnPwOYKiK3ufOxQD/3WNV6c2S/NxNEZA3Oa/8QTkdxXVT1PXf9IpzEBE6CeF5E/onTj5An6nTNsByYIiIvA+cAt+P0Ouv1Paw2SURuB+KBrjhJ/FV33Yvu+d4XkURx6lkael1840sHrvF6PT72AscdxX6mEZYIwtNlQA9glKqWi9M7Z6zvBu5/7NNwPkAWisgcYD/wlqpO93COWar6cvWMiEyubyNV3eSWkZ8NPCAi/1XV2V4uQlVLReRd4ExgGs6gJeCM3HSjqr7p5xAlqpoqIvE4fdlcDzyKM5jNO6r6A3Eq1t9tYH/B+Xb6RWPnoM5ri1NHcG7NQUQ6N7L/OTjftqcAd4nIsEa2rWsxcANOMUu6qha6xTpe30NEJBZ4AufubIeI3Eft66nbR43SwOsiIr2aEHtDYnFeU9OCrI4gPHUG9rpJYBJwxPjF4oxpvEdVnwT+hjN03sfAeBGpLvPvKCIDPZ7zA+B8EYkXkY44xTofiMhxQLGqPofTMV59Fafl7p1JfZbgdLpVfXcBzof6tdX7iMhA95z1UmfktpuAW+Wbbsmru/Wd4bNpIU4RWbU3gRury8zF6eG1rk04xRwNUtUCYL+49TDAFcB74oypcLyqvoNThNMZp1jNV92YfL2H83r+H98kyaa+h9Uf+nluXULdlkTVdTrfxukFswBvr8vRGgjUO5avOXqWCMLT80CaiKwDfgR8Xs82E4G1bhHGNODPqpqL88H4oohk4hQpDPJyQlX9DKfc+VOcOoO/qeoaYBjwqVtEcy/wQD27zwcyxa0sruM/OMUdb6szlCE4iWsD8Jk4TRD/ip+7XzeWTJxBTn4H/Na9dt/93gGGVFcW49w5RLuxrXfn6x73ELC1+oO3EVfiFKdl4rROmo1Td/Gc+z6tAR7VIweYWQzMcitlT6xz7krgNeD77m+a+h6653sS58P3TZwiQ1+l7us0D6cIEDy8LuI0BPhbfecUp/fNlcDJIpIjIle7y6NxGh6EclfibZL1PmpMgInID3CK4e4OdiyhzH0dT1HVXwU7lvbG6giMCTBVfUVEQnlM7LYiCvhDsINoj+yOwBhjwpzVERhjTJizRGCMMWHOEoExxoQ5SwTGGBPmLBEYY0yY+3+M+HxG6OJfLAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "create_roc_auc_plot(model, X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "4ce51f26",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T03:25:20.718735Z",
     "start_time": "2023-02-07T03:25:20.326397Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\furqan ali\\desktop\\python\\env\\lib\\site-packages\\sklearn\\utils\\deprecation.py:87: FutureWarning: Function plot_confusion_matrix is deprecated; Function `plot_confusion_matrix` is deprecated in 1.0 and will be removed in 1.2. Use one of the class methods: ConfusionMatrixDisplay.from_predictions or ConfusionMatrixDisplay.from_estimator.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUUAAAEGCAYAAADyuIefAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAdKklEQVR4nO3deZgV1bnv8e+vmedRCAIKKuohGIcQQM01RjyCJjeYPMYh8chjzDU4Z7pGT/KEqDHTzYlDjHqIkqgxGs1w1GhAJRo1URQ1joggDoCANo3MU3e/949a3b0h0L1307u7d/fv8zz1sGvVqqp3d8PLWrWqVikiMDOzTFlLB2Bm1po4KZqZ5XBSNDPL4aRoZpbDSdHMLEfHlg4g18D+HWLE8E4tHYYV4PUXu7d0CFaAzWxga2zR7hxj0id7xKqKqrzqPvviltkRMXl3ztfcWlVSHDG8E0/PHt7SYVgBJu15SEuHYAWYG3N2+xjlFVXMnT0sr7qdhrwxcLdP2MxaVVI0s1IQVEV1SwdRNE6KZlaQAKppuw99OCmaWcGqcUvRzAyAINjm7rOZWSaAKnefzczq+JqimVkSQFUbnl3LT7SYWcGq81waImmmpPckvZxT1l/SQ5IWpj/7pXJJulbSIkkvSjosZ5+pqf5CSVNzyj8q6aW0z7WSGrxx3UnRzAoSBFV5Lnn4NbDjEy+XAHMiYhQwJ60DHA+MSsvZwA2QJVFgOjAeGAdMr0mkqc7/ydmvwadrnBTNrCARsC3PpeFjxWNAxQ7FU4Bb0udbgBNzym+NzFNAX0lDgEnAQxFRERGrgYeAyWlb74h4KrLZtG/NOdYu+ZqimRVIVJH349MDJc3LWZ8RETMa2GdwRCxPn1cAg9PnocCSnHpLU1l95Ut3Ul4vJ0UzK0gA1fmPs5RHxNhGnysiJDXrqI67z2ZWsKrUWmxoaaSVqetL+vO9VL4MyJ0xZlgqq6982E7K6+WkaGYFyW7eLmpSvBeoGUGeCtyTU35GGoWeAKxJ3ezZwHGS+qUBluOA2WnbWkkT0qjzGTnH2iV3n82sIAFsi6ZpT0m6Azia7NrjUrJR5B8Bd0k6C3gbODlVfwA4AVgEbATOBIiICklXAM+kepdHRM3gzblkI9zdgL+kpV5OimZWkEBUNVEnMyJO28WmiTupG8B5uzjOTGDmTsrnAWMKiclJ0cwKVh27NXl3q+akaGYFqbmm2FY5KZpZgURVE11TbI2cFM2sINnM206KZmYARIit0aGlwygaJ0UzK1i1rymamWWygRZ3n83MEg+0mJnV8kCLmdkOqnzztplZJhDbou2mjrb7zcysKDzQYmaWI5C7z2ZmuTzQYmaWROBbcszMamQDLX7Mz8yslgdazMySQJ5k1swsl1uKZmZJ9t5nJ0Uzs2S3Xl/a6jkpmllBslecevTZzAzIZt5299nMLIdv3jYzS7L5FH1N0cws8czbZma1slty3FI0MwP87LOZ2b/w1GFmZkk2dZi7z2ZmtXxN0cwsyWbJcffZzAyoeczPSbHd+6+vDWfuw73pO7CSGY8sAGDt6g78YNoIVi7tzOBhW/n2f79Fr75VvPCPnnzvzJF8aPhWAI484QNO//pKAM4YN5puPasoK4MOHYPrZr0OwC8v35OnHupNp87BkL238I2rltCzT1XLfNl2oqws+Pms11m1vBPfnboPnzmznM9++X32HLmVz4/5MGsrsn8eh09awxn/d0V2La1S3Dh9T155umcLR9+S2nZLsajfTNJkSQskLZJ0STHPVWzHnVLBlbcv3q7srusGcejH1/Grv8/n0I+v43fXDardNmb8em54eAE3PLygNiHW+Mndi7jh4QW1CRHgsKPWMeOR17hxzgKG7rOFO38+CCuuE79czpKFXWvXX3mmO5ecsi8rlnTart7zj/fknGP359x/P4CffX04X/vp0uYOtdWpRnktDZH0NUmvSHpZ0h2SukoaKWluyhu/k9Q51e2S1hel7SNyjnNpKl8gadLufLeiJUVJHYBfAMcDo4HTJI0u1vmK7aAJG+jVb/uW25Oz+3DsyRUAHHtyBU/O6tPo43/06HV0SO32f/voRsqXd6p/B9stA4dsZdzEtfzlt/1ry954uTsrl3b+l7qbN3aA9A+8a/dqIporytapZvQ5n6U+koYCFwJjI2IM0AE4FfgxcFVE7AesBs5Ku5wFrE7lV6V6pLxyKvBhYDJwfco/jVLMluI4YFFELI6IrcCdwJQinq/ZrS7vxIDBlQD0H1TJ6vK6RDb/2R5MO/YAvv3FfXhrQV1rBAX/edq+nDdpfx74zYCdHnf2Hf352DHrihp7ezftsne56ftDiOr8RlGPmLyGmx57jStufZOffX14kaNr/aqjLK8lDx2BbpI6At2B5cAxwO/T9luAE9PnKWmdtH2iJKXyOyNiS0S8CSwiyz+NUsxrikOBJTnrS4HxO1aSdDZwNsBeQ0v3EqcEUtaE2O+gjdz29Kt061HN03N6cdmXRvKrv88H4Gf/s4iBQ7bxQXlHLjl1X4bvt5mDJmyoPc5vrxlMh47BMZ9b3SLfoz0Yf+xaPijvyKKXuvORw9fntc8/ZvXhH7P6MGb8eqZevIJLTtm3yFG2XgW+o2WgpHk56zMiYgZARCyT9FPgHWAT8CDwLPBBRFSm+kvJcgnk5JSIqJS0BhiQyp/KOUfuPgVr8aulETEjIsZGxNg9BpTWo0P9Bm5j1coska9a2ZG+A7LfY49e1XTrUQ3AuInrqNom1qzKvtvAIdsA6DuwkiMnr+G157vXHu/B3/Xn6Yd7863r3kZt9zawFjf6YxuYcNxabpn7Kpfe8DYHf3w9F//87bz2fXluTz6011Z6969suHIbFUBllOW1AOU1/77TMqPmOJL6kbXyRgJ7Aj3Iur8tqphJcRmQ288YlsrajAnHreXhu7JrUg/f1Z/DJ60BoOK9jrXXnV57vjvV1dC7fxWbN5axcX32I9+8sYxn/9aLEQduBuCZR3px9/WD+N6vF9O1ezu/aFVkv/rhEE4fO5qp40fzw3P25oUnevKTC/beZf09R2whSwVZL6BT52rWVpTWf+BNrYm6z8cCb0bE+xGxDfgjcCTQN3WnYfu8UZtT0vY+wCqaONcUs7/6DDBK0kiyAE8FvlDE8xXVD8/Zmxef7Mmaio588aOj+Y9vrOCU81dy5bQRzLpzAIOGZrfkADz+5778+dYBdOgIXbpWc+kNbyHB6vc7ctlZIwGoqoRPfvYDPvbJ7NrhL749jG1bxKWn7AfAgR/dwEU/9ihnc5py1vt8/pz36T9oGzc+vICn/9qbq785nI9/ag3HnlRBZaXYsqmMH5yzN7Th+QQbFE32itN3gAmSupN1nycC84BHgJPIxiGmAvek+vem9SfT9r9GREi6F/itpJ+RtThHAU83NihFEYfSJJ0AXE02qjQzIq6sr/7Yg7vG07N9EbuUTNrzkJYOwQowN+awNip2K6P1O3BQHDPzpLzq/vHIG56NiLG72i7pMuAUoBJ4Hvgy2fXAO4H+qez0iNgiqStwG3AoUAGcGhGL03G+DXwpHeerEfGXRn694t68HREPAA8U8xxm1vya6tnniJgOTN+heDE7GT2OiM3A53dxnCuBehtd+Srd4V4zaxGeZNbMLEcgKqtb/MaVonFSNLOC+cVVZmY1wt1nM7NavqZoZrYDJ0UzsyQQVR5oMTOr44EWM7MkPNBiZra9cFI0M6vRZBNCtEpOimZWMLcUzcySCKjK8zUOpchJ0cwK5tFnM7MkcPfZzCyHB1rMzLbTlt997aRoZgVz99nMLMlGn/3ss5lZLXefzcxyuPtsZpYEclI0M8vVhnvPTopmVqCA8GN+ZmZ13H02M8vRLkefJf2cei4dRMSFRYnIzFq19vzs87xmi8LMSkcA7TEpRsQtueuSukfExuKHZGatXVvuPjf4rI6kwyW9CryW1g+WdH3RIzOzVkpEdX5LKcrnAcargUnAKoCIeAE4qogxmVlrF3kuJSiv0eeIWCJtl/WrihOOmbV60X4HWmoskXQEEJI6ARcB84sblpm1aiXaCsxHPt3nacB5wFDgXeCQtG5m7ZbyXEpPg0kxIsoj4osRMTgi9oiI0yNiVXMEZ2atVHWeSwMk9ZX0e0mvSZqfBnb7S3pI0sL0Z79UV5KulbRI0ouSDss5ztRUf6Gkqbvz1fIZfd5H0n2S3pf0nqR7JO2zOyc1sxJWc59iPkvDrgFmRcSBwMFkl+YuAeZExChgTloHOB4YlZazgRsAJPUHpgPjgXHA9JpE2hj5dJ9/C9wFDAH2BO4G7mjsCc2s9EXkt9RHUh+yO1luzo4ZWyPiA2AKUHOf9C3AienzFODWyDwF9JU0hOzumIcioiIiVgMPAZMb+93ySYrdI+K2iKhMy2+Aro09oZm1AfnfkjNQ0ryc5eyco4wE3gd+Jel5STdJ6gEMjojlqc4KYHD6PBRYkrP/0lS2q/JGqe/Z5/7p418kXQLcSfY1TwEeaOwJzawNyP+WnPKIGLuLbR2Bw4ALImKupGuo6ypnp4kISc061l3fLTnPkiXBmm//lZxtAVxarKDMrHVrojS1FFgaEXPT+u/JkuJKSUMiYnnqHr+Xti8DhufsPyyVLQOO3qH80cYGtcvuc0SMjIh90p87Lh5oMWuvQlCd51LfYSJWkN0HfUAqmgi8CtwL1IwgTwXuSZ/vBc5Io9ATgDWpmz0bOE5SvzTAclwqa5S8nmiRNAYYTc61xIi4tbEnNbMS13Qd2guA2yV1BhYDZ5I11u6SdBbwNnByqvsAcAKwCNiY6hIRFZKuAJ5J9S6PiIrGBtRgUpQ0naxpOjoFdTzwBOCkaNZeNVFSjIh/Aju75jhxJ3WDXTw4EhEzgZlNEVM+o88nkQW4IiLOJLuXqE9TnNzMSlQ7nxBiU0RUS6qU1JvsoufwhnYyszaqvU4ym2OepL7AL8lGpNcDTxYzKDNr3Zr3Jpnm1WBSjIhz08cbJc0CekfEi8UNy8xatfaYFHMftt7Ztoh4rjghmVlr115biv9Vz7YAjmniWFj4Ug+O32dCUx/Wiqisez5jddZaaFMT/b7a4zXFiPhkcwZiZiWihEeW85HXzdtmZttxUjQzq6M8JpAtVU6KZla4NtxSzGfmbUk6XdJ30/peksYVPzQza40U+S+lKJ+hqOuBw4HT0vo64BdFi8jMWr+mex1Bq5NP93l8RBwm6XmAiFidZrQws/aqRFuB+cgnKW6T1IH0Y5C0B3m9p8vM2qpS7RrnI5+keC3wJ2CQpCvJZs35TlGjMrPWK9r56HNE3C7pWbLpwwScGBHzix6ZmbVe7bmlKGkvsllu78sti4h3ihmYmbVi7TkpAvdT9wKrrmSvJVwAfLiIcZlZK9aurylGxEG562n2nHN3Ud3MrKQV/ERLRDwnaXwxgjGzEtGeW4qSvp6zWkb28up3ixaRmbVu7X30GeiV87mS7BrjH4oTjpmVhPbaUkw3bfeKiG82Uzxm1sqJdjrQIqljRFRKOrI5AzKzEtAekyLwNNn1w39Kuhe4G9hQszEi/ljk2MysNSrhGXDykc81xa7AKrJ3stTcrxiAk6JZe9VOB1oGpZHnl6lLhjXa8P8TZtaQ9tpS7AD0ZPtkWKMN/0jMrEFtOAPUlxSXR8TlzRaJmZWGdvw2v9KcNtfMiq69dp8nNlsUZlZa2mNSjIiK5gzEzEpHe3/Mz8ysTju+pmhm9i9E2x5wyOcVp2Zm24s8lzxI6iDpeUl/TusjJc2VtEjS72reHiqpS1pflLaPyDnGpal8gaRJu/PVnBTNrGA7e/H9zpY8XQTkvvfpx8BVEbEfsBo4K5WfBaxO5VelekgaDZxK9jaAycD1aTKbRnFSNLPCNVFLUdIw4FPATWldZI8U/z5VuQU4MX2ektZJ2yem+lOAOyNiS0S8CSwCxjX2qzkpmllh0iSz+SzAQEnzcpazdzja1cDF1D1NPQD4ICIq0/pSYGj6PBRYApC2r0n1a8t3sk/BPNBiZoXLv2tcHhFjd7ZB0qeB9yLiWUlHN01gu89J0cwK1kRPtBwJfEbSCWSzcfUGrgH61sznCgwDlqX6y4DhwFJJHYE+ZDN41ZTXyN2nYO4+m1nhmuCaYkRcGhHDImIE2UDJXyPii8AjwEmp2lTgnvT53rRO2v7XiIhUfmoanR4JjCKbD7ZR3FI0s4IV+dnnbwF3Svo+8Dxwcyq/GbhN0iKggiyREhGvSLoLeJXsPVLnRURVY0/upGhmhQmafJLZiHgUeDR9XsxORo8jYjPw+V3sfyVwZVPE4qRoZgVpty+uMjPbJSdFM7M6irabFZ0UzawwniXHzGx7vqZoZpbDk8yameVyS9HMLClsWrCS46RoZoVzUjQzy/jmbTOzHai67WZFJ0UzK4zvU7SG9OhVyVd/tJi9999EBFz1rX048cwVDNtnMwA9e1eyfm1Hzv/0Qez/kfVc+IM3AZDg9muG8o8H+7dk+O3O0JGbuPSa12vXh+y1hduuHk75ys6cfuEShu+7ia9+7iAWvtwTgI6dqrngisWMOmg9US1u/P4IXprbp6XCbxV8S04jSJoJ1MysO6ZY52kNpn33beb9rS9Xnrc/HTtV06VrNT+6cFTt9i//59tsXJe9R+ft17tx4ZQxVFeJfnts5fr7X+KpOf2ormrLL41sXZa92Y3zP3MwAGVlwW1/f5Z/PNifLt2quOLcA7jw+4u3qz/5lPcAOPdTh9Cn/zaumDmfiz57EBHt+HfWhluKxZxk9tdkb9Zq07r3qmTMuHXMvmsPACq3lbFhXe7/NcFRJ1Tw6H0DAdiyuUNtAuzcpbot/90qCYccsYbl73TlvXe7sOSN7ix7s9u/1Nlrv4288FTWMlxT0YkNazsw6qD1zR1qq9LEb/NrVYqWFCPiMbKJINu0Dw3bwpqKjnz9J4u57r6XuOiHi+nSrW5+yzEfW8fqVZ14962utWUHHLyeG2e9yA1/eYnrvjPSrcQW9IlPlfO3Pw+ot86b83swYWIFZR2CwcM2s9+YDewxZGszRdgKBRCR31KCWvx1BJLOrnnT11a2tHQ4BevQMdjvwxu4//bBnP+/D2LzxjJOnvZu7fajP7OKv927/T+6BS/0ZNrkj3DRiWM4+Zx36dS5DV+gacU6dqpm/MTVPP5A/Ulx9u8HUb6iC9f+6UW+8p23mP9cr3b/H1kBb/MrOS2eFCNiRkSMjYixnenS0uEUrHx5Z8pXdGbBC9lF+Sdm9We/MRsBKOsQHDGpgsfu3/lAypI3urFpQwdGHLCx2eK1OmM/8QFvvNqDD1Z1rrdedZWYceUIzv/MwVw+7UB69K5kWU7Lv72puU/R3WfbqdXlnXl/eReGjtwEwCFHrOWdhdl1qUOPXMPSN7pRvqIu2Q8etpmyDtnflkF7bmH4vptYubT0/jNoC47+dHnttd76dOlaVXtJ5NAjP6CqUryzqHuxw2u98u06l2j32bfkNIEbvrc3F1/9Bp06VbP8na5cdfE+AHzi06t49L7tu2YfHruOk6e9TmWliGr4xXdHsHZ1p5YIu13r0q2KQ49cw7Xf2ae27Ih/X8U509+iT/9tXHbTayye353vnDmaPgO2ceWv5lNdLVat7MxPvzmqniO3D6XaCsyHokjZXNIdwNHAQGAlMD0ibq5vnz5lA2JC1xOKEo8VSZk7G6XkqU33s6aqfLcuiPbqOywOPeqivOo+ft/Fz0bE2N05X3MrWksxIk4r1rHNrGW15Zaiu89mVpgAqtpuVnRSNLOCuaVoZparREeW8+GkaGYFc0vRzKyGpw4zM6sjQB5oMTOrI19TNDNL3H02M8tVus8158NJ0cwK5tFnM7NcbimamSXRtkefPcWJmRUu8lzqIWm4pEckvSrpFUkXpfL+kh6StDD92S+VS9K1khZJelHSYTnHmprqL5Q0dXe+mpOimRVMEXktDagEvhERo4EJwHmSRgOXAHMiYhQwJ60DHA+MSsvZwA2QJVFgOjAeGAdMr0mkjeGkaGaFa4KZtyNieUQ8lz6vA+YDQ4EpwC2p2i3AienzFODWyDwF9JU0BJgEPBQRFRGxGniI3XiTqK8pmllhAsj/pVQDJc3LWZ8RETN2rCRpBHAoMBcYHBHL06YVwOD0eSiwJGe3palsV+WN4qRoZgUReXWNa5Q3NPO2pJ7AH4CvRsRaqW5i8IgIqXlvAHL32cwKV12d39IASZ3IEuLtEfHHVLwydYtJf76XypcBw3N2H5bKdlXeKE6KZlaYmu5zPks9lDUJbwbmR8TPcjbdC9SMIE8F7skpPyONQk8A1qRu9mzgOEn90gDLcamsUdx9NrOCNdGEEEcC/wG8JOmfqew/gR8Bd0k6C3gbODltewA4AVgEbATOBIiICklXAM+kepdHREVjg3JSNLPCNUFSjIgnyGYi25mJO6kfwHm7ONZMYOZuB4WTopkVzBNCmJnV8dv8zMy250lmzcxyOSmamSUBVDspmpklHmgxM9uek6KZWRJAVf4zQpQaJ0UzK1BAOCmamdVx99nMLPHos5nZDtxSNDPL4aRoZpZEQFVVS0dRNE6KZlY4txTNzHI4KZqZ1QiPPpuZ1QoI37xtZpbDj/mZmSUReb2+tFQ5KZpZ4TzQYmZWJ9xSNDOr4UlmzczqeEIIM7M6AYQf8zMzS8KTzJqZbSfcfTYzy9GGW4qKVjSKJOl94O2WjqMIBgLlLR2EFaSt/s72jog9ducAkmaR/XzyUR4Rk3fnfM2tVSXFtkrSvIgY29JxWP78O2u/ylo6ADOz1sRJ0cwsh5Ni85jR0gFYwfw7a6d8TdHMLIdbimZmOZwUzcxyOCkWkaTJkhZIWiTpkpaOxxomaaak9yS93NKxWMtwUiwSSR2AXwDHA6OB0ySNbtmoLA+/BkrqZmNrWk6KxTMOWBQRiyNiK3AnMKWFY7IGRMRjQEVLx2Etx0mxeIYCS3LWl6YyM2vFnBTNzHI4KRbPMmB4zvqwVGZmrZiTYvE8A4ySNFJSZ+BU4N4WjsnMGuCkWCQRUQmcD8wG5gN3RcQrLRuVNUTSHcCTwAGSlko6q6Vjsublx/zMzHK4pWhmlsNJ0cwsh5OimVkOJ0UzsxxOimZmOZwUS4ikKkn/lPSypLsldd+NY/1a0knp8031TVYh6WhJRzTiHG9J+pe3vu2qfIc66ws81/ckfbPQGM125KRYWjZFxCERMQbYCkzL3SipUe/xjogvR8Sr9VQ5Gig4KZqVIifF0vU4sF9qxT0u6V7gVUkdJP0/Sc9IelHSVwCUuS7N7/gwMKjmQJIelTQ2fZ4s6TlJL0iaI2kEWfL9Wmql/i9Je0j6QzrHM5KOTPsOkPSgpFck3QSooS8h6X8kPZv2OXuHbVel8jmS9khl+0qalfZ5XNKBTfLTNEsa1bKwlpVahMcDs1LRYcCYiHgzJZY1EfExSV2Av0t6EDgUOIBsbsfBwKvAzB2OuwfwS+CodKz+EVEh6UZgfUT8NNX7LXBVRDwhaS+yp3b+DZgOPBERl0v6FJDP0yBfSufoBjwj6Q8RsQroAcyLiK9J+m469vlkL5SaFhELJY0HrgeOacSP0WynnBRLSzdJ/0yfHwduJuvWPh0Rb6by44CP1FwvBPoAo4CjgDsiogp4V9Jfd3L8CcBjNceKiF3NK3gsMFqqbQj2ltQzneNzad/7Ja3O4ztdKOmz6fPwFOsqoBr4XSr/DfDHdI4jgLtzzt0lj3OY5c1JsbRsiohDcgtSctiQWwRcEBGzd6h3QhPGUQZMiIjNO4klb5KOJkuwh0fERkmPAl13UT3SeT/Y8Wdg1pR8TbHtmQ2cI6kTgKT9JfUAHgNOSdcchwCf3Mm+TwFHSRqZ9u2fytcBvXLqPQhcULMi6ZD08THgC6nseKBfA7H2AVanhHggWUu1RhlQ09r9Alm3fC3wpqTPp3NI0sENnMOsIE6Kbc9NZNcLn0svX/pvsh7Bn4CFadutZDPBbCci3gfOJuuqvkBd9/U+4LM1Ay3AhcDYNJDzKnWj4JeRJdVXyLrR7zQQ6yygo6T5wI/IknKNDcC49B2OAS5P5V8EzkrxvYJf8WBNzLPkmJnlcEvRzCyHk6KZWQ4nRTOzHE6KZmY5nBTNzHI4KZqZ5XBSNDPL8f8BQNRBLbNBNVkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "create_confusion_matrix_plot(model, X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc429fce",
   "metadata": {},
   "source": [
    "### MLFlow work Starts from here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "e5756180",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T05:11:17.242446Z",
     "start_time": "2023-02-07T05:11:17.215410Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'accuracy': 0.91, 'precision': 0.64, 'recall': 0.52, 'entropy': 0.2}\n"
     ]
    }
   ],
   "source": [
    "experiment_name = \"basic_classifier\" ##basic classifier\n",
    "run_name=\"term_deposit\"\n",
    "run_metrics = get_metrics(y_test, y_pred, y_pred_prob)\n",
    "print(run_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "af260494",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T10:02:34.636337Z",
     "start_time": "2023-02-07T10:02:30.991667Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Run - term_deposit is logged to Experiment - basic_classifier\n"
     ]
    }
   ],
   "source": [
    "create_experiment(experiment_name,run_name,run_metrics,model,'confusion_matrix.png', 'roc_auc_curve.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "828a00fb",
   "metadata": {},
   "source": [
    "### Function to create an experiment in MLFlow and log parameters, metrics and artifacts files like images etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "26f2c095",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T15:07:30.753807Z",
     "start_time": "2023-02-07T15:07:30.734255Z"
    }
   },
   "outputs": [],
   "source": [
    "def create_experiment(experiment_name,run_name, run_metrics,model, confusion_matrix_path = None, \n",
    "                      roc_auc_plot_path = None, run_params=None):\n",
    "    import mlflow\n",
    "    #mlflow.set_tracking_uri(\"http://localhost:5000\") #uncomment this line if you want to use any database like sqlite as backend storage for model\n",
    "    mlflow.set_tracking_uri(\"http://localhost:5001\")\n",
    "    mlflow.set_experiment(experiment_name)\n",
    "    \n",
    "    with mlflow.start_run():\n",
    "        \n",
    "        if not run_params == None:\n",
    "            for param in run_params:\n",
    "                mlflow.log_param(param, run_params[param])\n",
    "            \n",
    "        for metric in run_metrics:\n",
    "            mlflow.log_metric(metric, run_metrics[metric])\n",
    "        \n",
    "        mlflow.sklearn.log_model(model, \"model\")\n",
    "        \n",
    "        if not confusion_matrix_path == None:\n",
    "            mlflow.log_artifact(confusion_matrix_path, 'confusion_materix')\n",
    "            \n",
    "        if not roc_auc_plot_path == None:\n",
    "            mlflow.log_artifact(roc_auc_plot_path, \"roc_auc_plot\")\n",
    "        \n",
    "        mlflow.set_tag(\"tag1\", \"Random Forest\")\n",
    "        mlflow.set_tags({\"tag2\":\"Randomized Search CV\", \"tag3\":\"Production\"})\n",
    "            \n",
    "    print('Run - %s is logged to Experiment - %s' %(run_name, experiment_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00c5b47c",
   "metadata": {},
   "source": [
    "### Create another experiment after tuning hyperparameters and log the best set of parameters for which model gives the optimal performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "5a0d0f40",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T15:10:38.019379Z",
     "start_time": "2023-02-07T15:07:34.248197Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 100 candidates, totalling 500 fits\n",
      "Random grid:  {'n_estimators': [5, 21, 51, 101], 'max_features': ['auto', 'sqrt'], 'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120], 'min_samples_split': [2, 6, 10], 'min_samples_leaf': [1, 3, 4], 'bootstrap': [True, False]} \n",
      "\n",
      "Best Parameters:  {'n_estimators': 21, 'min_samples_split': 6, 'min_samples_leaf': 3, 'max_features': 'sqrt', 'max_depth': 90, 'bootstrap': True}  \n",
      "\n"
     ]
    }
   ],
   "source": [
    "import mlflow\n",
    "experiment_name = \"optimized model\"\n",
    "run_name=\"Random_Search_CV_Tuned_Model\"\n",
    "model_tuned,best_params = hyper_parameter_tuning(X_train, y_train)\n",
    "run_params = best_params\n",
    "\n",
    "y_pred = predict_on_test_data(model_tuned,X_test) #will return the predicted class\n",
    "y_pred_prob = predict_prob_on_test_data(model_tuned,X_test) #model.predict_proba(X_test)\n",
    "run_metrics = get_metrics(y_test, y_pred, y_pred_prob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "163fd982",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T15:11:57.582599Z",
     "start_time": "2023-02-07T15:11:57.567263Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.91, 'precision': 0.61, 'recall': 0.58, 'entropy': 0.2}"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "2d8bafa5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T15:11:58.098932Z",
     "start_time": "2023-02-07T15:11:58.088307Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n_estimators 21\n",
      "min_samples_split 6\n",
      "min_samples_leaf 3\n",
      "max_features sqrt\n",
      "max_depth 90\n",
      "bootstrap True\n"
     ]
    }
   ],
   "source": [
    "for param in run_params:\n",
    "    print(param, run_params[param])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "f520e4c9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-02-07T15:12:04.752307Z",
     "start_time": "2023-02-07T15:11:59.108667Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023/02/08 00:12:01 INFO mlflow.tracking.fluent: Experiment with name 'optimized model' does not exist. Creating a new experiment.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Run - Random_Search_CV_Tuned_Model is logged to Experiment - optimized model\n"
     ]
    }
   ],
   "source": [
    "create_experiment(experiment_name,run_name,run_metrics,model_tuned,'confusion_matrix.png', 'roc_auc_curve.png',run_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3a466fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Above we are utilizing the file as a models data storage. But if we want to store all these results into our database\n",
    "# then we need to follow the steps given below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6aaa3cb1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7da88ce8",
   "metadata": {},
   "source": [
    "\n",
    "### if want to use the model registry feature, we need a database.\n",
    "\n",
    "#### _If you have MySQL installed then you can use the below command:_\n",
    "\n",
    "1. Create a database to use as an MLflow backend tracking server.\n",
    "\n",
    "`CREATE DATABASE mlflow_tracking_database;`\n",
    "\n",
    "2. Start MLflow tracking server using MySQL as a backend tracking store.\n",
    "` mlflow server \\\n",
    "   --backend-store-uri  mysql+pymysql://root@localhost/mlflow_tracking_database \\ \n",
    "   --default-artifact-root  file:/./mlruns \\\n",
    "   -h 0.0.0.0 -p 5000`\n",
    "\n",
    "\n",
    "3. Set the MLflow tracking uri (within code section).\n",
    "\n",
    "  mlflow.set_tracking_uri(\"http://localhost:5000\")\n",
    "\n",
    "#### _If you have sqlite installed then you can use the below command:_\n",
    "\n",
    "1. Start MLflow tracking server using sqlite as a backend tracking store.\n",
    "\n",
    "`mlflow server --backend-store-uri sqlite:///mlflow.db --default-artifact-root ./artifacts --host 0.0.0.0 --port 5000`\n",
    "\n",
    "\n",
    "2. Set the MLflow tracking uri (within code section).\n",
    "    \n",
    "    mlflow.set_tracking_uri(\"http://localhost:5000\")\n",
    "\n",
    "\n",
    "You can also follow the official documentation for more information on backend database for model registry\n",
    "\n",
    "https://www.mlflow.org/docs/latest/model-registry.html#model-registry-workflows"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c51dadd7",
   "metadata": {},
   "source": [
    "## Next steps:\n",
    "\n",
    "Using MLFlow\n",
    "* How deploy models from model registry\n",
    "* Model serving both batch serving and online serving\n",
    "* MLFlow pipelines\n",
    "* Model Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dea7af37",
   "metadata": {},
   "source": [
    "# Thank You "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ac6b8d7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}