Mod4-1-ML-SL-PerceptronAdaBoostANN.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"celltoolbar":"Slideshow","colab":{"name":"Mod4-1-ML-SL-PerceptronAdaBoostANN.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.5"}},"cells":[{"cell_type":"markdown","metadata":{"id":"Ji-lASE7BLUo"},"source":["\n","# <font color=#770000>ICPE 639 Introduction to Machine Learning </font>\n","\n","## ------ With Energy Applications\n","\n","Some of the examples and exercises of this course are based on several books as well as open-access materials on machine learning, including [Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/)\n","\n","\n","<p> &#169; 2021: Xiaoning Qian </p>\n","\n","[Homepage](http://xqian37.github.io/)\n","\n","**<font color=blue>[Note]</font>** This is currently a work in progress, will be updated as the material is tested in the class room.\n","\n","All material open source under a Creative Commons license and free for use in non-commercial applications.\n","\n","Source material used under the Creative Commons Attribution-NonCommercial 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc/3.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA."]},{"cell_type":"markdown","metadata":{"id":"h8CLzQygBLUt"},"source":["# Perceptrons, Boosting, \\& Artificial Neural Networks (ANNs)\n","\n","\n","Here we cover some basics that may serve as the introduction to deep neural networks. This section will cover the content listed below: \n","\n","- [1 Perceptron](#1-Perceptron)\n","- [2 AdaBoost](#2-AdaBoost)\n","- [3 ANNs](#3-ANNs)\n","- [4 Hands-on Exercise](#4-Hands-on-Exercise)\n","- [Reference](#Reference) \n","\n"]},{"cell_type":"code","metadata":{"id":"P78hJOpbBLUu","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1632780853929,"user_tz":300,"elapsed":1067,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"1266ad98-b45f-453e-d14c-e22f5750db5c"},"source":["# required modules\n","import pandas as pd\n","import numpy as np\n","import matplotlib.pyplot as plt\n","\n","from sklearn.ensemble import AdaBoostClassifier\n","\n","from sklearn.tree import DecisionTreeClassifier, export_graphviz, export_text\n","from sklearn.externals.six import StringIO  \n","import pydot\n","from IPython.display import Image\n","from IPython.core.display import HTML\n","from sklearn.metrics import accuracy_score\n","\n","import seaborn as sns\n","from sklearn import neighbors, datasets\n","from matplotlib.colors import ListedColormap\n","from sklearn.model_selection import train_test_split\n","from sklearn.neighbors import KNeighborsClassifier\n"],"execution_count":1,"outputs":[{"output_type":"stream","name":"stderr","text":["/usr/local/lib/python3.7/dist-packages/sklearn/externals/six.py:31: FutureWarning: The module is deprecated in version 0.21 and will be removed in version 0.23 since we've dropped support for Python 2.7. Please rely on the official version of six (https://pypi.org/project/six/).\n","  \"(https://pypi.org/project/six/).\", FutureWarning)\n"]}]},{"cell_type":"markdown","metadata":{"id":"CzW3wV_zKTbk"},"source":["## 1 Perceptron\n","\n","\n"]},{"cell_type":"markdown","metadata":{"executionInfo":{"elapsed":1294,"status":"ok","timestamp":1616527812431,"user":{"displayName":"Xiaomeng Yan","photoUrl":"","userId":"11420231255337685715"},"user_tz":300},"id":"agkk34mwfSW1","outputId":"dfc8c64f-d800-428a-a4aa-6b6de7147a9a"},"source":["### 1.1 Basics\n","\n","Perceptrons are simply linear halfspaces for binary classification: \n","$$sign(w^Tx),$$\n","where $w$ are model parameters (often including the intercept $b$ if needed to make explicit) to consider all features $x$ to predict class label $y\\in\\{-1, 1\\}$. \n","\n","The intuition is that if all the data points are predicted correctly, then we have $y_n w^Tx_n \\geq 0$. Hence, we can just construct the constrained optimization formulation to solve for $w$ under these constraints. Note that if we set the objective function to be $|w|^2$, it is similar as SVM. "]},{"cell_type":"markdown","metadata":{"id":"hW7sE500KdST"},"source":["#### A simple example with a few lines of Python code\n","\n","[reference link](https://github.com/MaviccPRP/perceptron/blob/master/perceptron.ipynb)"]},{"cell_type":"code","metadata":{"id":"zRIFOwDsidt7","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1632781915625,"user_tz":300,"elapsed":194,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"b63c3ea7-697e-4848-c74b-73ada28a0c1e"},"source":["X = np.array([\n","    [-2,4,-1],\n","    [4,1,-1],\n","    [1, 6, -1],\n","    [2, 4, -1],\n","    [6, 2, -1],\n","\n","])\n","\n","y = np.array([-1,-1,1,1,1])\n","\n","def perceptron_sgd(X, Y):\n","    w = np.zeros(len(X[0]))  # initialize to 0's\n","    eta = 1                  # learning rate\n","    epochs = 20              # number of iterations\n","\n","    for t in range(epochs):\n","        for i, x in enumerate(X):\n","            if (np.dot(X[i], w)*Y[i]) <= 0:\n","                w = w + eta*X[i]*Y[i]            # update if not correctly classified\n","\n","    return w\n","\n","w = perceptron_sgd(X,y)\n","print(w)"],"execution_count":2,"outputs":[{"output_type":"stream","name":"stdout","text":["[ 2.  3. 13.]\n"]}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":282},"id":"JHerhFCOijnp","executionInfo":{"status":"ok","timestamp":1632781931279,"user_tz":300,"elapsed":376,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"6dec3d5a-8cbf-4c93-829e-49c7e8387d3e"},"source":["# visualize \n","for d, sample in enumerate(X):\n","    # Plot the negative samples\n","    if d < 2:\n","        plt.scatter(sample[0], sample[1], s=120, c=\"r\",marker='_', linewidths=2)\n","    # Plot the positive samples\n","    else:\n","        plt.scatter(sample[0], sample[1], s=120, c=\"b\", marker='+', linewidths=2)\n","\n","# Print a possible hyperplane, that is seperating the two classes.\n","plt.plot([-2,6],[6,0.5])"],"execution_count":3,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[<matplotlib.lines.Line2D at 0x7f895d305810>]"]},"metadata":{},"execution_count":3},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3iUZfr28e+VRuhdpIfeq6EGElQCSFVsWLALKjVZ111X13WL7eduaCoq9gKoCIiiQHAhIfSE3ptUKUF6b/f7R2Bf1gVJIJNnJjk/xzEHITN55jwgnNy553muMeccIiLiv4K8DiAiIr9NRS0i4udU1CIifk5FLSLi51TUIiJ+LsQXBy1VqpSLiIjwxaFFRHKltLS0vc650pe6zydFHRERQWpqqi8OLSKSK5nZlsvdp60PERE/p6IWEfFzKmoRET+nopZs165dxk1EsoeKWkTEz2WqqM2smJmNM7M1ZrbazFr5OpiIiGTI7Ol5w4Apzrk7zCwMKODDTCIicpErFrWZFQWigYcAnHOngFO+CDP8x/XE1CxNo4rFfHF48ZFf70cnJV368zNn5kAYkVwoM1sfVYB04EMzW2xm75lZwV8/yMz6mFmqmaWmp6dnOciBY6cYPX8rt701m5e/X83xU2ezfAwRkdzIrvTGAWYWCcwDopxz881sGHDIOffny31NZGSku5orEw+dOM0r369hzIKtRJQswCs9G9KqWsksH0e8dWElrRW0SOaZWZpzLvJS92VmRb0d2O6cm3/+9+OAptkV7mJFwkN5pWcDRj/eAgfcM2oef5qwnEMnTvvi6UREAsIVi9o5twvYZma1zn/qZmCVL0O1rlaKKYOiebxtFcYu2EqHhGR+XL3bl08pIuK3Mnse9QDgczNbBjQGXvZdpAz5w4J5rktdxj8VRdH8oTz6cSoDxyzmlyMnff3UIiJ+5Yp71FfjaveoL+fUmXO8NXMDb87YQOHwUP7SrS7dG5XDzLLtOUREvHSte9SeCwsJYnD7mnw3oC0VSxRg0NglPPZxKjsPHvc6moiIzwVEUV9Q6/rCjH+yNc93qcPsjXvpkJDM6PlbOXcu+38qEBHxFwFV1ADBQcZjbasydXA09csX5U8TlnPve/PYvPeo19FERHwi4Ir6gsolCzL68Ra82rMBK3ccouPQZN5N3siZs+e8jiYikq0CtqgBzIxezSuRGB9D2xqlePn7Ndw+cg5rdh3yOpqISLYJ6KK+4Pqi4Yx6IJIR9zRh+/7jdB2eQkLiOk6e0WXoIhL4ckVRQ8bqulujciTGx9C1YVmG/7iebiNSWLx1v9fRRESuSa4p6gtKFAxjaK8mfPBQJIdPnKHnyDn8/btVHDt1xutoIiJXJdcV9QU31S7DtLho7mtRifdTfqLj0GRmb9jrdSwRkSzLtUUNUDg8lH/c2oCxfVoSbMZ9783nj18v4+BxDXkSkcCRq4v6gpZVSzJlcDR9Y6ryZeo2YhOSmLZyl9exREQyJU8UNUB4aDDP3lKHif2iKFEwjD6fptF/9CL2asiTiPi5PFPUFzSsUIxJ/dvwu9iaTFu5m/YJSUxYvB1fDKcSEckOea6oIWPI04CbazB5YBuqlCpI3BdLeeSjhfx8QEOeRMT/5MmivqBGmcKMe6I1L3Sty7xN+4hNSOLTeVs05ElE/EqeLmrIGPL0SJsqTIuLpkml4vx54gp6vTuPTelHvI4mIgKoqP+jYokCfPpoc/7v9oas3nWIW4bN4u0kDXkSEe+pqC9iZtzVrCLT42OIqVmaV39Yw61vzWbVzxryJCLeUVFfQpki4bzT+wbeuq8puw6eoPsbKfxr2loNeRIRT6ioL8PM6NygLIlxMXRvXI4R/95Al+EppG3Z53U0EcljVNRXULxgGAl3Neajh5tx/NRZ7nh7Li9OWsnRkxryJCI5Q0WdSe1qXcfUuGh6t6zMR3M203FoMrPWp3sdS0TyABV1FhTKF8LfetTny76tCAsOovf7C/j9V0s5eExDnkTEd1TUV6F5lRJ8P6gtT7WrxvjFO2g/JIkpKzTkSUR8Q0V9lcJDg3mmU22+6RdF6UL5eOKzNJ76PI09h094HU1EchkV9TWqX74o3/SP4vcdazF99R5iE5IZl6YhTyKSfVTU2SA0OIh+N1bn+4FtqX5dIZ7+aikPfriQ7fuPeR1NRHKBTBW1mW02s+VmtsTMUn0dKlBVv64QX/VtxV+71yN18z46DEnm4zmbNeRJRK5JVlbUNzrnGjvnIn2WJhcICjIebB3BtLhoIiNK8JdJK7nrnbls1JAnEblK2vrwkQrFC/Dxw834552NWL/nCLcMm8WbMzZwWkOeRCSLMlvUDphmZmlm1udSDzCzPmaWamap6em6EAQyLkO/44YKJMZH077Odbw+dS093pjNih0HvY4mIgHEMnN2gpmVd87tMLPrgERggHMu+XKPj4yMdKmp2sr+tSkrdvL8xJXsP3aKPtFVGXRzDcJDg72OJSJ+wMzSLre1nKkVtXNux/lf9wATgObZFy/v6FS/LD/Gx9CzSXlGztxI52GzWLhZQ55E5LddsajNrKCZFb7wMdABWOHrYLlV0QKhvH5nIz55pDknz5zjzrfn8sI3KziiIU8ichmZWVGXAVLMbCmwAJjsnJvi21i5X3TN0kyLi+ah1hF8Om8LHYckk7ROe/si8r8ytUedVdqjzpq0Lft4ZtwyNqYfpWfT8rzQtS7FCoR5HUtEctA171GLb91QuQSTB7al/43VmbTkZ9onJPH98p26DF1EABW13wgPDebpjrX4pn8U1xcN56nPF/HEZ2nsOaQhTyJ5nYraz9QrV5SJT0Xxh061mbE2nfYJSXyZuk2ra5E8TEXth0KCg3iyXTWmDGpL7euL8My4ZfR+fwHb9mnIk0hepKL2Y1VLF2Jsn5b8/db6LN66nw5Dkvlw9k+c1ZAnkTxFRe3ngoKM3i0rMy0+hhZVS/DXb1dx59tz2LDnsNfRRCSHqKgDRPli+fnwoWYMubsRm/YepfOwFEb8uF5DnkTyABV1ADEzbmtSgenxMcTWK8O/EtfRbUQKy7dryJNIbqaiDkClCuXjzXub8k7vG9h39BQ93kzhlR9Wc+L0Wa+jiYgPqKgDWMd615MYH8NdkRV5J2kTtwybxfxNv3gdS0SymYo6wBXNH8qrtzfk88dacObcOe5+dx7PT1zO4ROnvY4mItlERZ1LRFUvxdTB0Tzapgqfz99KxyHJzFizx+tYIpINVNS5SIGwEP7ctS5fP9magvlCePijhcR9sYR9R095HU1EroGKOhdqWqk43w1sw8Cba/Dt0p+JTUji26U/6zJ0kQClos6l8oUEEx9bk28HtKF88fwMGLOYxz9JY7eGPIkEHBV1LlenbBHGP9maP3Wuzaz1GUOexi7YqtW1SABRUecBIcFB9ImuxtTB0dQtW4Q/jl/Ofe/NZ+svGvIkEghU1HlIRKmCjHm8JS/f1oBl2w/SYWgS783apCFPIn5ORZ3HBAUZ97aoRGJ8NK2rleIfk1fTc+Qc1u7SkCcRf6WizqPKFs3P+w9GMqxXY7btO0bXEbMYOn0dp85oyJOIv1FR52FmRo/G5UmMi6Zzg7IMnb6ebiNSWLrtgNfRROQiKmqhZKF8DOvVhPceiOTg8dPc9tZsXpq8iuOnNORJxB+oqOU/2tctw7T4aHo1r8SoWT/RaVgyczdqyJOI11TU8l+KhIfy8m0NGP14CwDuGTWPZ8cv55CGPIl4RkUtl9S6WimmDIqmT3RVvli4ldiEJKav2u11LJE8SUUtl5U/LJg/da7D+KeiKJY/jMc+SWXgmMX8cuSk19GuSrt2GTeRQKOilitqXLEY3w5oQ1z7mvywYiftE5L4ZskOXYYukkMyXdRmFmxmi83sO18GEv8UFhLEoPY1mDywLZVLFmTQ2CU89nEqOw8e9zqaSK6XlRX1IGC1r4JIYKhZpjBfP9ma57vUYfbGvcQmJPP5/C2c02XoIj4TkpkHmVkFoAvwEhCfbc+e2Q3DmTOz7Snl2gUHGY+1rUqHutfzx/HLeG7CCr5d+jOv9mxIRKmCXsf7j19/eyUlXfrz+vYSf5fZFfVQ4BngstcXm1kfM0s1s9T09PRsCSf+rVLJAnz+WAte7dmAlTsO0XFoMu8mb+TMWV2GLpKd7EovCJlZV6Czc+4pM2sHPO2c6/pbXxMZGelSU1OzL6X4vV0HT/D8xBVMX72bhhWK8trtDalTtojXsf7LhZW0VtDij8wszTkXean7MrOijgK6m9lmYCxwk5l9lo35JBe4vmg4ox64gTfubcKO/cfpNiKFhMR1nDyjy9BFrtUVi9o596xzroJzLgLoBfzbOXe/z5NJwDEzujYsx/T4GLo1KsfwH9fTdXgKi7bu9zqaSEDTedSS7YoXDGPI3Y358KFmHDl5httHzuHv363i2KkzXkcTCUhX3KO+GtqjlgsOnzjNa1PW8Nm8rVQskZ9XezYkqnopr2OJ+J1r3aMWuWqFw0P5x60N+KJPS0KCgrjvvfn8YdwyDh7XkCeRzFJRS45oUbUkPwxqyxMx1Ri3aDuxCUlMW7nL61giAUFFLTkmPDSYP95Sm4lPRVGyUD76fJpGv9GLSD8cmEOeRHKKilpyXIMKRZnUP4qnO9QkceVuYockMWHxdg15ErkMFbV4IjQ4iP431eD7QW2oWqogcV8s5eGPFrLjgIY8ifyailo8Vf26wnz1RGv+0q0u8zfto0NCEp/O3awhTyIXUVGL54KDjIejqjAtLpqmlYvz529W0uvdeWxKP+J1NBG/oKIWv1GxRAE+eaQ5r9/RkDW7DtFp2CxGztSQJxEVtfgVM+POyIpMj4/hxlqleW3KGm59azarfj7kdTQRz6ioxS9dVyScd3pHMvK+puw6eJLub6Twz6lrOXFaQ54k71FRi1+7pUFZpsdH06Nxed6YsYEuw2eRtmWf17FEcpSKWvxesQJh/OuuRnz8SHNOnD7HHW/P5cVJKzl6UkOeJG9QUUvAiKlZmqlx0TzQsjIfz91MhyHJJK/TuwlJ7qeiloBSKF8If+1Rny/7tiJfaBAPfLCAp79aysFjGvIkuZeKWgJSs4gSfD+wLU+1q8aExTtoPySJKSt2eh1LxCdU1BKwwkODeaZTbb7pF0XpQvl44rNFPPlZGnsOn/A6mki2UlFLwKtfvijf9I/i9x1r8eOaPcQmJDMuTUOeJPdQUUuuEBocRL8bq/P9wLbUuK4QT3+1lAc+WMC2fce8jiZyzVTUkqtUv64QX/Ztxd961GPRlv10HJrMR7N/0pAnCWgqasl1goKMB1pFMDUumsiIErz47SruemcuG/ZoyJMEJhW15FoVihfg44eb8a87G7F+zxE6D5vFmzM2cFpDniTAqKglVzMzbr+hAtPjY2hf9zpen7qWHm/MZsWOg15HE8k0FbXkCaUL5+Ot+27g7fubkn7kJD3enM1rU9ZoyJMEBBW15Cmd6pdlelwMtzctz8iZG+k8bBYLN2vIk/g3FbXkOUULhPJ/dzTis0dbcOrsOe58ey4vfLOCIxryJH5KRS15VpsapZg6OJqHoyL4dN4WOg5JZubaPV7HEvkfKmrJ0wrmC+Ev3eox7onW5A8L5qEPFxL/5RL2Hz3ldTSR/7hiUZtZuJktMLOlZrbSzP6aE8FEctINlYszeWAbBtxUnUlLfiZ2SBKTl+3UZejiFzKzoj4J3OScawQ0BjqZWUvfxhLJeflCgvldh1pM6t+GskXz02/0Ivp+msaeQxryJN66YlG7DBcu6Qo9f9MyQ3KtuuWKMOGp1jx7S22S1qVzc0ISXy7cptW1eCZTe9RmFmxmS4A9QKJzbv4lHtPHzFLNLDU9Xe+6IYEtJDiIvjHV+GFQW+qULcIzXy+j9/sa8iTesKysEsysGDABGOCcW3G5x0VGRrrU1NRsiCfivXPnHKMXbOXVH9Zw9pzj9x1r8WDrCIKDzOtokouYWZpzLvJS92XprA/n3AFgBtApO4KJBIKgIOP+lpWZFhdNi6ol+Nt3q7jz7Tms333Y62iSR2TmrI/S51fSmFl+IBZY4+tgIv6mXLH8fPhQM4be3Zif9h6ly/AURvy4nlNnNORJfCszK+qywAwzWwYsJGOP+jvfxhLxT2bGrU3KkxgfQ8f61/OvxHV0fyOFZdsPeB1NcrEs7VFnlvaoJa9IXLWb5ycuJ/3wSR5vW5W42JqEhwZ7HUsCULbtUYvIf4utW4ZpcTHc3awi7yRvotPQZOZt+sXrWJLLqKhFrlHR/KG80rMhox9rwTkHvd6dx3MTlnP4xGmvo0kuoaIWySatq5diyuC2PNamCmMWbKXDkGRmrNGQJ7l2KmqRbFQgLITnu9bl6ydbUyhfCA9/tJDBYxezT0Oe5BqoqEV8oEml4nw3sA2Dbq7B5OU7aZ+QxKSlP+sydLkqKmoRH8kXEkxcbE2+HdCGisXzM3DMYh7/JI1dBzXkSbJGRS3iY7WvL8L4p6J4rnMdUjakE5uQxJgFW7W6lkxTUYvkgOAg4/HoqkwZFE298kV4dvxy7h01ny2/HPU6mgQAFbVIDoooVZDRj7Xk5dsasGLHQToOTea9WZs4e06ra7k8FbVIDgsKMu5tUYlp8dFEVSvFPyavpufIOazdpSFPcmkqahGPlC2an/cejGT4PU3Ytu8YXUfMYuj0dRryJP9DRS3iITOje6NyTI+PoXODsgydvp5uI1JYsk1DnuT/U1GL+IESBcMY1qsJ7z8YycHjp+n51mxemryK46fOeh1N/ICKWsSP3FynDNPio+nVvBKjZv1Ex6HJzNm41+tY4jEVtYifKRIeysu3NWDM4y0xg3tHzefZ8cs5pCFPeZaKWsRPtapWkimDoukbXZUvFm4lNiGJ6at2ex1LPKCiFvFj+cOCebZzHSb2i6J4gTAe+ySVAWMW88uRk15HkxykohYJAA0rFGNS/zbEx9ZkyoqMIU/fLNmhy9DzCBW1SIAICwli4M01mDywLZVLFmTQ2CU8+nEqPx847nU08TEVtUiAqVmmMF8/2Zo/d63L3I2/0GFIMp/P38I5XYaea6moRQJQcJDxaJsqTB0cTaOKRXluwgruGTWPn/ZqyFNupKIWCWCVShbgs0db8NrtDVi18xCdhibzTtJGzpzVZei5iYpaJMCZGXc3q8T0+Biia5bmlR/W0HPkHFbvPOR1NMkmKmqRXKJMkXDe7X0Db97blJ8PHKfbiBQSpq3l5Bldhh7oVNQiuYiZ0aVhWRLjYujeqBzD/72BrsNTWLR1v9fR5BqoqEVyoeIFw0i4uzEfPtyMoyfPcPvIOfzt21UcO3XG62i5Vrt2GTdfUFGL5GI31rqOqXHR3N+iMh/MzhjylLJeQ54CzRWL2swqmtkMM1tlZivNbFBOBBOR7FE4PJS/31qfL/u2IiQoiPvfn88z45Zy8LiGPAWKzKyozwC/c87VBVoC/cysrm9jiUh2a16lBD8MasuT7arx9aIdxCYkMXXlLq9jSSaEXOkBzrmdwM7zHx82s9VAeWCVj7OJSDYLDw3mD51q07l+WZ75ehl9P02jS4OyvNi9HqUL5/M6XkD59X50UtKlPz9z5rU/V5b2qM0sAmgCzL/EfX3MLNXMUtPT0689mYj4TIMKRZnUP4rfd6xF4qrdxA5JYvyi7Rry5Kcss38xZlYISAJecs6N/63HRkZGutTU1GyIJyK+tmHPYZ4Zt4xFWw8QU7M0L/dsQPli+b2OFXAurKSvdgVtZmnOuchL3ZepFbWZhQJfA59fqaRFJLBUv64wXz3Rmhe71WXh5n10SEjik7mbNeTJj2TmrA8D3gdWO+cSfB9JRHJacJDxUFTGkKemlYvzwjcrufvduWxMP+J1NCFzK+oooDdwk5ktOX/r7ONcIuKBiiUK8MkjzXn9joas3XWYW4bN4q2ZGzTkyWOZ3qPOCu1RiwS+PYdP8MLElUxZuYv65Yvw2u0NqVeuqNexcq1r3qMWkbznusLhvN37Bkbe15RdB0/S/Y3ZvD51DSdOa8hTTlNRi8hvuqVBWabHR3Nbk/K8OWMjXYbPInXzPq9j5SkqahG5omIFwvjnnY345JHmnDh9jjvfmcuLk1Zy9KSGPOUEFbWIZFp0zdJMi4vmwVYRfDx3Mx2GJJO8The4+ZqKWkSypGC+EF7sXo+v+rYiX2gQD3ywgKe/WsqBY6e8jpZrqahF5KpERpTg+4Ft6XdjNSYs3kH7hGR+WL7T61i5kopaRK5aeGgwv+9Ym0n9oyhTJB9Pfr6IJz9LY8/hE15Hy1VU1CJyzeqVK8rEflH8oVNtflyzh9iEZL5K3aYhT9lERS0i2SI0OIgn21Xjh0FtqVmmEL8ft4wHPljAtn3HvI4W8FTUIpKtqpUuxBd9WvH3HvVYtGU/HYcm89HsnzTk6RqoqEUk2wUFGb1bRTA1LppmESV48dtV3PnOXDbsOex1tICkohYRn6lQvAAfPdyMhLsasTH9CJ2HpfDmjA2c1pCnLFFRi4hPmRk9m1YgMS6G2HpleH3qWnq8MZsVOw56HS1gqKhFJEeULpyPN+9tyju9byD9yEl6vDmb16ZoyFNmqKhFJEd1rHc90+NiuKNpBUbO3EjnYbNY8JOGPP0WFbWI5LiiBUJ57Y6GfPZoC06dPcdd78zlzxNXcERDni5JRS0inmlToxTT4qJ5JKoKn83fQoeEJGas3eN1LL+johYRTxUIC+GFbnUZ90RrCuQL4eEPFxL/xRL2H9WQpwtU1CLiF26oXJzJA9sw8KbqTFr6M7FDkpi8bKcuQ0dFLSJ+JF9IMPEdavHtgDaULZqffqMX0ffTNHYfyttDnlTUIuJ36pQtwoSnWvPsLbVJWpdO+4Qkvli4Nc+urlXUIuKXQoKD6BtTjSmDo6lTtgh/+Ho5978/n62/5L0hTypqEfFrVUoVZOzjLfnHrfVZuu0gHYcm837KT5zNQ0OeVNQi4veCgoz7W1ZmWlw0LauW4O/freKOt+ewfnfeGPKkohaRgFGuWH4+eKgZw3o1ZvPeo3QZnsLwH9dz6kzuHvKkohaRgGJm9GhcnunxMXSsfz0Jievo/kYKS7cd8Dqaz6ioRSQglSyUjxH3NGHUA5HsP3aK296azSvfr+b4qdw35OmKRW1mH5jZHjNbkROBRESyIrZuGRLjY7i7WUXeSd7ELcOSmbfpF69jZavMrKg/Ajr5OIeIyFUrEh7KKz0bMvqxFpxz0OvdeTw3YTmHT5z2Olq2CLnSA5xzyWYW4fsoIuKpdu0y97iZM32Z4pq0rl6KqYOjSUhcy/spP/HvNXt46bb63FS7jNfRrkm27VGbWR8zSzWz1PT09Ow6rIhIluQPC+a5LnUZ/1QURcJDeeSjVAaNXcwvR056He2qWWYuyTy/ov7OOVc/MweNjIx0qamp15ZMROQanTpzjrdmbuDNGRsoHB7Ki93r0a1hWczM62j/w8zSnHORl7pPZ32ISK4VFhLE4PY1+W5AWyqWKMDAMYt5/JNUdh0MrCFPKmoRyfVqXV+Y8U+25vkudUjZsJfYhCTGLAicIU+ZOT1vDDAXqGVm283sUd/HEhHJXsFBxmNtqzJ1cDT1yxfl2fHLuXfUfLb8ctTraFeUqT3qrNIetYj4M+ccYxdu4+XJqzl97hy/i63FI22qEBzk3d619qhFRC5iZtzTvBKJ8TG0qV6Kl75fTc+3ZrN2l38OeVJRi0iedX3RcEY9EMmIe5qwff9xuo6YxZDEdX435ElFLSJ5mpnRrVE5EuNj6NKgLMN+XE/XEbNY4kdDnlTUIiJAiYJhDO3VhA8eiuTwiTP0fGs2//hulV8MeVJRi4hc5KbaZZgWF809zSvxXspPdByazJyNez3NpKIWEfmVwuGhvHRbA8b2aUmQwb2j5vPs+GUcPO7NkCcVtYjIZbSsWpIpg6PpG1OVLxZuo8OQJBJX7c7xHCpqEZHfEB4azLO31GFivyiKFwjj8U9S6T96EXtzcMiTilpEJBMaVijGpP5t+F1sTaat3E1sQhITF+/IkcvQVdQiIpkUFhLEgJtrMHlgGyJKFWTwF0t49ONUfj5w3KfPq6IWEcmiGmUKM+6J1rzQtS5zN/5ChyHJfDZvC+fO+WZ1raIWEbkKwUHGI22qMC0umsYVi/H8xBX0GjWPY6fOZPtzXfGtuERE5PIqlijAp48256vU7aRt2U+BsOyvVRW1iMg1MjPualaRu5pV9MnxtfUhIuLnVNQiIn5ORS0i4udU1CIifk5FLSLi51TUIiJ+TkUtIuLnVNQiIn7OfDH5yczSgS1X+eWlAG/fTuHSlCtrlCtrlCtrcmOuys650pe6wydFfS3MLNU5F+l1jl9TrqxRrqxRrqzJa7m09SEi4udU1CIifs4fi/pdrwNchnJljXJljXJlTZ7K5Xd71CIi8t/8cUUtIiIXUVGLiPg5vyxqM3vdzNaY2TIzm2BmxbzOBGBmd5rZSjM7Z2aenhpkZp3MbK2ZbTCzP3qZ5WJm9oGZ7TGzFV5nuZiZVTSzGWa26vzf4SCvMwGYWbiZLTCzpedz/dXrTBeYWbCZLTaz77zOcjEz22xmy81siZmlep3nAjMrZmbjznfXajNrlV3H9suiBhKB+s65hsA64FmP81ywAugJJHsZwsyCgTeBW4C6wD1mVtfLTBf5COjkdYhLOAP8zjlXF2gJ9POTP7OTwE3OuUZAY6CTmbX0ONMFg4DVXoe4jBudc4397FzqYcAU51xtoBHZ+Gfnl0XtnJvmnLvwDpHzgApe5rnAObfaObfW6xxAc2CDc26Tc+4UMBbo4XEmAJxzycA+r3P8mnNup3Nu0fmPD5Pxj6i8t6nAZThy/reh52+ev8JvZhWALsB7XmcJBGZWFIgG3gdwzp1yzh3IruP7ZVH/yiPAD16H8DPlgW0X/X47flA6gcLMIoAmwHxvk2Q4v8WwBNgDJDrn/CHXUOAZ4JzXQS7BAdPMLM3M+ngd5rwqQDrw4fntovfMrGB2Hdyzojaz6Wa24hK3Hhc95jkyfmT93J9ySeAys0LA18Bg59whr/MAOOfOOucak/GTY3Mzq+9lHjPrCuxxzqV5meM3tHHONSVj66+fmUV7HYiMNwpvCp7atPoAAAGUSURBVIx0zjUBjgLZ9tqRZ+9C7pxr/1v3m9lDQFfgZpeDJ3tfKZef2AFc/HbHFc5/Tn6DmYWSUdKfO+fGe53n15xzB8xsBhl7/F6+GBsFdDezzkA4UMTMPnPO3e9hpv9wzu04/+seM5tAxlagp68bkfFT7faLfhoaRzYWtV9ufZhZJzJ+7OrunDvmdR4/tBCoYWZVzCwM6AVM8jiTXzMzI2P/cLVzLsHrPBeYWekLZzWZWX4gFljjZSbn3LPOuQrOuQgyvrf+7S8lbWYFzazwhY+BDnj7nxoAzrldwDYzq3X+UzcDq7Lr+H5Z1MAbQGEg8fwpOG97HQjAzG4zs+1AK2CymU31Isf5F1r7A1PJeFHsS+fcSi+y/JqZjQHmArXMbLuZPep1pvOigN7ATee/p5acXzF6rSwww8yWkfEfcKJzzq9Oh/MzZYAUM1sKLAAmO+emeJzpggHA5+f/LhsDL2fXgXUJuYiIn/PXFbWIiJynohYR8XMqahERP6eiFhHxcypqERE/p6IWEfFzKmoRET/3/wBPguoPvA4IdgAAAABJRU5ErkJggg==\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":297},"id":"WlKMz3h_KjiN","executionInfo":{"status":"ok","timestamp":1632672163467,"user_tz":300,"elapsed":364,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"a542c510-fe0c-4c92-e2b2-2e97fcd45614"},"source":["# visualize the training procedure\n","def perceptron_sgd_plot(X, Y):\n","    '''\n","    train perceptron and plot the total loss in each epoch.\n","    \n","    :param X: data samples\n","    :param Y: data labels\n","    :return: weight vector as a numpy array\n","    '''\n","    w = np.zeros(len(X[0]))\n","    eta = 1\n","    n = 30\n","    errors = []\n","\n","    for t in range(n):\n","        total_error = 0\n","        for i, x in enumerate(X):\n","            if (np.dot(X[i], w)*Y[i]) <= 0:\n","                total_error += (np.dot(X[i], w)*Y[i])\n","                w = w + eta*X[i]*Y[i]\n","        errors.append(total_error*-1)\n","        \n","    plt.plot(errors)\n","    plt.xlabel('Epoch')\n","    plt.ylabel('Total Loss')\n","    \n","    return w\n","\n","\n","print(perceptron_sgd_plot(X,y))"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["[ 2.  3. 13.]\n"]},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYAAAAEGCAYAAABsLkJ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3xc9Znv8c+jZhVbslVsy5ZVXAgYA8Y2BAcuJBA2QDaB0Fl7QxJ2KQFCbhqQ3dzsctkUsskuScBALtlAqKEkZCkhhFCX6gI2Nt0Vy8Zykatky9Jz/5gzRhiNNJJndGbmfN+vl14anWnPYbC+Or9q7o6IiERPXtgFiIhIOBQAIiIRpQAQEYkoBYCISEQpAEREIqog7AKSUV1d7Y2NjWGXISKSVebNm7fe3WsS3Z8VAdDY2MjcuXPDLkNEJKuY2Yre7lcTkIhIRCkAREQiSgEgIhJRCgARkYhSAIiIRJQCQEQkohQAIiIRpQCQlNvd2cVdL61kd2dX2KWISC8UAJJy//PuBq64fxHPvL0+7FJEpBcKAEm59zbtAGD5hu0hVyIivVEASMo1t7YBsGLDjpArEZHeKAAk5Zpb2wFYuVEBIJLJFACSch9cAagJSCSTKQAk5Zo3xwJg1cY2Ors85GpEJBEFgKRUZ5ezdnM7lWVF7OrsYu2W9rBLEpEEFACSUuu37aSj0zlifCWgZiCRTJbTAfDYkve548WVYZcRKauD9v+Z46sAWKmRQCIZKyt2BBsId+d3c1fxxBvr2G/UUGY0VoZdUiTEO4CnNYygMN9YoZFAIhkrZ68AzIx/P+MQxo4o4eI75tOydWfYJUVCPADGVZYybkSprgBEMljOBgBARUkhN8yezua2Di65Y77WphkEza3tDBtSQHlxIfVVpZoNLJLBcjoAAA6oLecHXziIF5dt5JpH3wy7nJy3urWNMcNLAGiojF0BuGsoqEgmyvkAADh1Wh2zj6jnpqeX8siiNWGXk9OaW9sYM7wYgPqqMrbu3M2mHR0hVyUiPYlEAAB8728nM3XccL5970LebdkWdjk5q3mvKwDQUFCRTJX2ADCzfDNbYGYPBj83mdmLZvaOmd1tZkXprgFgSEE+18+aRlFBHhf+dh7bd+4ejLeNlLZdnWza0bEnABqrYwGgNYFEMtNgXAFcBrze7ecfA//h7hOBTcB5g1ADAGOGl/CLcw7l3ZZtXH7fQrVNp1h8CYixQQDUjSjFTKuCimSqtAaAmdUBnwX+X/CzAccC9wYPuQU4JZ017O3IidV86zMf48GFa/iv/1k+mG+d8+JDQONXAMWF+YwuL9ZIIJEMle4rgP8EvgPEx19WAa3uHm9/eQ8Y29MTzex8M5trZnNbWlpSWtRFx0zg+Mmj+MHDr/Py8o0pfe0o+yAAivccq6/UXACRTJW2ADCzvwXWufu8gTzf3W9y9xnuPqOmpibVtfHTMw+hbkQJF98+n3VbtWBZKqxubccMRpV/EAANVaWaDSySodJ5BXAk8HkzWw7cRazp51pguJnFl6CoA1ansYaEyosLmTN7OlvaO7jkjgV0aJLYPmtubWPUsGIK8z/436qhqoyWrTvZsUud7iKZJm0B4O5XunuduzcCZwN/dfdZwBPA6cHDzgUeSFcNfTmgtpwfnnoQLy3byDV/eiOsMnJG9zkAcQ1VGgkkkqnCmAdwOfANM3uHWJ/AzSHUsMcXDq3jizMb+NUzy3hooSaJ7YvucwDiGirLAFi+XgEgkmkGZTVQd38SeDK4vRQ4fDDeN1n//NnJLFq9me/c+yofGz2UiSOHhV1S1unqcpo3t/OZA0d/6Hj9nisAjQQSyTSRmQncm6KCPK6fNY3iwnwuvG0+2zRJrN82bN/Frt1dH7kCqCgpZHhpoeYCiGQgBUCgtiI2SWxpyzYuv1eTxPpr7zkA3TVUlqoPQCQDKQC6+cTEar5zwv48tGgNNz+7LOxyskpPcwDiGqrKdAUgkoEUAHu54OjxfObAUfzwkTd4aZkmiSWreXNsLsXYnq4AqkpZ3dqmobYiGUYBsBcz4ydnHEJ9ZSkX3zGfdVs0SSwZza1tlBblU1FS+JH76itL6exyVm9qC6EyEUlEAdCD8uLYTmLb2ndz8R3z9ZdrEuJDQGPLPX1YQ1VsKKhmBItkFgVAAh8bPYwfnXYQLy/fxI8e0SSxvjS3tlFb8dH2f+g2GUyLwolkFAVAL06eOpYvfaKRm59dxoMLm8MuJ6Otbm3vsf0fYOSwIRQX5qkjWCTDKAD68N2TDmB6wwi+c+9C3n5/a9jlZKT2jk7Wb9vZ4xBQiPWrNFSWqQlIJMMoAPpQVJDHdX83jdKifC64bR5b27W/7d7WBiOAEgUAxGYEa2tIkcyiAEjC6IpifnHONFZs2MF3NEnsI3qbAxAXnwym/3YimUMBkKSZE6q4/ISP8chra/nVM0vDLiejrG798FaQPWmoKqW9o4t1W3cOVlki0gcFQD/84/8az4lTRvPjP73JC0s3hF1OxmhujTUBjU4wCgigPj4UVB3BIhlDAdAPZsY1px9MQ1Upl9wxf0/bd9Q1t7ZRM2wIQwryEz6moTI2FFT9ACKZQwHQT8OKC7lx9nR27Ork4jvms2u3Jok1b/7oPgB7GzuihPw806JwIhlEATAAk0YN48enHcy8FZv4wcOvh11O6Jpb2xjbSwcwQGF+HmOHl7BcTUAiGUMBMECfO2QMXzmyid88t5wHXgllW+OM4O40t7YzpqL3KwCIdQRrNrBI5lAA7IMrT9qfwxpHcMV9i3hzbTQnibXu6KCto5PaPpqAILYonCaDiWQOBcA+KMyPTRIbWlzARbfNY0sEJ4l9MAS09yYgiF0BtO7oYHNb9P47iWQiBcA+GllezHV/N40VG3fw7XtejdxEp952AttbfFXQleoHEMkICoAUOLypkitP3J9HF7/PjU9Ha5JY/wIgNhR0ufoBRDKCAiBFzjuqic8eVMs1f3qD595ZH3Y5g6Z5cztFBXlUlRX1+dj6YC6AhoKKZAYFQIqYGT8+/WCaqsu49M4FrNkcjd2vVre2MTbBRjB7Ky0qoGbYEE0GE8kQCoAUGjqkgBv/fjrtHZ189fZoTBKL7QTWdwdwXENlqZaDEMkQCoAUmzhyGD854xAWrGzl6oeWhF1O2jW3tiU1ByCuvqpUTUAiGUIBkAYnHVTLPxzVxK3Pr+Ccm17g0cVr6ezKvdFBu3bHVvdMpgM4rrGqjDWb22nv6ExjZSKSjIKwC8hVV5y4PzXDhnDr8yu44LfzGDu8hC/ObOCsw8YxvLTvDtNs8P6Wdtx7XwZ6b/GRQKs27mDSqGHpKk1EkqArgDQpyM/jgmMm8NS3P8kNs6cxrrKEHz7yBkf88HGuvH8hb6zdEnaJ+6w/Q0Dj6vesCqpmIJGw6QogzQry8zhhSi0nTKnl9TVbuOW55dw/fzV3vrSKI8ZX8qVPNPHpA0ZSkJ99WdwcjHSq7U8ncHxfAPUDiIQu+37rZLEDasv50WkH88KVx3HFifuzamMbF942j2N+8iR3v7wy7PL6Lb4RTH86gUeUFjJsSIEWhRPJAAqAEIwoK+LCPc1D06mtKOby+xbxL39cnFWdxatb26gsK6KkKPFGMHszs9gG8boCEAmdAiBEseah0dx9wUz+8X/FlpY+/9a5bN+5O+zSktLfOQBxjVVl6gMQyQAKgAyQn2f802cnc/UpU3jyrRbOuOH5rJhJ3N85AHH1VaW8t2lHVl3tiOQiBUAGmX1EA7/+0mGs3LiDU677H15bvTnskhJyd1Zv6nsryJ40VJbS0el7RhGJSDgUABnmmP1quPeimRTk5XHGDc/zlyXvh11Sj7a072b7rs5+zQGIq6/SonAimUABkIH2H13O7y/+BJNGDeUffzuXXz+7LOP2GRjIHIC4PUNB1Q8gEqq0BYCZFZvZS2b2qpktNrN/DY43mdmLZvaOmd1tZrkxLTbFRg4r5u7zZ/I3k0dx1YNL+P4fF7O7M3MWl/sgAPrfCVxbXkxRQR4rNmooqEiY0nkFsBM41t0PAaYCJ5jZEcCPgf9w94nAJuC8NNaQ1UqK8pkzazrnHz2eW59fwT/cOpdtGTJCqHlzbA7AQJqA8vKMcSNKWLFeVwAiYUpbAHjMtuDHwuDLgWOBe4PjtwCnpKuGXJCXZ3z3pAP4ty9M4Zm313P6nOdYv21nyl6/dccurn/ynX4HS3NrG4X5RvXQIQN634aqMs0FEAlZWvsAzCzfzF4B1gGPAe8Cre4e/23zHjA2wXPPN7O5Zja3paUlnWVmhVkfj40QWrZ+O5fesSAlzUGdXc7X7nqFa/70Jre/sKJfz21ubWN0RTF5eX1vBNOT+spSVm7YnnF9GyJRktYAcPdOd58K1AGHA/v347k3ufsMd59RU1OTthqzyTH71XD1KVN4fukG/v3Pb+3z6137+Ns8/VYLw0sLuXvuqn79Mh7oHIC4hqpStu/qZMP2XQN+DRHZN4MyCsjdW4EngJnAcDOLL0JXB6wejBpyxRkzxnHO4fXc8NS7PLp47YBf54k31vHzx9/m9Ol1fPfEA1jasp25KzYl/fzm1vYBtf/HxZeF1kggkfCkcxRQjZkND26XAMcDrxMLgtODh50LPJCuGnLV9z83mYPrKvjW715lacu2vp+wl5UbdnDZXQuYXFvO1adM4bMH1zJ0SAF3vbQqqefv7uxi7Zb2AQ0BjftgKKhGAomEJZ1XALXAE2a2EHgZeMzdHwQuB75hZu8AVcDNaawhJxUX5nP9rGkU5BsX3TafHbuS78Bt7+jkwtvmAXDD7OkUF+ZTNqSAzx1Sy8OL1rClvaPP11i3dSedXb5PAVA3ogQzXQGIhCmdo4AWuvuh7n6wu09x96uC40vd/XB3n+juZ7h76oa0REjdiFKuPftQ3lq3lSvvX5RU+727870/vMaSNVv4z7On7pmRC3DWYfW0dXTy36829/k6+zIHIG5IQT5jKko0G1gkRJoJnMWO3q+Gb3x6Px54pZlbn+97FM9dL6/innnv8bXjJnHs/qM+dN8hdRXsP3oYd7/cdzPQ6iAA9qUPAGIjgdQEJBIeBUCWu/hTEzlu/5H83weXMG/FxoSPe3VVK99/YDFH71fDZcdN+sj9ZsZZh41j4XubWdLc+3aV8Y1gavcxABqqSnUFIBIiBUCWy8szfnbWVMYML+Grt8+nZetHW9Q2bt/FV2+fT82wIVx71lTyE4zdP2XqWIry8/jd3N6vAppb26goKWTokH3bUbShqoz123ZlzOxmkahRAOSAipJC5syeRuuODi69c/6HJol1djmX3bWAlm07mTN7GiPKEi+9NKKsiM9MGc3vF6ymvaMz4ePWbB7YMtB7+2AoqJqBRMKgAMgRB46p4AdfOIgXlm7kJ39+c8/xa//yFs+8vZ6rPn8gB9cN7/N1zj5sHJvbOnqdY7C6tZ0xFQPvAI6rrwyWhdZIIJFQ7Ns1vGSU06bXMX/lJm58aimHjhtOYX4eP//rO5w5o46zD69P6jVmjq9iXGUJd7+8ipOn9rhKB82tbcxoGLHP9e65AlA/gEgoFAA55v98bjKvNW/hW/csJM9gythyrjp5StLPz8szzpw+jp8+9hYrNmzfM2ErbtvO3Wxu60hJE9Cw4kIqy4o0F0AkJGoCyjFDCvKZM2saRQV5mBlzZsUme/XH6TPqyDO4Z+57H7lvTQrmAHTXVF3Gi0s30LYrcZ+DiKRHnwFgZkeaWVlwe7aZ/czMGtJfmgzUmOElPHDxkTxw8ZGMqyzt+wl7qa0o4Zj9arhn3qqPrDqaqjkAcV87bhLLNmznn36f3GQ2EUmdZK4A5gA7zOwQ4JvElnS+Na1VyT4bV1lKY3VZ3w9M4KzD6nl/y06efvvDS3HH5wCkogkIYiuc/u9P78f9C1Zz24srU/KaIpKcZAJgt8f+NDsZ+KW7XwcMS29ZErbjDhhJ9dCijywQ19zaRn6eMXLYwDaC6ckln5rIsfuP5Kr/Xsz8lcmvSCoi+yaZANhqZlcCs4GHzCyP2O5eksMK8/M4bXodj7+xjnVb2/ccb25tY3R5MQX5qes+yssz/uPMqYyuKObi2+ezIYU7nolIYsn8Kz6L2P6+57n7WmJr+P8krVVJRjhzxjg6u5z753+wZcPq1raUdQB3V1FayJxZ09m4fReX3pmaHc9EpHdJXQEA17r7M2a2H7EN3u9Mb1mSCSbUDOXwxkrufvmD3cKaUzQLuCdTxlZw9SlTeO7dDfz0sX3f8UxEepdMADwNDDGzscCfgb8HfpPOoiRznHXYOJat385LyzbS1eWs3bxvG8H0Jb7j2Zwn3+XP+7DjmYj0LZkAMHffAZwKXO/uZwDJzyySrHbSQbUMG1LA3XNXsX7bTjo6PSXLQPQmvuPZN3/3KsvWa50gkXRJKgDMbCYwC3ioH8+THFBSlM/np47h4UVreH3tViB1Q0AT6b7j2YW/ndevHc9EJHnJ/CL/OnAl8Ht3X2xm44nt6ysRcfZh9bR3dHHDk+8C6Q8AGNiOZyLSP30GgLs/5e6fB64zs6HBlo5fG4TaJENMGVvOAbXlPL90AzA4AQD93/FMRPonmaUgDjKzBcBiYImZzTOzA9NfmmQKM+Psw8YBMHRIAeXFg7eGYHzHs6sfWsK8FZokJpJKyTQB3Qh8w90b3L2e2HIQv0pvWZJpTpk6lqKCPMYML8as5x3F0iG+41ltRQlfvX1ejzueicjAJBMAZe6+p83f3Z8EBr7IjGSlitJCvnbsRE45tOc9AtL63iWF3DB7eo87nonIwCUTAEvN7Htm1hh8/TOwNN2FSea55NhJfPWTE0N578ljynvc8UxEBi6ZAPgKUAPcD9wHVANfTmdRIj05bXodsz5ez41PLeVPr60JuxyRrNdnb567bwI+NOrHzO4mtkaQyKDqvuPZpFHDmFAzNOySRLLWQCd0zUxpFSJJGlIQmyRWVJDHRbdpkpjIvtCMXsk6Y4eX8POzD+Wdddu44j5NEhMZqIRNQGY2LdFdaD8ACdlRk6r55t98jJ88+iaH1g/ny0c2hV2SSNbprQ/gp73c90aqCxHpr4uOmcCCla3820Ovc9DYCmY0VoZdkkhWsWy4fJ4xY4bPnTs37DIkA21u6+DkXz5LW0cn/33pUYwclt6VSkWyiZnNc/cZie5XH4BktYqSQubMns7mtg4uvUM7iYn0hwJAst4BteX88NSDeHHZRq55VJPERJKlAJCc8IVD6/jizAZuenopjyzSJDGRZAxkFBAA7j4/9eWIDNw/f3Yyi1Zv5lv3vMqkUcOYOFKTxER6k7AT2Mx62/TF3f3Y9JT0UeoElmSt2dzG3/78WeqrSrn/ok8M6sqlIpmmr07ghFcA7v6p9JQkkj61FSV8/fj9+N4fXuOFpRuZOaEq7JJEMlZSfQBmNsXMzjSzL8a/knjOODN7wsyWmNliM7ssOF5pZo+Z2dvB9xH7ehIi3Z0xvY7qoUXc8NS7YZciktGS2RHs+8Avgq9PAdcAn0/itXcD33T3ycARwMVmNhm4Anjc3ScBjwc/i6RMcWE+Xz6yiafeamFx8+awyxHJWMlcAZwOHAesdfcvA4cAFX09yd3XxDuK3X0r8DowFjgZuCV42C3AKQOoW6RXs49oYOiQAm58SltXiCSSTAC0uXsXsNvMyoF1wLj+vImZNQKHAi8Co9w9Pk5vLTAqwXPON7O5Zja3paWlP28nQkVJIbM+Xs+DC5tZuWFH2OWIZKRkAmCumQ0ntg/wPGA+8Hyyb2BmQ4ltJPN1d9/S/T6PDUHqcRiSu9/k7jPcfUZNTU2ybyeyx1eOaqIgL4+bnlFfgEhP+gwAd/+qu7e6+w3A8cC5QVNQn8yskNgv/9vd/f7g8PtmVhvcX0vsikIk5UaVF3PqtLHcM/c9bSYv0oNkOoEfj9929+XuvrD7sV6eZ8DNwOvu/rNud/0RODe4fS7wQP9KFkne+UePZ1dnF795blnYpYhknIQBYGbFZlYJVJvZiGD4ZmXQnj82idc+Evh74FgzeyX4Ogn4EXC8mb0NfDr4WSQtxtcM5YQDR/Pb51ewtb0j7HJEMkpv+wFcAHwdGEOs3T9uC/DLvl7Y3Z8ltnlMT45LtkCRfXXhMRN45LW13PnSSs4/ekLY5YhkjIRXAO5+rbs3Ad9y96ZuX4e4e58BIJIpDhk3nE9MqOLmZ5exc3dn2OWIZIxkRgHdaGZfM7N7g69Lgs5dkaxx0Scn8P6WnfxhweqwSxHJGMkEwPXA9OB7/PacdBYlkmpHTaxmythybnx6KZ1dmb8Lnshg6K0TON4/cJi7n+vufw2+vgwcNjjliaSGmXHhMRNY2rKdx5asDbsckYzQ2xXAS8H3TjPb03NmZuMBNaRK1jlxSi0NVaXMefJdsmEvbJF06y0A4iN4vgU8YWZPmtmTwF+Bb6a7MJFUy88zzj96PK++t5nnl24IuxyR0PUWADVm9g1gKnAjsV/8fyW2JMShg1CbSMqdNq2O6qFDmPOklocQ6S0A8oGhwDBi8wUs+CoIjolkneLCfM47qoln3l7Pa6u1VLREW28Twda4+1WDVonIIJl1RD3XP/EONzz1Lr/8u163vhbJacn0AYjklPLiQmYd0cDDi9awYsP2sMsRCU1vAaDlGiRnfeXIRgry87jpaW0YI9HV21IQGwezEJHBNLK8mNOm1XHPvPdYv01LRUs0JbUpvEguOmNGHbt2d7FgZWvYpYiEQgEgkTWheigAy9ZvC7kSkXAoACSyKkoLqSwrYtl67Rks0aQAkEhrrCpl+XqNBJJoUgBIpDVVD2WZAkAiSgEgkdZUXcraLe3s2LU77FJEBp0CQCKtKegIXq5+AIkgBYBEWmN1KQDLNSNYIkgBIJHWWFUGoH4AiSQFgERa2ZACRpUPUQBIJCkAJPIaq8o0FFQiSQEgkddUXaY+AIkkBYBEXlN1Geu37WJLe0fYpYgMKgWARF5jdawjWM1AEjUKAIm88dUaCSTRpACQyBtXWYqZAkCiRwEgkVdcmM+YihI1AUnkKABEgPE1ZboCkMhRAIgQmwuwbP123D3sUkQGjQJAhNhIoC3tu9m0Q0NBJToUACJ0Hwmk7SElOhQAInwwF0DbQ0qUKABEgLoRJRTkma4AJFIUACJAYX4e4ypLtTGMREraAsDMfm1m68zstW7HKs3sMTN7O/g+Il3vL9JfjVWlGgoqkZLOK4DfACfsdewK4HF3nwQ8HvwskhGaqoeyfIOGgkp0pC0A3P1pYONeh08Gbglu3wKckq73F+mvpupSduzqZN3WnWGXIjIoBrsPYJS7rwlurwVGJXqgmZ1vZnPNbG5LS8vgVCeRFh8JtLRFzUASDaF1AnvsOjvhtba73+TuM9x9Rk1NzSBWJlHVFF8WWpvDSEQMdgC8b2a1AMH3dYP8/iIJjakooaggT4vCSWQMdgD8ETg3uH0u8MAgv79IQnl5RkNlKUsVABIR6RwGeifwPPAxM3vPzM4DfgQcb2ZvA58OfhbJGE3V2iBeoqMgXS/s7uckuOu4dL2nyL5qqi7jybda6Oxy8vMs7HJE0kozgUW6aaouY9fuLppb28IuRSTtFAAi3TRqJJBEiAJApJsmbRAvEaIAEOlm5LAhlBblKwAkEhQAIt2YGY1VGgkk0aAAENlLU7U2iJdoUACI7KWpuoxVm9ro6OwKuxSRtFIAiOylsbqMzi7nvU0aCiq5TQEgspcmbRAvEaEAENlLkzaIl4hQAIjsZURpIeXFBboCkJynABDZi5nRVDNUG8RLzlMAiPSgSRvESwQoAER60FhdRvPmNto7OsMuRSRtFAAiPWiqLsMdVm5UM5DkLgWASA+0KJxEgQJApAeNCgCJAAWASA/KiwupHlqkReEkpykARBJorCrTBvGS0xQAIglog3jJdQoAkQQaq8tYt3Un23fuDrsUkbRQAIgkoJFAkusUACIJNGmDeMlxCgCRBBqrgiuAFgWA5CYFgEgCJUX5jC4vZpmuACRHKQBEeqGRQJLLFAAivWjUBvGSwxQAIr0YX13Gph0dbN7REXYpIimnABDpxZ41gdQPIDlIASDSi6bqUkAbxEtuUgCI9GJcZSl5pg3iJTcpAER6MaQgn7EjStQRLDlJASDSh8YqDQWV3KQAEOnD+GAugLuHXYpISikARPrQWF3G1p27Wb9tV9iliKSUAkCkD1oUTnJVKAFgZieY2Ztm9o6ZXRFGDSLJ0rLQkqsGPQDMLB+4DjgRmAycY2aTB7sOkWSNHV5CQZ4pACTnFITwnocD77j7UgAzuws4GVgSQi0ifSrIz6O+qpTbXljBX5a8H3Y5EjE3n3sY9VWlaXntMAJgLLCq28/vAR/f+0Fmdj5wPkB9ff3gVCaSwCWfmshfXtcvfxl8RQXpa6gJIwCS4u43ATcBzJgxQ+PvJFSnTqvj1Gl1YZchklJhdAKvBsZ1+7kuOCYiIoMojAB4GZhkZk1mVgScDfwxhDpERCJt0JuA3H23mV0CPArkA79298WDXYeISNSF0gfg7g8DD4fx3iIiEqOZwCIiEaUAEBGJKAWAiEhEKQBERCLKsmGNczNrAVYM8OnVwPoUlpMJcu2cdD6ZL9fOKdfOB3o+pwZ3r0n0hKwIgH1hZnPdfUbYdaRSrp2Tzifz5do55dr5wMDOSU1AIiIRpQAQEYmoKATATWEXkAa5dk46n8yXa+eUa+cDAzinnO8DEBGRnkXhCkBERHqgABARiaicDoBc23zezJab2SIze8XM5oZdz0CY2a/NbJ2ZvdbtWKWZPWZmbwffR4RZY38kOJ9/MbPVwef0ipmdFGaN/WFm48zsCTNbYmaLzeyy4Hg2f0aJzikrPyczKzazl8zs1eB8/jU43mRmLwa/7+4Oltvv/bVytQ8g2Hz+LeB4YttOvgyc4+5Zu/ewmS0HZrh71k5gMbOjgW3Are4+JTh2DbDR3X8UBPUId788zDqTleB8/gXY5u7/HmZtA2FmtUCtu883s2HAPOAU4Etk72eU6JzOJAs/JzMzoMzdt5lZIfAscBnwDeB+d7/LzG4AXnX3Ob29Vi5fAezZfN7ddwHxzeclRO7+NLBxr8MnA7cEt28h9o8zKyQ4n6zl7mvcfbs0n88AAAONSURBVH5weyvwOrF9vLP5M0p0TlnJY7YFPxYGXw4cC9wbHE/qM8rlAOhp8/ms/dADDvzZzOaZ2flhF5NCo9x9TXB7LTAqzGJS5BIzWxg0EWVNc0l3ZtYIHAq8SI58RnudE2Tp52Rm+Wb2CrAOeAx4F2h1993BQ5L6fZfLAZCLjnL3acCJwMVB80NO8VibZLa3S84BJgBTgTXAT8Mtp//MbChwH/B1d9/S/b5s/Yx6OKes/ZzcvdPdpxLbU/1wYP+BvE4uB0DObT7v7quD7+uA3xP74HPB+0E7bby9dl3I9ewTd38/+AfaBfyKLPucgnbl+4Db3f3+4HBWf0Y9nVO2f04A7t4KPAHMBIabWXyXx6R+3+VyAOTU5vNmVhZ0YGFmZcDfAK/1/qys8Ufg3OD2ucADIdayz+K/KANfIIs+p6CD8WbgdXf/Wbe7svYzSnRO2fo5mVmNmQ0PbpcQG+jyOrEgOD14WFKfUc6OAgIIhnX9Jx9sPv9vIZc0YGY2nthf/RDby/mObDwfM7sT+CSxpWvfB74P/AH4HVBPbNnvM909KzpWE5zPJ4k1KziwHLigW/t5RjOzo4BngEVAV3D4u8TazLP1M0p0TueQhZ+TmR1MrJM3n9gf8b9z96uC3xF3AZXAAmC2u+/s9bVyOQBERCSxXG4CEhGRXigAREQiSgEgIhJRCgARkYhSAIiIRJQCQAQws85uq0K+ksrVY82ssftqoSKZoqDvh4hEQlswtV4kMnQFINKLYA+Ga4J9GF4ys4nB8UYz+2uwkNjjZlYfHB9lZr8P1mp/1cw+EbxUvpn9Kli//c/BDE6RUCkARGJK9moCOqvbfZvd/SDgl8RmlgP8ArjF3Q8Gbgd+Hhz/OfCUux8CTAMWB8cnAde5+4FAK3Bams9HpE+aCSwCmNk2dx/aw/HlwLHuvjRYUGytu1eZ2Xpim4x0BMfXuHu1mbUAdd2n4AdLED/m7pOCny8HCt396vSfmUhiugIQ6ZsnuN0f3ddk6UT9b5IBFAAifTur2/fng9vPEVthFmAWscXGAB4HLoI9m3ZUDFaRIv2lv0JEYkqCHZbi/uTu8aGgI8xsIbG/4s8Jjl0K/JeZfRtoAb4cHL8MuMnMziP2l/5FxDYbEck46gMQ6UXQBzDD3deHXYtIqqkJSEQkonQFICISUboCEBGJKAWAiEhEKQBERCJKASAiElEKABGRiPr/UMPwVUDAZ0sAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"mFy14PRRKkhC"},"source":["### 1.2 Perceptron Algorithm\n","\n","As shown in the previous example, here is the summary of the algorithm: \n","\n","  1. Initialize $w$ (often set it to all zero).\n","  2. Update: $w \\leftarrow w + \\eta y_n x_n$ if $x_n$ is predicted wrong, where $\\eta$ is the learning rate. \n","\n","Note that this is an **online** learning algorithm. One important theoretical results is that if the training set is linearly separable, the perceptron algorithm converges in a finite number of iterations.  \n","\n","One possible derivation is to consider the *hinge loss* of misclassification: \n","$$ l(y_n, x_n; w) = (- y_n w^T x_n)_+ $$\n","As we are minimizing the loss, we would update by **gradient descent**: \n","$$ w \\leftarrow w - \\eta \\nabla_w l(y_n, x_n; w), $$\n","where if we have straight-through simplification for computing the hinge-loss gradient, we can have the updating rule: \n","$$ w \\leftarrow w + \\eta y_n x_n, \\mbox{ if wrong prediction}. $$\n","\n","**<font color=blue>[Note]</font>** Compare with the updating equation for deriving **logistic regression (LR)** (Note the different outspace: Here, $y\\in\\{-1, 1\\}$; while in LR, $y\\in\\{0, 1\\}$. "]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"I1iRjLuW8wL3","executionInfo":{"status":"ok","timestamp":1632782436438,"user_tz":300,"elapsed":322,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"a5d0b711-c4bd-4868-dcfd-afb3ab0e28ae"},"source":["from sklearn.linear_model import Perceptron\n","\n","clf = Perceptron(fit_intercept=False, random_state=2)#(tol=1e-5, fit_intercept=False)#, random_state=0)\n","# Not sure why the results change for different random_state as supposed to converge easily in this case...\n","\n","clf.fit(X, y)\n","Perceptron()\n","\n","clf.score(X, y)\n","\n","print(clf.coef_)\n","\n","print(X)\n","\n","print(clf.predict(X))\n","print(y)\n"],"execution_count":4,"outputs":[{"output_type":"stream","name":"stdout","text":["[[ 2.  3. 13.]]\n","[[-2  4 -1]\n"," [ 4  1 -1]\n"," [ 1  6 -1]\n"," [ 2  4 -1]\n"," [ 6  2 -1]]\n","[-1 -1  1  1  1]\n","[-1 -1  1  1  1]\n"]}]},{"cell_type":"markdown","metadata":{"id":"3e5tNaqGKq3L"},"source":["\n","### 1.3 An Application "]},{"cell_type":"markdown","metadata":{"id":"zDZygSpqHqsh"},"source":["#### Background\n","* `Heart` dataset contains a binary outcome `HD` for 303 patients who presented with chest pain. An outcome value of `Yes` indicates the presence of hear disease based on an angiographic test, while `No` means no heart disease. \n","\n","* There are 13 predictors including `Age`, `Sex`, `Chol` (a cholesterol measurement), `Thal`( Thallium stress test) and other heart and lung function measurements. \n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"yZ_XqXdcKvz7","executionInfo":{"status":"ok","timestamp":1632783408065,"user_tz":300,"elapsed":694,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"872db53c-2b77-4599-af2a-1247545eb892"},"source":["Heart = pd.read_csv('https://raw.githubusercontent.com/XiaomengYan/MachineLearning_dataset/main/Heart.csv').drop('Unnamed: 0', axis=1).dropna()\n","Heart.info()"],"execution_count":5,"outputs":[{"output_type":"stream","name":"stdout","text":["<class 'pandas.core.frame.DataFrame'>\n","Int64Index: 297 entries, 0 to 301\n","Data columns (total 14 columns):\n"," #   Column     Non-Null Count  Dtype  \n","---  ------     --------------  -----  \n"," 0   Age        297 non-null    int64  \n"," 1   Sex        297 non-null    int64  \n"," 2   ChestPain  297 non-null    object \n"," 3   RestBP     297 non-null    int64  \n"," 4   Chol       297 non-null    int64  \n"," 5   Fbs        297 non-null    int64  \n"," 6   RestECG    297 non-null    int64  \n"," 7   MaxHR      297 non-null    int64  \n"," 8   ExAng      297 non-null    int64  \n"," 9   Oldpeak    297 non-null    float64\n"," 10  Slope      297 non-null    int64  \n"," 11  Ca         297 non-null    float64\n"," 12  Thal       297 non-null    object \n"," 13  AHD        297 non-null    object \n","dtypes: float64(2), int64(9), object(3)\n","memory usage: 34.8+ KB\n"]}]},{"cell_type":"code","metadata":{"id":"O8Yq_fFqji8F","executionInfo":{"status":"ok","timestamp":1632783410909,"user_tz":300,"elapsed":341,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}}},"source":["Heart.ChestPain = pd.factorize(Heart.ChestPain)[0]\n","Heart.Thal = pd.factorize(Heart.Thal)[0]\n","X2 = Heart.drop('AHD', axis=1) # explanatory variables\n","y2 = pd.factorize(Heart.AHD)[0] # response variables AHD"],"execution_count":6,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"IbFFfNSjjlNk","executionInfo":{"status":"ok","timestamp":1632783413889,"user_tz":300,"elapsed":551,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"44999a62-c938-48f6-926b-c240893764fe"},"source":["clf = Perceptron(fit_intercept=False, random_state=2)\n","clf.fit(X2,y2)\n","clf.score(X2,y2)"],"execution_count":7,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.6666666666666666"]},"metadata":{},"execution_count":7}]},{"cell_type":"markdown","metadata":{"id":"J4FBmcX0I-Wo"},"source":["## 2 AdaBoost\n","\n","**Reminder** Ensemble learning: Ensemble learning methods are meta-algorithms that combine several *weak* learners (homogenous or heterogeneous) into a single predictive model to improve prediction performance. Ensemble methods can decrease variance using bagging, reduce bias by boosting, or improve predictions using stacking.\n","\n","\n","\n","**<font color=blue>[Note]</font>** **Boosting**, **bagging**, and **stacking**\n","\n","* *Boosting*: an ensemble method for improving the model predictions. The idea is to train weak learners sequentially, each trying to correct its predecessor. When an input is misclassified, its contribution to the training loss is increased so that it's more likely to be classified correctly. Models in boosting are related to the previous derived models. \n","* *Bagging*: (Bootstrap Aggregating) a machine learning ensemble strategy to improve performance, which uses bootstrap to get samples from the original training data, builds the models on each sampled dataset, and aggregates the results of all the models. Bagging can be parallelized as models are trained by different bootstrapped samples. \n","* *Stacking*: a strategy that combines multiple base models' predictions into a new data set, which serves as the input data for another model to predict. (similar as deep network architectures)\n","\n","\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":432},"id":"RmDwYGCtNhDJ","executionInfo":{"status":"ok","timestamp":1631919776787,"user_tz":300,"elapsed":11,"user":{"displayName":"Ziyu Xiang","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"09056639542636885950"}},"outputId":"41078b12-bb7c-4dc4-867f-7db8d66fa9ad"},"source":["Image(url= \"https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1542651255/image_2_pu8tu6.png\")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<img src=\"https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1542651255/image_2_pu8tu6.png\"/>"],"text/plain":["<IPython.core.display.Image object>"]},"metadata":{},"execution_count":9}]},{"cell_type":"markdown","metadata":{"id":"NnDbAfJOkjsk"},"source":["\n","\n","### 2.1 Algorithm\n","\n","\n","Take classification to illustrate the algorithmic procedure for AdaBoost (Adaptive Boosting). \n","\n","1. Initially, AdaBoost selects a training subset randomly.\n","2. It iteratively trains the weak learner $h_t$ by selecting the training set based on the accurate prediction of the last training.\n","3. It assigns the weight to the trained classifier in each iteration according to the accuracy of the classifier. The more accurate classifier will get high weight: \n","$$ \\alpha_t = \\frac{1}{2}\\ln{(\\frac{1}{\\epsilon_t}-1)},$$\n","where $\\epsilon_t = \\sum_n \\rho_n^t \\mathbb{1}(y_n \\neq h_t(x_n))$ denotes the weighted training classification error. \n","4. It assigns the higher weight to wrong classified observations so that in the next iteration these observations will get the high probability for classification: \n","$$ \\rho^{t+1}_n = \\frac{\\rho^{t}_n e^{-y_n \\alpha_t h_t(x_n)}}{\\sum_i \\rho^{t}_i e^{-y_i \\alpha_t h_t(x_i)}}. $$\n","5. This process iterate until the complete training data fits without any error or until reached to the specified maximum number of estimators.\n","6. Output the final prediction $f(x_n) = \\sum_t \\alpha_t h_t(x_n)$. \n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":321},"id":"lpXd-ix3Od3j","executionInfo":{"status":"ok","timestamp":1632672190070,"user_tz":300,"elapsed":92,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"2794cb04-bbdb-4090-9788-64f54d2f96cd"},"source":["Image(url= \"https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1542651255/image_3_nwa5zf.png\")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<img src=\"https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1542651255/image_3_nwa5zf.png\"/>"],"text/plain":["<IPython.core.display.Image object>"]},"metadata":{},"execution_count":9}]},{"cell_type":"markdown","metadata":{"id":"6UMo8YxQg3t5"},"source":["### 2.2 AdaBoost with the `Heart` dataset\n","\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"KyMbSRqZMR7n","executionInfo":{"status":"ok","timestamp":1632783423829,"user_tz":300,"elapsed":231,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"b89e7c8d-1a08-48e0-c9c5-ad6326d626a9"},"source":["clf = AdaBoostClassifier(n_estimators=50, learning_rate=1)\n","clf.fit(X2,y2)"],"execution_count":8,"outputs":[{"output_type":"execute_result","data":{"text/plain":["AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None, learning_rate=1,\n","                   n_estimators=50, random_state=None)"]},"metadata":{},"execution_count":8}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"hA4Wg9qWMYfJ","executionInfo":{"status":"ok","timestamp":1632672195889,"user_tz":300,"elapsed":96,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"2415fe55-8806-494f-98aa-9aa5e5edb1b8"},"source":["clf.score(X2,y2) #Return the mean accuracy on the given data and labels."],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.9259259259259259"]},"metadata":{},"execution_count":11}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"vFU4hTnwM0GV","executionInfo":{"status":"ok","timestamp":1632783481799,"user_tz":300,"elapsed":292,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"0a2bd5b4-0db5-46ef-c88e-89637c1ecdd8"},"source":["# Create adaboost classifer object with Perceptron\n","abc = AdaBoostClassifier(n_estimators=50, algorithm='SAMME', base_estimator=Perceptron(), learning_rate=1) #algorithm='SAMME.R', \n","abc.fit(X2,y2)\n","abc.score(X2,y2)\n"],"execution_count":9,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.7037037037037037"]},"metadata":{},"execution_count":9}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"5DR1xoYoPTIh","executionInfo":{"status":"ok","timestamp":1632783920453,"user_tz":300,"elapsed":384,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"5a2faae8-e352-40c1-eea9-6d3f6092b519"},"source":["from sklearn import datasets\n","\n","# Load data\n","iris = datasets.load_iris()\n","X = iris.data\n","y = iris.target\n","\n","# Split dataset into training set and test set\n","X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3) # 70% training and 30% test\n","\n","# Create adaboost classifer object\n","abc = AdaBoostClassifier(n_estimators=50, learning_rate=1)\n","\n","# Train Adaboost Classifer\n","model = abc.fit(X_train, y_train)\n","\n","#Predict the response for test dataset\n","y_pred = model.predict(X_test)\n","\n","# Model Accuracy, how often is the classifier correct?\n","print(\"Accuracy:\", accuracy_score(y_test, y_pred))"],"execution_count":13,"outputs":[{"output_type":"stream","name":"stdout","text":["Accuracy: 0.9111111111111111\n"]}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"JZt9oR7PlBHl","executionInfo":{"status":"ok","timestamp":1632783925219,"user_tz":300,"elapsed":673,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"a08fb9d8-b529-4a64-edc6-a3b9f26fafea"},"source":["# Create adaboost classifer object\n","abc = AdaBoostClassifier(n_estimators=50, algorithm='SAMME', base_estimator=Perceptron(), learning_rate=1) #algorithm='SAMME.R', \n","\n","# Train Adaboost Classifer\n","model = abc.fit(X_train, y_train)\n","\n","#Predict the response for test dataset\n","y_pred = model.predict(X_test)\n","print(y_pred)\n","\n","# Model Accuracy, how often is the classifier correct?\n","print(\"Accuracy:\", accuracy_score(y_test, y_pred))"],"execution_count":14,"outputs":[{"output_type":"stream","name":"stdout","text":["[1 0 2 1 0 0 0 2 1 2 2 0 1 0 1 1 2 2 0 2 2 0 0 0 2 2 2 2 0 0 2 1 1 1 0 1 1\n"," 0 1 0 2 2 0 2 1]\n","Accuracy: 0.9111111111111111\n"]}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"hIhxRmqVn-Gp","executionInfo":{"status":"ok","timestamp":1632783937259,"user_tz":300,"elapsed":154,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"906f0bb1-a5ec-4385-a367-0ae7fc423f3b"},"source":["# Create Perceptron classifer object\n","clf = Perceptron(tol=1e-3, random_state=0)\n","\n","# Train Perceptron Classifer\n","model = clf.fit(X_train, y_train)\n","\n","#Predict the response for test dataset\n","y_pred = model.predict(X_test)\n","print(y_pred)\n","\n","# Model Accuracy, how often is the classifier correct?\n","print(\"Accuracy:\", accuracy_score(y_test, y_pred))"],"execution_count":16,"outputs":[{"output_type":"stream","name":"stdout","text":["[0 0 2 0 0 0 0 2 0 2 2 0 0 0 2 0 2 2 0 2 2 0 0 0 2 2 2 2 0 0 2 0 2 0 0 0 2\n"," 0 0 0 2 2 0 2 0]\n","Accuracy: 0.6666666666666666\n"]}]},{"cell_type":"code","metadata":{"id":"AVCKuLtcgewd","executionInfo":{"status":"ok","timestamp":1632783963589,"user_tz":300,"elapsed":538,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}}},"source":["Perceptron??"],"execution_count":17,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"luWvkSDo-HxG"},"source":["**SAMME** (Stagewise Additive Modeling using a Multi-class Exponential loss function): https://web.stanford.edu/~hastie/Papers/samme.pdf\n","\n","http://article.sapub.org/10.5923.j.ajis.20130302.02.html"]},{"cell_type":"markdown","metadata":{"id":"zXYjeWDeyZcv"},"source":["\n","### 2.3 Math derivations\n","\n","We will just make hand-waving explanation here: If we have seens that AdaBoost assumes the prediction model: \n","$$f(x) = \\sum_t \\alpha_t h_t(x),$$\n","and for each iteration, the model updates: \n","$$f_{t}(x) = f_{t-1}(x) + \\alpha_t h_t(x), $$\n","the model training is to minimize the training loss with the exponential loss function (upperbounding the zero-one loss): \n","$$ \\min_{\\alpha, h}\\sum_n e^{-y_n f_t(x_n)}.  $$\n","\n","The choice of the expential loss also provides the theoretical proof that AdaBoost will achieve infinitely small empricial risk asymptotically. Another interesting theoretical result is that AdaBoost somehow is immune to overfitting. "]},{"cell_type":"markdown","metadata":{"id":"hquPBahNBLU3"},"source":["## 3 ANNs\n","\n"]},{"cell_type":"markdown","metadata":{"id":"w-SDYByDBLU3"},"source":["### 3.1 Basics\n","\n","We have introduced the perceptron algorithm, which is the basic unit for most of the artificial neural networks, including many popular deep network models. Here we focus on the **multilayer perceptron (MLP)**. MLP is a feed-forward ANN model that maps sets of input data onto a set of appropriate outputs. MLP often consists of multiple layers and each layer is fully connected to the following one. The nodes of the layers are neurons using nonlinear activation functions, except for the nodes of the input layer. There can be one or more non-linear hidden layers between the input and the output layer. The training is through backpropagation (chain rule to compute gradients for involved network parameters) based on the corresponding loss function(s). \n","\n","MLP can be mathematically represented as: \n","$$ h^l = g (W^l h^{l-1}), $$\n","where $W^l$ denotes the weighting parameters at the $l$th layer and $g(\\cdot)$ is the activation function. The hidden variables $h^l$ can be considered as the output for each hidden layer, taking the previous layer's output $h^{l-1}$ as the input. $h^0 = X$ as the input feature vector and the output of the output layer is often used to predict the outcome of interest: \n","$$\\sum_i W^{L}_i h^{L-1}_i \\rightarrow \\hat{y}$$ for regression or \n","$$\\sigma(\\sum_i W^{L}_i h^{L-1}_i) \\rightarrow \\hat{y}$$ for classification. \n","\n","**<font color=blue>[Note]</font>** All the deep network architectures, convolutional NNs (CNNs), recurrent NNs (RNNs), and graph NNs (GNNs) can all be considered as more \"structured\" neural networks, customized to the input feature relationships. "]},{"cell_type":"markdown","metadata":{"id":"Al_jGUMAs9mL"},"source":["### 3.2 Backpropagation\n","\n","Essentially, training neural networks is mostly based on (stochastic) gradient descent algorithms. Recently, there have also been high-order methods. \n","\n","Similar as training perceptrons, the solution of the network parameters $W^l$'s can be solved by (error) **backpropagation** with gradients of the corresponding loss functions: \n","$$\\mathcal{L}(\\hat{y}, y; X, W^l),$$\n","where $\\nabla_{W^l} \\mathcal{L}$ is always computed from the last layer to the input layer through the chain rule with the hidden variables $h^l$. Note that the updating $W^L$ is easy (We have covered it in linear regression or logistic regression).  $\\nabla_{h^{L-1}} \\mathcal{L}$ can also be computed easily.  \n","\n","Let's first denote $z^l = W^{l}h^{l-1}$ and $h^{l} = g(z^l)$ as defined above in MLP, we can write $\\nabla_{W^l}\\mathcal{L} = \\nabla_{z^l}\\mathcal{L} \\nabla_{W^l} z^l = \\nabla_{z^l}\\mathcal{L} \\bigodot h^{l-1}$ by the chain rule. Note that $\\nabla_{z^l}\\mathcal{L}$ is easy to compute by backpropagation: \n","$$\\nabla_{z^l}\\mathcal{L} = \\nabla_{h^{l}}\\mathcal{L}\\nabla_{z^l} h^{l} = \\nabla_{h^{l}}\\mathcal{L} \\bigodot g'(z^{l}),$$\n","where $g'(\\cdot)$ is the derivative of the activation function. Note that \n","$$\\nabla_{h^{l}}\\mathcal{L} = \\nabla_{z^{l+1}}\\mathcal{L} \\nabla_{h^{l}} z^{l+1} = \\nabla_{z^{l+1}}\\mathcal{L} W^{l+1}$$\n","and $W^{l+1}$ has been updated at the layer $l+1$ and $\\nabla_{z^{l+1}}\\mathcal{L}$ can be again backpropagated from the layer $l+2$. Therefore, \n","$$\\nabla_{W^l }\\mathcal{L} = \\nabla_{z^{l+1}}\\mathcal{L} W^{l+1}  \\bigodot g'(z^{l})\\bigodot h^{l-1}.$$\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":329},"id":"tJ9fYUIWWYBy","executionInfo":{"status":"ok","timestamp":1631919777228,"user_tz":300,"elapsed":10,"user":{"displayName":"Ziyu Xiang","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"09056639542636885950"}},"outputId":"567d8752-d12a-4db7-fef2-5669e8875aef"},"source":["Image(url= \"https://www.python-course.eu/images/mlp_example_layer_800w.webp\", width=500)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<img src=\"https://www.python-course.eu/images/mlp_example_layer_800w.webp\" width=\"500\"/>"],"text/plain":["<IPython.core.display.Image object>"]},"metadata":{},"execution_count":17}]},{"cell_type":"markdown","metadata":{"id":"qp6Y7x7QWUbn"},"source":["### 3.3 A naive example"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":265},"id":"lPZm0GaBBLU4","executionInfo":{"status":"ok","timestamp":1632785441549,"user_tz":300,"elapsed":482,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"3ed34aff-8dc3-4f6f-eeff-cddedfc3e58c"},"source":["# simulated data in two-dimensional space\n","from sklearn.datasets import make_blobs\n","\n","n_samples = 200\n","blob_centers = ([1, 1], [3, 4], [1, 3.3], [3.5, 1.8])\n","data, labels = make_blobs(n_samples=n_samples, \n","                          centers=blob_centers, \n","                          cluster_std=0.5,\n","                          random_state=0)\n","\n","\n","colours = ('green', 'orange', \"blue\", \"magenta\")\n","fig, ax = plt.subplots()\n","\n","for n_class in range(len(blob_centers)):\n","    ax.scatter(data[labels==n_class][:, 0], \n","               data[labels==n_class][:, 1], \n","               c=colours[n_class], \n","               s=30, \n","               label=str(n_class))\n","    \n","\n","datasets = train_test_split(data, \n","                            labels,\n","                            test_size=0.2)\n","\n","train_data, test_data, train_labels, test_labels = datasets"],"execution_count":18,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO2dfYyd5Xnmr/t4JibYZkD1DErMuFNprWAKDF2GmURekg1SdgNELCooapEjU1VhqGCUSpVG2z+8Utd/dbaKKtF2MWlSt8G0QmmSooSwigRNKE3HHrdjHDAt2QI+QLM2BczYBjPjc+8f73k9Z955v7+e533O9ZOGw5yP9zzn2L7e+73uj0dUFYQQQuylZXoBhBBC4qFQE0KI5VCoCSHEcijUhBBiORRqQgixnIEqDrp161YdGxur4tCEEOIkR44ceUtVh8Meq0Sox8bGsLCwUMWhCSHESUTktajHaH0QQojlUKgJIcRyKNSEEGI5FGpCCLGcVMlEEXkVwBKACwBWVHWiykURQghZJUvVx2dV9a3KVkKIC5xtA8fngLfmga1TwM5ZYNOo6VWRhlNJeR4hfcnZNvDkOLByBtBl4J1F4JWDwG1H14t1VkHnCaCvkTRjTkXkFQDvAFAA+1X1kZDn3AfgPgDYvn37ja+9FlkSSIibLMwAL+/3RNpHBoEd08DEQ6v3BQVdBoGBzeGCHvZ8tAARYGw3cP0+CrYjiMiRKFs5bTLxP6nqfwRwK4AHROTTwSeo6iOqOqGqE8PDoc01hLjNW/NrRRrwfv/3Q2vvOz7XI7rd56yc8e4PI/h8dAC9ALzyF56An22X+jGIfaQSalV9o3t7EsB3AExWuShCGsnWKS867kUGgV8I/HNJK+hxz/deFC/wxBkShVpENonIFv//AfwXAD+temGENI6ds56F4Yu1b2nsnF37vLSCHvd8nziBB7xoe2EGeGrSu2X03UjSRNRXAvg7ETkK4BCA76vqU9Uui5AGsmnU85l3THuiu2M63HdOK+jB54f9c40TeN/bfnk/8PZh75ZWSSNJlUzMysTEhHIoEyEx+FUc/37IE9o0VR/P7wVeeRRAB4AmJyHTJjeJFcQlE1meR4gJNo2urwRZmIkuv9s0CnzqgFflkVbgs3rhxFoo1ISYJkv9dVDg49g65R0rGFFHWSUu0/A6dM76IMQ0Wcv10pLVC3cVB7x6CjUhpqnKokib3HSdqk6ENULrgxDTVGlRZLFKXCXNidBya4QRNSGmscmicLHuOqluvQHWCIWaENPYYlE0QLBykXQibIA1QuuDEBuwwaKIEyzTayuCfyKMKmtsQBkjhZoQ4tEAwcpN3ImwAWWMtD4IqRObPeCsM0hcwaYcQQSMqAmpiyyNLSbYOeutJzgn2yLBqoQka8QCKNSE1IXtHnADBKsybMgRxEChJqQoaWtwm+ABWy5YRrCgxppCTUgRstgZDUhaZcICAUuk6BotsauYTCSkCGF2xvJpbyRpkAYkrVLThJrrMtZoSY01hZqQIoRuk9Xx5kYHBaHKxpa6q0miBOz5vfZUtZQhspbYVbQ+CCnC1ing7SPwhvn30glPElbhAZu4PI8SsFceBaRlR1VLGSJriV3FiJqQIuycBURCHlBPEPJGulleZ+LyPHQfxxa8HdItacUuoy7cErvKKaFut4GZGWBy0rttW2SXEUfZNAqM7QYQEGsZBC7bmc8jzeqtFokc855IwgRMBEBgaz+TVS1liKwlc1icsT7abWB8HDhzBlheBhYXgYMHgaNHgVHLEtHEMa7fB7z+xPpGESBf3XTWeuu8l+dFLJOwmuvlJeDVx4zbBLFrzFOZYkHJojNCPTe3KtKAd3vmjHf/QywLJVUSJQjP3pUu0g2WkJ18NluEnLejsGgDTti+j2EnLJNVLRaIbBk4I9Tz86si7bO8DByyqJeAOEyYIKSJdMOiWrQAGQB0Jfp1wffOEzmWXdHQz52NFeOMUE9NeXZHr1gPDnp+NSGFyNs0kSbSDYtqMQC0NgCQ9JFpnsixiooGRyJY2xBVTX5WRiYmJnRhYaH048YR9KgHB4HNm+lRk4IEI15fNNMmlHyRj4own5r0EoZBLh8HRm6uNjIt+tl6j2N7h2IDEJEjqjoR9pgzEfXoqCfKc3Oe3TE5CczOUqRJQcr2cYNERbUjN1cfmZZhVVjSYu06zgg14IkyE4ekVKruTDM9WrSoVZHmRJYm4mZUHotTQk1I6ST5uEUFpukJuKQTWZqIm1F5Ik41vBBSOnFNE2UNJvKj2v867902SZySuv/SdE1aMvjIZijUBmEnZQOI60yjwCR3/6WxjiwZfGQztD4MwU7KBhHl41Jgkq2bMOsILa+93seSwUc2w4jaEHGdlKQhxF321zGMyRbirJuds8DApYEXdID2d1c/myWDj2wmtVCLyAYR+ScR+V6VC+oX2EnpAFECM7a7mmFMTRXxq+7EuqFVF86tWkSWDD6ymSzWx1cAHAdwWUVr6SvYSekAUZf9eWuv4163c7a5lRGnX0TiVD12NMaSKqIWkasA3A7gT6tdTv8wO+t1Tg52gzG/k3K2AVd7TIL2EHbZn9e7jntdkxOXZcyF7nPSWh9/CGAW67exuIiI3CciCyKycOrUqVIW5zJ+J+X0tCd409PNSCT6SdD9+4HDh73b8fE+F+sgeYVp6xTWX+QOeK+zJXGZx3656FP7ctPyfqcHnZpEoRaRLwA4qapH4p6nqo+o6oSqTgwPD5e2wKaRJdr0Oynn571b20UaYBI0FXmTY2O7AVwI3HnBu9+GqDToof/LHwN/80vAT+5NFmy9+B/vtvwRQ06TJqLeBeAOEXkVwF8BuEVEHq10VQ3FZLRZlx3BJGgK8ibHXn0UkA1r75MB734bKiOC9gsUwAXglW/GJ0uPz3nJw16h7k0mkkQShVpVf1dVr1LVMQC/BuBpVd1d+coaiKlos84TxNTUqq/ukyUJ2jf+dp5uw7fm186gBlbtDRsqI0J3XAeATrxfbott02BYR10ipqLNOk8QRZKg9LcTSLI3TLeah25o2yVOeG2wbRpOJqFW1b9V1S9UtZimUzTazEvUCeLZZ8t/ryJJUPrbCdhgb8Thry9YEw3EC6/tn6sBOLNxgA2Y2rxgZgZ4+GFgJXDVvHEj8PLL9iQpJye9SDrs/vn5+tdTmKjJeUUm6iVtNGCas23g+b2eb64KoJNuwwHbP5cFxG0cQKEumXa7/s0L2m1gxw7g/Pm19w8OehGvLTO6Z2Y8uyPY5GPTGlMTtTvKZ38APHNr8V1TbIfCWzoU6j7ghhu8yD2ITdGqU9ulLcx4ZWrBQUKXXQ2899L6+3dMs/OOxBIn1EwmFsSWKoabbzbjj2ehqU0+oURVMpz5V1Y4kNJhRF0AmyJEm9bSFzQhoja9vZXp928YjKgrwqYqhrKi1TRXCLZcRRglqpJh6mvZKhyqmohX1u4zTX1/x2BEXQDXqhjSROVFInc/0To/75UyNn6X+KiEWtpEW1RCMinxmCZSjYr464rsTb9/A4mLqLnDS5c8ItKUUaVxn633sQ8+AJaWVsv8eq8Q/KqMuKuIuMoNJ3e0iRrNmXZkZ9hEvOV3gL+9HfjP308n7lHjTk13A5p+f8egUCO/iMzOes8LRpc2jSqN+2zA2sfCCHZW5u2+zCvwThPVkn36mCfGYZF12lnXpre3Mv3+jkGPGvm95qqrGMrwguM+W/CxMIJXCHm7LznMKYS4luyo2RlpI1XT3YCm398xGFGjmIj4o0rLpiyrIO6zqSaLdPAKIe9VRFNsolrZOevZFsvvrH8syiZIG6kmbTpbNabf3zEYUcPcjI4w/Ch6YgJ49921kfA77wC3354usvaP89prgARGM/ifLexzDwwA110XfYWQ9yrCxI421len+GI2dN36x6JsgiyRqukhTqbf3yFY9QF7apCD64jiiivi1xZ3nN7PBtT7uetsr7flzzQVWas/XGrfZq31ReKqPqCqpf/ceOON2jROnFB98EHVyUnv9sSJ+tfw4IOqg4OqnikR/TM46D03y3FaLdWRkfWfrezP7R/vppvMfY+q4d9B0vdmlDMnVA8/qPrUpHd7xtAXF1zT3+9R/daI9/P3e8pd15kTqo9fofrYoOpBeLePX2HHZzcAgAWN0FRG1BYRVZcd9dyoWu2o44yPe63mVdUx2xTFulbjXog8UevZNvD964CV02vvHxwCbjtWTtTLWus19GVnovX+ZAhhnnEYSf55lPf80kvVDu23qVPTpryDUfJ2CB6fA1beW3//8lJ5W2ix1jo1Tgp1U3cSCUu4DQ15P1mScGHH2bABuHChWhG1qQTPRPLSSuLqruN4ax7hO9B2yhNS7vySGieF2qbILgthFRXHjnk//n333APccQdw113RVwphx7n66vUbC6QR0SxXJjZFsU5N6itC3qh16xRCd3JBqzwhZa11apz0qPP6k7bPoijiAecZ2p/1/WzyqPuCKmd+1OFR936GKitYGlJZ0ndVH3ky/idOqF5xxerrBge9301VLYRRpJIhz+fL+z2arp7pC9JWTBSprKi66qMOGlRZgpiqDyetjzz+pA12SZLNULSDMqsVkOf9/E7N+XnvtgmRdBMTz6m9Z7+pZse0F7HumE6/LdimUeBTB4C7/p/386kDVkaiseT16C3DyRZyX5SyNFeYToSlaRkv2oYd1u4eZ/dccw1w5AjQ6eR7vybQ2Kl+WbzntNP8XMSRyhInI2oge2RnOhGWJqIvu5Ihrjqm3Qa++921Ig0Al17qVuVELVdSVWwOwIqJdDjyPTkr1FkxXc6VJqIvu5IhabLeuXNrny8C3Hmn5ZFmRiq/kqpqpxMbKiaq2p2mTGz4nkqAQt3FdDnXNddED08CVn3Uu+7yfv/Wt4p7wHEiFfaYKnD8eP73s5HKr6Sq8kiLeM9l0JSttkx/TyXhZHle02i3vYl1pwOVUENDXg01kG6LrKTSwuBzlpaAxx4LL9kDspfzNZHKSwqfmvSELMgvTHpT5ZoK279Lh1txGSJtXXaYzdBqrdoMMzPxu6OkSYiFPefSS72fc+fCZ0vbvntNGeRJPGfC1Z1OHEnSNQUKdUX4UfLSkpeQO3IE+OY3vQg5KAJPP73eZuh0Vm2GJB81zTZXYc85d87rdNyyJVykKhUwi6hq8wcAq5sDBEeYNswjXYerJyBLoVBXxN69a62MTsf7fe9e4MCB1fvn54EXX1z/+oGBVZ80qSwvTUIs6jnHj0d3a6YRMNu7OY3j6k4nrp6AbCWqE6bIj+nOxLJImq0c9/jISPgs6ZGRtcf4xCfCn9dqrR4vqaswTQdhFfOZm9DNSSrExhnaDQYxnYkU6giSRCjp8bRCPTAQ/rwNG9avJ6o1O41g5hHVpBNV44bzE2IxhYQawCUADgE4CuAFAL+X9BoXhDpJhJIe37MnXID37Fn7PpdcEv68Sy7Jtt40MzayzOFII+w33RS+9snJbGsnhBQXagGwufv/gwDmAXwy7jUuCHWSCCU9fuKE6tCQqoh3v4j3e1Ac7747/Dh3313v5w1iyk4hpF+JE+rEhpfuMc50fx3s/pRffG0ZSY0QSY+PjnoVHg884N33wAPhFR9f/apXddHLli3e/SZJk6A03c1JSN8QpeC9PwA2AFgEcAbA70c85z4ACwAWtm/fXttZqCqKetRZ38u2DWbTRssca0pCOaGqD6rqTd1b/r1IBGVtbisilwP4DoAZVf1p1PNc6Uz0S8+i6oiTHjdBWZ123ATAIA0ZdB9JG8A4vLBuGd41+GZ4Wa4GfYy6ietMzNxCLiL/A8A5Vf2DqOe4ItRNJM9OLlHYeCJyHn+GRrA+uUnzKWYA7Icn0j6DAKYBsLs8kkIt5CIyDGBZVd8VkY8C+ByA3y95jaQkypwGV2nHHgknbohTU2ZozGOtSKP7O7vLc5Nmet7HADwjIs8DOAzgh6r6vWqXRfJieq42KYgLMzSm4EXQvQwC4N/B3KSp+nheVX9FVa9X1WtV9X/WsTCSD1ZiNBwXBt3PwvOk/Y/he9T8O5gbzqN2DNNztZuKNfsmujDofhRe4nAaXhQ9DSYSC8J51KTvsa7Cxa/6cGmIE0mE86gJiSHNmNha6efNaEkotD5I32N6B3pCkqBQk76HlTLEdijUpO9hpQyxHQq1IaypMiDNrpRpw+sEnOzelvH3qIpjkkKw6sMA1lUZkGZSxUwNzukwRlzVByPqGvGj6IkJb//EsCoDQlIzh1VBRff2TPd+m45JCsPyvJoIRtFBWGVAMlPFTA3O6bASRtQ1EazVDcIqA5KZKmZqcE6HlVCoayKsVtenKVUGTIBaRhUzNTinw0oo1DURVqsrAoyMZKsyMCWWvnWzfz9w+LB3Oz5OsTZKFTM1OKfDSlj1URNlVHqYrBYpc0MCQsh6WPVhAWXU6sbNpKgatlkTYg5WfdRI0R1TTIrl1BSwuLg+omYClJDqYUTdIEzOpGCbNWkkjnRZUqgbhEmxbHSbNTGDaZH0uyz3w9tEcH/39waKNZOJDYM7g5NGYEMresN2Q+fGAQ7BncFJI4hrRa/r769DXZa0PojVsMmmZsqyK2wQSYe6LBlRE2sJ1o0vLgIHD9Ibr4ygXbEI4CCy2xVtAB+E3F+3SM7CW3/QfmlgApwRNbEWk3XjfUkZk/N8sT8euH8A9YtkFV2WhhKkjKiJtbDJpmbKsCt8sV8J3L8TwPdRfyv6KMrzxMu64sgBI2piLdzLsGbK8HTDxB4APormzwsxOKubQk2shU02NVPG5DyHEnjrMJggpVCTymifbmPmyRlMfm0SM0/OoH06m6HHJpuaSePpJnm0Lo9JNXgSYsMLqYT26TbGHx7HmQ/PYLmzjMHWIDZ/ZDOO3n8Uo0MWK20b3qXsPLx/mLNo/iV7WaRtYvG/w0PwRMyV77DiJh5OzyO1M/fc3EWRBoDlzjLOfHgGc89ZXLLhUMtxJaT1aP0E3nz31gWRBozO6mbVB6mE+TfmL4q0z3JnGYfesLhkw4ZuOpspy6Nt8lVLmVUkGUiMqEVkVESeEZEXReQFEflKHQsjzWZq2xQGW2sNvcHWICa3WZxVMtlNZ3qAURrK8Gh51ZKLNNbHCoDfUdVrAHwSwAMick21yyJNZ3bXLDZ/ZPNFsfY96tldFmeVTCWLfPF6GJ54/RGAHfBOHDZRRqLQYIlbk0kUalX9N1X9x+7/L8HrOdpW9cKqpmhFAolndGgUR+8/iukbpzH58UlM3zhtfyLRVMXCHIAlrG0SOQ/gM7Ar0izDoy3zqiXLVUgTrlhiyFT1ISJjAH4M4FpVfS/qebZXfTS2IqHPaJ9uY+65Ocy/MY+pbVOY3TVb/Z+PiYqFSXiRdBgPwi1/vKzRo1kqMGwYuZqCUqo+RGQzgL8G8NthIi0i94nIgogsnDp1Kv9qa6CRFQk1YNOkOv9kuv/Ifhx+8zD2H9mP8YfHq7/yMVGxMBXzmMW511yUddWSxUJxwG5JJdQiMghPpA+q6rfDnqOqj6jqhKpODA8Pl7nG0mlkRUIOstg7/qS6/fuBw4e92/Fxc2LdVyfTWQAbQ+4fgP0dfVnthzl4J7+r4UW5eUvcslgodSSJK7ZWEsvzREQAfB3AcVX9arlvb4apbVNY/PniGrG2viIhI0F7Z/Hnizh47GCkvRM3qc7ERgX9cjIF4InUj+B50ue79w0A2AK7O/qyDCmKsh/yWktT3fcLWihh/4SzPDcPNQxrShNR7wLwJQC3iMhi9+e2ct7eDI2sSMhI1ojUtkl1jSzvK8IUgJfhedKTAO6HdR7qOvYCeBdm7IcsFkrVSeIarJXEiFpV/w6AlPeW5vErEuaem8OhNw5hcttkPYmqGskakU5NeYP5e8Xa5KS62V2zOHjs4LqEr0sn03UYaqbIRRvAowCCtQh12Q9+BUqaxG+W5+ahBmulbzsTR4dG8dBtTflXkZ2s9s7srLd7im9/mJ5U1w8n00Yzh/UiDXghXV32Q/DE5vvEYR2PVZ4Eq7ZWwKFMzpKnBJE7nJPURJUUbgDwCuovkTNZglfSe3MX8j4kLCLdff3u2LpkW3c4N1JPTeIJiyJbAHYjWpzuAPCD7v/fCmBfzHOzYnJOS9XWChhR9w1NbfJp6rqdx7aGk6gIfxL2teJHwDGnpLF1yU1dt/NkaSevoioiWLf8ixHP21ngPSyC1kef0NS65Kau22mCY0q/hfjIuOyqiLC65ai6tH+AJ+ZNG6cagBF1n9DUuuSmrtsoVXbJ5RlTGjVrM2+0Gxahfxjx3H+GE+NUKdQNoIxJf01t8mnquo1R9bznNDZG8ERxtqT39ona6TyOBs736IXWh+VkbQWPwsa65DTVHDau22qqrn5IsjHCbIlOxLGO51xDVN1yq/teUSJe1yYQFUChtpy4ZFrWhh2bmnyynIBsWnepVLElVdVdcknNHWEnihY8D1kjXpOVWXizNHqrSC4F8Dl4Q5gB4AoA/xdrZ3wH37NBW4LR+iiRKjYjcDWZ1vfVHFVZFGXtUhPlcyfNzQg7UXTgKU1ZszaCFSf3dO//GwAnAbwD4OcANsW8Z8O2BKNQl0RV85NdTaa5egJKTVWDfMoYQBQnYklleVEnit0xr8lD79zwLQDOYe13eQ7AnTHv2bAZ1RTqkqgqQqwimWbDNmSunoBSU5VFUcZ2WUkiFre5QtSJYl/Ma4oS9V0ej3lPkxsZ54AedUlUFSGWnUwrKzlZlL6cjtdLlYN8ig4gKiJiwXZqvwTvLlTnA+f5LrO8xgIvmy3kBfErFx5/8XGcOnsK2pMxGWwNYvrGaasSYTNPzmD/kf3rpuqZWKf/3fVlNUcdQ4ryiouJfQ2LkOd90r6mxmFPHMpUEcHotBdbI0SbvGFnqznSUOUgn6I7joRVVST53GEnhroGJeX5LtO+xuSwpx4o1Dm4GEW/8DhOf3AanZ5C0RZa2HrpVnzxl79oZYTYD9uQNYaqZiQXERdfcLdjtVrjZsQLX9SJYTvq84HzfJdpXmOJl02hzkhcFA0AHXQwdvmYtZFilDe8+/rdmHlyhqNEXSCvuOTd1zDqxNDpHiOrD2+BJ3yRGjYFSIM1Qt2UmcPB6o4gtkenUXOqbz14q/EEIymJvOKSNxKPOjG04Al9Vgul4o1iM5HHBqoAK4TalkqENIR5vD62+tJBgt7wzJMzpXU/EgvIKy55I/GoE4NvmWTxjucALGG1o3C5+3vNnvBFatgUIA1W1FE3qUstrP5XIBjZNILpG6etPLkkYVOCsZFUOa0uD3lrqfN2NcY12cTVXIfxY6xt+0b392cTXlclWT9DBVgRUTdJKKI83oUvLzROoH2YYCyAbZfqPnmSa3kj8TKjzqhq4ajBTn2CFRG1yS61rF16vsc7feM0Jj8+2dgouheOEi1Aw1qRYynS1TgKT5z9ra/mkO/KIkqRrFAqc1jR8GJqXzzX9uPzE7LPnngWHe1ARPDp7Z++KLhxydomN58YTUQ7sFdfKZTVGDID4GGsn3qXttmmDWAvqttEt0LiGl6sEGrAjFDU1aVXh5D4J52lD5ew0ln9Wz4gA9j0kU0AgLMfnsWKeo9t3LARP7r3R5i6aqrUddSN8ZNtWV18NpGnPM6GbsY2gOsAnA7cPwTgWIrXG6YRnYkmutTq8MbrqmjxE7K9Ig0AK7qC0+eDf3OB8xfO4zMHPoOXZ15et46mlEoC5c7rzoUl5VulkddzL6sxpIjfPQfgvZD7TVaNlERfOz91eON1VbTElQ1Gcf7C+XXrqGpca1UYT0SXMa2uCGVXnOT13Muagw3kr7KYR3gysgNrp+Klpa+Fuo4kWl1CEnbSSUNwHU0qlQQsGZdqqnyriuH3eSPjMuZgF2UK4buRt1B7J2HZ9LVQ11HBUaaQxFWo+CedgVZ6NytsHcYj1Iz0dcVKmRUnfmT+GtarQprI2PSVBeCdFC4LuX8LmmtFdbEmmegqZSW70hwnWPXx5tKbePv9t9eMXvWJWkfaBGv7dBt7n96LH/zsB4AAt/6HW7Hvs/uMeNlNrlgpRNaKk6gkYdCX7qXCsZ6VwKqP9FCoV2mfbmPvM11B066g3ZJd0MIEtIUWvjT+JRy480Dke/eK+4AMYENrA67eejVu3n5zqKClPSFc97+vW5ekHNo4hGO/daw/RNIGslRaxFVTzIUcRwAMA/girN701SXihDrR+hCRb4jISRH5aflLcxtf9B479hhOnj2Jdz54B0/8yxO5jhVmSXTQwaPPPxqZ6AtaO/dP3I+XZ17G4v2LeOi2h0IFNY0dNPfcHN47vz69vnR+yVov20my+MJxNkmYL60AxmCsZToTtrXwV0AaQ/MAgD8C8BfVLsU9yiwdm9o2hSNvHlkz+xoAOtqJPV6essek18y/MR9qp3TQsdbLdpIspWxxScIiozxNjyS1tYW/ZBIjalX9MYC3a1iLc5SZmJvdNQuR9SlthdYujlPbpiAh6fUWWpwPUjdpK07iyufyVmzkrTopMwJ2qYU/htKqPkTkPhFZEJGFU6dOlXXYRlNmxcfo0Ch2X7d7nUCaGJ40u2sWl21cn17fsnHLxWoLG3Y6Jz0kTbjLU7GRRyTLLim0ZAeWqilNqFX1EVWdUNWJ4eHhsg7baMouHdt3yz5cfsnlmY5XhWCODo3i2G8dw57r92Dk0hGMbBrBnvE9FxOJSU0zFHEDJIlxnlrwPCJZdgRcZqONxaSq+hCRMQDfU9Vr0xy0aVUfVbZMl106luV4puZgxJX4ze6azbSmJrWz9x155nuUPcSqxl3Cq6ZweZ7LQm18qE8K8opVXUOngkx+bRKH31z/r3Hy45OY3DaZek1N+LPpa/KIZBVDrPyEpsEdWMqgaHneXwL4CYBPiMjrIvKbZS/QJLa3TBeZvWGqyzDOm8+yJtv/bPqePN52Fa3mFuzAUjVpqj5+XVU/pqqDqnqVqn69joXVhemW6fbpNu797r248g+uxJX/60rc+51714hwEbGKEsz3V96v1B+O8+azJFhN/9mQFGQVSRtazRuINWNOTWFyG6qwDr8/f/7P8e2Xvo1f3fmrePHUi3jt3ddyi1XYtmErnRUcP3UcK7pS2cjVsJ3OfbsmaiuzsIQotwg0uYEAAAX6SURBVAhzlDzbhPU5fd9CbtIHnXlyBn9y+E/WNbEAXk1yB52Lt71k8Zl7k4/vr7x/UaTzHKss0iZE6VGTfoKzPhIwNdQnKukWhkCg0EJiFZfkm/+ynftG9e3AJdJ3NGKHF5NUvbtMVNVGVFt4GMObhjE2NFZIrJpoJZjY+YcQ22BEXTFxl+8AQqfQBSnLnqCVQIi9FCrPI8WIq9q42OE3vgcjm0YwcukI7r76bgxtHKpkEH6ayXjsGiTEPhhRV0weX9iUL1s04mYXISH5oUdtkDy+sClftshY1rp2WyekH6H1UTFN2NPPtzv+bPHPctdss4uQkOpgRF0xcc0fNhCMhIOkrQphFyEh1UGhrgGbS8yCkXAvWaL/Jpb+EdIUaH30OWGRMABsGtwUWhUShS0WD6tWiIswou5zoiLh37jhNzJdBdhg8TChSVyF5Xl9jktNMKbmbxNSBmx4IZGkaYJpCkxoEleh9WEhdTeO2JzszAITmsRVaH1YhktWRBhV70/p8ndH3IbWR4NwuXGkyLZiaXDJxiGkF1ofluGyz1qkRT0trtg4hPTCiNoysuwp2DRcPgkRUiUU6pIp2nBhS+NIFdh6EmKTDLEdJhNLpKxklqvbT9mY7LNxTaQ/YTKxJspKBPo+6/yX5/HQbQ8ZEYwqokwbk30uJ2+JOzCZWCKueLBVtmLbluxz5c+MuA0j6hKx1YPNSj9Fma78mRG3oVCXiCuJwH6KMl35MyNuQ6EukSQPtinVBf0UZdromxMShFUfNdGk6oImrZUQV2DVhwU0yfdllEmIXbDqoyaa5vvaVp1BSD/DiLom+sn3JYSUSyqhFpHPi8g/i8jPROS/V70o12ifbmPpwyV0tAOBAGB1ASEkPYnWh4hsAPDHAD4H4HUAh0XkCVV9serFuUBvYu6CXkALLbSkhXuuvQf7btlH35cQkkiaiHoSwM9U9V9V9UMAfwXgv1W7LHcIJhE76KAlLWzZuIUiTQhJRRqh3gagt+D39e59axCR+0RkQUQWTp06Vdb6Gk/TkoiEEPsoLZmoqo+o6oSqTgwPD5d12MbDJCIhpChphPoNAL3X6Fd17yMpYIsyIaQoaYT6MIAdIvJLIvIRAL8G4Ilql+UObB4hhBQlsepDVVdE5EEA/wfABgDfUNUXKl+ZQ7B5hBBShFSdiar6JIAnK14LIYSQENiZSAghlkOhJoQQy6FQE0KI5VCoCSHEcirZOEBETgF4rfQDZ2MrgLcMr8EG+D148HtYhd+Fh23fwy+qami3YCVCbQMishC1W0I/we/Bg9/DKvwuPJr0PdD6IIQQy6FQE0KI5bgs1I+YXoAl8Hvw4PewCr8Lj8Z8D8561IQQ4gouR9SEEOIEFGpCCLEcJ4Wam/ECIvINETkpIj81vRaTiMioiDwjIi+KyAsi8hXTazKBiFwiIodE5Gj3e/g902syiYhsEJF/EpHvmV5LGpwT6p7NeG8FcA2AXxeRa8yuyggHAHze9CIsYAXA76jqNQA+CeCBPv37cB7ALao6DuAGAJ8XkU8aXpNJvgLguOlFpMU5oQY34wUAqOqPAbxteh2mUdV/U9V/7P7/Erx/nOv2/HQd9TjT/XWw+9OXlQQichWA2wH8qem1pMVFoU61GS/pP0RkDMCvAJg3uxIzdC/3FwGcBPBDVe3L7wHAHwKYBdAxvZC0uCjUhKxDRDYD+GsAv62q75lejwlU9YKq3gBv39NJEbnW9JrqRkS+AOCkqh4xvZYsuCjU3IyXrEFEBuGJ9EFV/bbp9ZhGVd8F8Az6M4exC8AdIvIqPFv0FhF51OySknFRqLkZL7mIiAiArwM4rqpfNb0eU4jIsIhc3v3/jwL4HICXzK6qflT1d1X1KlUdg6cNT6vqbsPLSsQ5oVbVFQD+ZrzHATzej5vxishfAvgJgE+IyOsi8pum12SIXQC+BC9yWuz+3GZ6UQb4GIBnROR5eMHMD1W1EaVphC3khBBiPc5F1IQQ4hoUakIIsRwKNSGEWA6FmhBCLIdCTQghlkOhJoQQy6FQE0KI5fx/qS/OvACNEjAAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"f6Rxvq_wVAq6","executionInfo":{"status":"ok","timestamp":1632785495666,"user_tz":300,"elapsed":494,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"ac804459-f931-4a11-b7d3-06aa4adb51d9"},"source":["\n","from sklearn.neural_network import MLPClassifier\n","\n","clf = MLPClassifier(solver='lbfgs', \n","                    alpha=1e-5,\n","                    hidden_layer_sizes=(6,), \n","                    random_state=1)\n","\n","clf.fit(train_data, train_labels)     \n","\n","clf.score(train_data, train_labels)"],"execution_count":19,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.9875"]},"metadata":{},"execution_count":19}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Q9PaIbUSVULx","executionInfo":{"status":"ok","timestamp":1632785537131,"user_tz":300,"elapsed":168,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"cf4e60ec-354f-4548-fc62-0a3b07b0a30b"},"source":["predictions_train = clf.predict(train_data)\n","predictions_test = clf.predict(test_data)\n","train_score = accuracy_score(predictions_train, train_labels)\n","print(\"score on train data: \", train_score)\n","test_score = accuracy_score(predictions_test, test_labels)\n","print(\"score on train data: \", test_score)\n","\n","predictions_train[:20]"],"execution_count":20,"outputs":[{"output_type":"stream","name":"stdout","text":["score on train data:  0.9875\n","score on train data:  0.975\n"]},{"output_type":"execute_result","data":{"text/plain":["array([2, 3, 2, 0, 0, 2, 0, 0, 2, 1, 1, 0, 2, 3, 3, 0, 2, 2, 3, 1])"]},"metadata":{},"execution_count":20}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"lmxlsU8UVf7E","executionInfo":{"status":"ok","timestamp":1632672234724,"user_tz":300,"elapsed":84,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"0eb09853-465a-4cfb-970f-c2fb7006614b"},"source":["## A really naive example\n","\n","X = [[0., 0.], [0., 1.], [1., 0.], [1., 1.]]\n","y = [0, 0, 0, 1]\n","clf = MLPClassifier(solver='lbfgs', alpha=1e-5,\n","                    hidden_layer_sizes=(5, 2), random_state=1)\n","\n","print(clf.fit(X, y))  "],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["MLPClassifier(activation='relu', alpha=1e-05, batch_size='auto', beta_1=0.9,\n","              beta_2=0.999, early_stopping=False, epsilon=1e-08,\n","              hidden_layer_sizes=(5, 2), learning_rate='constant',\n","              learning_rate_init=0.001, max_fun=15000, max_iter=200,\n","              momentum=0.9, n_iter_no_change=10, nesterovs_momentum=True,\n","              power_t=0.5, random_state=1, shuffle=True, solver='lbfgs',\n","              tol=0.0001, validation_fraction=0.1, verbose=False,\n","              warm_start=False)\n"]}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"IBm3Ejv_Vitv","executionInfo":{"status":"ok","timestamp":1632672238415,"user_tz":300,"elapsed":103,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"fd4c55f2-9d57-4129-8352-81d5afdf067c"},"source":["print(\"weights between input and first hidden layer:\")\n","print(clf.coefs_[0])\n","print(\"\\nweights between first hidden and second hidden layer:\")\n","print(clf.coefs_[1])"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["weights between input and first hidden layer:\n","[[-0.14203691 -1.18304359 -0.85567518 -4.53250719 -0.60466275]\n"," [-0.69781111 -3.5850093  -0.26436018 -4.39161248  0.06644423]]\n","\n","weights between first hidden and second hidden layer:\n","[[ 0.29179638 -0.14155284]\n"," [ 4.02666592 -0.61556475]\n"," [-0.51677234  0.51479708]\n"," [ 7.37215202 -0.31936965]\n"," [ 0.32920668  0.64428109]]\n"]}]},{"cell_type":"markdown","metadata":{"id":"9eiGT5R4BLU4"},"source":["## 4 Hands-on exercise: Iris data "]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":205},"id":"jeJdQzy6lm87","executionInfo":{"status":"ok","timestamp":1632672243768,"user_tz":300,"elapsed":174,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"969cea54-1263-4671-9e1e-2f9fd46ebd01"},"source":["from sklearn import datasets\n","\n","iris = datasets.load_iris()\n","iris_data = pd.DataFrame({\n","    'sepal length':iris.data[:,0],\n","    'sepal width':iris.data[:,1],\n","    'petal length':iris.data[:,2],\n","    'petal width':iris.data[:,3],\n","    'species':iris.target\n","})\n","iris_data.head()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sepal length</th>\n","      <th>sepal width</th>\n","      <th>petal length</th>\n","      <th>petal width</th>\n","      <th>species</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>5.1</td>\n","      <td>3.5</td>\n","      <td>1.4</td>\n","      <td>0.2</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>4.9</td>\n","      <td>3.0</td>\n","      <td>1.4</td>\n","      <td>0.2</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>4.7</td>\n","      <td>3.2</td>\n","      <td>1.3</td>\n","      <td>0.2</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>4.6</td>\n","      <td>3.1</td>\n","      <td>1.5</td>\n","      <td>0.2</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>5.0</td>\n","      <td>3.6</td>\n","      <td>1.4</td>\n","      <td>0.2</td>\n","      <td>0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   sepal length  sepal width  petal length  petal width  species\n","0           5.1          3.5           1.4          0.2        0\n","1           4.9          3.0           1.4          0.2        0\n","2           4.7          3.2           1.3          0.2        0\n","3           4.6          3.1           1.5          0.2        0\n","4           5.0          3.6           1.4          0.2        0"]},"metadata":{},"execution_count":22}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"TXHLWciBBLU5","executionInfo":{"status":"ok","timestamp":1632672246309,"user_tz":300,"elapsed":86,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"18237e61-1387-49e2-baec-e5d2edfd7c61"},"source":["iris = datasets.load_iris()\n","X = iris.data[:, 2:4]\n","y = iris.target\n","\n","# Split dataset into training set and test set\n","X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3) # 70% training and 30% test\n","\n","cmap_light = ListedColormap(['orange', 'azure', 'green'])\n","cmap_bold = ['darkorange', 'darkblue', 'darkgreen']\n","\n","weights='uniform'\n","\n","# we create an instance of Perceptron to fit the data: \n","\n","# Create Perceptron classifer object\n","clf = Perceptron(tol=1e-8, random_state=0)\n","\n","# Train Perceptron Classifer\n","model = clf.fit(X_train, y_train)\n","\n","clf.score(X_test,y_test)\n","  "],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.6444444444444445"]},"metadata":{},"execution_count":23}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"RQmxxV9TOqQS","executionInfo":{"status":"ok","timestamp":1632672250811,"user_tz":300,"elapsed":201,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"9f5ddce0-3b9d-468a-fc8e-564162248fb6"},"source":["# Create adaboost classifer object\n","abp = AdaBoostClassifier(n_estimators=50, algorithm='SAMME', base_estimator=Perceptron(), learning_rate=1) #algorithm='SAMME.R', \n","\n","# Train Adaboost Classifer\n","model = abp.fit(X_train, y_train)\n","\n","abp.score(X_test,y_test)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.9555555555555556"]},"metadata":{},"execution_count":24}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"cK6fDlEwPcBh","executionInfo":{"status":"ok","timestamp":1632672256704,"user_tz":300,"elapsed":215,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"7d182651-9cd5-4aaa-dd8f-f73a962a3906"},"source":["# Create adaboost classifer object\n","abc = AdaBoostClassifier(n_estimators=50, learning_rate=1)\n","\n","# Train Adaboost Classifer\n","model = abc.fit(X_train, y_train)\n","\n","#Predict the response for test dataset\n","y_pred = model.predict(X_test)\n","\n","# Model Accuracy, how often is the classifier correct?\n","print(\"Accuracy:\", accuracy_score(y_test, y_pred))"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Accuracy: 0.9777777777777777\n"]}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"LN7dk_3LQJc1","executionInfo":{"status":"ok","timestamp":1632672262297,"user_tz":300,"elapsed":99,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"7bfec726-0284-415e-db95-5f98583a357e"},"source":["# Create MLP\n","\n","mlp = MLPClassifier(solver='lbfgs', alpha=1e-5,\n","                    hidden_layer_sizes=(15, 12), random_state=1)\n","\n","model = mlp.fit(X_train, y_train)\n","\n","y_pred = model.predict(X_test)\n","\n","print(\"Accuracy:\", accuracy_score(y_test, y_pred))"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Accuracy: 0.8666666666666667\n"]}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":404},"id":"bayzbi7RN7Jq","executionInfo":{"status":"ok","timestamp":1632672268119,"user_tz":300,"elapsed":555,"user":{"displayName":"Xiaoning Qian","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06431978792501680815"}},"outputId":"4c82c660-5ccd-4a01-d9d0-9b6766892854"},"source":["\n","# Plot the decision boundary. For that, we will assign a color to each\n","# point in the mesh [x_min - 0.5, x_max + 0.5]x[y_min - 0.5, y_max + 0.5].\n","h = 0.01\n","xx, yy = np.meshgrid(np.arange(X[:, 0].min() - 0.5, X[:, 0].max() + 0.5, h), np.arange(X[:, 1].min() - 0.5, X[:, 1].max() + 0.5, h))\n","Z = mlp.predict(np.c_[xx.ravel(), yy.ravel()])\n","\n","# Put the result into a color plot\n","Z = Z.reshape(xx.shape)\n","plt.figure(figsize=(8, 6))\n","plt.contourf(xx, yy, Z, cmap = cmap_light)\n","\n","# Plot also the training points\n","#n_neighbors = 8\n","sns.scatterplot(x = X[:, 0], y = X[:, 1], hue = iris.target_names[y], palette = cmap_bold, alpha = 0.8, edgecolor = \"black\")\n","plt.xlim(xx.min(), xx.max())\n","plt.ylim(yy.min(), yy.max())\n","plt.title(\"3-Class classification\")\n","plt.xlabel(iris.feature_names[2])\n","plt.ylabel(iris.feature_names[3])\n","\n","plt.show()"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAfEAAAGDCAYAAAA72Cm3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3hU1dbA4d+aTHqlk9A7KF2woNjFiuVaULH7Ye+9XcGGXa8Ny1WsoCBWsF0bVpTepNdAEkp6IHXm7O+PM4FMJpNeZpL1Pk+eZPbMPmdP0KzZ+6yzthhjUEoppVTwcTT1AJRSSilVOxrElVJKqSClQVwppZQKUhrElVJKqSClQVwppZQKUhrElVJKqSClQVypJiYik0Tkg+Y+DhH5R0SO9vwsIvK2iGSJyHwRGS0iaxvgnF1FZI+IhNT3sZUKBBrElaohEflARNJEJFdE1onI/1Wjz4UistATUNJE5BsROaIxxhsojDEHGmPmeh4eAZwAdDbGHGyM+c0Y06+u5xCRLSJyfJlzJhtjYowx7roeW6lApEFcqZp7HOhujIkDTgceFZGD/L1YRG4D/gNMBjoAXYEpwBmNMNZA1Q3YYozZ29QDUSqYaRBXqoaMMf8YY4pKH3q+elX0WhGJBx4GrjfGfGqM2WuMKTHGzDbG3Omnz8ciskNEckTkVxE5sMxzp4jIKhHJE5EUEbnD095WROaISLaIZIrIbyJS4f/fInKgiHzved1OEbmvMcZROksWkSuBN4HDPCsTD4nI0SKyvczxu4jIpyKyW0QyRORlT3svEfnJ05YuItNEJMHz3PvYH5Bme457l4h0FxEjIk7Pa5JE5EvP2DaIyIQy55wkIjNF5D3P+/pHREZU9LtRKlBoEFeqFkRkiojkA2uANOBrPy89DIgAPqvB4b8B+gDtgcXAtDLPvQVcbYyJBQYCP3nabwe2A+2wZ/v3YX+4KD/uWOAH4FsgCegN/NiY4zDGvAVcA8zzLHVPLDfGEGAOsBXoDnQCPip9GnslJAkYAHQBJnmOezGQDIz1HPepCt7TR57xJQHnAJNF5Ngyz5/ueU0C8CXwsp/fjVIBQYO4UrVgjLkOiAVGA58CRX5e2gZIN8a4anDsqcaYPM9sfxIwxDOjBygBDhCROGNMljFmcZn2RKCbZ6b/m6l4Y4TTgB3GmGeNMYWe8/zdBOOozMHYQfZOz8pFoTHmd8+YNhhjvjfGFBljdgPPAUdV56Ai0gU4HLjbc8yl2CsCl5R52e/GmK8919DfB4bUcOxKNSoN4krVkjHG7QkunYFrATwJa3s8X+OBDKBt6XJuVUQkRESeEJGNIpILbPE81dbz/WzgFGCriPwiIod52p8GNgD/E5FNInKPn1N0ATYGwDgq0wXYWtEHHxHpICIfeZbwc4EPyoypKklApjEmr0zbVuyZfqkdZX7OByKq+2+nVFPQIK5U3TnxXBM3xpzsWcqNMcZMA+Zhz9LPrOaxLsROeDseiMdeTgZ7GRljzAJjzBnYS9yfAzM97XnGmNuNMT2xl4RvE5HjKjj+NqBnAIyjMtuArn6C52Ts5flBnsTCi0rH5FHZrD8VaO25pFCqK5BSw/EpFTA0iCtVAyLSXkTOF5EYz2z1ROAC/FxXNsbkAA8Cr4jImSISJSKhInKyiFR0zTYWO+hnAFHYQav03GEiMl5E4o0xJUAuYHmeO01EeouIADmAu/S5cuYAiSJyi4iEi0isiBzSBOOozHzsPIMnRCRaRCJE5PAy49oD5IhIJ6B8cuBO/HxIMcZsA/4EHvccczBwJfZsXqmgpEFcqZox2Evn24Es4BngFmPMl347GPMscBvwALAbe6Z5A/YMtrz3sJd4U4BVwF/lnr8Y2OJZSr4GGO9p74OdsLYHe/Y/xRjzcwVjycO+P3ss9tLxeuCYxh5HZTzXo8diJ90lY/+ux3mefggYjv0B4SvsfISyHgce8GTH31HB4S/AXlVIxU42nGiM+aEm41MqkEjNc06UUkopFQh0Jq6UUkoFKQ3iSimlVJDSIK6UUkoFKQ3iSimlVJDSIK6UUkoFqaCrRNQ2Vkz3dk09ioblbu13QyylVBNYmrqoqYegWro00o0xPtEv6IJ493aw8NGmHkXDyrlwYVMPQSlVRsJDUvWLlGpIk9haUbMupyullFJBSoO4UkopFaQ0iCullFJBKuiuiVekJKQV2ztNojC8N0jwfy6xVq9u0vOHRkTQpnNnnKGhTToOpZRSlWsWQXx7p0nEdjqY7rFOpBnkn7jaDGiycxtjyMzIIGP7djr06NFk41BKKVW14J+2AoXhvWnTTAI4gDOj6bLTRYTWbdpQUljYZGNQSilVPc0iiCOOZhPAA4HoL1MppYJC8wjiQeadD2eTmra7qYehlFIqyGkQbwLvfDiH1B0axJVSStVNywziG7+Bj8fCOwfb3zd+U+dD7t1bwKnn38KQoy5k4BHjmPHZ/1i0dDVHjb2Kg469mBPPvZG0HenM+vJHFi5bzfhr/s3Qoy+koKCQH3+dz7BjxjNo9PlccdPDFBUVA3DfPfcw+IADGDZ4MHfdcQcAc2bPZtQhhzBi2DBOPP54du7cWeexK6WUCk7NIju9RjZ+A39OBrcncWvvDvsxQK+Ta33Yb3+aR1LHtnz10X8AyMndw8njbuKL95+lXdtWzPjsf9w/eQpTX3yQl9+cyTMP3cyIYQdQWFjEZTc8xI+fTqFv725cct1EXn17Fhdc048vPvuMlWvWICJkZ2cDcPgRR/DHX38hIrz15ps889RTPP3ss3X6lSilKpc90WjpVRWQWl4QXzxlfwAv5S602+sQxAcN6MXtD/6Hux96idPGHEGrhFhWrt7ECedcb5/CbZHYoa1Pv7UbttKjayf69u4GwKXnn8orb33MNXfHEx4RwYQrr+TU007j1NNOA2D79u1cOG4caWlplBQX011vA1NKqRar5S2n7/Wz/OyvvZr69u7G4p/eZ9CAXjww+VU+mf0zB/bvydK501k6dzorfvuI/816udrHczqdzJs/n7PPOYev5szh1JNOAuCWG2/kuhtuYOmKFUx5/XUK9VYwpRpF9kTT1ENQykfLC+LRHWrWXk2pabuJiozgovNO4c4bLubvxSvZnZHFvAXLASgpcfHPmo0AxMZEkbcnH4B+vbuxZVsqGzZtA+D9mV9z1KjhFG79lZycHE4+5RSeff55li9bBkBOTg5JnToB8N6779ZpzEoppYJby1tOH36d9zVxgJAIu70OVqzewJ2TXsThEEKdTl59+h6czhBuuvdZcvL24HK5uOXqCziwfy8uu2As19zxOJGR4cz7Zipvv/Qg5155Dy6Xm5HDDuCay84mMyuXM849jcLCQowxPP3ccwA8OGkSF5x7LgmtWnHMsceyZfPmOo1bKaVU8BJjgmuJaERPMeX3E1/d9xsG9PC93uzXxm/sa+B7d9oz8OHX1el6eENxtRnRZOdeu3o1nQc0XflXpQKRJrepJjOJRcYYn6DQ8mbiYAfsAAzaSqnAplnqKtC0vGviQaQpa6grpZQKfBrElVJKqSClQVwppWpAbzVTgUSDuFJKKRWkNIgrpVQN6WxcBQoN4gHqwcdf44df/q5xctsvc+dyhqdEq1JKqeatZd5iFiCMMRhjcDh8P0s9fO81jTIGl8uF06n/GSilVDBqkTPxabN20n3oPBzt5tJ96Dymzapb3fR7Hn6JV96aue/xpCff4JmX3+fpl95n5PGXMPjIC5j4xOsAbElOpd8hZ3PJdRMZeMQ4tqXs5LIbJjHwiHEMGn0+z786HYDLbpjErC9/BGDBggWMHjWK4UOGcNjBB5OXl0dhYSFXXn45QwcNYsSwYcz9+WefcWVmZnL2mWcybPBgDj/0UJYvt0vAPjxpEpdefDFHHn44l118cZ3eu1ItlS6pq0DQ4qZg02bt5Kpb15JfYAGwdXsRV926FoDx59Sufvq4M8dwy/3Pcv2V5wEw84sfuPumS/jj72XM//5djDGcPv52fv1zMV07d2T9pm28+8okDh0xiEVLV5OStpuVv88AIDsnz+vYxcUljB83nmkzZjBy5Ehyc3OJjIzkpRdeQERYumIFa9as4ZQxY1i1bp1X34cmTmTosGF88vnn/PzTT1x+ySUsWroUgNWrVvHL778TGRlZq/eslFKq6bW4mfj9j27aF8BL5RdY3P/oplofc9jgfuxKzyI1bTfLVq6jVUIsK1Zt4H9z/2bYMeMZfuxFrNmwhfWeTU66dUnk0BGDAOjZvRObtqZw4z1P8+2PfxIXG+117LUbtpLYLpaRI0cCEBcXh9Pp5I/ff2f8RRcB0L9/f7p268a6ckH8j99/Z7xnpn3MsceSmZFBbm4uAGNPP10DuFJKBbkWNxNPTimqUXt1nXv6ccya/SM7dmUw7swT2LotjXtvvoyrL/uX1+u2JKcSHRWx73GrhDiWzZ3Odz//xWvvfMrML35g6osP1mks1REdHV31i5RSldIyrKqptbiZeNdO4TVqr65xZ53AR5/9j1lf/sS5px/PiccextTpX7LHs+VoStoudu3O9OmXnpGNZSzOHnssj953DYuXr/F6vl/vbqTtTGfBggUA5OXl4XK5OHz0aKZPmwbAunXr2JacTL9+/bz6HjF6NB96XvPL3Lm0aduWuLi4Or1PpZRSgaPFzcQfe6Cn1zVxgKhIB4890LNOxz2wfy/y9uTTKbEdiR3bktixLavXbeawk68AICY6ig9efZiQEO/PTSlpu7j8xoexjD2exx+43uv5sLBQZvx3MjfceCMFBQVERkby3Q8/cO1113H9tdcydNAgnE4nb73zDuHh3h9EHpw0iQlXXMGwwYOJiopiqu4/rlS909m4akotcivSabN2cv+jm0hOKaJrp3Aee6BnrZPaGktjb0uqW5EqVX0axFWD061I9xt/ToeAD9rlOTMWNun+4kop/3Q2rppKi7smrpRSSjUXGsSVUkqpIKVBPIjUtI66Ukqp5k2DuFJK1QMtw6qaggZxpZRSKkhpEG8gqWm7Oefyu2vc75Tzb/apn16WM2Mhkx58kB9/+KEuw1NKNQCdjavG1iJvMWsMSYntmPX2kz7tVW39+fVHL1R57EkPP1ynsSmllGoeWuRMfNqSb+j+xFgc9xxM9yfGMm3JN3U6nr+tSAceMQ6Adz6czenjb+PYM6/luLOuIz+/kPOuvJcDRp3HWZfcySFjLmPhklUAdB92OukZ2WxJTmXAYecy4ZZHOfDw8xhzzg0UFBQC8H8XnMYns2YBFW9TumXLFo4ePZqRw4czcvhw/vzzzzq9P6WUUoGpxQXxaUu+4apPJ7M1ewcGw9bsHVz16eQ6BfJxZ45h5uf7l7dnfvEDhxw00Os1i5evZdbbT/DL7DeYMvVjWsXHsurPmTxy79UsWram/CEBWL9pG9dfeS7//DGThPhYPpn9k9fzxcXFjB83judeeIHFy5bx3Q8/EBkZSfv27fn2++9ZsHgx02fM4Nabbqr1e1NK1YwuqavG1GBBXES6iMjPIrJKRP4RkZsreM3RIpIjIks9Xw2+fdf9300hv6TQqy2/pJD7v5tS62NWtBVpl07eFeFOOPpgWreKB+D3v5dx/lljABg4oDeDD+hd4XF7dE1i6CB7U5ODhvRny7Y0r+fXrl1Lx8REn21KS0pKuHrCBIYOGsT5557L6lWrav3elFLNRBGwClgO+E+7qR4DpABLgK1A2d2dC4CVwAogv47nUVVqyGviLuB2Y8xiEYkFFonI98aY8hHlN2PMaQ04Di/J2Ttr1F5d5bciLS86quZ7d4eHh+77OcThoMDl3v84byPQr4Je8MLzz9OhQwcWL1uGZVnERERU+DqlVMMIuDKsacA6oAv2X/2VQCugfy2O5Qb+8PRvD+QAvwKjPOdJ9pxHsIN8IlC3/aVUJRpsJm6MSTPGLPb8nAesBjo11Pmqq2tCxTXT/bVXV/mtSCtz+MFDmPmFvfy+au0mVqzeUKtz9uvXjx1paT7blObk5JCYmIjD4eCD99/H7XZXcSSlVLNlsAN4PyAaCMcOqhlAYSX9/FkDJGEHcCfQBuiBPfPeCvQFIoEIoDewHXtKpxpEo1wTF5HuwDDg7wqePkxElonINyJyYEOP5bETryMq1HtmGhUawWMnXlen45bfirQy111xDrszsjhg1Hk8MPlVDuzfk/i4mBqfMypvOdNmzOCWG29k+JAhnHTCCRQWFnLNddfx3rvvMnzIENauWUN0dHRt35ZSKtjtBaKwZ8ZltcNeEq+pXCC2XFsEkAnEVfD6VsCuWpxHVUuDb0UqIjHAL8BjxphPyz0XB1jGmD0icgrwgjGmTwXHuAq4CqBrWw7aWu4urBpvRbrkG+7/bgrJ2TvpmtCBx068jvHDTq7pW6s1t9tNSYmLiIhwNm7ezvFnX8/av2YRFhZadedyGmpnM92KVKm6CZjl9CJgIfasuKxd2LPopBoe7y+gG95TQIN9vT3G81xZ27Fn/q1qeB7lrSm2IhWRUOATYFr5AA5gjMkt8/PXIjJFRNoaY9LLve4N4A2w9xOv67jGDzu5UYN2efn5hRxz5rWUuFwYY5jy1F21CuBKqcAVMNfFw7EDbgH2MjfY17XTgdqsffYCNgHdy7SlAn2ALUAxEOZpL8FeCdAA3mAaLIiLiABvAauNMc/5eU1HYKcxxojIwdj/qWU01JgCRWxsNAt/fK9ejqX7jCulqjQSWIw9Kwf7L+1IandBtR121vk67CV6C3s23wXo4DlPiee1IcDBtR61qoaGnIkfDlwMrBCRpZ62+4CuAMaY14BzgGtFxIX9OfF809Dr+0op1UgCZjbuxA6mxvNV12yobp4vq9yxwoBD6/E8qkoNFsSNMb/jm0pR/jUvAy/X/WQWxoAEwP8rTaG+Z+P6OUqpZkqo4q9yDfkL0vV9HuVXs/icFFG0gYw8Fxp76s4YQ2ZGBqF6b7lSSgW8ZrEBSueUSWxnErvDe4M0i88lNbf5W6zo8mmhtRMaEUGbzp3r5VhKtSTffjeHz36YSWFJAb0T+3HztXc17pJ6CjAf+1q0CzsjfXDjnFo1jWYRxEPdWfRI9qnq2uLkXKhLEUo1lZmzpjFl7vPQ0X68ae8G1t69indemll5x/qShV2JYwR2ELewk8+WAkMbZwiq8bXQaatSStWvz36euS+AA4SEhbAlfCPf//Bt4wxgHvasO8Tz2IFdPW1T45xeNQ0N4s1I/HTNJFGqKRhjyCnM8ml3JISwau3yxtnZrJj994HvGwCgJSiaNQ3iSilVRyJC+5iOPu0m3TD6sGMaZxAx2JuRlGWx/55t1SxpEG9mdDauVNOYMO4GQjeFYrnsfTldmSUcFHMII0ceCjTCPuNHYO9OVrrNaDF24ZXhDXta1bSaRWKbUko1tcNHHclbSR/yzodvkFeQy2GjR3PG6ecgjVXAIgIYC/yGXVEtBHt70HaNc3rVNDSIK6VUPenavTsP3ju56QYQCYxputOrxqfL6c2QLqkrFZgaJcFNtSgaxJVSqh7t2rmT9WvX4nK5vNqLi4tZt2aNvatXeXnYu4pZ5dpdwG7s5fH6kIO9xVT581QmG3uv8Op+/jDY96zXpE8gs7D/bXKremHT0OX0Zip+umjxF6UaUX5+Pvc8dDP/ZC2n0FFIRyuRa8fdwvHHncQXsz/hnTmvs8uxk5HuQ1mw7C97QxIXsAB7u9Aw4B/svbe7YBdq2QnEYW8PZbD7hFR09irsxd5TPNrTfwUwAHvXMX/ygEXYWe8OYDkwEGhbSZ8cYAkQi107fTn2veutazHmQJCK/e+QgJ3lX4C9+1v5W/makAZxpZSqB5OffZAloQsJ6RpCKKFkkM7T0x4lqUNnXvr8aVw9XIQSSgkldvBeip1B3oP993InAmux/zJnYRdrKZUPLKN22eaLsff7Lv0AkAisBtrgPwosAfqxf702CTv7/UgqXsM1nj792b/5SRJ2IPfXJ5CVAOuxP+yUcmH/Lg9vkhFVKNh+rUopFXCMMaxIXkpIhPc0eW/HPTz05L0Udy727hCBPauz8C3G0hk7wHcp1x5F7ZbVCzznKDs0Adpj11qvSB72bLNshBDsWfgOP32y2T8DL9snAXs5Othsxf6wU1bpB55y/5xNSYN4M6YJbko1Hsv4XmgWh2C5rYq35fR3tUs8z9XX/77+jiX4vzZe1dhqep5gvLJX2fsJIBrElVKqjkSEA5IG4i52e7VHpkVy762TCE3xnm4Paj3Uno0L9hJtWduxryNvL9deiN2npqKAInwD9i58Z/ul4rBn/WWDr8FOsvMtTGdrhZ38Vb5PFsF5r3o3IK1cmxv79xjW+MPxR4N4M6ezcaUax/13PMoBeQMx2wyFaQXEbY3n+jNvZ/jIkfzfCdcTmxxLQWo+bIMhRcPtncWGAxuwl7V3AWuwl9M7YyehrccOnMnANmq/G9lQ7GvtqWXO05vKs6KGYF83T8NOsFuDfb3bX2KdYH/4WI295L7D02cgwRlpwrDzFdZi/85Kk9wCbEc4MSa41jlG9BSz8NGmHkVw0Sx1pRrPls2bycrIYMDAgURE7J865+fns3bVKtq2b0+Xrl0B9u8znok9026Pd2Atwr6eHAPE13FgBvv2shLPeaqT5W4853d7+lQnGJfekmWwZ+DBGMDLcmMH8TDsLPummhdNYpExZkT5Zs1OV0qpetS9Rw+69+jh0x4VFcWwET5/g23+bsEKBzrV08BKE9Nq2qemS+EO7IDfXITgm+AWQIL9M5KqBl1SV0qp5kmDuFJKNZF6L8Pqxr6G+zf2PeWF1ejjwr52/Tf2Pd1F9TukgJKK/T4XYCfcNQMaxFsInY0r1cy5sHcwE6A79v3Z86i8XGiJp0+Ip08s8CewpwHH2VSWYF/b7oZ9iWItsKlJR1QvNIgrpVQTqrfZ+Abs4FSaABeBnU3+TyV91mIHtTjP4yjsKm0r62dIAWMP9gpDIvaHnBDszPPt1KyOfADSIN6C6GxcqWYsh/3BuJSDyoPUHuxb2coKqaJPMNqBXWK2vCgq3pAmiGgQV0qp5iCMisuBVjbRd2IvqZd/fXO7KzWOioN1bQvoBBAN4kop1cTqZUm9H7AR7wC8DXu5vLI+m8r12Yq91NyctMNeqSibtJeLXR++fO36IKP3ibcwukWpUs1UFHY1sVXsn013xX9pVbAT2QZiV1nD06c79u5jzYkAh2Fn7Bd5HscBw5pyUPVDg7hSSjUXCcCoGvZpTUBtrdlgwrD3Am9mdDm9BdIEN6UCT73fM65aBA3iSimlVJDS5XSllAoQ2RPN/k1R0rGvVZfux90O+75vC/vabmlBFgf2tfBo7GIma9l/a1lHoA92Jbel2NuLgv2Xfyh2Ypc/Lk+fAs/jUE+fQM/m3ohdma30d9CbgK59XlcaxFsoTXBTKoDlYxdc6c/+XbN2Y2+FmYt97bs0+cyNXUp0BHYA71Omzw7s7PN07M1POnvaXZ4+R+F/V64F2MGvNNCXlOkTqJKxfz99y7RtwH4PCU0yogany+lKKRVo1gE98Q6w7bCDcgF2VnmpEKADdtDtUa5PR+yqZCV4F3VxAq2w9wmvSOnsu+xMPRQ7ozu9Bu+jsW3DN7O+B/bvs5nSIN6CaYKbUgGqCDubuiIV3dcciV3opaLnjJ9jVVatrBB7G9TyIti/JB8smmMFujI0iCulVADJnmjsmXVGuSfc2DPoInwrqu3CrptefmcuN3YAryjwpuP/WnE8FW+Ckok9tkAVim8FutLLD82UBnGllAo0XYFs7OVugx1Q1wKDsK/3rsOeLbuxl5AjsYu27MYO6AbIw95idDDQy9OnyNNnK/aSfJSf8zuwK71twJ7hu4DN2PeUVzRDDxSDsN9nDvbvIB07ya1PUw6qYYkxwZXcNKKnmIWPNvUomhdNcFMq8CRMEkgB0oAY7Czr0uXyAmA9doDtjp20Bvay8Xbs4B/r6VOavpyPHeBc2NfbW1djEHs857GwPwgEw4zWhZ3Ml4OdR9CV5jFdncQiY8yI8s2ana6UUoFIsLPJO1fwXCT2DLs8B3bQ6lrBc6VlWWsihuArTerEOzu9mWsOn09UHWmCm1JKBacGC+Ii0kVEfhaRVSLyj4jcXMFrREReFJENIrJcRIY31HiUUqomjDEsWriAOXM+IzPDO8ssKzOTr+Z8zqIF87Gs/anPlmWxYP5ffDXnc7KzvLPMMtLTmT3nUxYvWkh1LmNqGVaPDOwl/ex6OFYx9lJ7MnZuQDPQkMvpLuB2Y8xiEYkFFonI98aYVWVeczJ2ykEf4BDgVc93pZRqMtnZ2dx8/wQ2mvW4I93EfR7HOaMuYMIVN/DO+2/w4S/vkRufQ0hRCD3e7sWLj/4Xt9vNzf++iq3OzbjD3MR9Hs+Fx1zGpeP/j9fffJFP/v6IvLg8QgpD6PN2P16Y/F/i4uKa+q0GLjcwD/vSQTx2IHcDB1O76edm7CTADthJgb8CQ6hebkAAa7TENhH5AnjZGPN9mbbXgbnGmA89j9cCRxtj0vwdRxPbGo4muCllu3vizczjNxyh+6OFJAuPjH+aBz+4G6vb/mmc5bIYUXIolnGzOGwBDuf+PiFbQ5g4/gkenH6n13Vqq9jiiJCjmDzx+SrHsq8Ma0uzHLtATUyZtizsAF7Ta97FwF/Y+6eXMtiJfkfWYYyNyU9iW6NcExeR7tjpEX+Xe6oT9mejUts9bUop1WTW7VjtFcAB3ElunnllMq4k7xuRHU4H63euZsOudV4BHKC4YzHPv/o4ppP3B2RHmIM1qatQldiDdwAHu8pc+fvnq2M7vve3C/btcv4K3gSJBg/iIhIDfALcYozJreUxrhKRhSKycHde/Y5P7acJbkrZnA7fK43GZYiJisEq8S3/FRoSVmEfq8QiJjoWy1VxH1WJihYGa7tYGErF18At7IpuQaxBg7iIhGIH8GnGmE8reEkK0KXM486eNi/GmDeMMSOMMSPaxZZ/Viml6tfB/UbhynN5tUWnxvDIg88Qt8P7OrZrj4uRfQ9jRO9Dce3x7pOwK4GH7n+KqJRo7z65Lg4dcHi1xtJiE9w6YBeuKSuVim+fq0on9hfOKVWCHcQDfVe2KjRkdroAb6j9sfMAACAASURBVAGrjTHP+XnZl8Alniz1Q4Gcyq6HK6VUY7j1hnsYk3AK0ckxOLY5SExN4v4rHqFHz578+6rJJKZ0wrEthKjkaI6NHcMdN93PXbf8m6OjTyBqaxSObSEkpXRm4rVP0rNXL+697CE6piTh2OYgOjmGMa1P5abr7mzqtxnYenu+r8POJl+Hfa97bS64lm7XuhbYgp2hvhU4qM6jbHINltgmIkcAvwEr2F9+/j48n6OMMa95Av3LwEnY9YQuN8YsrOy4mtjW8DTBTSlbfn4+e/fsoW27dth/rmzGGNJ37yY6JoaoKO/apXv37iV/716fPpZlkb57NzGxsT59qtJik9vAvs+pEDtLvT6WvvOxg3qwzcD9JLZp2VXlQ4O4UoGnRQdy1bTZ6Sq4aIKbUkoFB62drpRSNVBUVMTb773Oio1LiQqL5uKzr2TwELso+YwZ03hh6pMUWoUM7DKYKS++Q0xM+fukvC1Y8BcfffEeha5CDhpwCBdfeAWhob4bg2dPNIE/GzfY15t3eh63w762HeDDDma6nK780mV1pbxZlsXVt17CqsgVhMaEYiyDc3so953/EAsW/sWLPzyN1ctCHIKVa9FxcyKL5q7D6ax4vvT5l7N44asnsTrZfVw5JQyTkbz45H+9rqeXCvggvgT7dq52nscZ2DuuNYMEsiany+lKKVU3P/38PWtYSWiMPVMWh+Du6uK9z9/kndlvYPoYxGEHWkecg50d0nh1ygsVHssYw4zv3sd02d/HGR/KsvzFLFo4v3HeUH0qwk4aa1emrQ12tbT8JhlRi6BBXCmlqmnh0r9wtPFNkc7I300+e3w7tIWffv+uwmMVFRWRWexbfswkGBYvqziIB/Q94znYe5iXF0v9bF6iKqRBXPmlCW5KeTto8MG4M10+7a2j2hJJtG+HdDjqsOMqPFZ4eDitQivYfSMbhg72WTUNfHFQ0ecY9gAJjTyWFkSDuFJKVdNxx51IP/cBuPbagdxYBse2EC4aewUXnXw5bGDfNqPWHov2aR244cbbKzyWiHDumAuRFNnXx5VXwsCwIYwceWjjvKH6FOH5Kru4kIWdPl2z2+JVDWhim6qSJrgptV9BQQFvvj2Ff7YuJzo0mgvOvJQRI+wdlN97701eef95ikwh/RMP4LWX3ichofJp6J9//sbMr6ZRWFLAsD4Hcfml1xAWVnld9YBNcDPABmC353Eb7B3HAnS4QUWLvaja0iCuVGAJ2CCuGo5mpyullFLNiwZxpVSL5na7WbF0KakpPhsokpuby+aNG7Es361E/SksLCQrM5OGXOXMnmjsfbCzKnjSwr6lyzf/zr/SPhVt11mffSpT5PlSNaIV21SV4qeLLqmrZunZZ57gqbcepaRVMVIktClpx19zlxMTE8MZ445nVcYKSsJKiC2M5Y7L/s2E/7vO77FKSkp47JkHWbBpHkVSRMfQRG678l6GDx9Zr2NOT0/nzPHH27d0ObG/Dwd6ApuBbUA49labMcAQKr8mvQFIA8Kw7+mOBwZV0WcdsMNznmKgNXBgLd/QHuwiMSGec7qwi8NoMly16DVxVW0ayFVzsi05maGn9ME90uyrjmYKLBJXd2LAgH78GfIrEr9/sdK5zMmv7y6mR69eFR7v8Wcm8nX6l4TE2PeRG2OI2RzLRy/MJja2ohuoa2f0ScNYl7QGR6SDgpJ8e0Y8Hzgce5vNHmVenIX9/AF+DrYbO/B3K9OWgf3hoK+fPmnY+3p3LtO2C4jG/iBREwb4xXOu0tvv3cB64KgaHqu502viSim13/XXT8A9wO1V3lQiHew2O1i6bZFXAAco7lvMAw9XfLuYMYb56+ftC+Bg30KW3T6Lzz6fUW9jzs7OZkvBJhyRZcbmwK5P/huejZ7LaEXFS+6lNgFdyrW1YX92eUW24Lund3vsmXlNZWLP/MvWzwnB/kCQU4vjtUAaxJVSLVJWbiaE+q4ZmxCD2/he6JVQyN2b6/d4xe5inzZHmIOcvPorV1aQn48VUsH1+VDsZeiK/qJXtoBm+elTlfpKji+m4ou6IdiXA1SVNIgrpVqkRyc+iWz2jkbGMkQWRNMpugumxDtYyhbhpgl3VngsEaFX275YLu8+oamhnDV2XL2NOTEpidYlrTFWuci8FRiM7wy6CLsAiz/t8S7OAvaGJZGV9GmNbxnVvVBRwboqtafilYIcz3lUlTSIq2rTMqyqOTnq2GMZGX8YssJgct2YXRZh85xMfW4a0179jITlrbBSLawcC8caB8ckncgJJ57s93gP3PoIndK64NpZQnFOMWFbwrj4yCvp3KX8enXdvPLo20Qvjsba6SZ8bzgsxQ6gBwCF2AF9L/by9mbs4O5PTyAPSPb0SWP/BwJ/+mAvg2/39EkFUoCBtXgzIZ4xrMEO3NnAaqAfGp2qSRPbVI1pgptqTpYuWsQjkyfRvUt3Hn38KSIj7Wmoy+XinbffYN3GNVxx8dX0P7Dq9GvLspj35+/s2JXK0UceT5u2bRtkzIWFhbz++kvs3r2Dl/c+512bPBt7P+94oAPVW/rOwk5OS8CeHVenTyb2zL+Vp09dFGNn1YN9Xd93O3WlFdtUfdEgrlRg0QpuLYBmpyullFLNiwZxVWN6bVw1peLiYj77fCZPPfcI8+fPa9DKaP6sWvEPTz/6EtPfmUl+fn6jn7+8gN5nXDUordimlAoaGenpXHfvZWyPTcaZEMqc9z7jkC9G8eQjL+JwNM6c5PFJL/Dt7N0Y0wu3O42Z0+7m+dduo0evHlV3Vqqe6Uxc1YrOxlVTePaVyaR1SiW0dRjiEBwdHczb8xs//PBto5z/n+Ur+XZ2OiL9cDichIa2JitzBE8/+najnF+p8qoVxEWklYgcKCI9RUQDv1KqSWxO34jD6f0nKKStk1/++qFRzv/1lz9TvraoiIPkLXsa5fyV0SX1lsnvcrqIxAPXAxdgl8bfjV02oIOI/AVMMcb83CijVEopINLpuyuGVWyR0KZxKoMkdW6Py7WD0FDv80VGhfjpoVTDqmxWPQv7zr3Rxph+xpgjjDEjjDFdgCeAM0TkykYZpQpIuqSuGtvYo86y72f2MMYQkxLDpRdMaJTzn3XeWNq334gx+yuzud27OHaMvx1GlGpYfmfixpgTKnluEbCoQUaklFJ+nHXmeeQX5jPn18/IK8mlY3QS1994K+07dGiU80dFRfHca7fy9CNTSd66l6goB8eOOZCrb7qsUc5fleyJRu8Zb2GqVexFRAYD3SkT9I0xnzbcsPzTYi+BR4u/KBU4NIg3U36KvVR5i5mITMWupPsP9p43YO+L0yRBXCmllFK26twnfqgxRi/4KKVUENAl9ZalOkF8nogcYIxZ1eCjUUEpfrrokrpqNL/89DsfvvsdebluEpMiufXeK+jUuVOlfX749mdmTvuRvXssOneN4dZ7rqBjYsd6H9sXs+Yw+7O/KMy36NE7ntvvu4qEVq3q/Tz1Kgt757DS/4V7AElNNxxVM1VeExeRo4AvsTe2K8Le38YYYyrbrK7B6DXxwKRBXDWGuT/+xiP3fYfb3Q8AY9wktFrMe588Qnx8fIV9vvnyfzz96B9YVm8ALMtF23ZLeP/Tx4mOrs0m2BWb9s4s3nhpDSLdPOcponPXlbw36xmczsYvjlmt2fgeYCHQH89fduytSLsCiQ04OFVzddgA5S3gYuAkYCxwmue7Uvvo7WaqMUx/Z38ABxAJISN9AB9MneW3z8zpP+8L4AAOh5OdO/ow44P6TeuZ/enf+wK4fZ5wkjd34rs539freerVWqA3+7ceFaAbsKnJRqRqqDpBfLcx5ktjzGZjzNbSrwYfmVJKlZOX6/ZpczpjSNmW7r9PjuXTFhqawNZNO+ttXMYYcrN9x+YIacuaVQEcEUvw3btbqN5+4iogVCeILxGR6SJygYj8q/SrwUemlFLldEyK8Cq0AlBSvJvhI/tW0ifcZ6cztyuNQw4fVG/jEhE6JIVX8Mw2jjnh0Ho7T01UqwxrDFB+EzY3uqtGEKnOP1Uk9rXwMdjL6KVL6kp50SV11dBuvvNS4uIX4XbbkaekZDcDBu3gjHP8/0m68Y6LiI5ZhNtd4Omzk4FDszjx1OPrdWzX3nw24eGLsawiAFwlKRwyyjBsxPB6PU+96gdswb42DlCIvcR+YFMNSNVUtYq9BBJNbAtsmuCmGlpWZibvvvkxO1KzGDaiD2edN5awsLBK+6Tv3s27//2Y3btyGXlof84457QGSTZLTUnh3f9+Qm52PqOPHcZJp53QaFukVqRayW0u7MCdh707Rn/PdxVY/CS2VSc7/V3gZmNMtudxK+BZY8wVDTLQKmgQD3wayJUKHHrPeDNRh+z0waUBHMAYkwUMq8+xKaWUUqrmqhPEHZ7ZNwAi0prqFYlRLZReG1flWZbFmlWr2LRhg0+SWW18MHUqV19yCcuXLvVqd7lcrFq5kq1btvj02ZGWxsrlyykoKKj2eZYsXMjns2aRm5vr1V5cXMzK5ctJ2b7dp8/2bdtYuXw5xcXF1T6PUrVVnWD8LHbVto89j88FHquqk6fm+mnALmPMwAqePxr4AtjsafrUGPNwdQatlAoeSxctY/KD77B9WxwhIRbde+bz5Iu3k9Sp5mXB0tLSOLDzcVhWf6A9M96/g9at09mUsZS5P/7Gi099SlpqAmFhxfTuV8IzL99DdEwM997yBEsWFZC/J4J2HfO44JLDuPDSs/2eJz09nZOOuITU7W1wlcQTFf0u5198EE+8MInZcz7lzS9eZaczjUh3JAfED+KpSS9hjOGuSTeyOncFBSGFdHQlctXZN3LKSafX4bdXd1qGtXmr7i5mBwDHeh7+VJ0SrCJyJHbO43uVBPE7jDE1ynTXa+LBQa+LK4CSkhLGnXYnGenDEbEDiTFuevddzVsfPlnj4yVGH0BB/llA2VKmizjjvHDSdyawd89B+1otq5hhI7aR2Kk1cz5z4nTG7ntOZCVvfXgVPXv3qvA8xx92HiuWDsHh2F/RzSF/8P6nE3hixiSKe+yfZbsL3YwOOxZjWfzunktIeMi+58I2h/Phk1/Qpm3bGr/X+qRBvBmo6TVxEYkp/dkYs8oY87Lna1VFrynPGPMrkFmHISulgtzCvxeQltJuXwAHu8ra5k0Odu/aVePjFeQn4B3AAYYye9ZfZGV282p1OMJYt2YPi+YnewVwALe7Px+9N9vveTatL/AK4HafEdx1930UJHovx4dEhLAyeSkrU5Z7BXCA/I57mfX5h9V7cw2oWveMq6BU2TXxL0TkWRE5UkT2/dcsIj1F5EoR+Q67FGtdHCYiy0TkGxHxe2eiiFwlIgtFZOHuvDqeUTUKvS6uSpUN4PvaqOegUov/3CoYVq1PU9F7VKox+A3ixpjjgB+Bq4F/RCRXRDKAD4COwKXGGP8Fi6u2GOhmjBkCvAR8XslY3jDGjDDGjGgX6+9VKtBoIFcjDhlJx067vJLZjHHTvaehXfv2NT5eZFQ2vgt8Sznr3FG0bu1dDdqyiuk7IIaDDu6K2+X96d/pXMP5l/i/Vt2rTySWtderLSRkIU8+OZnItEivdnehm0FdhzGo82DcRd6lV6PSojn3rAur9+aUqoVKE9uMMV8DXzfEiY0xuWV+/lpEpohIW2OM/yLISqmgEhoayoOTL2fyg++QUiax7dFn76jV8RZv+JEDOx+LZfXDnkusp3XrDN76cCm//vwHLzw1i7Tt8YSFF9NvgMVDT9xNVHQ06bufYMmCZPL3RtKuQy4XXnY4PXr19HueGV+9xomHX0xKcivc7ngio5K58LKRHDtmDPkle3nLk9gW4Y5kYNwQ7rvvISzL4u6HbmL1rpUUOAro4OrI1efdSOs2bWr3y6tnmuDWPDVoxTYR6Q7M8ZPY1hHYaYwxInIwMAt7Zl7pgDSxLbhogpsC+xazDevWERYWRrcePeq8/Pzhu+/y688/c+0ttzB46NB97S6Xiw3r1hEdE0OXrl29+uzauZPdu3bRq08fIiKqV5Js2eLFJG/dyjEnnEBMzP4UoOLiYjasW0er1q1JTPLOsk9NSSE7K4vefftWWUmusWkQD2K1rdhWWyLyIXA00BbYCUzEs1+OMeY1EbkBuBa76F8BcJsx5s+qjqtBPPhoIFcqMGgQD2J+gniDFW0xxlxQxfMvAy831PmVUkp50yX15qdaQVxEQoAOZV9vjEluqEGp5iV+uuhsPMjk5eXxwdSP2bJpJ737JnHhZecQHR1ddcd6YlkWLz3zCrM+/JnwcCf3PnQVx51Y+a5jxhh++O4nfvx2PlFRYVx85Vn7rnuv+ecf7rrpCTIz8jlkVF8eeebfREVFYVkW3371Pb/8sJjYuAguufJfdO3erdLzBLLi4mI+njWdFeuX0L51IpePv4pWrVs39bBUA6rOBig3Yi+F7wRKN/I1xpjBDTy2CulyenDSIB48srOymDB+EmkpfXGGxlFSkkuXrut5Y9rDxMXFNcoYTjnqAhbNb4NIL8CNw7GYCdcP5KEn7/PbZ+LdT/HTdy4cIZ0xxk1Y2D/cM+lUSoqLuOrilykpPgJxRGC5d5DYaTHzV8/h33c+xR9znThDO2FZJURE/MO/J5/N6KNHNcr7rE/FxcVcfdslrI9cjTM+FHehm4SUVrwyaSpdu3X3eq3OxoNQHTZAuRnoZ4w50BgzyPPVJAFcKdXwpvznXXakDcIZagfs0NA4UrYP4I2X3m+U8y/4+2+WLgrB4eiNiCDixJiDmfbO77hcrgr7bFy/gV9/yiHE2QURweFw4nIN4e3Xv+H+21+ipOQ4xGEnszlCOrIjdTC3XXsXf/1egjO0k93uCKW4eChvvvJlo7zP+jbrk+n7AjjYRWhye+Tw4n+fbuKRqYZUnSC+Dchp6IGo5k3vGQ8eyZuzCQkJ92oLCYli88aMRjn/px99hdvV3ac9f28bNq5fX2Gf339ZQElJJ5/29F1CVqZvMRZxdGbu94uwrC4+fXbtKPH7YSGQLV+3dF8ALyUOIS03tYlGpBpDZWVXbxOR24BNwFwRube0zdOulGqG4luFYYzl1WZZLlq1DvfTo36NPGQI4tjt0x4alkunLr5BF6Bv/x4gvlWeo2PcREb5vt6ysujRuyPG8i1LERsnhISE+HYKcIltk3AV+H74iA33vQSiZVibj8pm4rGer2TgeyCsTJvfmulK+aOz8eDwf9edS3jEsn1V1oyxiIpezoTrxzXK+c887yw6Jm7GsvbsazMmhZGHtPG6V7usQ0YdTN9+Objd++uaW1YqJ542kPGXH4OwfP+xrBKio//kzekv0aP3LtzuojLn2capZx0UlGVUL71wAq3T2mDcpf9uBuc2JxedeUUTj0w1pOoktp1rjPm4qrbGooltwU0T3ILDimUr+e/Ls8jKKKF121CuuXkcAw4c0Gjnz8zM5JKzb2DDuj04nYYjj+nDy1OfweHwP+/Yu3cv/3nyDdb8s4vQMAdjThnOuIv+hYjw7GMv8t7U/1FYEEKHjiFMeedhBg4eSG5uLs8//gYb1mUSEeHglDMO4azzxjba+6xv27Yl8+J/n2ZHTgqx4XGMP/MKDh91ZIWv1eS2IFPbYi8istgYM7yqtsaiQTz4aSBXKjBoIA8iNS32IiInA6cAnUTkxTJPxWFXWVNKKaVUE6qs2EsqsAg43fO9VB5wa0MOSimllFJV8xvEjTHLgGUiMs0YU9KIY1LNnFZwq38FBQU8Mellli3egdsyHHBgG+5/9MZ6L86ydvVqTj7yCnKyohCB1m0LmbtwJkmdOnHf7Y/wyYd/UZDvJCbWxTU3n8ZNd1zL1s2bOf/0m0ndbnA4oE+/KGZ9+1/i4uK44/p/8+WniykscBIX5+Lme85lwnWXkb57N48+MIWN63NwOoWDR3XjjvuvJTQ0lGnvzGLOp/PJzXXRMTGc6287j+Ejh5GWmsbjE19j04Y8wsMdHHpED26795qgzDRvLFqGNfj5vSYuIisAv39ptWKbqgsN4vXr9use5u8/2+N02qVRLauIvgPW8+a0J+v1PEkxI8nfeybisG83M9Ze4hO+5o77L2XyxMXA/g0LHTKPqTOv4NarnyEz/QTEEeYZWy59+y/mpNNG8fKzm0D67+sTEvIrM7+6nZef/ZjtyUNwOOx5htudy7Fj8hk8rB8vPbMCsHcoM8YQGbWYqR/dyS1XP8XOtKGI2H1crixOPcNw70M31+vvoLnRIB4kaprYJiKlBYSv93wvLdd0EXbZ1XvqfZDVoEG8+dBAXj8y0tM5f+zTlJR47/hrWWuZ+tEl9O7bt17OM+ODD7jmku9Ahni1C/PomJhNZuZpXrdmGeOmQ8fZ7Ejrh8PhndkeEvIH8Ql5ZGac5NVurBJ69vkJhxyOM9S7hnl0zHLatBOSNw/yane7izhw0EpWLOtIaKh3wZe4+KV88u0ThIZ6F0FR3jSQB4Gall01xmw1xmwFTjDG3GWMWeH5uhsY05BjVUpVX25ODoWFvvtWFxdFkpFef1XW1qxagzHxPu2WFU9BgfGtiiYh7N1TgrF8l/Td7hiKCiyfdnGEkptdhGX5brZSWOhkb16hT7vDEcbOHVnYJSzK9SlwUFRU5NOuVHNRnbKrIiKHl3kwqpr9lFKNoGv37nRIzPdpb9c+k0FDh1TQo3YmXH8tISFrfdqdoRs5YFAHjLXXq91y72b0MQMJj9jg0yc6Zgfde8dhLO+gbFkpnHDKQcTGpXi1G2PomOii74BELMs7RcftSmHcxScTGbnVp09SZ4ffAjFqP63gFryqE4yvBKaIyBYR2QpMAbQEkKozreBWP0JCQrj25rFERCzG5dqD211AaOgyLr3qSKKiKqg5WktJnTpxzAmJwC8YKx9j7UH4nrPPH8LUj56ndZufsaxtGOPGsjbSqctCXnn7ec48ty8OmYexCrGsPJzOH7nt3rN4e8ZzxCd8j2WlYIwLY62jW49/ePLFx7jg0uE4HCuwrCJKSnKJiV3ELfecz13/vpqkTssoKdmJMS4sayOjjnJx3vjzOOeCAxDHKiyrmJKSbOLiF3HbfRfV2/tXKhBVWexl3wtF4gGMMU26GYpeE29e9Lp4/cnKzOTj6V9SXFzCuReOpUPHjg1ynt/nzuWO6x/FGRrCC288wkEHHwzYW2E+O/k//P3Hck4aO5qrbpiwr8LakoULefLhKURGRTLp8Tvo1qMHAIWFhTz50DMsWbiGsWcfy+VXXbavz7bkZD6b8TWxcTGcfcHp+zLti4uL+Wb2/1i/ZgvHnHAow0fuL5O6ZdNmPvv4W9q0SeCscacRG+u7xK4qptfFA1wtEtsuMsZ84G+zE2PMc/U8xGrRIN78aCBXqulpEA9wtdhPvDSzJNbPl1JKqWZCr4sHp8qKvbzu+fFJY4xvSqhS9USLvwSmVSv+YdmS1QwbMZD+B+y/l3vv3r389N1cQpwhHDvmaCIiIqo8Vl5eHj99N5eIyHCOOeFowsJ8s+nLy8nJ4afv5hITG81Rxx1ZrT7+WJbF/HkLSN6SwhFHH0JSJ9+9x5UKRtXZAGUDsBP4zfP1e1NeF9fl9OZJg3jgcLlc3HnDIyxeAC5XW0JDdzHy0FCeeOF+fv3pD56d/BkZ6Z0RMbRtl8y/J1/KyEMO8nu8b2d/z0vPfENmZmcEN+07bueRZyYwaMhAv30+n/UVr73wA7k5XYFiOiam8sQLN9C3f58av5+cnBxu+r+H2bg+AWPFExGZyqlndee2e66p8bFaAl1WD1C1WE4HwBjTG7gAWAGcil2KdWn9j1ApFQjeeWM6C+a1RqQ3oaEJQF/m/RbD+1M/4oWnPmNP3nDCw9sTFtaBnOwRPPPINCzL955vgPz8fF55/mv27h1GeHg7wsI7kpV5EI9PfBt/E4jc3FzeeOknCguGExbWlrCwJDLShzP5wf/W6v08+fCrbN44AKezO6FhrXC7D+SLj1NZulj/jKngV2UQF5HOwOHAaGAY8A8wo4HHpVoYvd0scCz8eyPO0NZebc7Qdnw3+1d2prXxahcRUlOi2bp5c4XHWvDXQnbv6uDTJy0lnB1paRX2+e2nP8jKTCrXx8H2bUJWZmZN3w4b1mTi8JSJ3a8XX8z6scbHUirQVLaLWalkYAEw2Rij609KNXNhYQ6MMeVKqBoiosJxhvruhRQa6iLSz/3osbHRhIT49nGGuv1eS4+Nj8Ehxb59nBZh4eWDcdVCw3znKpZVQmxsZI2P1RLopijBpTrFXoYB7wEXisg8EXlPRK5s4HGpFkhn44HhjHOPQsR7Zi2ygQnXX0CPXnswxrWv3bKK6dtf6JiYWOGxhh40jK7dMzFm/3K7213EgAPDadW6dYV9Ro0+jKTOO8r1KWDQkDiio33LsVbliKP64nane7VFRa9i/OX/qvGxlAo0Vc7EjTHLRGQjsBF7Sf0i4CjgrQYem1KqCRw35mhSt+/ii4/nk53tJKGVi3MuGMWo0YfRq09PJt71Aps3liAOQ9/+UTz89F1+j+VwOHjmlTuYdM/LbN3kwhFiOGBgLA89daffPk6nk6dfvpVHHniNbVssnE6LQUNbMfGJ22v1fq6+6TL27HmV335eRkG+k3Yd3Fx9078arBiOUo2pOtnpC4Fw4E88GeqejVGahGanN3+aqR4Y3G43uTk5xCck7KuiViovL4+QkJAalXXNzc3F6XQ2eB9/iouLyd+7l/iEBJ/NWpQvXVIPMH6y06tzTfxkY8zuBhiSUiqAhYSE+F3yrk0509KyqQ3dx5+wsLA63WuuVCCqzi1mGsCVUkqpAFSdmbhSjSqQK7j9+vPvfPLhjxQVWQw9qBtXXHNR0M7ujDF88tEX/Pz9UoyBo48fwrkXnomIkLI9hVf/8wG7dxXQoWMU19x8EUmdkqo+qGo2NEs9OGgQV6qaZnzwGa/+ZynG9EJEWLk0i+VLJ/HKW48F5TXWpx5+iTmfFRDikG5lnQAAH7pJREFU7AXAiiXr2bjuRS6ZcA5XXfQkebnDcDicrF5ZwpKFT/Dm9Ps0GUypAON3OV1E/lXZV2MOUqmmZlkWn370J9B7X8B2hrZi5dIIlixc3LSDq4Xs7Gx+/mE7Ic79s+sQZyK//JjC84+/sS+AAzgcoeRkD+H1l6Y31XBVE9FNUQJfZTPxsZU8Z4BP63ksSu0TaEvqBQUFZGf5zrYtqx1LFq1i+Ej/tcMD0fbkZLKzYihfbyUrK4ZtW1NxOHp7tTscYexM29OII1RKVUdlu5hd3pgDUSqQRUZGktDasLNcpVCHYycjDhnVNIOqg67du9OqdR4F+d7trVrvoXuvJNJSS3A4Qve1W1YRSZ10B2KlAk11KrYhIqeKyF0i8mDpV0MPTKlAquDmcDg4/+KjEFmzr5KYqySD4SMtBg8d3MSjq7m4uDhOOKUXbmv7vjbL2s7xJ3XnlrsnEJ+wFMsq9rQX0brNcq6+6aKmGq5qQrqkHtiqTGwTkdeAKOAY4E3gHGB+A49LqYBz1nmn0alLB2Z88B3FRW5GHNqH8ZfdEJRJbQC33n01/Qd8x/++no8xMOaUkZxyxkmICG99eD+vvzSdnWl5JHaK45qbHqRN27ZNPWSlVDnVqdi23BgzuMz3GOAbY8zoxhmiN63Y1rIE0nVxpVoqvdUsANR2P3GgwPM9X0SSgBKg4t0OlKpngbSkrlRLpUvqgas6QXyOiCQATwOLgS3Ahw05KKVaiuLiYpYvXUp2dna1+xQWFrJs8WJyc3Or3aekpIS01FSKi323+KyNzIwMsrOyfNqLiopIS03F5XJV0EspVd+qU+zlKWNMEfCJiMwBIoDCqjqJyFTgNGCXMWZgBc8L8AJwCpAPXGaMCb4bbpWqpXtvfZgZH8ynIL8NYWHZHHRIK2Z9PdVns5Gybrn6Pr78ZDkFBW0ID8/m0CPaM/3z1yvt896bH/H5xwvJygwnLqGIU88YxFU3XFqrMadsT+HBO19ky2ZwiKFXHyePPXcbrdu0YcrzU/nf16vJyw2nVesizr7gMC689OxanUcFHq3gFpiqMxOfV/qD+f/27jzKqurM+/j3qQGKQRABFQQpBGQICAiCKCrirAn6ttIqaseOxnQ7t8mr0QxG23Trim+njYn6IioOoOKAokKcBQFFBhEZFZBZZJ6p+ek/zqFrrroFde+pe+/vs1Yt6+579tnPva7FU3s4e7vnu/vOsmU1GAucX8P7FwDdwp8bgMdjuKekoVQcUp/85js8O3oZ+/edDfSnoOBMZk47kht/9qtq64wfO56Xn9/A/v1Bnfz8M/nk/ebcddu91db59JMZPP3EErZt7Yt7D3Zu78u4Z9Yy5a336hyzu3PnLf+P5d/0orioN4WFfVi8sBt33fYwE195i5df2MTOHX0pKenB1i19efLRL5n7xdw6tyMisatpx7ajzWwA0MTM+pvZieHPMILV6jVy92nAthouuRh4zgOfA4ebmebapUqplsgf/o+xlPiJ5crM2jP942+rrfP4I6/glH+czTI68d7kr6qtM3HCR7h3qdBOJ955Y2adY166eDGrVh6GWek/GxkZWaz41njzlWlkZHQod32Jd2PCC+/WuR0RiV1Nw+nnAdcCHYD/KlO+C7inHto+Blhb5vW6sOz7ihea2Q0EvXWO1VMukgIKC4qp6m/o4uLq/1gpLPRyCfR/69Qw/VxU6FU+AldUVPeFSnn78ygpzoLs8uXFRVnk5++vdL1ZBoVFJXVuRxouDak3PNX2xN39WXc/k2Cu+swyPxe7e0K3XHX30e4+0N0HttWmUZICRl41HHx5ubKSkt1079my2joXXjwYL1lVrsxLtnNCvyOrrTN0WG+KijaWKysq3MqgU7pWU6N6vfueQLtjyg+uuTvtO+zljOEnUFRUfnFeUeF6hp9TfrRBROpXLHPiM8zsKTObAmBmvczsunpoez3QsczrDmGZSMq7+Y4bOWnIDjIyvqC4aBP4Io7pMIPnXvtbtXV+++930af/BjJsTlCHBRzbeTZjXvxLtXUuu/IShp1dTEbmUvLzN2O2lCGn7+an14+qc8zZ2dn88jeX0/LwueTnb6Agfx2t287j7vuu5fqbrmHQkK2YfUN+/mYyM5dw9gWZXHTJBXVuRxo2PW7WsMSy2csU4BngN+7e18yygC/dvU+tNzfLBd6uZnX6RcDNBKvTBwN/cfdBtd1Tm72kt1Tb/GXB/PlMfPkdThrSjwtHXBRTnbmzZ/P26+8xdNggzjrvnJjqrFm1mq/nL6Zn7+M5rmuX2ivUID8/nxlTZ5CZlcUppw0hO7t0fH3Ft8tZuuhb+g7oTYeOHWu4iyQzDalHoJrNXmJJ4rPd/SQz+9Ld+4dl8929Xy31XgSGAW2AH4B7CWfT3P2J8BGzvxKsYN8H/LO7z6ntcyiJp7dUS+IiyUhJPALVJPFYnhPfa2atCY4fxcxOBnbWVsndr6zlfQduiqF9kf/V0I4oFRGJUixz4ncAk4AuZjYDeA64Ja5RiYhIg6V58Yaj1p64u88zszOA7oABy9y9MO6RiYiISI1q7YmbWQ5wK/DvwH3ATWGZSCRSbeMXkWSk3njDEMuc+HPAbuDR8PUo4HlgZLyCEhERkdrFksR7u3uvMq8/NrPF8QpIREREYhPLwrZ54Yp0AMxsMFDro2Ai8aQhdZHoaUg9erH0xAcAM81sTfj6WGCZmX1N8KTYCdVXFRERkXiJJYnXdJyoiIikMR2KEq1YHjFbnYhAROpKG7+ISLqLZU5cREREGiAlcUlqWuAmEj0tcIuOkriIiEiSUhIXERFJUkrikvQ0pC4SPQ2pR0NJXEREJEkpiUtKUG9cJHrqjSeekriIiEiSUhKXlKHeuIikGyVxERGRJKUkLiIi9Ubz4omlJC4pRUPqIpJOlMRFRKReqTeeOEriknLUGxeRdKEkLiIikqSUxEVEpN5pSD0xlMQlJWlIXUTSgZK4iIjEhXrj8ackLilLvXERSXVK4iIiIklKSTwF7NgL46fBU+/DS9Nh9/6oI2o41BsXiZaG1ONLSTzJbd4JY9+HizrDdf3hnI7w5N9h176oIxMRkXhTEk9yk2bDLwZByybB69bN4LqB8OYX0cbVkKg3LiKpSkk82RVDk0bli1o2gYL8aMIREalIQ+rxoySe5IqB4pLyZXmFYJmRhCMiIgmkJJ7kzu4Hz8+FkjCRF5fAs3Ph/BOjjauh0ZC6SLTUG4+PrKgDkENz3FFQ3A9Gz4NsoAg4fzC0PyLqyEREJN6UxFNAt3bBj9Ss5Xhj5yj1BkQkdWg4XUREEkJD6vVPSTwNlJSUzpmnO82Ni0gqietwupmdDzwCZAJj3P3BCu9fC/wJWB8W/dXdx8QzpnSybQ+8NA0aAe7gWXD1MGjaOOrIRESkPsQtiZtZJvA34BxgHTDbzCa5++IKl77s7jfHK4505Q5jP4AbB0NOdlC2Kw+e+RBuujDa2KKmuXGR6Oy41zn8Po2I1Zd4DqcPApa7+0p3LwBeAi6OY3tSxrINcOLRpQkcoEUOHJUDW3ZHF5eIiNSfeCbxY4C1ZV6vC8squtTMFpjZq2bWsaobmdkNZjbHzOZsVgKKyc59cESTyuWtcmCXDkjR3LiIpISoF7a9BeS6+wnA+8CzVV3k7qPdfaC7D2x7WELjS1p9joUv1lcuX7oVctskPh4RkQO0Sr3+xDOJrwfK9qw7ULqADQB33+ruB3b5HgMMiGM8aaVpYziuIzw3F7buhY27YPQsGNwTMqL+001EROpFPP85nw10M7POZtYIuAKYVPYCMyu7RckIYEkc40k7w/vAhafAJxtg1ha44kwY2DXqqBoODamLREe98foRt9Xp7l5kZjcD7xI8Yva0uy8ys/uBOe4+CbjVzEYQ7Ba6Dbg2XvGkqzaHwaVDoo6i4dJKdRFJZnF9TtzdJwOTK5T9vszvdwN3xzMGERGRVKW90yOyNw8+XQIlDkN7QIumtdf55nv405uQYXDnxdDl6KDcHRashhUboUcH6NWhtM7OfTB9CWRlwmk9Y9voZfveoE7j7KBOxfPKRUSkYdASpwjMWwnPvQ+9mkHfFjDhE5i5tOY6f3gZxk6GO0+FXw6BJ9+GP74GBUXwyFuwdzMMaweb1sGj7wRHkk5bBK99Av0Phx5NYex78NV3Nbfz4QKY9CkMbAVdG8OYKbB4XX198oZJc+Mi0dC8+KFTTzzBSkpgxkK45ZTSsusHwWOfBYvOGlXxf2TnHtixHf67zFY5D14Et0yElz6FK3rD0S2C8mFd4djDYeIXsG0r3DC4tM6NQ+AvM6FPp6pXqO/Jg9Xr4WcnlZbd3Boe/Qx6HgOmXCci0qCoJ55gKzZBnyMrlw/uAAvXVF1nzFT4Sa/K5Rf0gM+WlSbwA45rA8vWwMkdKtfp2RpWb6m6nbkrYGin8mVmkNsSfthZdZ1Uod64SDTUGz80SuIJ1rwx7MyrXL5jPxxWzbx4h1aweW/l8i17oWlOMCdeVkkJZGbCjira2V0AzXOqbuewJkEcFe3Jh6aaFxcRaXCUxBOsXStYvRv2FZSWFRTBV5ug61FV17n8FHhzEezNLy3bkw9TlsGlJ8O0FeWvn7IULhgAc74P7l22zvq90LZCz/2Afrnw6RooKi4t27kfdhTFtvBORORgqDd+8DQnHoFrz4Kxn0BTC+amdxfBT4fXPOf8q3+An78GvY4MVrQv3gT3jIQTOsGUefDELDi8EWzPh+6doG8utG8Fo6dDi6ygd74/bLs6GRkw6gx44jNomQWFJVCYUXOdVKJnxkUk2ZhXHItt4AYeZz7ngaijqB8FRUFyzanDUPXmcG66bcvy5SUlsL8geBys4qK1vIKgrKpFc9XZXwCZdayTCpTERaKh40lr8QfmuvvAisUaTo9Qo6y6JXAIknfFBA5Bkm6WU/Wq85xGdU/GTQ6iTirQAjeRaGhI/eAoiYuIiCSpNOxrRa+kBB58A7ZtD143aw6/uxSysmDSbHhxKrRpBlv2wT+eBv9nUFDn7bmwcWtQp30buPDEmk8kKy6BN7+ALTuC1x2PhPP763nv2mhuXESShZJ4BH71LFzVGwaEG74s2Qi3PwM/OQmmz4exl0PjrGDO/N73INNgy044rT2M6BzUWfYDPPcJXDu8+nbGvA/n5UJueHLZwg3w4qcw6vR4fjoRkYOz417X3HgdaTg9wdZthWOawoAyJ633PBr6tIHHp8B95wUJHII56XvPgec+huYG3cpsEtP9KGhUHOxzXpUN2+GoRpDburSsd3so2h/s2y4109y4iCQDJfEEW7AWflTFjm392kNOFjTJLl+ekw0tGkOXIyrX6dIK1m+rup1Vm+D4tpXLj20JG1N89zURkXShJJ5gg46DeRsql3+2GvKKYXeFXvKe/GCXtSWbK9dZthU6tam6nW7tYOEPlctXbg+eHxcRaYi0Sr1ulMQTrE0L2GswZUmwWK2kBKaugHX74J7L4P++A9v2Bddu3wd3vQO3XAQZOfDFmmCLVXeY+R3kNA+2Sq1K2xaQlwHz1gav3WHqcmjVSkeLxkpD6iLS0Gmzl4g89REsWAEGdOsI/3pusNJ87kp4+A1ong17CuFXl8CA44IkPGMpLA4PSendCU7pUXMb7jB1MXyzFjDodxwM6hbvT5ZatEpdJBpa4FZBNZu9KImL1EKJXCTxlMQr0I5tIgdHw+oi0lApiddBQRGs2ly3R7T25MH0pbBlV+X33pwLL82sXD59CYx8BD5cWPm930+A+1+tXL5wLdz/OqzYWL7cPXjcbFMdVqS7B6veN1cRs4hIImiBW2y02UuMpsyDtd/Dca1g2m6wxnD1GTXvfnb/BCjJgxPawadzYXsxPHg1vDUXxn0EZ3WD7AwY9TCcdSJcNxxO/y0M7QyXd4ePZsH9L8HUB+Cu8fDdWji3e3CK2ciHoNuxcN9l8OP/hEEdoG97GPM2fLURJv8WVm+GNz6DLodDYTGs3QNXnl713usHLP8eJs+Grq0grwjW7YFrhkOrZvX/nYqIyKHRnHgMFq6BNavhwp6lZd9sgqV7YcRJVdcZ9yk0y4dLepeWzVoFUzfCnGXw/KjSTV2Ki+GfXoJvt8B958IFZdr54Bu4+x3o0gZeGAVZmUF5QRFcPR52FsA9Z8IZXUrrTFoI476GwZ3h9lNLt2YtKILHv4DbRlQdc1ExPDoJbh9a+sdJXiGMngO3/iTmrytlaW5cJLE0L16G5sQP3mdL4fzu5cuOPxI2bqm+zqylMKJX+bLBuTB1Efy4V2kCB8jMhCv6QbvD4LwK7Zx9PLRrCSP7liZwCHZz+4c+UFJcPoFDcP/CAhjSofze6o2yoHNL+H571THPXQlndi4/upCTDUc3hW17qv+sIiLxoCH12imJx8C97oeGZFD14STFxdA0u3J5s8bB+d1V1cm04P2KmjQKHlGr1HYGZFn5PxQOaJQZDK1XpaCo6uNHszOCXnq60wI3EWlolMRj0CcXZq0uX7ZxFzSvYZ44tx18vqp82beboW8uvLU42OSlrJfnw3fbYN668uVfbYA12+GVr8rXKSmBN76GgmJYUmFntllrYHshzFxbvtwdlm6Fjq2p0kldYOp35ctKSmDNbjiyhnl0EZF4UW+8ZlrYFoOTj4cXpsKar6BfO1i+FVbughvOq77OrRfCvz0NCzbCkGNh/gaYsRb+8nP489tww6twzQDIyoBx8+CI1vDxNXDpn+GyE+CU3CAZT/gKPvwNXD8G/uW1oI4Dz8+BfQ6T74ZLH4ZLT4CTOsLMVfDGQph8T7Bi/dEZcHpu0Pv+dDWMGFL9qEJOIxjQAx77DE7rBPsLYfoauGxo/X+nIiJy6LSwrQ427YRlG4KebG4Vh5hUZcHq4BGz/rkwpMx89548uHNcMEz9H5dDmzI93RufhhmLYXB3GP3z0vJ1m+GyRyDDYMKt0KHMASd/mhTMt180EP71nNLygiL48rtgPr1fbjBkX5v8Qpj3XTAf3rdTzWeWpyMtcBNJLC1wQzu2idQnJXKRxEr7RK7V6SIiIqlFc+L1YMM2+GABFBXB0F5wfPuDv9eePPjrFFi/BY5uDbddCM1zgvemLYLXPg/mtEeeAqfWcgCKiIikNvXED9G0xfDJHBh5PFzTB75dDhM/P7h7bdkF/zYGLu4Mj/4ELu0Kv3wKNu6AB1+Hb5bDQ+fCf54DC5fAw2/W72eR2OlxM5HE0ir1qimJH4LiEli4AkadGDyznZ0JF/WCvbtgx9663+/B1+GP50HPo4PXPY4MXj/wKhTuhesHB4vNmmTDL4bA7p2wa1/9fiYREUkeSuKHYP22YI/xigYeA1+vqfv9MkrgyMPKl7VpDgV5MOy4yteflgsfLqp7O1I/1BsXkagpiR+Cw5vC5ip6wut3wlEHsTnK/qLKO6MVF0MRsHJb5eu/2w7djq57OyIiyUhD6pUpiR+CFk1hv8GGMsd87sqDBZsPbnHbiJPhsQpHk/7/z2HkqfDZOti0u7T8+10wbwP07nhwsYuISPLTc+KHqKgYXp4Oe8M58IxsuPI0aJZzcPd7eTrMXAytcmBHHgzuBVcODea+fzcemofPE+wtgQdGla5cl+jomXGRxErLZ8areU5cj5gdoqxMuOqM+rvf5UODn4paNIVHrq+/dkREJPlpOF3kEGmBm4hERUlcRESSiha4lYprEjez881smZktN7NfV/F+YzN7OXx/lpnlxjMekXhRb1xEohC3JG5mmcDfgAuAXsCVZtarwmXXAdvdvSvwZ+CheMUjIiKSauLZEx8ELHf3le5eALwEXFzhmouBZ8PfXwXOMqvutGuRhk29cZHE0ZB6IJ5J/BhgbZnX68KyKq9x9yJgJ9C64o3M7AYzm2NmczbvrviuiIhIekqKhW3uPtrdB7r7wLaH1X69SFTUGxeRRIpnEl8PlN1PrENYVuU1ZpYFtAS2xjEmERFJERpSj28Snw10M7POZtYIuAKYVOGaScBPw98vAz7yZNtCTkREJCJxS+LhHPfNwLvAEmCCuy8ys/vNbER42VNAazNbDtwBVHoMTSTZaEhdJHHSvTce121X3X0yMLlC2e/L/J4HjIxnDCJRaDnetKe6iMRdUixsExERqU4698aVxEVERJKUkrhInGhuXETiTUlcREQkSSmJi8SReuMiiZGu8+JK4iIiIklKSVwkztQbF0mMdOyNK4mLiIgkKSVxkQRQb1xE4kFJXEREUka6DakriYskiHrjIlLflMRFRESSlJK4SAKpNy4Sf+k0pK4kLiIikqSUxEUSTL1xEakvSuIiIpJy0mVIXUlcJALqjYtIfVASFxGRlJQOvXElcZGIqDcuIodKSVwkQkrkIvGV6r1xJXEREZEkpSQuIiKSpJTERSKmIXUROVhK4iINgBK5SPyk8ry4kriIiEiSUhIXaSDUGxeJn1TtjSuJi4iIJCklcZEGRL1xEakLJXEREUkLqTikriQu0sCoNy4isVISFxGRtJFqvXElcZEGSL1xEYmFkrhIA6VELiK1MffkGlows83A6ojDaANsiTiGKKX75wd9B6DvAPQdgL6DRH3+Tu7etmJh0iXxhsDM5rj7wKjjiEq6f37QdwD6DkDfAeg7iPrzazhdREQkSSmJi4iIJCkl8YMzOuoAIpbunx/0HYC+A9B3APoOIv38mhMXERFJUuqJi4iIJCkl8Tows6fNbJOZLYw6liiYWUcz+9jMFpvZIjO7LeqYEs3McszsCzP7KvwO7os6piiYWaaZfWlmb0cdSxTMbJWZfW1m881sTtTxRMHMDjezV81sqZktMbMhUceUSGbWPfz/f+Bnl5ndnvA4NJweOzM7HdgDPOfuvaOOJ9HMrB3Qzt3nmdlhwFzgEndfHHFoCWNmBjRz9z1mlg1MB25z988jDi2hzOwOYCDQwt1/HHU8iWZmq4CB7p62z0eb2bPAp+4+xswaAU3dfUfUcUXBzDKB9cBgd0/oPibqideBu08DtkUdR1Tc/Xt3nxf+vhtYAhwTbVSJ5YE94cvs8Cet/hI2sw7ARcCYqGORaJhZS+B04CkAdy9I1wQeOgtYkegEDkricpDMLBfoD8yKNpLEC4eS5wObgPfdPd2+g/8G7gRKog4kQg68Z2ZzzeyGqIOJQGdgM/BMOK0yxsyaRR1UhK4AXoyiYSVxqTMzaw68Btzu7ruijifR3L3Y3fsBHYBBZpY2Uytm9mNgk7vPjTqWiA119xOBC4Cbwqm2dJIFnAg87u79gb3Ar6MNKRrhVMII4JUo2lcSlzoJ54FfA8a5++tRxxOlcPjwY+D8qGNJoFOBEeGc8EvAcDN7IdqQEs/d14f/3QRMBAZFG1HCrQPWlRmFepUgqaejC4B57v5DFI0riUvMwkVdTwFL3P2/oo4nCmbW1swOD39vApwDLI02qsRx97vdvYO75xIMIX7k7ldHHFZCmVmzcGEn4RDyuUBaPbHi7huBtWbWPSw6C0ibBa4VXElEQ+kQDIlIjMzsRWAY0MbM1gH3uvtT0UaVUKcC1wBfh3PCAPe4++QIY0q0dsCz4WrUDGCCu6flY1Zp7ChgYvA3LVnAeHf/e7QhReIWYFw4nLwS+OeI40m48I+4c4BfRBaDHjETERFJThpOFxERSVJK4iIiIklKSVxERCRJKYmLiIgkKSVxERGRJKUkLpJkzOxaM2sfw3VjzeyyWMvrIa57yvyeG+tpf2Z2u5n9Uz20f7OZ/exQ7yOSTJTERZLPtUCtSTwC99R+SXlmlgX8DBhfD+0/TfDsskjaUBIXiVDYY11qZuPCM5lfNbOm4XsDzGxqeMjGu2bWLuxBDyTYZGO+mTUxs9+b2WwzW2hmo8Od9WJtv1IbYfknZvZQeHb6N2Z2Wlje1MwmhGfKTzSzWWY20MweBJqEMY0Lb59pZk+G566/F+5wV9Fwgi0ri8L7dzWzD8Lz2ueZWRczGxbG+KaZrTSzB83sqjC2r82sC4C77wNWmVm6bYEqaUxJXCR63YHH3L0nsAu4Mdyj/lHgMncfQNDL/KO7vwrMAa5y937uvh/4q7ufFJ5x3wSI6Xzv6tooc0mWuw8CbgfuDctuBLa7ey/gd8AAAHf/NbA/jOmq8NpuwN/c/UfADuDSKsI4leBc+gPGhXX6AqcA34flfYF/AXoS7Bp4fBjbGMr3vucAp8Xy+UVSgbZdFYneWnefEf7+AnAr8HegN/B+2LHOpDShVXSmmd0JNAWOABYBb8XQbvda2jhwwM1cIDf8fSjwCIC7LzSzBTXc/zt3P7A9b9l7lNWO4Fx6wv3Ij3H3ieH988JygNnu/n34egXwXlj/a+DMMvfbBPSoISaRlKIkLhK9insfO2DAIncfUlNFM8sBHgMGuvtaM/sDkBNju7W1kR/+t5iD+7civ8zvxQSjBBXtJ7Z4y96rpMzrkgqx5YT3FEkLGk4Xid6xZnYgkY4CpgPLgLYHys0s28x+FF6zGzgs/P1AAtwSnvNel1XnNbVRnRnAP4bX9wL6lHmvMByir4slQFcAd98NrDOzS8L7Nz6wPqAOjifNThST9KYkLhK9ZcBNZrYEaAU87u4FBAn5ITP7CphPMEcMMBZ4IjxJLh94kiBxvQvMjrXRWtqozmMEiX8x8ADB0P3O8L3RwIIyC9tiMQU4vczra4Bbw2H6mcDRdbgXBHPs79exjkjS0ilmIhEys1zg7XBRWoMXHsGa7e554arwD4Du4R8EB3vPicCd7v7tIcbWH7jD3a85lPuIJBPNiYtIXTQFPg6HzQ248VASeOjXBAvcDimJA20IVsyLpA31xEVERJKU5sRFRESSlJK4iIhIklISFxERSVJK4iIiIklKSVxERCRJKYmLiIgkqf8BnwYYWGwcZbAAAAAASUVORK5CYII=\n","text/plain":["<Figure size 576x432 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"qwQ8Yb-0h__O"},"source":["## Reference \n","* [Perceptron](https://en.wikipedia.org/wiki/Perceptron)\n","* [A quick guide to boosting in Machine Learning](https://medium.com/greyatom/a-quick-guide-to-boosting-in-ml-acf7c1585cb5)\n","* [AdaBoost from DataCamp](https://www.datacamp.com/community/tutorials/adaboost-classifier-python)\n","* [A Primer to Ensemble Learning – Bagging and Boosting](https://analyticsindiamag.com/primer-ensemble-learning-bagging-boosting/)\n","* [ANNs with scikitlearn](https://www.python-course.eu/neural_networks_with_scikit.php)"]},{"cell_type":"markdown","metadata":{"id":"Jz7RJ9HgARGF"},"source":["# Questions? "]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":56},"id":"O1Y6HcZPBLU6","executionInfo":{"status":"ok","timestamp":1631919778483,"user_tz":300,"elapsed":10,"user":{"displayName":"Ziyu Xiang","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"09056639542636885950"}},"outputId":"540ad1a3-2458-4d10-c268-db9a76f7750a"},"source":["Image(url= \"https://mirrors.creativecommons.org/presskit/buttons/88x31/png/by-nc-sa.png\", width=100)\n"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<img src=\"https://mirrors.creativecommons.org/presskit/buttons/88x31/png/by-nc-sa.png\" width=\"100\"/>"],"text/plain":["<IPython.core.display.Image object>"]},"metadata":{},"execution_count":29}]},{"cell_type":"code","metadata":{"id":"euTnNjqSBLU6"},"source":[""],"execution_count":null,"outputs":[]}]}