Course_2/Week_2_2_Multiple_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 2_2: Multiple regression (gradient descent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first notebook we explored multiple regression using graphlab create. Now we will use graphlab along with numpy to solve for the regression weights with gradient descent.\n",
    "\n",
    "In this notebook we will cover estimating multiple regression weights via gradient descent. You will:\n",
    "* Add a constant column of 1's to a graphlab SFrame to account for the intercept\n",
    "* Convert an SFrame into a Numpy array\n",
    "* Write a predict_output() function using Numpy\n",
    "* Write a numpy function to compute the derivative of the regression weights with respect to a single feature\n",
    "* Write gradient descent function to compute the regression weights given an initial weight vector, step size and tolerance.\n",
    "* Use the gradient descent function to estimate regression weights for multiple features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] Unable to write current GraphLab Create license to /Users/jcj/.graphlab/config. Ensure that this user account has write permission to /Users/jcj/.graphlab/config to save the license for offline use.\n",
      "[INFO] This non-commercial license of GraphLab Create is assigned to chengjun@chem.ku.dk and will expire on January 27, 2017. For commercial licensing options, visit https://dato.com/buy/.\n",
      "\n",
      "[INFO] Start server at: ipc:///tmp/graphlab_server-36813 - Server binary: /usr/local/lib/python2.7/site-packages/graphlab/unity_server - Server log: /tmp/graphlab_server_1455003525.log\n",
      "[INFO] GraphLab Server Version: 1.8.1\n",
      "[WARNING] Unable to create session in specified location: '/Users/jcj/.graphlab/artifacts'. Using: '/var/tmp/graphlab-jcj/36813/tmp_session_e59be5a1-b40c-4cc7-b688-8090f1da9943'\n"
     ]
    }
   ],
   "source": [
    "import graphlab\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sales = graphlab.SFrame('kc_house_data.gl/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_data, test_data = sales.random_split(0.8, seed=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert to Numpy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 3.    1.  ]\n",
      " [ 3.    2.25]\n",
      " [ 2.    1.  ]\n",
      " ..., \n",
      " [ 3.    2.5 ]\n",
      " [ 3.    2.5 ]\n",
      " [ 2.    0.75]]\n",
      "[[ 221900.]\n",
      " [ 538000.]\n",
      " [ 180000.]\n",
      " ..., \n",
      " [ 360000.]\n",
      " [ 400000.]\n",
      " [ 325000.]]\n"
     ]
    }
   ],
   "source": [
    "print train_data[['bedrooms', 'bathrooms']].to_numpy()\n",
    "print train_data[['price']].to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_numpy_data(data_sframe, features, output):\n",
    "    \"\"\"\n",
    "    features is a list of features.\n",
    "    output is a list or a string.\n",
    "    \"\"\"\n",
    "    data_sframe['constant'] = 1\n",
    "    features = data_sframe[['constant']+features]\n",
    "    features_matrix = features.to_numpy()\n",
    "    \n",
    "    output_array = data_sframe[output].to_numpy()\n",
    "    return (features_matrix, output_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def predict_outcome(feature_matrix, weights):\n",
    "    predictions = np.dot(feature_matrix, weights)\n",
    "    return(predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "(example_features, example_output) = get_numpy_data(sales, ['sqft_living'], 'price')\n",
    "my_weights = np.array([1., 1.])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting output given regression weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_predictions = predict_outcome(example_features, my_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(21613,)\n"
     ]
    }
   ],
   "source": [
    "print test_predictions.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing the Derivative"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def feature_derivative(errors, feature):\n",
    "    feature_derivative = 2 * np.dot(errors, feature)\n",
    "    return (feature_derivative)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(21613,)\n",
      "(21613,)\n",
      "(21613,)\n",
      "<type 'numpy.float64'>\n",
      "-23345850022.0\n",
      "-23345850022.0\n"
     ]
    }
   ],
   "source": [
    "(example_features, example_output) = get_numpy_data(sales, ['sqft_living'], 'price') \n",
    "my_weights = np.array([0., 0.]) # this makes all the predictions 0\n",
    "test_predictions = predict_outcome(example_features, my_weights) \n",
    "print test_predictions.shape\n",
    "# just like SFrames 2 numpy arrays can be elementwise subtracted with '-': \n",
    "errors = test_predictions - example_output # prediction errors in this case is just the -example_output\n",
    "print errors.shape\n",
    "feature = example_features[:,0] # let's compute the derivative with respect to 'constant', the \":\" indicates \"all rows\"\n",
    "print feature.shape\n",
    "derivative = feature_derivative(errors, feature)\n",
    "print type(derivative)\n",
    "print derivative\n",
    "print -np.sum(example_output)*2 # should be the same as derivative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def regression_gradient_descent(feature_matrix, output, initial_weights, step_size, tolerance):\n",
    "    converged = False\n",
    "    weights = np.array(initial_weights)\n",
    "    while not converged:\n",
    "        # compute the predictions based on feature_matrix and weights:\n",
    "        # compute the errors as predictions - output:\n",
    "        errors = predict_outcome(feature_matrix, weights) - output\n",
    "    \n",
    "        gradient_sum_squares = 0 # initialize the gradient\n",
    "        # while not converged, update each weight individually:\n",
    "        for i in range(len(weights)):\n",
    "            # Recall that feature_matrix[:, i] is the feature column associated with weights[i]\n",
    "            # compute the derivative for weight[i]:\n",
    "            derivative = feature_derivative(errors, feature_matrix[:, i])\n",
    "            #print 'derivative:', derivative\n",
    "            # add the squared derivative to the gradient magnitude\n",
    "            gradient_sum_squares += derivative**2\n",
    "            #print 'gradient_sum_squares:', gradient_sum_squares\n",
    "            # update the weight based on step size and derivative:\n",
    "            weights[i] = weights[i] - step_size * derivative\n",
    "            #print weights\n",
    "        gradient_magnitude = np.sqrt(gradient_sum_squares)\n",
    "        print 'gradient_magnitude:', gradient_magnitude\n",
    "        #converged = True\n",
    "        if gradient_magnitude < tolerance:\n",
    "            print 'Congrats, it is converged!'\n",
    "            print 'tolerance:', tolerance\n",
    "            converged = True\n",
    "    return(weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the Gradient Descent as Simple Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_magnitude: 5.05515307744e+13\n",
      "gradient_magnitude: 1.31274510235e+13\n",
      "gradient_magnitude: 3.40899608253e+12\n",
      "gradient_magnitude: 885263580100.0\n",
      "gradient_magnitude: 229889265720.0\n",
      "gradient_magnitude: 59698688259.7\n",
      "gradient_magnitude: 15502826422.1\n",
      "gradient_magnitude: 4025844401.5\n",
      "gradient_magnitude: 1045449748.16\n",
      "gradient_magnitude: 271487891.86\n",
      "gradient_magnitude: 70504114.6976\n",
      "gradient_magnitude: 18320016.7508\n",
      "Congrats, it is converged!\n",
      "tolerance: 25000000.0\n"
     ]
    }
   ],
   "source": [
    "# quiz 9\n",
    "simple_features = ['sqft_living']\n",
    "my_output= 'price'\n",
    "(simple_feature_matrix, output) = get_numpy_data(train_data, simple_features, my_output)\n",
    "initial_weights = np.array([-47000., 1.])\n",
    "step_size = 7e-12\n",
    "tolerance = 2.5e7\n",
    "\n",
    "simple_weights = regression_gradient_descent(simple_feature_matrix, \n",
    "                                             output,\n",
    "                                             initial_weights, \n",
    "                                             step_size,                                             \n",
    "                                             tolerance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-46999.88716555,    281.91211912])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "simple_features = ['sqft_living']\n",
    "my_output= 'price'\n",
    "(test_simple_feature_matrix, test_output) = get_numpy_data(test_data, simple_features, my_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4229, 2)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_simple_feature_matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predicted_test_output = predict_outcome(test_simple_feature_matrix, simple_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "356134.44317092974"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predicted_test_output[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "310000.0"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_output[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  46134.44317093,  134640.86422788,  202069.83652353, ...,\n",
       "         52733.65300782,  204217.10799338, -161550.5256668 ])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predicted_test_output - test_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.75400047593e+14\n"
     ]
    }
   ],
   "source": [
    "rss = np.sum((predicted_test_output - test_output)**2)\n",
    "print rss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running a multiple regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_magnitude: 7.30720205489e+13\n",
      "gradient_magnitude: 2.26732209651e+13\n",
      "gradient_magnitude: 7.0607945821e+12\n",
      "gradient_magnitude: 2.27568239427e+12\n",
      "gradient_magnitude: 928984105638.0\n",
      "gradient_magnitude: 656307425178.0\n",
      "gradient_magnitude: 610615351821.0\n",
      "gradient_magnitude: 593078765307.0\n",
      "gradient_magnitude: 578705920128.0\n",
      "gradient_magnitude: 564945676163.0\n",
      "gradient_magnitude: 551538681425.0\n",
      "gradient_magnitude: 538452422879.0\n",
      "gradient_magnitude: 525676912708.0\n",
      "gradient_magnitude: 513204543690.0\n",
      "gradient_magnitude: 501028100319.0\n",
      "gradient_magnitude: 489140559101.0\n",
      "gradient_magnitude: 477535065233.0\n",
      "gradient_magnitude: 466204926754.0\n",
      "gradient_magnitude: 455143610499.0\n",
      "gradient_magnitude: 444344738312.0\n",
      "gradient_magnitude: 433802083366.0\n",
      "gradient_magnitude: 423509566576.0\n",
      "gradient_magnitude: 413461253090.0\n",
      "gradient_magnitude: 403651348867.0\n",
      "gradient_magnitude: 394074197340.0\n",
      "gradient_magnitude: 384724276147.0\n",
      "gradient_magnitude: 375596193956.0\n",
      "gradient_magnitude: 366684687348.0\n",
      "gradient_magnitude: 357984617786.0\n",
      "gradient_magnitude: 349490968654.0\n",
      "gradient_magnitude: 341198842358.0\n",
      "gradient_magnitude: 333103457507.0\n",
      "gradient_magnitude: 325200146157.0\n",
      "gradient_magnitude: 317484351114.0\n",
      "gradient_magnitude: 309951623311.0\n",
      "gradient_magnitude: 302597619241.0\n",
      "gradient_magnitude: 295418098450.0\n",
      "gradient_magnitude: 288408921097.0\n",
      "gradient_magnitude: 281566045563.0\n",
      "gradient_magnitude: 274885526122.0\n",
      "gradient_magnitude: 268363510666.0\n",
      "gradient_magnitude: 261996238480.0\n",
      "gradient_magnitude: 255780038082.0\n",
      "gradient_magnitude: 249711325096.0\n",
      "gradient_magnitude: 243786600194.0\n",
      "gradient_magnitude: 238002447070.0\n",
      "gradient_magnitude: 232355530477.0\n",
      "gradient_magnitude: 226842594301.0\n",
      "gradient_magnitude: 221460459683.0\n",
      "gradient_magnitude: 216206023185.0\n",
      "gradient_magnitude: 211076255005.0\n",
      "gradient_magnitude: 206068197225.0\n",
      "gradient_magnitude: 201178962108.0\n",
      "gradient_magnitude: 196405730433.0\n",
      "gradient_magnitude: 191745749866.0\n",
      "gradient_magnitude: 187196333380.0\n",
      "gradient_magnitude: 182754857697.0\n",
      "gradient_magnitude: 178418761783.0\n",
      "gradient_magnitude: 174185545366.0\n",
      "gradient_magnitude: 170052767496.0\n",
      "gradient_magnitude: 166018045138.0\n",
      "gradient_magnitude: 162079051799.0\n",
      "gradient_magnitude: 158233516183.0\n",
      "gradient_magnitude: 154479220884.0\n",
      "gradient_magnitude: 150814001109.0\n",
      "gradient_magnitude: 147235743424.0\n",
      "gradient_magnitude: 143742384541.0\n",
      "gradient_magnitude: 140331910127.0\n",
      "gradient_magnitude: 137002353639.0\n",
      "gradient_magnitude: 133751795195.0\n",
      "gradient_magnitude: 130578360464.0\n",
      "gradient_magnitude: 127480219586.0\n",
      "gradient_magnitude: 124455586117.0\n",
      "gradient_magnitude: 121502715998.0\n",
      "gradient_magnitude: 118619906550.0\n",
      "gradient_magnitude: 115805495494.0\n",
      "gradient_magnitude: 113057859988.0\n",
      "gradient_magnitude: 110375415697.0\n",
      "gradient_magnitude: 107756615874.0\n",
      "gradient_magnitude: 105199950471.0\n",
      "gradient_magnitude: 102703945270.0\n",
      "gradient_magnitude: 100267161028.0\n",
      "gradient_magnitude: 97888192650.6\n",
      "gradient_magnitude: 95565668382.2\n",
      "gradient_magnitude: 93298249012.9\n",
      "gradient_magnitude: 91084627107.5\n",
      "gradient_magnitude: 88923526251.5\n",
      "gradient_magnitude: 86813700314.9\n",
      "gradient_magnitude: 84753932733.7\n",
      "gradient_magnitude: 82743035808.5\n",
      "gradient_magnitude: 80779850019.8\n",
      "gradient_magnitude: 78863243358.8\n",
      "gradient_magnitude: 76992110675.6\n",
      "gradient_magnitude: 75165373040.8\n",
      "gradient_magnitude: 73381977124.6\n",
      "gradient_magnitude: 71640894588.5\n",
      "gradient_magnitude: 69941121492.5\n",
      "gradient_magnitude: 68281677716.8\n",
      "gradient_magnitude: 66661606395.9\n",
      "gradient_magnitude: 65079973367.3\n",
      "gradient_magnitude: 63535866632.7\n",
      "gradient_magnitude: 62028395832.0\n",
      "gradient_magnitude: 60556691730.4\n",
      "gradient_magnitude: 59119905716.4\n",
      "gradient_magnitude: 57717209313.2\n",
      "gradient_magnitude: 56347793700.5\n",
      "gradient_magnitude: 55010869248.6\n",
      "gradient_magnitude: 53705665062.4\n",
      "gradient_magnitude: 52431428537.5\n",
      "gradient_magnitude: 51187424926.0\n",
      "gradient_magnitude: 49972936913.0\n",
      "gradient_magnitude: 48787264202.4\n",
      "gradient_magnitude: 47629723114.0\n",
      "gradient_magnitude: 46499646188.5\n",
      "gradient_magnitude: 45396381803.0\n",
      "gradient_magnitude: 44319293795.4\n",
      "gradient_magnitude: 43267761097.0\n",
      "gradient_magnitude: 42241177375.0\n",
      "gradient_magnitude: 41238950682.7\n",
      "gradient_magnitude: 40260503117.9\n",
      "gradient_magnitude: 39305270490.1\n",
      "gradient_magnitude: 38372701994.6\n",
      "gradient_magnitude: 37462259895.7\n",
      "gradient_magnitude: 36573419215.8\n",
      "gradient_magnitude: 35705667433.4\n",
      "gradient_magnitude: 34858504186.9\n",
      "gradient_magnitude: 34031440986.6\n",
      "gradient_magnitude: 33224000933.0\n",
      "gradient_magnitude: 32435718441.4\n",
      "gradient_magnitude: 31666138973.8\n",
      "gradient_magnitude: 30914818776.8\n",
      "gradient_magnitude: 30181324625.4\n",
      "gradient_magnitude: 29465233573.7\n",
      "gradient_magnitude: 28766132710.5\n",
      "gradient_magnitude: 28083618921.5\n",
      "gradient_magnitude: 27417298657.0\n",
      "gradient_magnitude: 26766787704.6\n",
      "gradient_magnitude: 26131710967.9\n",
      "gradient_magnitude: 25511702250.0\n",
      "gradient_magnitude: 24906404042.8\n",
      "gradient_magnitude: 24315467320.2\n",
      "gradient_magnitude: 23738551337.5\n",
      "gradient_magnitude: 23175323434.3\n",
      "gradient_magnitude: 22625458843.3\n",
      "gradient_magnitude: 22088640502.6\n",
      "gradient_magnitude: 21564558873.0\n",
      "gradient_magnitude: 21052911759.4\n",
      "gradient_magnitude: 20553404136.9\n",
      "gradient_magnitude: 20065747980.3\n",
      "gradient_magnitude: 19589662098.2\n",
      "gradient_magnitude: 19124871970.9\n",
      "gradient_magnitude: 18671109592.0\n",
      "gradient_magnitude: 18228113313.7\n",
      "gradient_magnitude: 17795627696.4\n",
      "gradient_magnitude: 17373403361.1\n",
      "gradient_magnitude: 16961196845.6\n",
      "gradient_magnitude: 16558770464.0\n",
      "gradient_magnitude: 16165892170.0\n",
      "gradient_magnitude: 15782335422.9\n",
      "gradient_magnitude: 15407879056.9\n",
      "gradient_magnitude: 15042307153.7\n",
      "gradient_magnitude: 14685408917.8\n",
      "gradient_magnitude: 14336978555.2\n",
      "gradient_magnitude: 13996815154.7\n",
      "gradient_magnitude: 13664722572.0\n",
      "gradient_magnitude: 13340509316.2\n",
      "gradient_magnitude: 13023988440.4\n",
      "gradient_magnitude: 12714977432.6\n",
      "gradient_magnitude: 12413298111.7\n",
      "gradient_magnitude: 12118776523.7\n",
      "gradient_magnitude: 11831242842.3\n",
      "gradient_magnitude: 11550531270.3\n",
      "gradient_magnitude: 11276479944.3\n",
      "gradient_magnitude: 11008930841.3\n",
      "gradient_magnitude: 10747729687.6\n",
      "gradient_magnitude: 10492725870.0\n",
      "gradient_magnitude: 10243772348.6\n",
      "gradient_magnitude: 10000725572.3\n",
      "gradient_magnitude: 9763445395.81\n",
      "gradient_magnitude: 9531794999.21\n",
      "gradient_magnitude: 9305640808.63\n",
      "gradient_magnitude: 9084852419.46\n",
      "gradient_magnitude: 8869302521.1\n",
      "gradient_magnitude: 8658866823.56\n",
      "gradient_magnitude: 8453423985.81\n",
      "gradient_magnitude: 8252855545.78\n",
      "gradient_magnitude: 8057045852.06\n",
      "gradient_magnitude: 7865881997.24\n",
      "gradient_magnitude: 7679253752.75\n",
      "gradient_magnitude: 7497053505.37\n",
      "gradient_magnitude: 7319176195.12\n",
      "gradient_magnitude: 7145519254.73\n",
      "gradient_magnitude: 6975982550.45\n",
      "gradient_magnitude: 6810468324.33\n",
      "gradient_magnitude: 6648881137.87\n",
      "gradient_magnitude: 6491127816.97\n",
      "gradient_magnitude: 6337117398.19\n",
      "gradient_magnitude: 6186761076.33\n",
      "gradient_magnitude: 6039972153.21\n",
      "gradient_magnitude: 5896665987.68\n",
      "gradient_magnitude: 5756759946.78\n",
      "gradient_magnitude: 5620173358.17\n",
      "gradient_magnitude: 5486827463.54\n",
      "gradient_magnitude: 5356645373.21\n",
      "gradient_magnitude: 5229552021.85\n",
      "gradient_magnitude: 5105474125.1\n",
      "gradient_magnitude: 4984340137.41\n",
      "gradient_magnitude: 4866080210.72\n",
      "gradient_magnitude: 4750626154.21\n",
      "gradient_magnitude: 4637911394.97\n",
      "gradient_magnitude: 4527870939.63\n",
      "gradient_magnitude: 4420441336.86\n",
      "gradient_magnitude: 4315560640.81\n",
      "gradient_magnitude: 4213168375.35\n",
      "gradient_magnitude: 4113205499.26\n",
      "gradient_magnitude: 4015614372.13\n",
      "gradient_magnitude: 3920338721.13\n",
      "gradient_magnitude: 3827323608.6\n",
      "gradient_magnitude: 3736515400.34\n",
      "gradient_magnitude: 3647861734.68\n",
      "gradient_magnitude: 3561311492.31\n",
      "gradient_magnitude: 3476814766.79\n",
      "gradient_magnitude: 3394322835.78\n",
      "gradient_magnitude: 3313788132.94\n",
      "gradient_magnitude: 3235164220.48\n",
      "gradient_magnitude: 3158405762.43\n",
      "gradient_magnitude: 3083468498.48\n",
      "gradient_magnitude: 3010309218.43\n",
      "gradient_magnitude: 2938885737.33\n",
      "gradient_magnitude: 2869156871.08\n",
      "gradient_magnitude: 2801082412.77\n",
      "gradient_magnitude: 2734623109.42\n",
      "gradient_magnitude: 2669740639.4\n",
      "gradient_magnitude: 2606397590.29\n",
      "gradient_magnitude: 2544557437.34\n",
      "gradient_magnitude: 2484184522.4\n",
      "gradient_magnitude: 2425244033.35\n",
      "gradient_magnitude: 2367701984.03\n",
      "gradient_magnitude: 2311525194.65\n",
      "gradient_magnitude: 2256681272.66\n",
      "gradient_magnitude: 2203138594.06\n",
      "gradient_magnitude: 2150866285.15\n",
      "gradient_magnitude: 2099834204.79\n",
      "gradient_magnitude: 2050012926.95\n",
      "gradient_magnitude: 2001373723.78\n",
      "gradient_magnitude: 1953888549.02\n",
      "gradient_magnitude: 1907530021.87\n",
      "gradient_magnitude: 1862271411.15\n",
      "gradient_magnitude: 1818086619.93\n",
      "gradient_magnitude: 1774950170.46\n",
      "gradient_magnitude: 1732837189.48\n",
      "gradient_magnitude: 1691723393.87\n",
      "gradient_magnitude: 1651585076.68\n",
      "gradient_magnitude: 1612399093.43\n",
      "gradient_magnitude: 1574142848.76\n",
      "gradient_magnitude: 1536794283.43\n",
      "gradient_magnitude: 1500331861.58\n",
      "gradient_magnitude: 1464734558.32\n",
      "gradient_magnitude: 1429981847.59\n",
      "gradient_magnitude: 1396053690.35\n",
      "gradient_magnitude: 1362930523.01\n",
      "gradient_magnitude: 1330593246.15\n",
      "gradient_magnitude: 1299023213.51\n",
      "gradient_magnitude: 1268202221.23\n",
      "gradient_magnitude: 1238112497.38\n",
      "gradient_magnitude: 1208736691.66\n",
      "gradient_magnitude: 1180057865.46\n",
      "gradient_magnitude: 1152059482.05\n",
      "gradient_magnitude: 1124725397.05\n",
      "gradient_magnitude: 1098039849.12\n",
      "gradient_magnitude: 1071987450.91\n",
      "gradient_magnitude: 1046553180.11\n",
      "gradient_magnitude: 1021722370.88\n",
      "gradient_magnitude: 997480705.317\n",
      "Congrats, it is converged!\n",
      "tolerance: 1000000000.0\n"
     ]
    }
   ],
   "source": [
    "# model 2\n",
    "model_features = ['sqft_living', 'sqft_living15']\n",
    "my_output = 'price'\n",
    "(feature_matrix, output) = get_numpy_data(train_data, model_features,my_output)\n",
    "initial_weights = np.array([-100000., 1., 1.])\n",
    "step_size = 4e-12\n",
    "tolerance = 1e9\n",
    "\n",
    "multiple_weights = regression_gradient_descent(feature_matrix, \n",
    "                                             output,\n",
    "                                             initial_weights, \n",
    "                                             step_size,                                             \n",
    "                                             tolerance)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ -9.99999688e+04,   2.45072603e+02,   6.52795277e+01])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multiple_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "366651.412037\n"
     ]
    }
   ],
   "source": [
    "model_features = ['sqft_living', 'sqft_living15']\n",
    "my_output = 'price'\n",
    "(test_feature_matrix, test_output) = get_numpy_data(test_data, model_features,my_output)\n",
    "\n",
    "predicted_test_output = predict_outcome(test_feature_matrix, multiple_weights)\n",
    "print predicted_test_output[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.70263446465e+14\n"
     ]
    }
   ],
   "source": [
    "rss = np.sum((predicted_test_output - test_output)**2)\n",
    "print rss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "310000.0"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_output[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}