Course_1/Week_4_Retrieving_Wikipedia_articles.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 4: Retrieving Wikipedia articles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this module, we focused on using nearest neighbors and clustering to retrieve documents that interest users, by analyzing their text. We explored two document representations: word counts and TF-IDF. We also built an iPython notebook for retrieving articles from Wikipedia about famous people.\n",
    "\n",
    "In this assignment, we are going to dig deeper into this application, explore the retrieval results for various famous people, and familiarize ourselves with the code needed to build a retrieval system. These techniques will be key to building the intelligent application in your capstone project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learning outcomes\n",
    "- Execute document retrieval code with the iPython notebook\n",
    "- Load and transform real, text data\n",
    "- Compare results with word counts and TF-IDF\n",
    "- Set the distance function in the retrieval\n",
    "- Build a document retrieval model using nearest neighbor search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] This non-commercial license of GraphLab Create is assigned to chengjun@chem.ku.dk and will expire on January 27, 2017. For commercial licensing options, visit https://dato.com/buy/.\n",
      "\n",
      "[INFO] Start server at: ipc:///tmp/graphlab_server-25044 - Server binary: /usr/local/lib/python2.7/site-packages/graphlab/unity_server - Server log: /tmp/graphlab_server_1454423320.log\n",
      "[INFO] GraphLab Server Version: 1.8.1\n",
      "[WARNING] Unable to create session in specified location: '/Users/jcj/.graphlab/artifacts'. Using: '/var/tmp/graphlab-jcj/25044/tmp_session_73cf76c0-3994-489f-a377-fd84c2b28012'\n"
     ]
    }
   ],
   "source": [
    "import graphlab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and explore the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "people = graphlab.SFrame('people_wiki.gl/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">URI</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">&lt;http://dbpedia.org/resou<br>rce/Digby_Morrell&gt; ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Digby Morrell</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">digby morrell born 10<br>october 1979 is a former ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">&lt;http://dbpedia.org/resou<br>rce/Alfred_J._Lewy&gt; ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Alfred J. Lewy</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">alfred j lewy aka sandy<br>lewy graduated from ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">&lt;http://dbpedia.org/resou<br>rce/Harpdog_Brown&gt; ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Harpdog Brown</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">harpdog brown is a singer<br>and harmonica player who ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">&lt;http://dbpedia.org/resou<br>rce/Franz_Rottensteiner&gt; ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Franz Rottensteiner</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">franz rottensteiner born<br>in waidmannsfeld lower ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">&lt;http://dbpedia.org/resou<br>rce/G-Enka&gt; ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">G-Enka</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">henry krvits born 30<br>december 1974 in tallinn ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">&lt;http://dbpedia.org/resou<br>rce/Sam_Henderson&gt; ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Sam Henderson</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">sam henderson born<br>october 18 1969 is an ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">&lt;http://dbpedia.org/resou<br>rce/Aaron_LaCrate&gt; ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Aaron LaCrate</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">aaron lacrate is an<br>american music producer ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">&lt;http://dbpedia.org/resou<br>rce/Trevor_Ferguson&gt; ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Trevor Ferguson</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">trevor ferguson aka john<br>farrow born 11 november ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">&lt;http://dbpedia.org/resou<br>rce/Grant_Nelson&gt; ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Grant Nelson</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">grant nelson born 27<br>april 1971 in london  ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">&lt;http://dbpedia.org/resou<br>rce/Cathy_Caruth&gt; ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Cathy Caruth</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">cathy caruth born 1955 is<br>frank h t rhodes ...</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[10 rows x 3 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tURI\tstr\n",
       "\tname\tstr\n",
       "\ttext\tstr\n",
       "\n",
       "Rows: 10\n",
       "\n",
       "Data:\n",
       "+-------------------------------+---------------------+\n",
       "|              URI              |         name        |\n",
       "+-------------------------------+---------------------+\n",
       "| <http://dbpedia.org/resour... |    Digby Morrell    |\n",
       "| <http://dbpedia.org/resour... |    Alfred J. Lewy   |\n",
       "| <http://dbpedia.org/resour... |    Harpdog Brown    |\n",
       "| <http://dbpedia.org/resour... | Franz Rottensteiner |\n",
       "| <http://dbpedia.org/resour... |        G-Enka       |\n",
       "| <http://dbpedia.org/resour... |    Sam Henderson    |\n",
       "| <http://dbpedia.org/resour... |    Aaron LaCrate    |\n",
       "| <http://dbpedia.org/resour... |   Trevor Ferguson   |\n",
       "| <http://dbpedia.org/resour... |     Grant Nelson    |\n",
       "| <http://dbpedia.org/resour... |     Cathy Caruth    |\n",
       "+-------------------------------+---------------------+\n",
       "+-------------------------------+\n",
       "|              text             |\n",
       "+-------------------------------+\n",
       "| digby morrell born 10 octo... |\n",
       "| alfred j lewy aka sandy le... |\n",
       "| harpdog brown is a singer ... |\n",
       "| franz rottensteiner born i... |\n",
       "| henry krvits born 30 decem... |\n",
       "| sam henderson born october... |\n",
       "| aaron lacrate is an americ... |\n",
       "| trevor ferguson aka john f... |\n",
       "| grant nelson born 27 april... |\n",
       "| cathy caruth born 1955 is ... |\n",
       "+-------------------------------+\n",
       "[10 rows x 3 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "people.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obama = people[people['name'] == 'Barack Obama']\n",
    "cloony = people[people['name'] == 'George Clooney']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obama['word_count'] = graphlab.text_analytics.count_words(obama['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# why [[]] here?\n",
    "obama_word_count_table = obama[['word_count']].stack('word_count', new_column_name = ['word', 'count'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">word</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">the</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">in</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">and</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">of</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">to</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">his</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">obama</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">act</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">he</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">a</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[273 rows x 2 columns]<br/>Note: Only the head of the SFrame is printed.<br/>You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tword\tstr\n",
       "\tcount\tint\n",
       "\n",
       "Rows: 273\n",
       "\n",
       "Data:\n",
       "+-------+-------+\n",
       "|  word | count |\n",
       "+-------+-------+\n",
       "|  the  |   40  |\n",
       "|   in  |   30  |\n",
       "|  and  |   21  |\n",
       "|   of  |   18  |\n",
       "|   to  |   14  |\n",
       "|  his  |   11  |\n",
       "| obama |   9   |\n",
       "|  act  |   8   |\n",
       "|   he  |   7   |\n",
       "|   a   |   7   |\n",
       "+-------+-------+\n",
       "[273 rows x 2 columns]\n",
       "Note: Only the head of the SFrame is printed.\n",
       "You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns."
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obama_word_count_table.sort('count', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute TF-IDF for the corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "people['word_count'] = graphlab.text_analytics.count_words(people['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tfidf = graphlab.text_analytics.tf_idf(people['word_count'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "people['tfidf'] = tfidf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examine the TF-IDF for the Obama article"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">word</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">obama</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">43.2956530721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">act</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">27.678222623</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">iraq</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">17.747378588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">control</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14.8870608452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">law</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14.7229357618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">ordered</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14.5333739509</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">military</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13.1159327785</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">involvement</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12.7843852412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">response</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12.7843852412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">democratic</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12.4106886973</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[273 rows x 2 columns]<br/>Note: Only the head of the SFrame is printed.<br/>You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tword\tstr\n",
       "\tcount\tfloat\n",
       "\n",
       "Rows: 273\n",
       "\n",
       "Data:\n",
       "+-------------+---------------+\n",
       "|     word    |     count     |\n",
       "+-------------+---------------+\n",
       "|    obama    | 43.2956530721 |\n",
       "|     act     |  27.678222623 |\n",
       "|     iraq    |  17.747378588 |\n",
       "|   control   | 14.8870608452 |\n",
       "|     law     | 14.7229357618 |\n",
       "|   ordered   | 14.5333739509 |\n",
       "|   military  | 13.1159327785 |\n",
       "| involvement | 12.7843852412 |\n",
       "|   response  | 12.7843852412 |\n",
       "|  democratic | 12.4106886973 |\n",
       "+-------------+---------------+\n",
       "[273 rows x 2 columns]\n",
       "Note: Only the head of the SFrame is printed.\n",
       "You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns."
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obama = people[people['name'] == 'Barack Obama']\n",
    "obama[['tfidf']].stack('tfidf', new_column_name = ['word', 'count']).sort('count', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Manually compute distances between a few people"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "clinton = people[people['name'] == 'Bill Clinton']\n",
    "beckham = people[people['name'] == 'David Beckham']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.833985493688\n",
      "0.979130584475\n"
     ]
    }
   ],
   "source": [
    "print graphlab.distances.cosine(obama['tfidf'][0], clinton['tfidf'][0])\n",
    "print graphlab.distances.cosine(obama['tfidf'][0], beckham['tfidf'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a nearest neighbor model for document retrieva"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROGRESS: Starting brute force nearest neighbors model training.\n"
     ]
    }
   ],
   "source": [
    "knn_model = graphlab.nearest_neighbors.create(people, features=['tfidf'], label='name')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying the nearest-neighbors model for retrieval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROGRESS: Starting pairwise querying.\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n",
      "PROGRESS: | Query points | # Pairs | % Complete. | Elapsed Time |\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n",
      "PROGRESS: | 0            | 1       | 0.00169288  | 22.349ms     |\n",
      "PROGRESS: | Done         |         | 100         | 414.256ms    |\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">query_label</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">reference_label</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">distance</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">rank</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Barack Obama</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Joe Biden</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.794117647059</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Joe Lieberman</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.794685990338</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Kelly Ayotte</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.811989100817</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Bill Clinton</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.813852813853</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 4 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tquery_label\tint\n",
       "\treference_label\tstr\n",
       "\tdistance\tfloat\n",
       "\trank\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+-------------+-----------------+----------------+------+\n",
       "| query_label | reference_label |    distance    | rank |\n",
       "+-------------+-----------------+----------------+------+\n",
       "|      0      |   Barack Obama  |      0.0       |  1   |\n",
       "|      0      |    Joe Biden    | 0.794117647059 |  2   |\n",
       "|      0      |  Joe Lieberman  | 0.794685990338 |  3   |\n",
       "|      0      |   Kelly Ayotte  | 0.811989100817 |  4   |\n",
       "|      0      |   Bill Clinton  | 0.813852813853 |  5   |\n",
       "+-------------+-----------------+----------------+------+\n",
       "[5 rows x 4 columns]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_model.query(obama)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROGRESS: Starting pairwise querying.\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n",
      "PROGRESS: | Query points | # Pairs | % Complete. | Elapsed Time |\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n",
      "PROGRESS: | 0            | 1       | 0.00169288  | 23.254ms     |\n",
      "PROGRESS: | Done         |         | 100         | 398.965ms    |\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">query_label</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">reference_label</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">distance</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">rank</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Taylor Swift</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Carrie Underwood</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.76231884058</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Alicia Keys</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.764705882353</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Jordin Sparks</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.769633507853</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Leona Lewis</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.776119402985</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 4 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tquery_label\tint\n",
       "\treference_label\tstr\n",
       "\tdistance\tfloat\n",
       "\trank\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+-------------+------------------+----------------+------+\n",
       "| query_label | reference_label  |    distance    | rank |\n",
       "+-------------+------------------+----------------+------+\n",
       "|      0      |   Taylor Swift   |      0.0       |  1   |\n",
       "|      0      | Carrie Underwood | 0.76231884058  |  2   |\n",
       "|      0      |   Alicia Keys    | 0.764705882353 |  3   |\n",
       "|      0      |  Jordin Sparks   | 0.769633507853 |  4   |\n",
       "|      0      |   Leona Lewis    | 0.776119402985 |  5   |\n",
       "+-------------+------------------+----------------+------+\n",
       "[5 rows x 4 columns]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "swift = people[people['name'] == 'Taylor Swift']\n",
    "knn_model.query(swift)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "john = people[people['name'] == 'Elton John']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">word</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">the</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">in</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">and</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">of</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">a</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">has</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">john</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">he</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">on</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">award</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[255 rows x 2 columns]<br/>Note: Only the head of the SFrame is printed.<br/>You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tword\tstr\n",
       "\tcount\tint\n",
       "\n",
       "Rows: 255\n",
       "\n",
       "Data:\n",
       "+-------+-------+\n",
       "|  word | count |\n",
       "+-------+-------+\n",
       "|  the  |   27  |\n",
       "|   in  |   18  |\n",
       "|  and  |   15  |\n",
       "|   of  |   13  |\n",
       "|   a   |   10  |\n",
       "|  has  |   9   |\n",
       "|  john |   7   |\n",
       "|   he  |   7   |\n",
       "|   on  |   6   |\n",
       "| award |   5   |\n",
       "+-------+-------+\n",
       "[255 rows x 2 columns]\n",
       "Note: Only the head of the SFrame is printed.\n",
       "You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns."
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "john[['word_count']].stack('word_count', new_column_name=['word', 'count']).sort('count', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">word</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">furnish</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18.38947184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">elton</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">17.48232027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">billboard</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">17.3036809575</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">john</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13.9393127924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">songwriters</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11.250406447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">tonightcandle</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10.9864953892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">overallelton</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10.9864953892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">19702000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10.2933482087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">fivedecade</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10.2933482087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">aids</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10.262846934</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[255 rows x 2 columns]<br/>Note: Only the head of the SFrame is printed.<br/>You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tword\tstr\n",
       "\tcount\tfloat\n",
       "\n",
       "Rows: 255\n",
       "\n",
       "Data:\n",
       "+---------------+---------------+\n",
       "|      word     |     count     |\n",
       "+---------------+---------------+\n",
       "|    furnish    |  18.38947184  |\n",
       "|     elton     |  17.48232027  |\n",
       "|   billboard   | 17.3036809575 |\n",
       "|      john     | 13.9393127924 |\n",
       "|  songwriters  |  11.250406447 |\n",
       "| tonightcandle | 10.9864953892 |\n",
       "|  overallelton | 10.9864953892 |\n",
       "|    19702000   | 10.2933482087 |\n",
       "|   fivedecade  | 10.2933482087 |\n",
       "|      aids     |  10.262846934 |\n",
       "+---------------+---------------+\n",
       "[255 rows x 2 columns]\n",
       "Note: Only the head of the SFrame is printed.\n",
       "You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns."
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "john[['tfidf']].stack('tfidf', new_column_name=['word', 'count']).sort('count', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vb = people[people['name'] == 'Victoria Beckham']\n",
    "pm = people[people['name'] == 'Paul McCartney']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.956700637666\n",
      "0.825031002922\n"
     ]
    }
   ],
   "source": [
    "print graphlab.distances.cosine(john['tfidf'][0], vb['tfidf'][0])\n",
    "print graphlab.distances.cosine(john['tfidf'][0], pm['tfidf'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROGRESS: Starting brute force nearest neighbors model training.\n"
     ]
    }
   ],
   "source": [
    "knn_model_wc = graphlab.nearest_neighbors.create(people, features=['word_count'], distance='cosine', label='name')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROGRESS: Starting brute force nearest neighbors model training.\n"
     ]
    }
   ],
   "source": [
    "knn_model_tfidf = graphlab.nearest_neighbors.create(people, features=['tfidf'], distance='cosine', label='name')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROGRESS: Starting pairwise querying.\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n",
      "PROGRESS: | Query points | # Pairs | % Complete. | Elapsed Time |\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n",
      "PROGRESS: | 0            | 1       | 0.00169288  | 14.839ms     |\n",
      "PROGRESS: | Done         |         | 100         | 310.023ms    |\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">query_label</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">reference_label</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">distance</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">rank</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Elton John</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2.22044604925e-16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Cliff Richard</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.16142415259</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Sandro Petrone</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.16822542751</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Rod Stewart</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.168327165587</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Malachi O'Doherty</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.177315545979</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 4 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tquery_label\tint\n",
       "\treference_label\tstr\n",
       "\tdistance\tfloat\n",
       "\trank\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+-------------+-------------------+-------------------+------+\n",
       "| query_label |  reference_label  |      distance     | rank |\n",
       "+-------------+-------------------+-------------------+------+\n",
       "|      0      |     Elton John    | 2.22044604925e-16 |  1   |\n",
       "|      0      |   Cliff Richard   |   0.16142415259   |  2   |\n",
       "|      0      |   Sandro Petrone  |   0.16822542751   |  3   |\n",
       "|      0      |    Rod Stewart    |   0.168327165587  |  4   |\n",
       "|      0      | Malachi O'Doherty |   0.177315545979  |  5   |\n",
       "+-------------+-------------------+-------------------+------+\n",
       "[5 rows x 4 columns]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_model_wc.query(john)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROGRESS: Starting pairwise querying.\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n",
      "PROGRESS: | Query points | # Pairs | % Complete. | Elapsed Time |\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n",
      "PROGRESS: | 0            | 1       | 0.00169288  | 22.611ms     |\n",
      "PROGRESS: | Done         |         | 100         | 439.836ms    |\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">query_label</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">reference_label</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">distance</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">rank</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Elton John</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-2.22044604925e-16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Rod Stewart</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.717219667893</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">George Michael</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.747600998969</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Sting (musician)</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.747671954431</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Phil Collins</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.75119324879</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 4 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tquery_label\tint\n",
       "\treference_label\tstr\n",
       "\tdistance\tfloat\n",
       "\trank\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+-------------+------------------+--------------------+------+\n",
       "| query_label | reference_label  |      distance      | rank |\n",
       "+-------------+------------------+--------------------+------+\n",
       "|      0      |    Elton John    | -2.22044604925e-16 |  1   |\n",
       "|      0      |   Rod Stewart    |   0.717219667893   |  2   |\n",
       "|      0      |  George Michael  |   0.747600998969   |  3   |\n",
       "|      0      | Sting (musician) |   0.747671954431   |  4   |\n",
       "|      0      |   Phil Collins   |   0.75119324879    |  5   |\n",
       "+-------------+------------------+--------------------+------+\n",
       "[5 rows x 4 columns]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_model_tfidf.query(john)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROGRESS: Starting pairwise querying.\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n",
      "PROGRESS: | Query points | # Pairs | % Complete. | Elapsed Time |\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n",
      "PROGRESS: | 0            | 1       | 0.00169288  | 14.514ms     |\n",
      "PROGRESS: | Done         |         | 100         | 297.345ms    |\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">query_label</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">reference_label</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">distance</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">rank</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Victoria Beckham</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-2.22044604925e-16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Mary Fitzgerald (artist)</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.207307036115</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Adrienne Corri</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.214509782788</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Beverly Jane Fry</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.217466468741</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Raman Mundair</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.217695474992</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 4 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tquery_label\tint\n",
       "\treference_label\tstr\n",
       "\tdistance\tfloat\n",
       "\trank\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+-------------+--------------------------+--------------------+------+\n",
       "| query_label |     reference_label      |      distance      | rank |\n",
       "+-------------+--------------------------+--------------------+------+\n",
       "|      0      |     Victoria Beckham     | -2.22044604925e-16 |  1   |\n",
       "|      0      | Mary Fitzgerald (artist) |   0.207307036115   |  2   |\n",
       "|      0      |      Adrienne Corri      |   0.214509782788   |  3   |\n",
       "|      0      |     Beverly Jane Fry     |   0.217466468741   |  4   |\n",
       "|      0      |      Raman Mundair       |   0.217695474992   |  5   |\n",
       "+-------------+--------------------------+--------------------+------+\n",
       "[5 rows x 4 columns]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_model_wc.query(vb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROGRESS: Starting pairwise querying.\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n",
      "PROGRESS: | Query points | # Pairs | % Complete. | Elapsed Time |\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n",
      "PROGRESS: | 0            | 1       | 0.00169288  | 18.672ms     |\n",
      "PROGRESS: | Done         |         | 100         | 431.824ms    |\n",
      "PROGRESS: +--------------+---------+-------------+--------------+\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">query_label</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">reference_label</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">distance</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">rank</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Victoria Beckham</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.11022302463e-16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">David Beckham</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.548169610263</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Stephen Dow Beckham</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.784986706828</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Mel B</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.809585523409</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Caroline Rush</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.819826422919</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 4 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tquery_label\tint\n",
       "\treference_label\tstr\n",
       "\tdistance\tfloat\n",
       "\trank\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+-------------+---------------------+-------------------+------+\n",
       "| query_label |   reference_label   |      distance     | rank |\n",
       "+-------------+---------------------+-------------------+------+\n",
       "|      0      |   Victoria Beckham  | 1.11022302463e-16 |  1   |\n",
       "|      0      |    David Beckham    |   0.548169610263  |  2   |\n",
       "|      0      | Stephen Dow Beckham |   0.784986706828  |  3   |\n",
       "|      0      |        Mel B        |   0.809585523409  |  4   |\n",
       "|      0      |    Caroline Rush    |   0.819826422919  |  5   |\n",
       "+-------------+---------------------+-------------------+------+\n",
       "[5 rows x 4 columns]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_model_tfidf.query(vb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}