Staycurious

Predicting Car Prices using KNN

2019-09-05T00:00:00+00:00

Predicting Car Prices

The goal of this project is to practice machine learning workflow by predicting car prices using dataset containing car characteristics. The dataset comes from the UCI’s machine learning repository.

Disclaimer: This project is part of the guided project for a Data Scientist Path on dataquest.io.

Introduction

Let’s first read the dataset and explore it to get a better understanding of it. Since the original dataset doesn’t have column headers in it, we need to provide the list manually.

The numeric feature columns in the dataset are:

normalized-losses
wheel-base
length
width
height
curb-weight
engine-type
num-of-cylinders
engine-size
bore
stroke
compression-rate
horsepower
peak-rpm
city-mpg
highway-mpg

The target column is price.

# import necessary libraries
import pandas as pd
import numpy as np
import collections
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import mean_squared_error

# enable pandas to show all columns
pd.options.display.max_columns = 99


# list of columns names
columns = ['symboling', 'normalized-losses', 'make', 'fuel-type', 'aspiration',
           'num-of-doors', 'body-style', 'drive-wheels', 'engine-location', 'wheel-base',
           'length', 'width', 'height', 'curb-weight', 'engine-type', 'num-of-cylinders',
           'engine-size', 'fuel-system', 'bore', 'stroke', 'compression-rate', 'horsepower',
           'peak-rpm', 'city-mpg', 'highway-mpg', 'price']

# read the dataset
cars = pd.read_csv("imports-85.data", names=columns)

# check first few values
cars.head()

	symboling	normalized-losses	make	fuel-type	aspiration	num-of-doors	body-style	drive-wheels	engine-location	wheel-base	length	width	height	curb-weight	engine-type	num-of-cylinders	engine-size	fuel-system	bore	stroke	compression-rate	horsepower	peak-rpm	city-mpg	highway-mpg	price
0	3	?	alfa-romero	gas	std	two	convertible	rwd	front	88.6	168.8	64.1	48.8	2548	dohc	four	130	mpfi	3.47	2.68	9.0	111	5000	21	27	13495
1	3	?	alfa-romero	gas	std	two	convertible	rwd	front	88.6	168.8	64.1	48.8	2548	dohc	four	130	mpfi	3.47	2.68	9.0	111	5000	21	27	16500
2	1	?	alfa-romero	gas	std	two	hatchback	rwd	front	94.5	171.2	65.5	52.4	2823	ohcv	six	152	mpfi	2.68	3.47	9.0	154	5000	19	26	16500
3	2	164	audi	gas	std	four	sedan	fwd	front	99.8	176.6	66.2	54.3	2337	ohc	four	109	mpfi	3.19	3.40	10.0	102	5500	24	30	13950
4	2	164	audi	gas	std	four	sedan	4wd	front	99.4	176.6	66.4	54.3	2824	ohc	five	136	mpfi	3.19	3.40	8.0	115	5500	18	22	17450

# check columns and observations
cars.shape

(205, 26)

Data Cleaning

Machine Learning algorithms can’t treat missing values on their own. For that reason we need to treat missing values. From the output above we immediately can see that normalized-losses column contains missing values in the form of “?”. We need to first replace “?” values with numpy.nan across the dataset. This will help us to easier convert other columns to required datatypes.

# replace ? values with np.nan
cars = cars.replace("?", np.nan)

# check datatypes
cars.dtypes

symboling              int64
normalized-losses     object
make                  object
fuel-type             object
aspiration            object
num-of-doors          object
body-style            object
drive-wheels          object
engine-location       object
wheel-base           float64
length               float64
width                float64
height               float64
curb-weight            int64
engine-type           object
num-of-cylinders      object
engine-size            int64
fuel-system           object
bore                  object
stroke                object
compression-rate     float64
horsepower            object
peak-rpm              object
city-mpg               int64
highway-mpg            int64
price                 object
dtype: object

Now that we have replaced the “?” values, let’s convert columns containing int or float values to their correct data type

# list of columns to convert
cols = ["normalized-losses","bore","stroke","horsepower","peak-rpm","price"]

# convert the values to their appropriate datatype
for col in cols:
    cars[col] = pd.to_numeric(cars[col], errors='coerce')

Rerun the datatypes cell to check if the conversion was successful.

Now let’s look at the columns which have these missing values and decide on how to handle them.

# Check for missing values
cars.isnull().sum()

symboling             0
normalized-losses    41
make                  0
fuel-type             0
aspiration            0
num-of-doors          2
body-style            0
drive-wheels          0
engine-location       0
wheel-base            0
length                0
width                 0
height                0
curb-weight           0
engine-type           0
num-of-cylinders      0
engine-size           0
fuel-system           0
bore                  4
stroke                4
compression-rate      0
horsepower            2
peak-rpm              2
city-mpg              0
highway-mpg           0
price                 4
dtype: int64

We can see that there are 41 null values out of 205 total records in normalized-losses. The missing values are 25% of all records. To handle this column entirely we can fill the missing values with the average value or drop the column entirely. We won’t remove the missing rows since this is a big part of the dataset. To decide on this, let’s check the distribution of values in the normalized-losses.

cars['normalized-losses'].value_counts().plot(kind='bar', figsize=(16,8))

<matplotlib.axes._subplots.AxesSubplot at 0x1c3597dd898>

We can see that the distribution for this column is right skewed. Thus we won’t be imputing using average since this might skew our results and introduce bias into this column. One potential solution would be to use mean imputation after normalizing the column, which we will do.

# normalize the column
cars['normalized-losses'] = (cars['normalized-losses'] - cars['normalized-losses'].mean()) / cars['normalized-losses'].std()

# impute with average value
cars['normalized-losses'].fillna(cars['normalized-losses'].mean(), inplace=True)

Rerun, the cell showing missing values to ensure that the imputation was successful. We can also observe that there are few missing values for columns such as price, horsepower, peak-rpm, bore and stroke. We will also normalize them and impute with average value.

# columns to be treated as per above
cols = ['bore', 'stroke', 'peak-rpm', 'horsepower', 'price']

# normalize and fill missing values
for col in cols:
    cars[col] = (cars[col] - cars[col].mean()) / cars[col].std()
    cars[col].fillna(cars[col].mean(), inplace=True)

Rerun the cell generating null value count and make sure they are treated.

Univariate Model

Now let’s start with a univariate k-nearest neighbors model as part of our project for predicting car prices. Starting with simple models before moving to more complex models helps us structure your code workflow and understand the features better.

To enhance and simplify our model testing workflow, we will create a function that encapsulates the training and simple validation process.

# function to knn_model training and validation
def knn_train_test(train_name, target_name, data):
    # make a copy of the dataset
    df = data.copy()
    
    # number of observations for split
    obs = int(df.shape[0] / 2)
    
    # split the data into train and test
    train_df = df.iloc[:obs]
    test_df = df.iloc[obs:]
    
    # reshape the values since we have a single feature
    train_data = train_df[train_name].values.reshape(-1, 1)
    test_data = test_df[train_name].values.reshape(-1, 1)
    
    # instantiate  and fit the model
    knn = KNeighborsRegressor()
    knn.fit(train_data, train_df[target_name])
    
    # make predictions and evaluate
    preds = knn.predict(test_data)
    rmse = mean_squared_error(test_df[target_name], preds) ** (1/2)
    
    return rmse

# use the function to train a model on normalized-losses
knn_train_test('normalized-losses', 'price', cars)

1.0768788918340664

Resulting rmse is based on the model that uses only one feature normalized-losses. We can test additional features to see how our model performs.

# use numeric features
numeric_features = ['wheel-base', 'length', 'width', 'height',
                    'curb-weight', 'engine-size', 'compression-rate',
                    'city-mpg', 'highway-mpg', 'horsepower', 'peak-rpm']

# loop through models with individual feature
for col in numeric_features:
    result = knn_train_test(col, 'price', cars)
    print("RMSE for model with {}: {}".format(col, result))

RMSE for model with wheel-base: 1.214252890227465
RMSE for model with length: 1.1107969135114395
RMSE for model with width: 0.8581184488064216
RMSE for model with height: 1.3712276576274274
RMSE for model with curb-weight: 0.7526797743700095
RMSE for model with engine-size: 0.6066522674323407
RMSE for model with compression-rate: 0.9980929742948769
RMSE for model with city-mpg: 0.6071818370617673
RMSE for model with highway-mpg: 0.5904065996733864
RMSE for model with horsepower: 0.6212450466337898
RMSE for model with peak-rpm: 1.1696138374863996

While the models above are not indicative of performance, but it seems that model with city-mpg feature has the lowest error on the model.

Next, let’s modify our function to accept a k parameter and rerun our univariate model with the above features to see the performance. We will use k values of 1, 3, 5, 7 and 9 to see which model performs well.

# function to knn_model training and validation - modified 
def knn_train_test(train_name, target_name, data, k=5):
    # make a copy of the dataset
    df = data.copy()
    
    # number of observations for split
    obs = int(df.shape[0] / 2)
    
    # split the data into train and test
    train_df = df.iloc[:obs]
    test_df = df.iloc[obs:]
    
    # reshape the values since we have a single feature
    train_data = train_df[train_name].values.reshape(-1, 1)
    test_data = test_df[train_name].values.reshape(-1, 1)
    
    # instantiate  and fit the model
    knn = KNeighborsRegressor(n_neighbors=k)
    knn.fit(train_data, train_df[target_name])
    
    # make predictions and evaluate
    preds = knn.predict(test_data)
    rmse = mean_squared_error(test_df[target_name], preds) ** (1/2)
    
    return rmse

# use numeric features
numeric_features = ['wheel-base', 'length', 'width', 'height',
                    'curb-weight', 'engine-size', 'compression-rate',
                    'city-mpg', 'highway-mpg', 'horsepower', 'peak-rpm']

# dictionary to store model results
lowest_rmse = {}

# loop through models with individual feature
for col in numeric_features:
    for k in [1,3,5,7,9]:
        result = knn_train_test(col, 'price', cars, k=k)
        lowest_rmse[(col, k)] = result

lowest_rmse

{('wheel-base', 1): 1.0200915246848115,
 ('wheel-base', 3): 1.1330559244770682,
 ('wheel-base', 5): 1.214252890227465,
 ('wheel-base', 7): 1.1050644625745172,
 ('wheel-base', 9): 1.0706582345927034,
 ('length', 1): 1.2582151367719674,
 ('length', 3): 1.164040684899018,
 ('length', 5): 1.1107969135114395,
 ('length', 7): 1.0788091491246363,
 ('length', 9): 0.9649699176225501,
 ('width', 1): 1.173235330629889,
 ('width', 3): 0.776724402613776,
 ('width', 5): 0.8581184488064216,
 ('width', 7): 0.7920962270887307,
 ('width', 9): 0.7818084447558675,
 ('height', 1): 1.835270961051065,
 ('height', 3): 1.4522475457196613,
 ('height', 5): 1.3712276576274274,
 ('height', 7): 1.2536105424726978,
 ('height', 9): 1.1799262400433599,
 ('curb-weight', 1): 0.8977925627200231,
 ('curb-weight', 3): 0.7810203611189004,
 ('curb-weight', 5): 0.7526797743700095,
 ('curb-weight', 7): 0.7273112573379618,
 ('curb-weight', 9): 0.6962953604831444,
 ('engine-size', 1): 0.711130777393697,
 ('engine-size', 3): 0.6670315366668025,
 ('engine-size', 5): 0.6066522674323407,
 ('engine-size', 7): 0.5846523661450151,
 ('engine-size', 9): 0.5470224273794155,
 ('compression-rate', 1): 0.9995557239010212,
 ('compression-rate', 3): 1.0014666665882759,
 ('compression-rate', 5): 0.9980929742948769,
 ('compression-rate', 7): 0.994312348888388,
 ('compression-rate', 9): 0.9657803897563202,
 ('city-mpg', 1): 0.672247401307514,
 ('city-mpg', 3): 0.6692599843503406,
 ('city-mpg', 5): 0.6071818370617673,
 ('city-mpg', 7): 0.5947058619062482,
 ('city-mpg', 9): 0.6184289881119851,
 ('highway-mpg', 1): 0.65226957297654,
 ('highway-mpg', 3): 0.5514192730892143,
 ('highway-mpg', 5): 0.5904065996733864,
 ('highway-mpg', 7): 0.5316220529521046,
 ('highway-mpg', 9): 0.5045969744641035,
 ('horsepower', 1): 0.7424499390187735,
 ('horsepower', 3): 0.751441782151203,
 ('horsepower', 5): 0.6212450466337898,
 ('horsepower', 7): 0.5678990997984015,
 ('horsepower', 9): 0.51029924508396,
 ('peak-rpm', 1): 1.2547163045227205,
 ('peak-rpm', 3): 1.1322701001836366,
 ('peak-rpm', 5): 1.1696138374863996,
 ('peak-rpm', 7): 1.1987345300306547,
 ('peak-rpm', 9): 1.1998145119829764}

We can observe that there are multiple univariate models with low rmse. Nevertheles, we can find out the one with smallest rmse.

# find the feature and k with minimum rmse
minval = min(lowest_rmse.values())
res = [k for k, v in lowest_rmse.items() if v == minval]

print("Model with lowest rmse is based on feature:{} with k={}".format(res[0][0], res[0][1]))

Model with lowest rmse is based on feature:highway-mpg with k=9

Multivariate Model

Let’s modify the function we wrote to work with multiple columns.

# function to knn_model training and validation
# modified for multivariate modeling
def knn_train_test(train_name, target_name, data, k=5):
    # make a copy of the dataset
    df = data.copy()
    
    # number of observations for split
    obs = int(df.shape[0] / 2)
    
    if len(train_name) <= 1:
        return "Please provide a list of features"
    
    # split the data into train and test
    train_df = df.iloc[:obs]
    test_df = df.iloc[obs:]
    
    # instantiate  and fit the model
    knn = KNeighborsRegressor(n_neighbors=k)
    knn.fit(train_df[train_name], train_df[target_name])
    
    # make predictions and evaluate
    preds = knn.predict(test_df[train_name])
    rmse = mean_squared_error(test_df[target_name], preds) ** (1/2)
    
    return rmse

To test our new function, we will choose the best 2, 3, 4, 5 and 6 features sequentially from the univariate model exploration and train using the default value of k=5.

# features ordered by performance from univariate modeling
features = ['highway-mpg', 'horsepower', 'engine-size',
            'city-mpg', 'curb-weight']

# starting feature set
cols = features[:1]

# iterate and add each feature to the next model
for col in features:
    # add the next feature
    cols.append(col)
    
    # train the model
    res = knn_train_test(cols, 'price', cars)
    
    # print the rmse
    print("The model with {} features has rmse:{}".format(cols,res))

The model with ['highway-mpg', 'highway-mpg'] features has rmse:0.5904065996733864
The model with ['highway-mpg', 'highway-mpg', 'horsepower'] features has rmse:0.6622571328368921
The model with ['highway-mpg', 'highway-mpg', 'horsepower', 'engine-size'] features has rmse:0.5902520717106036
The model with ['highway-mpg', 'highway-mpg', 'horsepower', 'engine-size', 'city-mpg'] features has rmse:0.5614448491361091
The model with ['highway-mpg', 'highway-mpg', 'horsepower', 'engine-size', 'city-mpg', 'curb-weight'] features has rmse:0.7406898108805515

We can observe from the results above that the model with 5 features has the best rmse.

Hyperparameter Tuning

Last, but not least, we will perform hyperparameter tuning in order to find the best k parameter for our top 3 models from the last section. We will visualize the hyperparameter tuning for each model in order to find out the optimal set.

# list of features of top models
model1 = ['highway-mpg', 'highway-mpg', 'horsepower', 'engine-size', 'city-mpg']
model2 = ['highway-mpg', 'highway-mpg']
model3 = ['highway-mpg', 'highway-mpg', 'horsepower', 'engine-size']

# dataframe to hold the results
results = []

# iterate through various k values
for k in range(1,25):

    # calculate the rmse for models
    res1 = knn_train_test(model1, 'price', cars, k=k)
    res2 = knn_train_test(model2, 'price', cars, k=k)
    res3 = knn_train_test(model3, 'price', cars, k=k)
    
    # save the results to the df
    results.append(['model1', k, res1])
    results.append(['model2', k, res2])
    results.append(['model3', k, res3])
    
# convert list of list to df
df = pd.DataFrame(results, columns=['model', 'k', 'score'])

df.dtypes

model     object
k          int64
score    float64
dtype: object

# visualize the results
import seaborn as sns
fig, ax = plt.subplots(figsize=(16,8))
sns.barplot(x='k', y='score', hue='model', data=df)
plt.show()

The bar chart displays the rmse for each model across k values. We can observe that model 2 has lower rmse consistently but all three models have lower rmse around k=12-14. We can also see that past k values of 15 the rmse started going up again and this indicates that our models started to overfit in for those k values. Nevertheless, k values around 12-14 seem like a good choice for the k parameter.

Summary & Ideas

In this project we build a knn classifier and trained both univariate and multivariate models with varying k values. Some ideas on improving our function are implementing k-fold cross validations since currently we are using half of the data to train and the other half for testing, we could improve the performance and accuracy of the model by implementing k-fold cross validation and choose the k value just as we did with k value for knn.

That’s it for this project. Feel free to leave your feedback and comments here below.

For a complete jupyter notebook file please refer to the github page of the project.

Winning Jeopardy

2019-07-25T00:00:00+00:00

Jeopardy is a popular TV show in the US where participants answer questions to win money. It’s been running for a few decades, and is a major force in popular culture.

Let’s say we want to compete on Jeopardy, and we’re looking for any edge we can get to win. In this project, we’ll work with a dataset of Jeopardy questions to figure out some patterns in the questions that could help us win.

Project Goal In this project, we’ll work with a dataset of Jeopardy questions to figure out some patterns in the questions that could help us win.

This project was motivated as part of the guided project series on dataquest. The analysis is an extended version of the guided project.

Jeopardy Questions

Let’s explore the jeopardy questions. We will see that the dataset contains the following columns:

Show Number – the Jeopardy episode number of the show this question was in.
Air Date – the date the episode aired.
Round – the round of Jeopardy that the question was asked in. Jeopardy has several rounds as each episode progresses.
Category – the category of the question.
Value – the number of dollars answering the question correctly is worth.
Question – the text of the question.
Answer – the text of the answer.

Let’s start our exploration with first of all loading the complete Jeopardy Question csv file from this reddit post. The csv file was downloaded and added to the folder location.

import pandas as pd
from string import punctuation
import re
import matplotlib.pyplot as plt
from scipy.stats import chisquare

# read in the dataset
jeopardy = pd.read_csv('jeopardy.csv')
jeopardy.columns

Index(['Show Number', ' Air Date', ' Round', ' Category', ' Value',
       ' Question', ' Answer'],
      dtype='object')

We can see that the names of each column contain an additional space. Let’s remove those and keep the name only.

# clean column names from additional spaces
jeopardy.columns = jeopardy.columns.str.replace(' ', '')
jeopardy.columns

Index(['ShowNumber', 'AirDate', 'Round', 'Category', 'Value', 'Question',
       'Answer'],
      dtype='object')

jeopardy.shape

(216930, 7)

jeopardy.head()

	ShowNumber	AirDate	Round	Category	Value	Question	Answer
0	4680	2004-12-31	Jeopardy!	HISTORY	$200	For the last 8 years of his life, Galileo was ...	Copernicus
1	4680	2004-12-31	Jeopardy!	ESPN's TOP 10 ALL-TIME ATHLETES	$200	No. 2: 1912 Olympian; football star at Carlisl...	Jim Thorpe
2	4680	2004-12-31	Jeopardy!	EVERYBODY TALKS ABOUT IT...	$200	The city of Yuma in this state has a record av...	Arizona
3	4680	2004-12-31	Jeopardy!	THE COMPANY LINE	$200	In 1963, live on "The Art Linkletter Show", th...	McDonald's
4	4680	2004-12-31	Jeopardy!	EPITAPHS & TRIBUTES	$200	Signer of the Dec. of Indep., framer of the Co...	John Adams

Our dataset contains 216930 questions.

Normalizing Text

Before we start with our analysis, we need to normalize the text, that is ensure lowercase words and remove punctuation. There are two ways we can go about this. Let’s see which one is more efficient.

# function to normalize the text using python string library
def normalize_text(s):
    s = str(s).lower()
    s = s.translate(str.maketrans('', '', punctuation))
    return s

# function to normalize the text using regex
def normalize_text_reg(s):
    s = str(s).lower()
    s = re.sub("[^A-Za-z0-9\s]", "", s)
    return s

s = 'Hello.?! World.*'
%timeit normalize_text(s)
%timeit normalize_text_reg(s)

3.26 µs ± 38.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
3.26 µs ± 194 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

We can observe that using the function with regular expression seems to be much faster. We will use it to treat and normalize our Question and Answer columns.

# normalize our columns
jeopardy['clean_question'] = jeopardy['Question'].apply(normalize_text_reg)
jeopardy['clean_answer'] = jeopardy['Answer'].apply(normalize_text_reg)

jeopardy.head()

	ShowNumber	AirDate	Round	Category	Value	Question	Answer	clean_question	clean_answer
0	4680	2004-12-31	Jeopardy!	HISTORY	$200	For the last 8 years of his life, Galileo was ...	Copernicus	for the last 8 years of his life galileo was u...	copernicus
1	4680	2004-12-31	Jeopardy!	ESPN's TOP 10 ALL-TIME ATHLETES	$200	No. 2: 1912 Olympian; football star at Carlisl...	Jim Thorpe	no 2 1912 olympian football star at carlisle i...	jim thorpe
2	4680	2004-12-31	Jeopardy!	EVERYBODY TALKS ABOUT IT...	$200	The city of Yuma in this state has a record av...	Arizona	the city of yuma in this state has a record av...	arizona
3	4680	2004-12-31	Jeopardy!	THE COMPANY LINE	$200	In 1963, live on "The Art Linkletter Show", th...	McDonald's	in 1963 live on the art linkletter show this c...	mcdonalds
4	4680	2004-12-31	Jeopardy!	EPITAPHS & TRIBUTES	$200	Signer of the Dec. of Indep., framer of the Co...	John Adams	signer of the dec of indep framer of the const...	john adams

Normalizing Columns

Now that we have normalized the string columns, let’s also normalize Value and Air Date columns. We will remove the $ sign from the Value column and convert the column to numeric; we will also convert Air Date to datetime.

# function to normalize value column 
def normalize_value(s):
    s = re.sub("[^A-Za-z0-9\s]", "", s)
    try:
        s = int(s)
    except:
        s = 0
    return s

# clean value column
jeopardy['clean_value'] = jeopardy['Value'].apply(normalize_value)
jeopardy['AirDate'] = pd.to_datetime(jeopardy['AirDate'])

Answers in Questions

In order to figure out whether to study past questions, study general knowledge, or not study it all, it would be helpful to figure out two things:

How often the answer is deducible from the question.
How often new questions are repeats of older questions.

We can answer the first question by seeing how many times words in the answer also occur in the question. We can answer the 2nd question by seeing how often meaningful words occur. Meaningful in this context means words that are not stopwords or words that make sense. Let’s tackle the first question first.

# function to count words from answer in the question
def count_terms(row):
    # split the column values by space
    split_answer = row['clean_answer'].split()
    split_question = row['clean_question'].split()    
    
    # initialize count of words
    match_count = 0
    
    # remove 'the' from answers
    if 'the' in split_answer:
        split_answer.remove('the')
    
    # check if answer is 0
    if len(split_answer) == 0:
        return 0
    
    # count ocurrences of answer values in question
    for i in split_answer:
        if i in split_question:
            match_count += 1
    
    # return the calculated count
    return match_count / len(split_answer)
    
# count the words
jeopardy['answer_in_question'] = jeopardy.apply(count_terms, axis=1)

# calculate the mean
jeopardy['answer_in_question'].mean()

0.05792070323661065

jeopardy.head()

	ShowNumber	AirDate	Round	Category	Value	Question	Answer	clean_question	clean_answer	clean_value
0	4680	2004-12-31	Jeopardy!	HISTORY	$200	For the last 8 years of his life, Galileo was ...	Copernicus	for the last 8 years of his life galileo was u...	copernicus	200
1	4680	2004-12-31	Jeopardy!	ESPN's TOP 10 ALL-TIME ATHLETES	$200	No. 2: 1912 Olympian; football star at Carlisl...	Jim Thorpe	no 2 1912 olympian football star at carlisle i...	jim thorpe	200
2	4680	2004-12-31	Jeopardy!	EVERYBODY TALKS ABOUT IT...	$200	The city of Yuma in this state has a record av...	Arizona	the city of yuma in this state has a record av...	arizona	200
3	4680	2004-12-31	Jeopardy!	THE COMPANY LINE	$200	In 1963, live on "The Art Linkletter Show", th...	McDonald's	in 1963 live on the art linkletter show this c...	mcdonalds	200
4	4680	2004-12-31	Jeopardy!	EPITAPHS & TRIBUTES	$200	Signer of the Dec. of Indep., framer of the Co...	John Adams	signer of the dec of indep framer of the const...	john adams	200

We can see that on average the answer shows up in the question around 6% of the time. This tells us that its quite difficult to deduce the answer from the question.

Recycled Questions

Let’s move to tackling our second question, where we need to find out how often new questions are repeats of older ones.

We also should remove uninformative words (e.g stopwords) in order to better understand the question overlap. For this we will use nltk package to remove stopwords in the english language.

from nltk.corpus import stopwords

# sorty jeopary values in ascending order
jeopardy.sort_values('AirDate', inplace=True)

# stopwords and used terms
terms_used = set()
stop_words = set(stopwords.words('english'))

# function to calculate the number of times 
# a term from previous questions has been used in new ones
def calculate_overlap(row):
    """
    Returns a word match count (proportion) for each row containing 
    a question back to the 'question_overlap' columns. 
    The count is calculated used a set of words used in previous questions
    
    Parameters:
        row (pandas.Series): row from pandas dataframe       
    
    Returns:
        match_count (float): match count proportion
    
    """
    
    # split the question by space
    split_question = row.split()
    
    # keep only non-stop words
    split_question = [word for word in split_question if word not in stop_words]

    # initialize count
    match_count = 0
    
    # loop through words in split_question: count matches
    for w in split_question:
        if w in terms_used:
            match_count += 1

        # add words to terms used set
        terms_used.add(w)
            
    if len(split_question) > 0:
        match_count /= len(split_question)

    return match_count
        
# run the function on the clean_question column
jeopardy['question_overlap'] = jeopardy['clean_question'].apply(calculate_overlap)

# print the mean overlap
print("Average overlap of questions based on individual words is {}".format(jeopardy['question_overlap'].mean()))

Average overlap of questions based on individual words is 0.9285999315781669

Our results show that there is an 92% overlap between terms in new questions vs terms in old questions. This is quite a strong indicator that new questions tend to include terms from old questions. Nevertheless, this doesn’t help us as much since we are only looking at individual terms.

We could possibly investigate further and look at either phrases or find out how similar questions to each other. To keep this analysis brief, we will not cover it today.

Low value vs high value questions

Nevertheless, we can also look at questions that pertain high value in terms of money. This strategy can help us earn more money when on Jeopardy. We can actually figure out which terms correspond to high-value questions using a chi-squared test. We’ll first need to narrow down the questions into two categories:

Low value – Any row where Value is less than 800.
High value – Any row where Value is greater than 800.

Let’s write the code which will help us identify value pairs for terms.

# function to find value 
def calculate_value(row):
    value = 1 if row['clean_value'] > 800 else 0
    return value

# determine questions with high value
jeopardy['high_value'] = jeopardy.apply(calculate_value, axis=1)

# function to calculate value pairs (high, low count) for any word
def calc_word(word):
    high_count = 0
    low_count = 0
    
    for index, row in jeopardy.iterrows():
        q = row['clean_question'].split()
        if word in q:
            if row['high_value'] == 1:
                high_count += 1
            else:
                low_count += 1
    
    return high_count, low_count

# empty dictionary to hold the word and 
# counts of high and low value questions the term appears in
observed_expected = {}

# sample terms to run the value function on
comparison_terms = list(terms_used)[:10]

# calculate value pairs (high and low count) for each word in the sample
for word in comparison_terms:
    observed_expected[word] = calc_word(word)

observed_expected

{'flighty': (2, 6),
 'pennants': (0, 2),
 'miltons': (2, 3),
 'hrefhttpwwwjarchivecommedia20110602j12jpg': (0, 1),
 'civilizations': (4, 19),
 'gethsemane': (1, 6),
 'theatera': (1, 1),
 'hotheaded': (1, 1),
 'hrefhttpwwwjarchivecommedia20080403dj30jpg': (1, 0),
 'targetblankolivia': (0, 1)}

Applying chi-squared test

Now that we’ve found the observed counts for a few terms, we can compute the expected counts and the chi-squared value.

# find total sum of high and low value questions
high_value_count = jeopardy["high_value"].sum()
low_value_count = jeopardy.shape[0] - high_value_count

# dictionary to hold chisquare and p-value for each term
chi_squared = {}

# loop through the dictionary 
for key, value in observed_expected.items():
    
    # calculate total value of high and low
    total = value[0] + value[1]
    
    # calculate word occurence proportion across the dataset
    total_prop = total / jeopardy.shape[0]
    
    # find expected term for high and low value rows
    high_exp, low_exp = total_prop * high_value_count, total_prop * low_value_count
    
    # calculate chi-squared statistic value and associated p-value
    chi_squared[key] = chisquare(value, (high_exp, low_exp))

chi_squared

{'flighty': Power_divergenceResult(statistic=0.043292301416985354, pvalue=0.8351758561462266),
 'pennants': Power_divergenceResult(statistic=0.7899529284667026, pvalue=0.3741143592744989),
 'miltons': Power_divergenceResult(statistic=0.3363947754070794, pvalue=0.5619176551024535),
 'hrefhttpwwwjarchivecommedia20110602j12jpg': Power_divergenceResult(statistic=0.3949764642333513, pvalue=0.5296950912486695),
 'civilizations': Power_divergenceResult(statistic=1.3519672340861204, pvalue=0.24493449653850868),
 'gethsemane': Power_divergenceResult(statistic=0.6787070195906365, pvalue=0.4100323130964867),
 'theatera': Power_divergenceResult(statistic=0.46338644448358013, pvalue=0.49604555208958945),
 'hotheaded': Power_divergenceResult(statistic=0.46338644448358013, pvalue=0.49604555208958945),
 'hrefhttpwwwjarchivecommedia20080403dj30jpg': Power_divergenceResult(statistic=2.5317964247338085, pvalue=0.11157312838169751),
 'targetblankolivia': Power_divergenceResult(statistic=0.3949764642333513, pvalue=0.5296950912486695)}

The results of the chi-squared test show that all of the sample terms we chose are not statistical significant. So far this shows that none of the terms above correspond to high value questions if we defined test significance acceptance level of $p=0.05$

Exploring Categories

Apart from investigating the values of questions, we can also prepare for jeopardy by knowing the likelihood of certain categories appearning in each round. In our dataset we have a column Category which we will use to answer the following questions:

Which categories appear most often
What is the probability of each category appearing in each round.

Now, the first question is pretty straight forward. We simply need to count occurences of each category. There are many unique categories, thus we will only look at top 10.

# count number of categories
unique_cats = len(jeopardy["Category"].unique())

print("Number of unique categories {}".format(unique_cats))

Number of unique categories 27995

# count category ocurrences
cat_occurs = jeopardy["Category"].value_counts().sort_values(ascending=False).head(10)
fig, ax = plt.subplots(figsize=(18, 12))
ax.bar(cat_occurs.index, cat_occurs)
[ax.spines[a].set_visible(False) for a in ax.spines]
plt.xticks(rotation=45)
plt.show()

The above top 10 categories have more questions than the rest of categories. The category ‘Before & After’ seems to be the most popular one. For more information on what this category means, you can read here.

Now, to find the probability of each category appearing in each round, we need to first group the questions by round and then look at the categories and basically calculate the proportion of questions for a specific category in that round. Let’s get to it.

# list of possible rounds
jeopardy["Round"].value_counts()

Jeopardy!           107384
Double Jeopardy!    105912
Final Jeopardy!       3631
Tiebreaker               3
Name: Round, dtype: int64

Jeopardy has only three main rounds with a potential tiebreaker in case of a tie. Let’s calculate the occurence of each category by round.

jeopardy["Round"].value_counts()

Jeopardy!           107384
Double Jeopardy!    105912
Final Jeopardy!       3631
Tiebreaker               3
Name: Round, dtype: int64

# get number of categories by round
catbyround = jeopardy[["Round", "Category", "Question"]].groupby(["Round", "Category"]).agg({'Question': 'count'})

# calculate the proportion of questions for each category in that round
catbyround["Percentage"] = catbyround.groupby(level=0).apply(lambda x: 100 * x / float(x.sum()))

# check results
catbyround

Since we have lots of categories in each round, we need to think of a way to condense it into a more readable format. For that we can only look at the top 10 categories in each round in terms of probability of occurence.

# groupby Round then take the top 10 largest
catbyround['Percentage'].groupby(level=0, group_keys=False).nlargest(10)

Round             Category               
Double Jeopardy!  BEFORE & AFTER              0.424881
                  LITERATURE                  0.359733
                  SCIENCE                     0.279477
                  WORLD GEOGRAPHY             0.239822
                  OPERA                       0.236045
                  WORLD HISTORY               0.223771
                  BALLET                      0.217161
                  COLLEGES & UNIVERSITIES     0.207720
                  ART                         0.202999
                  ISLANDS                     0.202999
Final Jeopardy!   U.S. PRESIDENTS             1.377031
                  WORD ORIGINS                0.936381
                  STATE CAPITALS              0.853759
                  AUTHORS                     0.743597
                  WORLD LEADERS               0.716056
                  FAMOUS AMERICANS            0.660975
                  FAMOUS NAMES                0.660975
                  HISTORIC NAMES              0.660975
                  BUSINESS & INDUSTRY         0.633434
                  ORGANIZATIONS               0.523272
Jeopardy!         POTPOURRI                   0.237466
                  STUPID ANSWERS              0.237466
                  SPORTS                      0.235603
                  ANIMALS                     0.216978
                  AMERICAN HISTORY            0.211391
                  SCIENCE                     0.202079
                  STATE CAPITALS              0.195560
                  TELEVISION                  0.186247
                  U.S. CITIES                 0.181591
                  BUSINESS & INDUSTRY         0.172279
Tiebreaker        CHILD'S PLAY               33.333333
                  LITERARY CHARACTERS        33.333333
                  THE AMERICAN REVOLUTION    33.333333
Name: Percentage, dtype: float64

The above table is a very nice summary of the probability of each category ocurring within each round. Some notable observations are:

It seems tiebreakers only have three category which equal probability of ocurrence
US Presidents is a very popular topic in Final Jeopardy! Round with 1.37% chance of ocurrence, which relative to other categories is high
Literature, science ans sports seem to be quite popular topics touched in every round

Summary

Here, we have explored the wonderful world of jeopardy and tried to find interesting information that could give us an edge in better preparing for the game and ultimately win in it.

For a complete jupyter notebook file please refer to the github page of the project.

Convert Jupyter Notebook to Word

2019-05-15T00:00:00+00:00

During my Data Mining course at the KU Leuven university, we were asked to present our data analysis project in the form of a word document. The justification behind this was that this was the most convenient form for the teacher to provide feedback to our analysis. As we all usually do, I googled ‘How to convert jupyter notebook file (ipynb) to word’. To my surprise there were not many posts describing this.

The best suggestion I found was this one Using Jupyter Notebooks For Assessment – Export as Word (.docx) Extension. The only challenge with this post is that its a bit confusing and cumbersome since it requires you to deal with an extension. Despite that, in the code provided in the post one only needs two commands to run in order to convert the jupyter notebook file (ipynb) to word (docx).

Before that, make sure you have jupyter nbconvert and pandoc set up.

To convert the jupyter notebook to word, first we need to convert the notebook to html. You can perform these commands from cmd, powershell or terminal on mac.

jupyter nbconvert --to html my_notebook.ipynb

Then we need to convert html file generated to word (docx)

pandoc -s my_notebook.html -o my_notebook.docx

As a result, we should have a nicely formatted word document ready for consumption.

My Data Science Journey

2019-03-15T00:00:00+00:00

In this post, I have summarized my data science learning journey and experience so far. The post describes a quick history on how I got excited about data science. In the next post (link below) I have shared the chronological order of online courses, reference materials and basically content I have gone through in my path.

I am originally from Azerbaijan and my first degree was in Tourism Management. I took the degree on due to various factors such as lack of good BA degree courses as well as the quality of education in the country. Azerbaijan Tourism Institute (currently Azerbaijan Tourism Management University) was quite new and promising with high quality education and no shady mechanisms governing education whatsoever. Later on I got involved in the IMC Krems Program which is basically double degree program. I graduated in 2012 with 2 degrees and lots of tourism management related skills :)

Despite that, later on I have worked for 3-4 years as a IT Business Analyst across different domains and projects. My first major business analyst job was as part of the European Games which was organized in Baku, Azerbaijan back in 2015. Later on I moved on to doing similar job in Eurodesign, a regional IT service provider and consultancy. This experience really helped me to dive into the world of data even more.

Throughout my work experience I have done some data analysis across various projects, but never really went deeper. My journey into the world of data science has started in 2016 with Microsoft’s Professional Degree in Data Science. The link is also given in the timeline below. I was in the first iteration of the program, and Microsoft partnered with edx.org has just started this online degree. The degree introduced me into many aspects of data science. In the end all students had to do a Capstone Project (which I also have shared in the post below). Follwing the successful completion of the Professional Degree I decided to dive deeper into this wonderful area, that is constantly changing and evolving. I enrolled in University of Tartu’s “Innovation & Technology Management” Degree where I tried to focus on data analytics and took extra courses on Machine Learning, Math and Statistics. During those years I also participated in Swedbank’s Business Intelligence (BI) Academy program.

‘Data Science Learning Path’ basically describes all of the various online courses, university courses, books and reference materials I have gone through in my journey.

My Data Science Learning Track

2019-03-15T00:00:00+00:00

This post contains online & university courses, as well as various content, reference materials, subjects which I have covered throughout my data science journey for the past 2 years both studying my Master’s Degree. I have provided links and references where possible but some courses I took from the university don’t have an online reference, thus I have summarized my learning from them briefly.

General Data Science Learning

Microsoft Professional Degree in Data Science

MS Degree in Data Science is an online degree program, developed by Microsoft and edx.org. This degree was the first major introduction for me into the world of Data Science and this is how I got excited about the area. The degree covers major topics and subjects necessary to do data science projects. It consists of varios units and you can take either R or Python track or both. For more information, check out the program here.

Microsoft Professional Program in Data Science on edx.org

Dataquest.io

Dataquest is a great platform not just for beginners but also for those who have already had some data science education or learning. This is primarily due to the fact that the platform is very hands on and enables you to work with real-life datasets and learning by doing. One of the benefits of the platform is also the community. As a premium subscriber you get access to Slack and get to meet many interesting people also working towards their data science career. For users with premium subscription, dataquest offers calls with in-house Data Scientists, which is a great opportunity to ask questions. I have joined dataquest in 2018 and still enjoying it.

Dataquest

Mathematics

MIT Open Courseware (OCW)

Math is super important for understanding various machine learning algorithms and numerous topics in Statistics. When it comes to learning or refreshing your math background I think MIT OCW is a great resource. I personally have taken the following three courses from MIT OCW. Please note that I did not have college-level math degree as part of my Bachelor’s.

Single Variable Calculus

Single variable calculus by Prof. David Jerison is a great start since it covers topics such as differentiation, integration and other topics which are fundamental to learning data science. The course is structured very well and I would strongly advice to take it for those wishing to either refresh their knowledge as well as those starting to learn.

MIT OCW Single Variable Calculus

Multivariable Calculus

Multivariable calculus is an extension to single variable calculus in that the topics covered are similar but apply to cases of multivariate analysis. Prof. Denis Aroux is very good at explaining various topics. This was the second course I took after single variable.

MIT OCW Multivariable Calculus

Linear Algebra

Prof. Gilbert Strang is quite popular in the domain of linear algebra. His course on MIT OCW is one of the most popular and his books are used across various institutions to teach linear algebra. The course gives you a good background into understanding matrix algebra, various operations and concepts behind it. I strongly advice taking the course and following it up.

MIT OCW Linear Algebra

3Blue1Brown

3Blue1Brown is a youtube channel that offeres great video courses/explanations into the intution behind certain concepts in calculus and linear algebra. They are short and concise, but nevertheless they offer nice insights into the concepts. I quite enjoyed them and here are the three playlists I watched through.

3Blue1Brown - Essence of Calculus
3Blue1Brown - Essence of Linear Algebra
3Blue1Brown Channel

Other

As part of my Master’s Degree I have also taken Higher Mathematics, Mathematical Statistics & other courses in order to have necessary background for Math and Stats.

Machine Learning & Statistics

UT Machine Learning

University of Tartu (UT) is one of the leaders in computer science education in Europe and I would say the world. UT is ranked in the top 1.2 % of the best universities in the world with a very strong computer science, math & statistics departments. During my studies at UT I have followed a Machine Learning course, where we covered major topics and algorithms used in Data Science, starting from linear classification/regression to deep neural networks. As part of the courses we followed Peter Flach’s “Machine learning: The Art and Science of Algorithms that Make Sense of Data” book. The course was really hands and we implemented different algorithms in python from scratch. At the end of the course professor Meelis Kull has organized a private kaggle competition, where my team (‘Friends’) got in the top 10 (10th spot) out of almost 66 teams and individuals.

The course is publicly available since UT is a public university. By the time of the writing of the post the course is available via the link below.

UT Machine Learning
UT Kaggle Competition

ISLR

Introduction to Statistical Learning (ISLR) is an excellent resource to learn machine learning from more statistical perspective and dive into the inner workings on various machine learning algorithms. The book provides an excellent overview of various machine learning methods with practical examples and applications in R.

ISLR

UT Business Data Analytics

Introduction to Probability & Statistics

MIT 18-05 is a great introductory course on Probability & Statistics. It covers major topics such as probability distributions, central limit theorem, bayesian inference and regression.

MIT OCW Introduction to Probability & Statistics