# Predicting Car Prices using KNN

# Predicting Car Prices

The goal of this project is to practice machine learning workflow by predicting car prices using dataset containing car characteristics. The dataset comes from the UCI’s machine learning repository.

**Disclaimer:** This project is part of the guided project for a Data Scientist Path on dataquest.io.

## Introduction

Let’s first read the dataset and explore it to get a better understanding of it. Since the original dataset doesn’t have column headers in it, we need to provide the list manually.

The numeric feature columns in the dataset are:

- normalized-losses
- wheel-base
- length
- width
- height
- curb-weight
- engine-type
- num-of-cylinders
- engine-size
- bore
- stroke
- compression-rate
- horsepower
- peak-rpm
- city-mpg
- highway-mpg

The target column is `price`

.

```
# import necessary libraries
import pandas as pd
import numpy as np
import collections
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import mean_squared_error
# enable pandas to show all columns
pd.options.display.max_columns = 99
# list of columns names
columns = ['symboling', 'normalized-losses', 'make', 'fuel-type', 'aspiration',
'num-of-doors', 'body-style', 'drive-wheels', 'engine-location', 'wheel-base',
'length', 'width', 'height', 'curb-weight', 'engine-type', 'num-of-cylinders',
'engine-size', 'fuel-system', 'bore', 'stroke', 'compression-rate', 'horsepower',
'peak-rpm', 'city-mpg', 'highway-mpg', 'price']
# read the dataset
cars = pd.read_csv("imports-85.data", names=columns)
# check first few values
cars.head()
```

symboling | normalized-losses | make | fuel-type | aspiration | num-of-doors | body-style | drive-wheels | engine-location | wheel-base | length | width | height | curb-weight | engine-type | num-of-cylinders | engine-size | fuel-system | bore | stroke | compression-rate | horsepower | peak-rpm | city-mpg | highway-mpg | price | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

0 | 3 | ? | alfa-romero | gas | std | two | convertible | rwd | front | 88.6 | 168.8 | 64.1 | 48.8 | 2548 | dohc | four | 130 | mpfi | 3.47 | 2.68 | 9.0 | 111 | 5000 | 21 | 27 | 13495 |

1 | 3 | ? | alfa-romero | gas | std | two | convertible | rwd | front | 88.6 | 168.8 | 64.1 | 48.8 | 2548 | dohc | four | 130 | mpfi | 3.47 | 2.68 | 9.0 | 111 | 5000 | 21 | 27 | 16500 |

2 | 1 | ? | alfa-romero | gas | std | two | hatchback | rwd | front | 94.5 | 171.2 | 65.5 | 52.4 | 2823 | ohcv | six | 152 | mpfi | 2.68 | 3.47 | 9.0 | 154 | 5000 | 19 | 26 | 16500 |

3 | 2 | 164 | audi | gas | std | four | sedan | fwd | front | 99.8 | 176.6 | 66.2 | 54.3 | 2337 | ohc | four | 109 | mpfi | 3.19 | 3.40 | 10.0 | 102 | 5500 | 24 | 30 | 13950 |

4 | 2 | 164 | audi | gas | std | four | sedan | 4wd | front | 99.4 | 176.6 | 66.4 | 54.3 | 2824 | ohc | five | 136 | mpfi | 3.19 | 3.40 | 8.0 | 115 | 5500 | 18 | 22 | 17450 |

```
# check columns and observations
cars.shape
```

```
(205, 26)
```

## Data Cleaning

Machine Learning algorithms can’t treat missing values on their own. For that reason we need to treat missing values. From the output above we immediately can see that `normalized-losses`

column contains missing values in the form of “?”. We need to first replace “?” values with `numpy.nan`

across the dataset. This will help us to easier convert other columns to required datatypes.

```
# replace ? values with np.nan
cars = cars.replace("?", np.nan)
```

```
# check datatypes
cars.dtypes
```

```
symboling int64
normalized-losses object
make object
fuel-type object
aspiration object
num-of-doors object
body-style object
drive-wheels object
engine-location object
wheel-base float64
length float64
width float64
height float64
curb-weight int64
engine-type object
num-of-cylinders object
engine-size int64
fuel-system object
bore object
stroke object
compression-rate float64
horsepower object
peak-rpm object
city-mpg int64
highway-mpg int64
price object
dtype: object
```

Now that we have replaced the “?” values, let’s convert columns containing `int`

or `float`

values to their correct data type

```
# list of columns to convert
cols = ["normalized-losses","bore","stroke","horsepower","peak-rpm","price"]
# convert the values to their appropriate datatype
for col in cols:
cars[col] = pd.to_numeric(cars[col], errors='coerce')
```

Rerun the datatypes cell to check if the conversion was successful.

Now let’s look at the columns which have these missing values and decide on how to handle them.

```
# Check for missing values
cars.isnull().sum()
```

```
symboling 0
normalized-losses 41
make 0
fuel-type 0
aspiration 0
num-of-doors 2
body-style 0
drive-wheels 0
engine-location 0
wheel-base 0
length 0
width 0
height 0
curb-weight 0
engine-type 0
num-of-cylinders 0
engine-size 0
fuel-system 0
bore 4
stroke 4
compression-rate 0
horsepower 2
peak-rpm 2
city-mpg 0
highway-mpg 0
price 4
dtype: int64
```

We can see that there are 41 null values out of 205 total records in `normalized-losses`

. The missing values are 25% of all records. To handle this column entirely we can fill the missing values with the average value or drop the column entirely. We won’t remove the missing rows since this is a big part of the dataset. To decide on this, let’s check the distribution of values in the `normalized-losses`

.

```
cars['normalized-losses'].value_counts().plot(kind='bar', figsize=(16,8))
```

```
<matplotlib.axes._subplots.AxesSubplot at 0x1c3597dd898>
```

We can see that the distribution for this column is right skewed. Thus we won’t be imputing using average since this might skew our results and introduce bias into this column. One potential solution would be to use mean imputation after normalizing the column, which we will do.

```
# normalize the column
cars['normalized-losses'] = (cars['normalized-losses'] - cars['normalized-losses'].mean()) / cars['normalized-losses'].std()
# impute with average value
cars['normalized-losses'].fillna(cars['normalized-losses'].mean(), inplace=True)
```

Rerun, the cell showing missing values to ensure that the imputation was successful. We can also observe that there are few missing values for columns such as `price`

, `horsepower`

, `peak-rpm`

, `bore`

and `stroke`

. We will also normalize them and impute with average value.

```
# columns to be treated as per above
cols = ['bore', 'stroke', 'peak-rpm', 'horsepower', 'price']
# normalize and fill missing values
for col in cols:
cars[col] = (cars[col] - cars[col].mean()) / cars[col].std()
cars[col].fillna(cars[col].mean(), inplace=True)
```

Rerun the cell generating null value count and make sure they are treated.

## Univariate Model

Now let’s start with a univariate k-nearest neighbors model as part of our project for predicting car prices. Starting with simple models before moving to more complex models helps us structure your code workflow and understand the features better.

To enhance and simplify our model testing workflow, we will create a function that encapsulates the training and simple validation process.

```
# function to knn_model training and validation
def knn_train_test(train_name, target_name, data):
# make a copy of the dataset
df = data.copy()
# number of observations for split
obs = int(df.shape[0] / 2)
# split the data into train and test
train_df = df.iloc[:obs]
test_df = df.iloc[obs:]
# reshape the values since we have a single feature
train_data = train_df[train_name].values.reshape(-1, 1)
test_data = test_df[train_name].values.reshape(-1, 1)
# instantiate and fit the model
knn = KNeighborsRegressor()
knn.fit(train_data, train_df[target_name])
# make predictions and evaluate
preds = knn.predict(test_data)
rmse = mean_squared_error(test_df[target_name], preds) ** (1/2)
return rmse
```

```
# use the function to train a model on normalized-losses
knn_train_test('normalized-losses', 'price', cars)
```

```
1.0768788918340664
```

Resulting rmse is based on the model that uses only one feature `normalized-losses`

. We can test additional features to see how our model performs.

```
# use numeric features
numeric_features = ['wheel-base', 'length', 'width', 'height',
'curb-weight', 'engine-size', 'compression-rate',
'city-mpg', 'highway-mpg', 'horsepower', 'peak-rpm']
# loop through models with individual feature
for col in numeric_features:
result = knn_train_test(col, 'price', cars)
print("RMSE for model with {}: {}".format(col, result))
```

```
RMSE for model with wheel-base: 1.214252890227465
RMSE for model with length: 1.1107969135114395
RMSE for model with width: 0.8581184488064216
RMSE for model with height: 1.3712276576274274
RMSE for model with curb-weight: 0.7526797743700095
RMSE for model with engine-size: 0.6066522674323407
RMSE for model with compression-rate: 0.9980929742948769
RMSE for model with city-mpg: 0.6071818370617673
RMSE for model with highway-mpg: 0.5904065996733864
RMSE for model with horsepower: 0.6212450466337898
RMSE for model with peak-rpm: 1.1696138374863996
```

While the models above are not indicative of performance, but it seems that model with `city-mpg`

feature has the lowest error on the model.

Next, let’s modify our function to accept a `k`

parameter and rerun our univariate model with the above features to see the performance. We will use k values of `1`

, `3`

, `5`

, `7`

and `9`

to see which model performs well.

```
# function to knn_model training and validation - modified
def knn_train_test(train_name, target_name, data, k=5):
# make a copy of the dataset
df = data.copy()
# number of observations for split
obs = int(df.shape[0] / 2)
# split the data into train and test
train_df = df.iloc[:obs]
test_df = df.iloc[obs:]
# reshape the values since we have a single feature
train_data = train_df[train_name].values.reshape(-1, 1)
test_data = test_df[train_name].values.reshape(-1, 1)
# instantiate and fit the model
knn = KNeighborsRegressor(n_neighbors=k)
knn.fit(train_data, train_df[target_name])
# make predictions and evaluate
preds = knn.predict(test_data)
rmse = mean_squared_error(test_df[target_name], preds) ** (1/2)
return rmse
```

```
# use numeric features
numeric_features = ['wheel-base', 'length', 'width', 'height',
'curb-weight', 'engine-size', 'compression-rate',
'city-mpg', 'highway-mpg', 'horsepower', 'peak-rpm']
# dictionary to store model results
lowest_rmse = {}
# loop through models with individual feature
for col in numeric_features:
for k in [1,3,5,7,9]:
result = knn_train_test(col, 'price', cars, k=k)
lowest_rmse[(col, k)] = result
lowest_rmse
```

```
{('wheel-base', 1): 1.0200915246848115,
('wheel-base', 3): 1.1330559244770682,
('wheel-base', 5): 1.214252890227465,
('wheel-base', 7): 1.1050644625745172,
('wheel-base', 9): 1.0706582345927034,
('length', 1): 1.2582151367719674,
('length', 3): 1.164040684899018,
('length', 5): 1.1107969135114395,
('length', 7): 1.0788091491246363,
('length', 9): 0.9649699176225501,
('width', 1): 1.173235330629889,
('width', 3): 0.776724402613776,
('width', 5): 0.8581184488064216,
('width', 7): 0.7920962270887307,
('width', 9): 0.7818084447558675,
('height', 1): 1.835270961051065,
('height', 3): 1.4522475457196613,
('height', 5): 1.3712276576274274,
('height', 7): 1.2536105424726978,
('height', 9): 1.1799262400433599,
('curb-weight', 1): 0.8977925627200231,
('curb-weight', 3): 0.7810203611189004,
('curb-weight', 5): 0.7526797743700095,
('curb-weight', 7): 0.7273112573379618,
('curb-weight', 9): 0.6962953604831444,
('engine-size', 1): 0.711130777393697,
('engine-size', 3): 0.6670315366668025,
('engine-size', 5): 0.6066522674323407,
('engine-size', 7): 0.5846523661450151,
('engine-size', 9): 0.5470224273794155,
('compression-rate', 1): 0.9995557239010212,
('compression-rate', 3): 1.0014666665882759,
('compression-rate', 5): 0.9980929742948769,
('compression-rate', 7): 0.994312348888388,
('compression-rate', 9): 0.9657803897563202,
('city-mpg', 1): 0.672247401307514,
('city-mpg', 3): 0.6692599843503406,
('city-mpg', 5): 0.6071818370617673,
('city-mpg', 7): 0.5947058619062482,
('city-mpg', 9): 0.6184289881119851,
('highway-mpg', 1): 0.65226957297654,
('highway-mpg', 3): 0.5514192730892143,
('highway-mpg', 5): 0.5904065996733864,
('highway-mpg', 7): 0.5316220529521046,
('highway-mpg', 9): 0.5045969744641035,
('horsepower', 1): 0.7424499390187735,
('horsepower', 3): 0.751441782151203,
('horsepower', 5): 0.6212450466337898,
('horsepower', 7): 0.5678990997984015,
('horsepower', 9): 0.51029924508396,
('peak-rpm', 1): 1.2547163045227205,
('peak-rpm', 3): 1.1322701001836366,
('peak-rpm', 5): 1.1696138374863996,
('peak-rpm', 7): 1.1987345300306547,
('peak-rpm', 9): 1.1998145119829764}
```

We can observe that there are multiple univariate models with low rmse. Nevertheles, we can find out the one with smallest rmse.

```
# find the feature and k with minimum rmse
minval = min(lowest_rmse.values())
res = [k for k, v in lowest_rmse.items() if v == minval]
print("Model with lowest rmse is based on feature:{} with k={}".format(res[0][0], res[0][1]))
```

```
Model with lowest rmse is based on feature:highway-mpg with k=9
```

## Multivariate Model

Let’s modify the function we wrote to work with multiple columns.

```
# function to knn_model training and validation
# modified for multivariate modeling
def knn_train_test(train_name, target_name, data, k=5):
# make a copy of the dataset
df = data.copy()
# number of observations for split
obs = int(df.shape[0] / 2)
if len(train_name) <= 1:
return "Please provide a list of features"
# split the data into train and test
train_df = df.iloc[:obs]
test_df = df.iloc[obs:]
# instantiate and fit the model
knn = KNeighborsRegressor(n_neighbors=k)
knn.fit(train_df[train_name], train_df[target_name])
# make predictions and evaluate
preds = knn.predict(test_df[train_name])
rmse = mean_squared_error(test_df[target_name], preds) ** (1/2)
return rmse
```

To test our new function, we will choose the best 2, 3, 4, 5 and 6 features sequentially from the univariate model exploration and train using the default value of k=5.

```
# features ordered by performance from univariate modeling
features = ['highway-mpg', 'horsepower', 'engine-size',
'city-mpg', 'curb-weight']
# starting feature set
cols = features[:1]
# iterate and add each feature to the next model
for col in features:
# add the next feature
cols.append(col)
# train the model
res = knn_train_test(cols, 'price', cars)
# print the rmse
print("The model with {} features has rmse:{}".format(cols,res))
```

```
The model with ['highway-mpg', 'highway-mpg'] features has rmse:0.5904065996733864
The model with ['highway-mpg', 'highway-mpg', 'horsepower'] features has rmse:0.6622571328368921
The model with ['highway-mpg', 'highway-mpg', 'horsepower', 'engine-size'] features has rmse:0.5902520717106036
The model with ['highway-mpg', 'highway-mpg', 'horsepower', 'engine-size', 'city-mpg'] features has rmse:0.5614448491361091
The model with ['highway-mpg', 'highway-mpg', 'horsepower', 'engine-size', 'city-mpg', 'curb-weight'] features has rmse:0.7406898108805515
```

We can observe from the results above that the model with 5 features has the best rmse.

## Hyperparameter Tuning

Last, but not least, we will perform hyperparameter tuning in order to find the best `k`

parameter for our top 3 models from the last section. We will visualize the hyperparameter tuning for each model in order to find out the optimal set.

```
# list of features of top models
model1 = ['highway-mpg', 'highway-mpg', 'horsepower', 'engine-size', 'city-mpg']
model2 = ['highway-mpg', 'highway-mpg']
model3 = ['highway-mpg', 'highway-mpg', 'horsepower', 'engine-size']
# dataframe to hold the results
results = []
# iterate through various k values
for k in range(1,25):
# calculate the rmse for models
res1 = knn_train_test(model1, 'price', cars, k=k)
res2 = knn_train_test(model2, 'price', cars, k=k)
res3 = knn_train_test(model3, 'price', cars, k=k)
# save the results to the df
results.append(['model1', k, res1])
results.append(['model2', k, res2])
results.append(['model3', k, res3])
# convert list of list to df
df = pd.DataFrame(results, columns=['model', 'k', 'score'])
```

```
df.dtypes
```

```
model object
k int64
score float64
dtype: object
```

```
# visualize the results
import seaborn as sns
fig, ax = plt.subplots(figsize=(16,8))
sns.barplot(x='k', y='score', hue='model', data=df)
plt.show()
```

The bar chart displays the rmse for each model across k values. We can observe that model 2 has lower rmse consistently but all three models have lower rmse around k=12-14. We can also see that past k values of 15 the rmse started going up again and this indicates that our models started to overfit in for those k values. Nevertheless, k values around 12-14 seem like a good choice for the k parameter.

## Summary & Ideas

In this project we build a knn classifier and trained both univariate and multivariate models with varying k values. Some ideas on improving our function are implementing k-fold cross validations since currently we are using half of the data to train and the other half for testing, we could improve the performance and accuracy of the model by implementing k-fold cross validation and choose the k value just as we did with k value for knn.

That’s it for this project. Feel free to leave your feedback and comments here below.

For a complete jupyter notebook file please refer to the github page of the project.

## Leave a comment