Standardization

Standardizing your data before starting in on machine learning routines is paramount. Not only does it allow your algorithms to converge faster (by delta’ing over a much narrower scope of data), but it also prevents any features scaled arbitrarily larger from having an inflated weight on whatever your model winds up learning.

E.g. a “0, 1, 2 car garage” probably has more predictive power on a home value than “0-10,000” jelly beans could fit in the master bathtub. Probably.

Getting the Data

Loading one of the gimmie datasets from scikitlearn

from sklearn.datasets import load_boston

data = load_boston()
X = data['data']

Stuffing into a pandas DataFrame for easier inspection

import pandas as pd

df = pd.DataFrame(X, columns=data['feature_names'])
df.head()

And if you look at each attribute, there’s a huge spread of:

• Relative difference between max and min
• Numeric scale of the attribute

df.describe().T[['max', 'min']]

Approaches

And so we have two ways of resolving this data imbalance.

Normalization

This is essentially ensuring that each column will have values between 0 and 1.

We achieve this by finding out the range of values for each column

spread = df.max() - df.min()
spread

CRIM        88.96988
ZN         100.00000
INDUS       27.28000
CHAS         1.00000
NOX          0.48600
RM           5.21900
AGE         97.10000
DIS         10.99690
RAD         23.00000
TAX        524.00000
PTRATIO      9.40000
B          396.58000
LSTAT       36.24000
dtype: float64

And then, per row, subtracting the minimum value and then dividing by that spread.

normed_df = (df - df.min()) / spread
normed_df.head()

normed_df.describe().T[['min', 'max']]

However, this handles outlier data… poorly

%pylab inline

_ = normed_df.boxplot(figsize=(18, 10))

Populating the interactive namespace from numpy and matplotlib

Standardization

Instead, we’ll try standardization, which substracts the mean from each value and then divides by the standard deviation.

means = df.mean()
means

CRIM         3.593761
ZN          11.363636
INDUS       11.136779
CHAS         0.069170
NOX          0.554695
RM           6.284634
AGE         68.574901
DIS          3.795043
RAD          9.549407
TAX        408.237154
PTRATIO     18.455534
B          356.674032
LSTAT       12.653063
dtype: float64

stand_df = (df - means) / df.std()
stand_df.head()

This, of course, leads to data that falls out of our neat, [0:1] range

stand_df.describe().T[['min', 'max']]

But does a… marginally better job at handling outliers

stand_df.boxplot(figsize=(18, 10))

<matplotlib.axes._subplots.AxesSubplot at 0xc6c7978>

And forces each variable to follow a useful unit distribution with mean 0

stand_df.describe().T[['mean', 'std']]

Using Scikit-Learn

Of course, if we weren’t interested in taking our numpy data, piping it into a pandas.DataFrame, doing our transformations, and then .values‘ing our way back to numpy, sklearn provides a useful class to handle this.

# Again, for demonstration
df.describe().T[['mean', 'std', 'min', 'max']]

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
scaler.fit(X)

StandardScaler(copy=True, with_mean=True, with_std=True)

pd.DataFrame(scaler.transform(X)).describe().T[['mean', 'std', 'min', 'max']]