Data Scientist at

Customer Loyalty Prediction 2: Feature Engineering and Feature Selection

04 Apr 2019 - Tags: feature engineering and feature selection

Elo is a Brazillian debit and credit card brand. They offer credit and prepaid transactions, and have paired up with merchants in order offer promotions to cardholders. In order to offer more relevant and personalized promotions, in a recent Kaggle competition, Elo challenged Kagglers to predict customer loyalty based on transaction history. Presumably they plan to use a loyalty-predicting model in order to determine what promotions to offer to customers based on how certain offers are predicted to affect card owners’ card loyalty.

In a previous post, we loaded and cleaned the data, and performed some exploratory data analysis. In this post, we’ll engineer new features about the card accounts, compute aggregate statistics about transactions made with each card, and then select which features to use in a predictive model.


Let’s first load the packages we’ll use:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from scipy.stats import spearmanr
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import RobustScaler
from sklearn.impute import SimpleImputer

from catboost import CatBoostRegressor

# Plot settings
%matplotlib inline
%config InlineBackend.figure_format = 'svg'

# Encoders and permutation importance
!pip install git+http://github.com/brendanhasz/dsutils.git
from dsutils.encoding import one_hot_encode
from dsutils.encoding import TargetEncoderCV
from dsutils.printing import print_table
from dsutils.evaluation import permutation_importance_cv
from dsutils.evaluation import plot_permutation_importance

Feature Engineering

At this point we have four tables:

Eventually, we’ll want one large table where each row corresponds to a card account, and each column corresponds to a feature of that account. There’s not a whole lot of information in the cards table as is, so we’ll have to engineer features of each card using the information in the other three tables. The idea is that information about how often an individual is making transactions, when, for how much, with what merchants, with what types of merchants - and so on - will be informative as to how likely that individual is to have a high loyalty score.

Each transaction in hist_trans and new_trans has a card_id, which we can use to find all the transactions from a given card account in the cards dataset (there is also a card_id column in that table). Similarly, each transaction also has a corresponding merchant_id, which we can use to look up the information about the corresponding merchant in the merchants dataset (that table also has a merchant_id column).

The first thing to do is merge the transactions datasets with the merchants dataset on the merchant_id column. That is, for each transaction, we need to look up the information about the merchant which participated in that transaction in the merchants dataset, and append it to the transactions data.

The pandas merge function makes this pretty easy:

# Merge transactions with merchants data
hist_trans = pd.merge(hist_trans, merchants, on='merchant_id')
new_trans = pd.merge(new_trans, merchants, on='merchant_id')

Next, we’ll want to encode some of the simpler categorical columns. The category_2 and category_3 columns have only 5 and 3 unique values (and NaNs), so we’ll simply one-hot encode them.

# One-hot encode category 2 and 3
cat_cols = ['category_2', 'category_3']
hist_trans = one_hot_encode(hist_trans,
new_trans = one_hot_encode(new_trans,

Then, we can create some time-based features about the purchases. For example: the hour of the day the purchase was made, the day of the week, the week of the year, the month, whether the purchase was made on a weekend, the time of the purchase relative to when the card owner was first active, etc.

# Time-based features for purchases
ref_date = np.datetime64('2017-09-01')
one_hour = np.timedelta64(1, 'h')
for df in [hist_trans, new_trans]:
    tpd = df['purchase_date']
    df['purchase_hour'] = tpd.dt.hour.astype('uint8')
    df['purchase_day'] = tpd.dt.dayofweek.astype('uint8')
    df['purchase_week'] = tpd.dt.weekofyear.astype('uint8')
    df['purchase_dayofyear'] = tpd.dt.dayofyear.astype('uint16')
    df['purchase_month'] = tpd.dt.month.astype('uint8')
    df['purchase_weekend'] = (df['purchase_day'] >=5 ).astype('uint8')
    df['purchase_time'] = ((tpd - ref_date) / one_hour).astype('float32')
    df['ref_date'] = ((tpd - pd.to_timedelta(df['month_lag'], 'M')
                          - ref_date ) / one_hour).astype('float32')

    # Time sime first active
    tsfa = pd.merge(df[['card_id']], 
                    on='card_id', how='left')
    df['time_since_first_active'] = ((tpd - tsfa['first_active_month'])
                                     / one_hour).astype('float32')
    # Clean up
    del tsfa
    del df['purchase_date']

Finally, we need to convert the first_active_month column (a datetime) to a month. This way at this point all of our data will be in a numerical format.

cards['first_active_month'] = (12*(cards['first_active_month'].dt.year-2011) + 

Feature Aggregations

Now we can engineer features for each card account by applying aggregation functions on the transaction data corresponding to each card. First, we need to group the transactions by the card account which was used to make them:

# Group transactions by card id
hist_trans = hist_trans.groupby('card_id', sort=False)
new_trans = new_trans.groupby('card_id', sort=False)

We’ll also need to define some custom aggregation functions which will allow us to better extract information from the transactions. The first is a function which computes the entropy given some categorical data. A feature corresponding to entropy could be informative - for example, it could be that card accounts with high entropy over the merchants they use their card with are more likely to be more loyal card users than those who only use their card with a single merchant (and therefore have low entropy).

def entropy(series):
    """Categorical entropy"""
    probs = series.value_counts().values.astype('float32')
    probs = probs / np.sum(probs)
    probs[probs==0] = np.nan
    return -np.nansum(probs * np.log2(probs))

Another aggregation function which could be useful is one which computes the mean difference between consecutive items in a series. For example, given a column with the purchase date, this function would compute the mean time between purchases. This could conceivably be a good predictor of how likely an individual is to be a loyal card user: individuals who use their cards regularly and frequently are probably more likely to be loyal.

def mean_diff(series):
    """Mean difference between consecutive items in a series"""
    ss = series.sort_values()
    return (ss - ss.shift()).mean()

The period of a sequence of transactions could also be a useful feature. That is, the difference between the minimum and maximum value. For example, customers who have been making purchases over a long period of time (the difference between the date of their first and last purchases is large) may be more likely to be loyal card users.

def period(series):
    """Period of a series (max-min)"""
    return series.max() - series.min()

Finally, we’ll create a function to compute the mode (just because pandas’ default mode function doesn’t handle NaNs well, or cases where there are two equally most common elements).

def mode(series):
    """Most common element in a series"""
    tmode = series.mode()
    if len(tmode) == 0:
        return np.nan
        return tmode[0]

Now we can actually compute the aggregations. We’ll define a list of aggregation functions to perform for each datatype:

# Aggregations to perform for each predictor type
binary_aggs = ['sum', 'mean', 'nunique']
categorical_aggs = ['nunique', entropy, mode]
continuous_aggs = ['min', 'max', 'sum', 'mean', 'std', 'skew', mean_diff, period]

And then, using those lists, we’ll define a dictionary containing which aggregation functions to apply on which columns of the transactions data. I’ve occasionally added a mean() aggregation function when it seems like the variable could be ordinal (but we’re not sure because of the anonymous feature names!).

# Aggregations to perform on each column
aggs = {
    'authorized_flag':             binary_aggs,
    'city_id':                     categorical_aggs,
    'category_1':                  binary_aggs,
    'installments':                continuous_aggs,
    'category_3_nan':              binary_aggs,
    'category_3_0.0':              binary_aggs,
    'category_3_1.0':              binary_aggs,
    'category_3_2.0':              binary_aggs,
    'category_2_nan':              binary_aggs,
    'category_2_1.0':              binary_aggs,
    'category_2_2.0':              binary_aggs,
    'category_2_3.0':              binary_aggs,
    'category_2_4.0':              binary_aggs,
    'category_2_5.0':              binary_aggs,
    'merchant_category_id':        categorical_aggs,
    'merchant_id':                 categorical_aggs,
    'month_lag':                   continuous_aggs,
    'purchase_amount':             continuous_aggs,
    'purchase_time':               continuous_aggs + ['count'],
    'purchase_hour':               categorical_aggs + ['mean'],
    'purchase_day':                categorical_aggs + ['mean'],
    'purchase_week':               categorical_aggs + continuous_aggs,
    'purchase_month':              categorical_aggs + continuous_aggs,
    'purchase_weekend':            binary_aggs,
    'ref_date':                    continuous_aggs,
    'time_since_first_active':     continuous_aggs,
    'state_id':                    categorical_aggs,
    'subsector_id':                categorical_aggs,
    'merchant_group_id':           categorical_aggs,
    'numerical_1':                 continuous_aggs,
    'numerical_2':                 continuous_aggs,
    'most_recent_sales_range':     categorical_aggs + ['mean'], #ordinal?
    'most_recent_purchases_range': categorical_aggs + ['mean'], #orindal?
    'avg_sales_lag3':              continuous_aggs,
    'avg_purchases_lag3':          continuous_aggs,
    'active_months_lag3':          continuous_aggs,
    'avg_sales_lag6':              continuous_aggs,
    'avg_purchases_lag6':          continuous_aggs,
    'active_months_lag6':          continuous_aggs,
    'avg_sales_lag12':             continuous_aggs,
    'avg_purchases_lag12':         continuous_aggs,
    'active_months_lag12':         continuous_aggs,
    'category_4':                  binary_aggs,

Ok, phew, now we can actually compute the aggregations. This’ll take a while!

# Perform each aggregation
for col, funcs in aggs.items():
    for func in funcs:
        # Get name of aggregation function
        if isinstance(func, str):
            func_str = func
            func_str = func.__name__
        # Name for new column
        new_col = col + '_' + func_str
        # Compute this aggregation
        cards['hist_'+new_col] = hist_trans[col].agg(func).astype('float32')
        cards['new_'+new_col] = new_trans[col].agg(func).astype('float32')

After blindly doing a bunch of aggregations on a dataset, it’s usually a good idea to check for non-informative columns. That is, columns which are all NaN, only contain one unique value, etc. Let’s check for those and remove them from the dataset.

def remove_noninformative(df):
    """Remove non-informative columns (all nan, or all same value)"""
    for col in df:
        if df[col].isnull().all():
            print('Removing '+col+' (all NaN)')
            del df[col]
        elif df[col].nunique()<2:
            print('Removing '+col+' (only 1 unique value)')
            del df[col]

Removing new_authorized_flag_mean (only 1 unique value)
Removing new_authorized_flag_nunique (only 1 unique value)
Removing hist_category_3_nan_sum (only 1 unique value)
Removing new_category_3_nan_sum (only 1 unique value)
Removing hist_category_3_nan_mean (only 1 unique value)
Removing new_category_3_nan_mean (only 1 unique value)
Removing hist_category_3_nan_nunique (only 1 unique value)
Removing new_category_3_nan_nunique (only 1 unique value)
Removing hist_category_2_nan_sum (only 1 unique value)
Removing new_category_2_nan_sum (only 1 unique value)
Removing hist_category_2_nan_mean (only 1 unique value)
Removing new_category_2_nan_mean (only 1 unique value)
Removing hist_category_2_nan_nunique (only 1 unique value)
Removing new_category_2_nan_nunique (only 1 unique value)
Removing hist_active_months_lag3_max (only 1 unique value)
Removing hist_active_months_lag6_max (only 1 unique value)

Now we have one, giant table, where each row corresponds to a card account for which we want to predict the loyalty, and each column corresponds to a feature of that account. Unfortunately, what with all the aggregations we performed, we now have well over 400 features!

<class 'pandas.core.frame.DataFrame'>
Index: 325540 entries, C_ID_92a2005557 to C_ID_87e7979a5f
Columns: 459 entries, first_active_month to new_category_4_nunique
dtypes: float32(456), uint8(3)
memory usage: 569.7+ MB

Finally, we’ll split the data into test and training data, as well as X and y data.

# Test data
test = cards['target'].isnull()
X_test = cards[test].copy()
del X_test['target']

# Training data
y_train = cards.loc[~test, 'target'].copy()
X_train = cards[~test].copy()
del X_train['target']

# Clean up 
del cards

Feature Selection

Machine learning models don’t generally perform well when they’re given a huge number of features, many of which are not very informative. The more superfluous features we give our model to train on, the more likely it is to overfit! To prune out features which could confuse our predictive model, we’ll perform some feature selection.

Ideally, we’d fit our model a bunch of different times, using every possible different combination of features, and use the set of features which gives the best cross-validated results. But, there are a few problems with that approach. First, that would lead to overfitting to the training data. Second, and perhaps even more importantly, it would take forever.

There are a bunch of different ways to perform feature selection in a less exhaustive, but more expedient manner. Forward selection, backward selection, selecting features based on their correlation with the target variable, and “embedded” methods such as Lasso regressions (where the model itself performs feature selection during training) are all options. However, here we’ll use two different methods: the mutual information between each feature and the target variable, and the permutation-based feature importance of each feature.

Mutual Information

To get some idea of how well each feature corresponds to the target (the loyalty score), we can compute the mutual information between each feature and the target. Let’s make a function to compute the mutual information between two vectors.

def mutual_information(xi, yi, res=20):
    """Compute the mutual information between two vectors"""
    ix = ~(np.isnan(xi) | np.isinf(xi) | np.isnan(yi) | np.isinf(yi))
    x = xi[ix]
    y = yi[ix]
    N, xe, ye = np.histogram2d(x, y, res)
    Nx, _ = np.histogram(x, xe)
    Ny, _ = np.histogram(y, ye)
    N = N / len(x) #normalize
    Nx = Nx / len(x)
    Ny = Ny / len(y)
    Ni = np.outer(Nx, Ny)
    Ni[Ni == 0] = np.nan
    N[N == 0] = np.nan
    return np.nansum(N * np.log(N / Ni))

The mutual information represents the amount of information that can be gained about one variable by knowing the value of some other vairable. Obviously this is very relevant to the task of feature selection: we want to choose features which knowing the value of will give us as much information as possible about the target variable.

Practically speaking, the nice thing about using mutual information instead of, say, the correlation coefficient, is that it is sensitive to nonlinear relationships. We’ll be using nonlinear predictive models (like gradient boosted decision trees), and so we don’t want to limit the features we select to be only ones which have a linear relationship to the target variable. Notice how the sin-like relationship in the middle plot below has a high mutual information, but not a great correlation coefficient.

# Show mutual information vs correlation
x = 5*np.random.randn(1000)
y = [x + np.random.randn(1000),
     2*np.sin(x) + np.random.randn(1000),
     x + 10*np.random.randn(1000)]
plt.figure(figsize=(10, 4))
for i in range(3):    
    plt.subplot(1, 3, i+1)
    plt.plot(x, y[i], '.')
    rho, _ = spearmanr(x, y[i])
    plt.title('Mutual info: %0.3f\nCorr coeff: %0.3f'
              % (mutual_information(x, y[i]), rho))
    plt.gca().tick_params(labelbottom=False, labelleft=False)


We’ll use the mutual information of the quantile-transformed aggregation scores (just so outliers don’t mess up the mutual information calculation). So, we’ll need a function to perform the quantile transform, and one to compute the mutual information after applying the quantile transform:

def quantile_transform(v, res=101):
    """Quantile-transform a vector to lie between 0 and 1"""
    x = np.linspace(0, 100, res)
    prcs = np.nanpercentile(v, x)
    return np.interp(v, prcs, x/100.0)
def q_mut_info(x, y):
    """Mutual information between quantile-transformed vectors"""
    return mutual_information(quantile_transform(x),

Now we can compute the mutual information between each feature and the loyalty score.

# Compute the mutual information
cols = []
mis = []
for col in X_train:
    mi = q_mut_info(X_train[col], y_train)
# Print mut info of each feature
print_table(['Column', 'Mutual_Information'],
            [cols, mis])
    Column                                    Mutual_Information
    new_purchase_amount_sum                   0.054757
    new_purchase_amount_period                0.054365
    new_active_months_lag12_sum               0.053549
    new_merchant_id_nunique                   0.053511
    new_avg_sales_lag3_sum                    0.053497
    new_avg_purchases_lag3_sum                0.053303
    new_merchant_id_entropy                   0.05321
    new_authorized_flag_sum                   0.053192
    new_purchase_time_count                   0.053192
    new_active_months_lag3_sum                0.05312
    new_active_months_lag6_sum                0.05306
    new_merchant_category_id_nunique          0.053004
    new_avg_sales_lag6_sum                    0.052795
    new_purchase_week_nunique                 0.052637
    new_avg_purchases_lag6_sum                0.052512
    new_avg_sales_lag12_sum                   0.051847
    new_avg_purchases_lag12_sum               0.051539
    new_subsector_id_nunique                  0.050857
    new_merchant_category_id_entropy          0.050746
    new_purchase_hour_nunique                 0.050432
    new_ref_date_sum                          0.050098
    new_purchase_week_entropy                 0.049811
    new_merchant_group_id_nunique             0.048877
    new_purchase_time_sum                     0.048794
    new_purchase_hour_entropy                 0.048575
    new_month_lag_sum                         0.048569
    new_subsector_id_entropy                  0.047022
    new_purchase_day_nunique                  0.046599
    new_purchase_month_sum                    0.046455
    new_purchase_time_period                  0.046102
    new_time_since_first_active_period        0.046101
    new_merchant_group_id_entropy             0.045913
    new_purchase_amount_max                   0.045449
    new_purchase_week_period                  0.044815
    new_purchase_week_sum                     0.04389
    new_purchase_day_entropy                  0.043767
    new_most_recent_purchases_range_nunique   0.042917
    new_most_recent_sales_range_nunique       0.042466
    new_ref_date_period                       0.040804
    new_most_recent_purchases_range_entropy   0.040498
    new_most_recent_sales_range_entropy       0.039465
    new_purchase_month_mean                   0.039101
    new_time_since_first_active_sum           0.037902
    new_avg_sales_lag3_period                 0.037719
    new_avg_purchases_lag3_period             0.037481
    new_month_lag_mean                        0.037309
    new_avg_sales_lag6_period                 0.03696
    new_avg_purchases_lag6_period             0.03626
    new_purchase_month_entropy                0.03607
    new_avg_purchases_lag12_period            0.035874
    new_avg_sales_lag12_period                0.035528
    new_category_3_0.0_sum                    0.033882
    new_month_lag_period                      0.032239
    new_purchase_month_nunique                0.032239
    new_purchase_month_period                 0.032239
    new_purchase_week_mean                    0.030506
    new_numerical_1_period                    0.030269
    new_ref_date_min                          0.030071
    new_purchase_time_min                     0.030042
    new_numerical_2_period                    0.030026
    new_purchase_amount_std                   0.028056
    new_installments_sum                      0.028037
    new_month_lag_std                         0.027715
    new_purchase_time_mean                    0.027591
    hist_purchase_time_max                    0.027389
    new_purchase_weekend_mean                 0.02721
    new_month_lag_mean_diff                   0.027155
    new_ref_date_mean                         0.026726
    new_purchase_week_min                     0.02663
    new_purchase_month_mean_diff              0.026444
    new_purchase_amount_mean                  0.026406
    new_purchase_month_std                    0.026338
    new_city_id_nunique                       0.026197
    new_ref_date_max                          0.026059
    new_purchase_weekend_sum                  0.025851
    new_purchase_week_max                     0.02575
    new_purchase_week_std                     0.025149
    new_numerical_2_sum                       0.024955
    new_purchase_time_max                     0.02463
    new_city_id_entropy                       0.02454
    hist_month_lag_mean_diff                  0.024
    hist_purchase_time_mean_diff              0.023314
    hist_time_since_first_active_mean_diff    0.023314
    hist_ref_date_max                         0.022919
    new_category_2_1.0_sum                    0.022894
    new_purchase_weekend_nunique              0.022806
    new_purchase_month_max                    0.022756
    new_avg_purchases_lag12_min               0.022074
    new_numerical_1_sum                       0.022061
    new_purchase_time_std                     0.021825
    new_time_since_first_active_std           0.021825
    new_avg_sales_lag3_max                    0.021822
    new_avg_purchases_lag6_min                0.021538
    new_purchase_day_mean                     0.021385
    new_avg_purchases_lag3_max                0.021209
    new_avg_sales_lag12_min                   0.0212
    new_category_3_1.0_sum                    0.021029
    new_purchase_month_min                    0.020932
    new_avg_sales_lag6_min                    0.020785
    new_avg_purchases_lag3_min                0.020696
    hist_ref_date_sum                         0.020629
    new_avg_sales_lag3_min                    0.020611
    new_category_3_1.0_mean                   0.020608
    new_most_recent_purchases_range_mean      0.020578
    new_most_recent_sales_range_mean          0.020412
    new_category_4_sum                        0.020386
    new_avg_sales_lag6_max                    0.019944
    hist_purchase_time_sum                    0.019924
    new_purchase_week_mean_diff               0.019582
    hist_ref_date_min                         0.019142
    hist_ref_date_mean                        0.018984
    hist_purchase_week_period                 0.018924
    new_avg_purchases_lag6_max                0.018923
    new_merchant_id_mode                      0.018294
    new_installments_max                      0.018253
    new_numerical_2_max                       0.018197
    new_avg_sales_lag12_max                   0.018155
    hist_authorized_flag_sum                  0.017949
    new_numerical_1_max                       0.017822
    new_avg_purchases_lag12_max               0.017567
    new_installments_period                   0.017396
    hist_avg_sales_lag3_sum                   0.017185
    hist_active_months_lag12_sum              0.01716
    hist_active_months_lag6_sum               0.017157
    hist_purchase_time_count                  0.017145
    hist_active_months_lag3_sum               0.017143
    hist_avg_purchases_lag3_sum               0.017119
    new_category_4_mean                       0.017022
    hist_avg_sales_lag6_sum                   0.016936
    hist_avg_purchases_lag6_sum               0.016819
    new_purchase_month_skew                   0.016749
    hist_avg_sales_lag12_sum                  0.016663
    new_state_id_entropy                      0.016658
    new_purchase_time_mean_diff               0.016612
    new_time_since_first_active_mean_diff     0.016611
    hist_ref_date_mean_diff                   0.016578
    hist_avg_purchases_lag12_sum              0.016576
    hist_merchant_id_nunique                  0.016176
    new_purchase_week_mode                    0.0161
    new_month_lag_skew                        0.015899
    hist_category_3_0.0_sum                   0.015847
    hist_purchase_month_sum                   0.015736
    hist_purchase_week_sum                    0.015632
    new_numerical_2_skew                      0.015542
    new_category_4_nunique                    0.015535
    new_ref_date_std                          0.015378
    new_category_3_2.0_mean                   0.015284
    hist_merchant_category_id_nunique         0.015193
    hist_purchase_amount_sum                  0.015158
    new_numerical_1_skew                      0.015077
    hist_purchase_month_period                0.015014
    hist_merchant_group_id_nunique            0.014956
    new_installments_mean                     0.014832
    new_month_lag_min                         0.014805
    hist_purchase_month_mean_diff             0.014752
    new_numerical_2_mean                      0.014721
    new_purchase_hour_mean                    0.014616
    new_purchase_amount_skew                  0.014573
    hist_purchase_week_mean_diff              0.014502
    hist_avg_sales_lag12_mean_diff            0.014314
    hist_avg_sales_lag6_mean_diff             0.013997
    hist_purchase_month_std                   0.013978
    hist_purchase_week_entropy                0.01396
    new_merchant_group_id_mode                0.01394
    hist_avg_purchases_lag12_mean_diff        0.013919
    new_ref_date_mean_diff                    0.013887
    new_numerical_1_min                       0.013825
    hist_purchase_weekend_sum                 0.013781
    hist_avg_purchases_lag6_mean_diff         0.013741
    hist_purchase_week_std                    0.013535
    new_state_id_nunique                      0.013426
    hist_avg_sales_lag3_mean_diff             0.013339
    hist_category_3_1.0_mean                  0.013327
    hist_purchase_week_max                    0.013283
    hist_subsector_id_nunique                 0.01318
    new_purchase_month_mode                   0.01317
    hist_purchase_week_nunique                0.013119
    new_purchase_amount_mean_diff             0.013111
    new_active_months_lag12_mean              0.012946
    new_numerical_1_mean                      0.0129
    hist_category_3_1.0_sum                   0.012663
    hist_avg_purchases_lag3_mean_diff         0.01259
    hist_installments_mean                    0.012296
    new_installments_std                      0.012075
    hist_purchase_hour_nunique                0.011996
    new_purchase_week_skew                    0.011787
    new_category_3_1.0_nunique                0.011641
    hist_merchant_id_entropy                  0.011539
    new_active_months_lag12_mean_diff         0.011509
    hist_purchase_time_mean                   0.011486
    new_category_3_2.0_sum                    0.011464
    new_numerical_2_min                       0.011085
    new_category_3_2.0_nunique                0.010784
    new_installments_mean_diff                0.010686
    hist_purchase_month_entropy               0.010576
    hist_category_2_1.0_sum                   0.010354
    new_month_lag_max                         0.010285
    hist_purchase_hour_entropy                0.010221
    hist_installments_sum                     0.010082
    hist_ref_date_period                      0.01004
    new_avg_sales_lag3_std                    0.010001
    hist_active_months_lag12_mean_diff        0.0097951
    new_avg_purchases_lag3_std                0.0097771
    hist_purchase_month_max                   0.0095819
    hist_purchase_week_min                    0.0094769
    hist_category_3_0.0_mean                  0.0092955
    new_category_2_1.0_mean                   0.0092706
    new_category_1_mean                       0.0092675
    hist_purchase_time_period                 0.0092008
    hist_time_since_first_active_period       0.0092008
    new_installments_min                      0.0091653
    new_purchase_time_skew                    0.0090182
    new_time_since_first_active_skew          0.0090131
    new_active_months_lag12_skew              0.0089038
    new_avg_purchases_lag12_std               0.0087701
    new_avg_purchases_lag6_std                0.0087281
    hist_month_lag_sum                        0.0087162
    new_avg_sales_lag6_std                    0.0086593
    new_active_months_lag12_period            0.0085932
    hist_avg_sales_lag6_min                   0.0085211
    hist_merchant_group_id_entropy            0.0084834
    hist_avg_sales_lag12_period               0.0084003
    hist_month_lag_skew                       0.008345
    hist_avg_purchases_lag12_min              0.008329
    hist_avg_purchases_lag6_min               0.0082943
    hist_purchase_month_nunique               0.0082444
    hist_avg_sales_lag12_min                  0.0082438
    new_avg_sales_lag12_std                   0.0082004
    new_category_2_1.0_nunique                0.0081688
    hist_avg_sales_lag6_period                0.0081279
    new_avg_sales_lag12_mean_diff             0.0080849
    hist_avg_sales_lag3_period                0.0080384
    hist_month_lag_max                        0.0080244
    new_numerical_2_std                       0.0079489
    new_avg_sales_lag6_mean_diff              0.0079073
    hist_avg_purchases_lag12_period           0.0079065
    hist_time_since_first_active_sum          0.0078141
    hist_category_4_sum                       0.0077914
    hist_city_id_nunique                      0.007767
    hist_merchant_category_id_entropy         0.0077301
    new_avg_purchases_lag6_mean_diff          0.0077202
    hist_purchase_day_entropy                 0.007709
    hist_purchase_time_skew                   0.0077077
    hist_time_since_first_active_skew         0.0077077
    hist_avg_purchases_lag3_period            0.0076734
    hist_numerical_1_sum                      0.0076627
    new_numerical_2_mean_diff                 0.0076452
    hist_avg_purchases_lag6_period            0.0076331
    hist_month_lag_period                     0.0076249
    hist_purchase_time_min                    0.0075276
    hist_category_1_mean                      0.007517
    hist_numerical_2_sum                      0.0075015
    new_avg_purchases_lag12_mean_diff         0.0074972
    new_active_months_lag12_std               0.0074804
    new_numerical_1_std                       0.0074678
    new_ref_date_skew                         0.0072579
    hist_most_recent_purchases_range_nunique  0.0071494
    hist_most_recent_sales_range_nunique      0.0071452
    new_avg_sales_lag3_mean_diff              0.0071183
    hist_month_lag_min                        0.0071006
    hist_installments_mean_diff               0.0070159
    hist_authorized_flag_mean                 0.0069817
    hist_month_lag_mean                       0.0068269
    hist_purchase_month_mode                  0.0068044
    new_active_months_lag12_min               0.0067621
    hist_purchase_week_mode                   0.006748
    hist_purchase_month_min                   0.0065987
    hist_installments_min                     0.006581
    new_category_2_3.0_sum                    0.0064858
    hist_installments_std                     0.0064594
    hist_subsector_id_entropy                 0.0064114
    new_avg_purchases_lag3_mean_diff          0.0063822
    hist_avg_purchases_lag12_max              0.0063011
    hist_installments_skew                    0.0062587
    hist_avg_sales_lag12_max                  0.006192
    hist_numerical_2_skew                     0.0061363
    first_active_month                        0.0061322
    new_avg_sales_lag12_mean                  0.0061219
    hist_numerical_1_period                   0.006049
    hist_category_3_2.0_mean                  0.0059891
    hist_numerical_1_max                      0.005986
    new_numerical_1_mean_diff                 0.0059282
    hist_installments_max                     0.0058908
    hist_avg_sales_lag3_min                   0.0058696
    new_avg_sales_lag6_mean                   0.0058559
    hist_purchase_amount_mean_diff            0.0058548
    new_purchase_amount_min                   0.0058373
    hist_numerical_2_period                   0.0057976
    hist_numerical_2_max                      0.0057563
    hist_numerical_1_skew                     0.0056678
    hist_ref_date_std                         0.0056007
    hist_avg_purchases_lag3_min               0.0055813
    hist_avg_sales_lag6_max                   0.0055765
    new_category_2_5.0_mean                   0.0055583
    hist_active_months_lag12_mean             0.0054629
    new_avg_purchases_lag6_mean               0.0054613
    hist_purchase_day_nunique                 0.0054522
    hist_avg_purchases_lag6_max               0.0054462
    hist_active_months_lag12_skew             0.005362
    new_installments_skew                     0.0053338
    new_avg_purchases_lag12_mean              0.005224
    new_merchant_category_id_mode             0.0051252
    new_category_2_3.0_mean                   0.0050581
    new_category_2_5.0_sum                    0.0050188
    new_avg_purchases_lag3_skew               0.0050157
    hist_most_recent_sales_range_entropy      0.0050028
    new_time_since_first_active_min           0.0049678
    hist_month_lag_std                        0.0049616
    hist_avg_purchases_lag3_max               0.0049593
    hist_avg_sales_lag3_max                   0.0048811
    new_avg_sales_lag3_mean                   0.0048604
    hist_active_months_lag12_std              0.0047854
    hist_numerical_1_std                      0.0047417
    hist_most_recent_purchases_range_entropy  0.0047181
    hist_purchase_time_std                    0.0046897
    hist_time_since_first_active_std          0.0046897
    hist_purchase_amount_skew                 0.0046872
    hist_numerical_2_mean_diff                0.0046632
    hist_category_2_1.0_mean                  0.0046537
    hist_installments_period                  0.0045897
    hist_numerical_2_std                      0.0045355
    hist_numerical_1_mean_diff                0.004512
    hist_avg_sales_lag3_std                   0.0044769
    hist_ref_date_skew                        0.0044716
    hist_purchase_amount_std                  0.0044703
    hist_avg_sales_lag12_mean                 0.0044191
    hist_avg_purchases_lag12_mean             0.0044072
    hist_purchase_month_mean                  0.0044025
    hist_purchase_week_mean                   0.0043976
    new_avg_purchases_lag3_mean               0.004365
    hist_avg_purchases_lag3_std               0.0043399
    hist_purchase_amount_mean                 0.0042991
    new_subsector_id_mode                     0.0042553
    new_avg_purchases_lag12_skew              0.00424
    hist_purchase_amount_period               0.0042385
    hist_category_3_2.0_sum                   0.0042169
    hist_subsector_id_mode                    0.0041857
    hist_avg_purchases_lag6_mean              0.0041039
    hist_active_months_lag12_min              0.0040995
    hist_avg_sales_lag6_mean                  0.0040983
    new_avg_purchases_lag6_skew               0.0040965
    hist_numerical_1_mean                     0.0040761
    hist_time_since_first_active_max          0.0040418
    hist_category_4_mean                      0.0040307
    hist_purchase_amount_max                  0.0040161
    new_category_2_4.0_sum                    0.0039832
    hist_purchase_week_skew                   0.003942
    hist_most_recent_sales_range_mean         0.0039258
    new_avg_sales_lag6_skew                   0.0039179
    new_avg_sales_lag12_skew                  0.0038884
    hist_avg_purchases_lag3_skew              0.0038856
    hist_purchase_month_skew                  0.0038579
    hist_avg_sales_lag6_std                   0.0038449
    new_avg_sales_lag3_skew                   0.0038093
    hist_active_months_lag12_period           0.0037645
    hist_numerical_2_mean                     0.0037435
    hist_avg_purchases_lag6_std               0.0037286
    new_category_1_nunique                    0.0037167
    hist_category_1_sum                       0.0036509
    hist_avg_purchases_lag3_mean              0.0035984
    new_category_3_0.0_mean                   0.0035965
    new_time_since_first_active_mean          0.0035515
    hist_most_recent_purchases_range_mean     0.0035373
    hist_avg_purchases_lag12_std              0.0034711
    hist_purchase_hour_mean                   0.0034215
    hist_avg_sales_lag3_mean                  0.003387
    hist_avg_sales_lag12_std                  0.0033672
    hist_merchant_category_id_mode            0.0033652
    new_category_2_5.0_nunique                0.0033255
    new_category_2_4.0_mean                   0.0032615
    hist_state_id_nunique                     0.0032521
    hist_state_id_mode                        0.0031744
    hist_city_id_mode                         0.0031466
    hist_avg_sales_lag3_skew                  0.0030995
    hist_category_2_5.0_sum                   0.0030599
    hist_purchase_amount_min                  0.0030229
    hist_category_2_5.0_mean                  0.0029206
    new_time_since_first_active_max           0.002807
    new_active_months_lag6_mean_diff          0.0027727
    hist_category_3_1.0_nunique               0.0027499
    hist_avg_purchases_lag6_skew              0.0026864
    feature_1                                 0.0026344
    hist_avg_sales_lag6_skew                  0.0025855
    hist_category_2_3.0_sum                   0.0025425
    hist_category_4_nunique                   0.0025078
    hist_category_3_2.0_nunique               0.0024532
    new_active_months_lag6_std                0.002429
    new_category_2_3.0_nunique                0.0024156
    new_purchase_hour_mode                    0.002373
    hist_time_since_first_active_min          0.0023379
    new_category_1_sum                        0.0023012
    hist_time_since_first_active_mean         0.0022897
    hist_avg_sales_lag12_skew                 0.0022693
    hist_city_id_entropy                      0.0022269
    new_category_2_2.0_mean                   0.002184
    hist_category_2_3.0_mean                  0.0021569
    new_most_recent_purchases_range_mode      0.0021251
    hist_purchase_day_mean                    0.0021149
    hist_merchant_id_mode                     0.002114
    hist_avg_purchases_lag12_skew             0.0021017
    new_state_id_mode                         0.0020227
    hist_category_2_1.0_nunique               0.0019786
    new_city_id_mode                          0.0019017
    new_most_recent_sales_range_mode          0.0018793
    hist_state_id_entropy                     0.0018788
    hist_active_months_lag6_mean_diff         0.001866
    hist_category_2_4.0_sum                   0.0018313
    hist_merchant_group_id_mode               0.001791
    new_active_months_lag6_period             0.0017701
    hist_purchase_weekend_mean                0.0017441
    hist_active_months_lag6_std               0.0017381
    new_category_2_2.0_sum                    0.001606
    hist_category_2_4.0_mean                  0.0014574
    new_category_2_4.0_nunique                0.001447
    hist_category_2_2.0_mean                  0.0014226
    new_active_months_lag12_max               0.0013094
    new_active_months_lag6_min                0.0013084
    new_active_months_lag6_mean               0.0013084
    hist_category_2_5.0_nunique               0.0012898
    hist_category_1_nunique                   0.0012347
    feature_2                                 0.0012139
    hist_purchase_hour_mode                   0.0011777
    hist_purchase_day_mode                    0.0011775
    feature_3                                 0.0011729
    hist_active_months_lag3_mean_diff         0.0011687
    hist_category_2_2.0_sum                   0.0011176
    new_purchase_day_mode                     0.0011036
    hist_active_months_lag3_std               0.001044
    hist_authorized_flag_nunique              0.00092792
    new_active_months_lag3_std                0.00086299
    new_active_months_lag6_skew               0.00080047
    new_category_2_2.0_nunique                0.00077771
    hist_most_recent_purchases_range_mode     0.00077108
    hist_most_recent_sales_range_mode         0.00068015
    hist_purchase_weekend_nunique             0.00063084
    hist_active_months_lag6_period            0.00061179
    hist_numerical_1_min                      0.00058408
    new_active_months_lag3_mean_diff          0.00048793
    hist_category_3_0.0_nunique               0.00047484
    hist_active_months_lag6_min               0.00043478
    hist_active_months_lag6_mean              0.00043478
    hist_active_months_lag6_skew              0.00043441
    hist_numerical_2_min                      0.00029645
    new_category_3_0.0_nunique                0.00027655
    hist_category_2_3.0_nunique               0.00024557
    hist_active_months_lag3_period            0.00024185
    hist_active_months_lag12_max              0.00023522
    new_active_months_lag6_max                0.00023134
    hist_active_months_lag3_skew              0.00019978
    hist_active_months_lag3_min               0.00019977
    hist_active_months_lag3_mean              0.00019977
    new_active_months_lag3_period             0.00014555
    hist_category_2_4.0_nunique               0.00012009
    hist_category_2_2.0_nunique               0.00010438
    new_active_months_lag3_skew               8.4217e-05
    new_active_months_lag3_min                8.4195e-05
    new_active_months_lag3_mean               8.4195e-05
    new_active_months_lag3_max                3.6816e-05          

Let’s only bother keeping the features with the top 200 mutual information scores.

# Create DataFrame with scores
mi_df = pd.DataFrame()
mi_df['Column'] = cols
mi_df['mut_info'] = mis

# Sort by mutual information
mi_df = mi_df.sort_values('mut_info', ascending=False)
top200 = mi_df.iloc[:200,:]
top200 = top200['Column'].tolist()

# Keep only top 200 columns
X_train = X_train[top200]
X_test = X_test[top200]

Permutation-based Feature Importance

A different way to select features is to try and train a model using all the features, and then determine how heavily the model’s performance depends on the features. But, we’ll need to use a model which can handle a lot of features without overfitting too badly (i.e., an unregularized linear regression wouldn’t be a good idea here). So, we’ll use a gradient boosted decision tree, specifically CatBoost.

Let’s create a data processing and prediction pipeline. First, we’ll target encode the categorical columns (basically just set each category to the mean target value for samples having that category - see my previous post on target encoding). Then, we can normalize the data and impute missing data (we’ll just fill in missing data with the median of the column). Finally, we can use CatBoost to predict the loyalty scores from the features we’ve engineered.

# Regression pipeline
cat_cols = [c for c in X_train if 'mode' in c] 
reg_pipeline = Pipeline([
    ('target_encoder', TargetEncoderCV(cols=cat_cols)),
    ('scaler', RobustScaler()),
    ('imputer', SimpleImputer(strategy='median')),
    ('regressor', CatBoostRegressor(loss_function='RMSE', 

We can measure how heavily the model depends on various features by using permutation-based feature importance. Basically, we train the model on all the data, and then measure its error after shuffling each row. When the model’s error increases a lot after shuffling a row, that means that the feature which was shuffled was important for the model’s predictions.

The advantage of permutation-based feature importance is that it gives a super-clear view and a single score as to how important each feature is. The downside is that this score is intrinsically linked to the model. Whereas computing the mutual information between the features and the target only depends on the data, permutation-based feature importance scores depend on the data, the model being used, and the interaction between the two. If your model can’t fit the data very well, your permutation scores will be garbage!

Luckily CatBoost nearly always does a pretty good job of prediction, even in the face of lots of features! So, let’s compute the permutation-based feature importance for each feature (the complete code is on my GitHub).

# Compute the cross-validated feature importance
imp_df = permutation_importance_cv(
    X_train, y_train, reg_pipeline, 'rmse')

Then, we can plot the importance scores for each feature. These scores are just the difference between the model’s error with no shuffled features and the error with the feature of interest shuffled. So, larger scores correspond to features which the model needs to have a low error.

# Plot the feature importances
plt.figure(figsize=(8, 100))


Finally, we’ll want to save the features so that we can use them to train a model to predict the loyalty scores. Let’s save the top 100 most important features to a feather file, so that we can quickly load them back in when we do the modeling. First though, we need to figure out which features are the ones with the best importance scores.

# Get top 100 most important features
df = pd.melt(imp_df, var_name='Feature', value_name='Importance')
dfg = (df.groupby(['Feature'])['Importance']
       .sort_values('Importance', ascending=False))
top100 = dfg['Feature'][:100].tolist()

Then, we can save those features (and the corresponding target variable!) to a feather file.

# Save file w/ 100 most important features
cards = pd.concat([X_train[top100], X_test[top100]])
cards['target'] = y_train


Now that we’ve engineered features for each card account, the next thing to do is create models to predict the target value from those features. In the next post, we’ll try different modeling techniques to see which gives the best predictions.