Problem Statement

Business Context

A sales forecast is a prediction of future sales revenue based on historical data, industry trends, and the status of the current sales pipeline. Businesses use the sales forecast to estimate weekly, monthly, quarterly, and annual sales totals. A company needs to make an accurate sales forecast as it adds value across an organization and helps the different verticals to chalk out their future course of action.

Forecasting helps an organization plan its sales operations by region and provides valuable insights to the supply chain team regarding the procurement of goods and materials. An accurate sales forecast process has many benefits which include improved decision-making about the future and reduction of sales pipeline and forecast risks. Moreover, it helps to reduce the time spent in planning territory coverage and establish benchmarks that can be used to assess trends in the future.

Objective

SuperKart is a retail chain operating supermarkets and food marts across various tier cities, offering a wide range of products. To optimize its inventory management and make informed decisions around regional sales strategies, SuperKart wants to accurately forecast the sales revenue of its outlets for the upcoming quarter.

To operationalize these insights at scale, the company has partnered with a data science firm—not just to build a predictive model based on historical sales data, but to develop and deploy a robust forecasting solution that can be integrated into SuperKart’s decision-making systems and used across its network of stores.

Data Description

The data contains the different attributes of the various products and stores.The detailed data dictionary is given below.

• Product_Id - unique identifier of each product, each identifier having two letters at the beginning followed by a number.
• Product_Weight - weight of each product
• Product_Sugar_Content - sugar content of each product like low sugar, regular and no sugar
• Product_Allocated_Area - ratio of the allocated display area of each product to the total display area of all the products in a store
• Product_Type - broad category for each product like meat, snack foods, hard drinks, dairy, canned, soft drinks, health and hygiene, baking goods, bread, breakfast, frozen foods, fruits and vegetables, household, seafood, starchy foods, others
• Product_MRP - maximum retail price of each product
• Store_Id - unique identifier of each store
• Store_Establishment_Year - year in which the store was established
• Store_Size - size of the store depending on sq. feet like high, medium and low
• Store_Location_City_Type - type of city in which the store is located like Tier 1, Tier 2 and Tier 3. Tier 1 consists of cities where the standard of living is comparatively higher than its Tier 2 and Tier 3 counterparts.
• Store_Type - type of store depending on the products that are being sold there like Departmental Store, Supermarket Type 1, Supermarket Type 2 and Food Mart
• Product_Store_Sales_Total - total revenue generated by the sale of that particular product in that particular store

Installing and Importing the necessary libraries

#Installing the libraries with the specified versions
!pip install numpy==2.0.2 pandas==2.2.2 scikit-learn==1.6.1 matplotlib==3.10.0 seaborn==0.13.2 joblib==1.4.2 xgboost==2.1.4 requests==2.32.4 huggingface_hub==0.34.0 -q

   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 301.8/301.8 kB 5.0 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 223.6/223.6 MB 1.6 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 558.7/558.7 kB 14.6 MB/s eta 0:00:00

Note:

• After running the above cell, kindly restart the notebook kernel (for Jupyter Notebook) or runtime (for Google Colab) and run all cells sequentially from the next cell.

• On executing the above line of code, you might see a warning regarding package dependencies. This error message can be ignored as the above code ensures that all necessary libraries and their dependencies are maintained to successfully execute the code in this notebook.

import warnings
warnings.filterwarnings("ignore")

# Libraries to help with reading and manipulating data
import numpy as np
import pandas as pd

# For splitting the dataset
from sklearn.model_selection import train_test_split

# Libaries to help with data visualization
import matplotlib.pyplot as plt
import seaborn as sns

# Removes the limit for the number of displayed columns
pd.set_option("display.max_columns", None)
# Sets the limit for the number of displayed rows
pd.set_option("display.max_rows", 100)


# Libraries different ensemble classifiers
from sklearn.ensemble import (
    BaggingRegressor,
    RandomForestRegressor,
    AdaBoostRegressor,
    GradientBoostingRegressor,
)
from xgboost import XGBRegressor
from sklearn.tree import DecisionTreeRegressor

# Libraries to get different metric scores
from sklearn.metrics import (
    confusion_matrix,
    accuracy_score,
    precision_score,
    recall_score,
    f1_score,
    mean_squared_error,
    mean_absolute_error,
    r2_score,
    mean_absolute_percentage_error
)

# To create the pipeline
from sklearn.compose import make_column_transformer
from sklearn.pipeline import make_pipeline,Pipeline

# To tune different models and standardize
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler,OneHotEncoder

# To serialize the model
import joblib

# os related functionalities
import os

# API request
import requests

# for hugging face space authentication to upload files
from huggingface_hub import login, HfApi

Loading the dataset

# mount google drive
from google.colab import drive
drive.mount('/content/drive')

Mounted at /content/drive

file_path = '/content/drive/MyDrive/Colab Notebooks/'

kart = pd.read_csv(file_path + 'SuperKart.csv')
data = kart.copy()

Data Overview

data.head()

	Product_Id	Product_Weight	Product_Sugar_Content	Product_Allocated_Area	Product_Type	Product_MRP	Store_Id	Store_Establishment_Year	Store_Size	Store_Location_City_Type	Store_Type	Product_Store_Sales_Total
0	FD6114	12.66	Low Sugar	0.027	Frozen Foods	117.08	OUT004	2009	Medium	Tier 2	Supermarket Type2	2842.40
1	FD7839	16.54	Low Sugar	0.144	Dairy	171.43	OUT003	1999	Medium	Tier 1	Departmental Store	4830.02
2	FD5075	14.28	Regular	0.031	Canned	162.08	OUT001	1987	High	Tier 2	Supermarket Type1	4130.16
3	FD8233	12.10	Low Sugar	0.112	Baking Goods	186.31	OUT001	1987	High	Tier 2	Supermarket Type1	4132.18
4	NC1180	9.57	No Sugar	0.010	Health and Hygiene	123.67	OUT002	1998	Small	Tier 3	Food Mart	2279.36

data.tail()

	Product_Id	Product_Weight	Product_Sugar_Content	Product_Allocated_Area	Product_Type	Product_MRP	Store_Id	Store_Establishment_Year	Store_Size	Store_Location_City_Type	Store_Type	Product_Store_Sales_Total
8758	NC7546	14.80	No Sugar	0.016	Health and Hygiene	140.53	OUT004	2009	Medium	Tier 2	Supermarket Type2	3806.53
8759	NC584	14.06	No Sugar	0.142	Household	144.51	OUT004	2009	Medium	Tier 2	Supermarket Type2	5020.74
8760	NC2471	13.48	No Sugar	0.017	Health and Hygiene	88.58	OUT001	1987	High	Tier 2	Supermarket Type1	2443.42
8761	NC7187	13.89	No Sugar	0.193	Household	168.44	OUT001	1987	High	Tier 2	Supermarket Type1	4171.82
8762	FD306	14.73	Low Sugar	0.177	Snack Foods	224.93	OUT002	1998	Small	Tier 3	Food Mart	2186.08

# understand shape of the dataset
print(f"There are {data.shape[0]} rows without header and {data.shape[1]} columns.")

There are 8763 rows without header and 12 columns.

# check the data types of the columns for the dataset
data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 8763 entries, 0 to 8762
Data columns (total 12 columns):
 #   Column                     Non-Null Count  Dtype  
---  ------                     --------------  -----  
 0   Product_Id                 8763 non-null   object 
 1   Product_Weight             8763 non-null   float64
 2   Product_Sugar_Content      8763 non-null   object 
 3   Product_Allocated_Area     8763 non-null   float64
 4   Product_Type               8763 non-null   object 
 5   Product_MRP                8763 non-null   float64
 6   Store_Id                   8763 non-null   object 
 7   Store_Establishment_Year   8763 non-null   int64  
 8   Store_Size                 8763 non-null   object 
 9   Store_Location_City_Type   8763 non-null   object 
 10  Store_Type                 8763 non-null   object 
 11  Product_Store_Sales_Total  8763 non-null   float64
dtypes: float64(4), int64(1), object(7)
memory usage: 821.7+ KB

# statistical summary of the data
data.describe(include='all').T

	count	unique	top	freq	mean	std	min	25%	50%	75%	max
Product_Id	8763	8763	FD306	1	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Weight	8763.0	NaN	NaN	NaN	12.653792	2.21732	4.0	11.15	12.66	14.18	22.0
Product_Sugar_Content	8763	4	Low Sugar	4885	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Allocated_Area	8763.0	NaN	NaN	NaN	0.068786	0.048204	0.004	0.031	0.056	0.096	0.298
Product_Type	8763	16	Fruits and Vegetables	1249	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_MRP	8763.0	NaN	NaN	NaN	147.032539	30.69411	31.0	126.16	146.74	167.585	266.0
Store_Id	8763	4	OUT004	4676	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Establishment_Year	8763.0	NaN	NaN	NaN	2002.032751	8.388381	1987.0	1998.0	2009.0	2009.0	2009.0
Store_Size	8763	3	Medium	6025	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Location_City_Type	8763	3	Tier 2	6262	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Type	8763	4	Supermarket Type2	4676	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Store_Sales_Total	8763.0	NaN	NaN	NaN	3464.00364	1065.630494	33.0	2761.715	3452.34	4145.165	8000.0

# check for duplicates
data.duplicated().sum()

np.int64(0)

# find missing values
data.isnull().sum()

	0
Product_Id	0
Product_Weight	0
Product_Sugar_Content	0
Product_Allocated_Area	0
Product_Type	0
Product_MRP	0
Store_Id	0
Store_Establishment_Year	0
Store_Size	0
Store_Location_City_Type	0
Store_Type	0
Product_Store_Sales_Total	0

dtype: int64

Exploratory Data Analysis (EDA)

data.shape

(8763, 12)

Univariate Analysis

# function to plot a boxplot and a histogram along the same scale

def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
    """
    Boxplot and histogram combined

    data: dataframe
    feature: dataframe column
    figsize: size of figure (default (12,7))
    kde: whether to show the density curve (default False)
    bins: number
    """
    f2, (ax_box2, ax_hist2) = plt.subplots(
        nrows=2,  # Number of rows of the subplot grid= 2
        sharex=True,  # x-axis will be shared among all subplots
        gridspec_kw={"height_ratios": (0.25, 0.74)},
        figsize=figsize,
    )  # creating the 2 subplots
    sns.boxplot(
        data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
    )  # boxplot will be created and a star will indicate the mean value of the column
    sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
    ) if bins else sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2
    ) # for histogram
    ax_hist2.axvline(
        data[feature].mean(), color="green", linestyle="--"
    )  # Add a vertical line to the mean of the column / add mean to the histogram
    ax_hist2.axvline(
        data[feature].median(), color="black", linestyle="-"
    )  # Add a vertical line to the median of the column / add median to the histogram

# product weight
histogram_boxplot(data, 'Product_Weight')

# product allocated area
histogram_boxplot(data, 'Product_Allocated_Area')

# product mrp
histogram_boxplot(data, 'Product_MRP')

# product store sales total
histogram_boxplot(data, 'Product_Store_Sales_Total')

# barplots function
def labeled_barplot(data, feature, perc=False, n=None):
    """
    Barplot

    data: dataframe
    feature: dataframe column
    perc: whether to show percentages instead of count (default False)
    n: displays the top n category levels (default None, i.e., display all levels)
    """
    total = len(data[feature])  # length of the column
    count = data[feature].nunique()
    if n is None:
        plt.figure(figsize=(count + 1, 5))
    else:
        plt.figure(figsize=(n + 1, 5))
    plt.xticks(rotation=90, fontsize=10)
    ax = sns.countplot(
        data=data,
        x=feature,
        palette="Paired",
        order=data[feature].value_counts().index[:n].sort_values(),
    )
    for p in ax.patches:
        if perc == True:
            label = "{:.1f}%".format(
                100 * p.get_height() / total
            ) # percentage of each class of the category
        else:
            label = p.get_height() # count of each level of the category

        x = p.get_x() + p.get_width() / 2  # width of the plot
        y = p.get_height() # height of the plot
        ax.annotate(
            label,
            (x, y),
            ha="center",
            va="center",
            size=12,
            xytext=(0, 5),
            textcoords="offset points",
        )  # annotate the percentage
    plt.show() # show the plot

# product sugar content
labeled_barplot(data, 'Product_Sugar_Content', perc=True)

# product type
labeled_barplot(data, 'Product_Type', perc=True)

# store id
labeled_barplot(data, 'Store_Id', perc=True)

# store size
labeled_barplot(data, 'Store_Size', perc=True)

# store location city type
labeled_barplot(data, 'Store_Location_City_Type', perc=True)

Bivariate Analysis

# correlation matrix
cols_list = data.select_dtypes(include=np.number).columns.tolist()
plt.figure(figsize=(16, 6))
sns.heatmap(data[cols_list].corr(), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral")
plt.show()

# check distribution of product store sales total with numeric columns
plt.figure(figsize=[10, 8])
sns.scatterplot(x='Product_Weight', y='Product_Store_Sales_Total', data=data)
plt.show()

plt.figure(figsize=[10, 8])
sns.scatterplot(x='Product_Allocated_Area', y='Product_Store_Sales_Total', data=data)
plt.show()

plt.figure(figsize=[10, 8])
sns.scatterplot(x='Product_MRP', y='Product_Store_Sales_Total', data=data)
plt.show()

# see which product type the company is generating most of the revenue
revenue_df1 = data.groupby(['Product_Type'], as_index=False)[
    'Product_Store_Sales_Total'
    ].sum()
plt.figure(figsize=[10, 8])
plt.xticks(rotation=90)
a = sns.barplot(x=revenue_df1.Product_Type, y=revenue_df1.Product_Store_Sales_Total)
a.set_xlabel("Product Type")
a.set_ylabel("Revenue")
a.set_title("Product Type vs Revenue")
plt.show()

revenue_df2 = data.groupby(['Product_Sugar_Content'], as_index=False)[
    'Product_Store_Sales_Total'
    ].sum()
plt.figure(figsize=[10, 8])
plt.xticks(rotation=90)
b = sns.barplot(x=revenue_df2.Product_Sugar_Content, y=revenue_df2.Product_Store_Sales_Total)
b.set_xlabel("Product Sugar Content")
b.set_ylabel("Revenue")
b.set_title("Product Sugar Content vs Revenue")
plt.show()

# which stores and locations the revenue generated more
store_revenue_df = data.groupby(['Store_Id'], as_index=False)[
    'Product_Store_Sales_Total'
    ].sum()
plt.figure(figsize=[10, 8])
plt.xticks(rotation=90)
rate = sns.barplot(x=store_revenue_df.Store_Id, y=store_revenue_df.Product_Store_Sales_Total)
rate.set_xlabel("Store Id")
rate.set_ylabel("Revenue")
rate.set_title("Store Id vs Revenue")
plt.show()

revenue_df3 = data.groupby(['Store_Location_City_Type'], as_index=False)[
    'Product_Store_Sales_Total'
    ].sum()
plt.figure(figsize=[10, 8])
plt.xticks(rotation=90)
city = sns.barplot(x=revenue_df3.Store_Location_City_Type, y=revenue_df3.Product_Store_Sales_Total)
city.set_xlabel("Store Location City Type")
city.set_ylabel("Revenue")
city.set_title("Store Location City Type vs Revenue")
plt.show()

revenue_df4 = data.groupby(['Store_Size'], as_index=False)[
    'Product_Store_Sales_Total'
    ].sum()
plt.figure(figsize=[10, 8])
plt.xticks(rotation=90)
size = sns.barplot(x=revenue_df4.Store_Size, y=revenue_df4.Product_Store_Sales_Total)
size.set_xlabel("Store Size")
size.set_ylabel("Revenue")
size.set_title("Store Size vs Revenue")
plt.show()

revenue_df5 = data.groupby(['Store_Type'], as_index=False)[
    'Product_Store_Sales_Total'
    ].sum()
plt.figure(figsize=[10, 8])
plt.xticks(rotation=90)
type = sns.barplot(x=revenue_df5.Store_Type, y=revenue_df5.Product_Store_Sales_Total)
type.set_xlabel("Store Type")
type.set_ylabel("Revenue")
type.set_title("Store Type vs Revenue")
plt.show()

# Product_Store_Sale_Total with other categorical columns
plt.figure(figsize=[10, 8])
sns.boxplot(x='Store_Id', y='Product_Store_Sales_Total', data=data)
plt.xticks(rotation=90)
plt.title("Boxplot - Store_Id vs Product_Store_Sales_Total")
plt.xlabel("Store Id")
plt.ylabel("Product_Store_Sales_Total")
plt.show()

plt.figure(figsize=[10, 8])
sns.boxplot(x='Store_Size', y='Product_Store_Sales_Total', data=data)
plt.xticks(rotation=90)
plt.title("Boxplot - Store_Size vs Product_Store_Sales_Total")
plt.xlabel("Stores")
plt.ylabel("Product_Store_Sales_Total")
plt.show()

# relationship b/t the other columns
plt.figure(figsize=[10, 8])
sns.boxplot(data=data, x='Store_Location_City_Type', y='Product_Weight', hue='Store_Location_City_Type')
plt.xticks(rotation=90)
plt.title("Boxplot - Store_Location_City_Type vs Product_Weight")
plt.xlabel("Store Location City Type")
plt.ylabel("Product_Weight")
plt.show()

# weight of the product and sugar content
plt.figure(figsize=[10, 8])
sns.boxplot(data=data, x='Product_Sugar_Content', y='Product_Weight', hue='Product_Sugar_Content')
plt.xticks(rotation=90)
plt.title("Boxplot - Product_Sugar_Content vs Product_Weight")
plt.xlabel("Product Sugar Content")
plt.ylabel("Product_Weight")
plt.show()

# sugar content of different product types
plt.figure(figsize=[10, 8])
sns.heatmap(
    pd.crosstab(data["Product_Sugar_Content"], data["Product_Type"]), annot=True,
    fmt="g",
    cmap="Blues",
)
plt.xticks(rotation=90)
plt.title("Product_Sugar_Content vs Product_Type")
plt.xlabel("Product Sugar Content")
plt.ylabel("Product_Type")
plt.show()

# each of the product type sold vs each store name
plt.figure(figsize=[10, 8])
sns.heatmap(
    pd.crosstab(data["Store_Type"], data["Product_Type"]), annot=True,
    fmt="g",
    cmap="Blues",
)
plt.xticks(rotation=90)
plt.title("Store_Type vs Product_Type")
plt.xlabel("Store_Type")
plt.ylabel("Product_Type")
plt.show()

# product types have different prices. analyze them.
plt.figure(figsize=[10, 8])
sns.boxplot(data=data, x='Product_Type', y='Product_MRP', hue='Product_Type')
plt.xticks(rotation=90)
plt.title("Boxplot - Product_Type vs Product_MRP")
plt.xlabel("Product_Type")
plt.ylabel("Product_MRP")
plt.show()

# drive deeper and make a detailed analysis of the each of the stores

# OUT001
data.loc[data['Store_Id'] == 'OUT001'].describe(include='all').T

	count	unique	top	freq	mean	std	min	25%	50%	75%	max
Product_Id	1586	1586	NC7187	1	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Weight	1586.0	NaN	NaN	NaN	13.458865	2.064975	6.16	12.0525	13.96	14.95	17.97
Product_Sugar_Content	1586	4	Low Sugar	845	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Allocated_Area	1586.0	NaN	NaN	NaN	0.068768	0.047131	0.004	0.033	0.0565	0.094	0.295
Product_Type	1586	16	Snack Foods	202	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_MRP	1586.0	NaN	NaN	NaN	160.514054	30.359059	71.35	141.72	168.32	182.9375	226.59
Store_Id	1586	1	OUT001	1586	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Establishment_Year	1586.0	NaN	NaN	NaN	1987.0	0.0	1987.0	1987.0	1987.0	1987.0	1987.0
Store_Size	1586	1	High	1586	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Location_City_Type	1586	1	Tier 2	1586	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Type	1586	1	Supermarket Type1	1586	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Store_Sales_Total	1586.0	NaN	NaN	NaN	3923.778802	904.62901	2300.56	3285.51	4139.645	4639.4	4997.63

df_OUT001 = (
    data.loc[data['Store_Id'] == 'OUT001']
    .groupby(['Product_Type'], as_index=False)['Product_Store_Sales_Total']
    .sum()
)
plt.figure(figsize=[10, 8])
plt.xticks(rotation=90)
plt.xlabel("Product_Type")
plt.ylabel("Product_Store_Sales_Total")
plt.title("Product_Store_Sales_Total")
sns.barplot(x=df_OUT001.Product_Type, y=df_OUT001.Product_Store_Sales_Total)
plt.show()

• OUT001 has generated the highest revenue from sales of fruits and vegetables and Snack Foods. Both of categories contributed around 800,000 each.

# OUT002

data.loc[data['Store_Id'] == 'OUT002'].describe(include='all').T

	count	unique	top	freq	mean	std	min	25%	50%	75%	max
Product_Id	1152	1152	NC2769	1	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Weight	1152.0	NaN	NaN	NaN	9.911241	1.799846	4.0	8.7675	9.795	10.89	19.82
Product_Sugar_Content	1152	4	Low Sugar	658	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Allocated_Area	1152.0	NaN	NaN	NaN	0.067747	0.047567	0.006	0.031	0.0545	0.09525	0.292
Product_Type	1152	16	Fruits and Vegetables	168	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_MRP	1152.0	NaN	NaN	NaN	107.080634	24.912333	31.0	92.8275	104.675	117.8175	224.93
Store_Id	1152	1	OUT002	1152	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Establishment_Year	1152.0	NaN	NaN	NaN	1998.0	0.0	1998.0	1998.0	1998.0	1998.0	1998.0
Store_Size	1152	1	Small	1152	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Location_City_Type	1152	1	Tier 3	1152	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Type	1152	1	Food Mart	1152	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Store_Sales_Total	1152.0	NaN	NaN	NaN	1762.942465	462.862431	33.0	1495.4725	1889.495	2133.6225	2299.63

1. OUT002 is a food mart which is located in a Tier 3 city and has store size as small. It was estabished in 1998.

2. OUT002 has sold products whose MRP range from 31 to 225.

3. Fruits and vegetables are the most sold product.

4. The revenue generated from each product at OUT002 from 33 to 2300.

OUT00_df = (
    data.loc[data['Store_Id'] == 'OUT002']
    .groupby(['Product_Type'], as_index=False)['Product_Store_Sales_Total']
    .sum()
)
plt.figure(figsize=[10, 8])
plt.xticks(rotation=90)
plt.xlabel("Product_Type")
plt.ylabel("Product_Store_Sales_Total")
plt.title("Product_Store_Sales_Total")
sns.barplot(x=OUT00_df.Product_Type, y=OUT00_df.Product_Store_Sales_Total)
plt.show()

• OUT002 has generated the highest revenue from the sale of fruits and vegetables (~ 300,000) followed by snack foods (250,000).

# OUT003
data.loc[data['Store_Id'] == 'OUT003'].describe(include='all').T

	count	unique	top	freq	mean	std	min	25%	50%	75%	max
Product_Id	1349	1349	NC522	1	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Weight	1349.0	NaN	NaN	NaN	15.103692	1.893531	7.35	14.02	15.18	16.35	22.0
Product_Sugar_Content	1349	4	Low Sugar	750	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Allocated_Area	1349.0	NaN	NaN	NaN	0.068637	0.048708	0.004	0.031	0.057	0.094	0.298
Product_Type	1349	16	Snack Foods	186	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_MRP	1349.0	NaN	NaN	NaN	181.358725	24.796429	85.88	166.92	179.67	198.07	266.0
Store_Id	1349	1	OUT003	1349	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Establishment_Year	1349.0	NaN	NaN	NaN	1999.0	0.0	1999.0	1999.0	1999.0	1999.0	1999.0
Store_Size	1349	1	Medium	1349	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Location_City_Type	1349	1	Tier 1	1349	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Type	1349	1	Departmental Store	1349	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Store_Sales_Total	1349.0	NaN	NaN	NaN	4946.966323	677.539953	3069.24	4355.39	4958.29	5366.59	8000.0

1. OUT003 is a Departmental store which is located in a Tier 1 city and has store as medium. It was established in 1999.
2. OUT003 has sold products whose MRP range from 86 to 266.
3. Snack Foods have been sold the highest number of times in OUT003.
4. The revenue generated from each product at OUT003 ranges from 3070 to 8000.

OUT003_df = (
    data.loc[data['Store_Id'] == 'OUT003']
    .groupby(['Product_Type'], as_index=False)['Product_Store_Sales_Total']
    .sum()
)
plt.figure(figsize=[10, 8])
plt.xticks(rotation=90)
plt.xlabel("Product_Type")
plt.ylabel("Product_Store_Sales_Total")
plt.title("Product_Store_Sales_Total")
sns.barplot(x=OUT003_df.Product_Type, y=OUT003_df.Product_Store_Sales_Total)
plt.show()

• OUT003 has generated the highest revenue from the sale of snack foods folowed by fruits and vegetables, both the categories contributing around 800,000 each.

# OUT004
data.loc[data['Store_Id'] == 'OUT004'].describe(include='all').T

	count	unique	top	freq	mean	std	min	25%	50%	75%	max
Product_Id	4676	4676	NC584	1	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Weight	4676.0	NaN	NaN	NaN	12.349613	1.428199	7.34	11.37	12.37	13.3025	17.79
Product_Sugar_Content	4676	4	Low Sugar	2632	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Allocated_Area	4676.0	NaN	NaN	NaN	0.069092	0.048584	0.004	0.031	0.056	0.097	0.297
Product_Type	4676	16	Fruits and Vegetables	700	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_MRP	4676.0	NaN	NaN	NaN	142.399709	17.513973	83.04	130.54	142.82	154.1925	197.66
Store_Id	4676	1	OUT004	4676	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Establishment_Year	4676.0	NaN	NaN	NaN	2009.0	0.0	2009.0	2009.0	2009.0	2009.0	2009.0
Store_Size	4676	1	Medium	4676	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Location_City_Type	4676	1	Tier 2	4676	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Store_Type	4676	1	Supermarket Type2	4676	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Product_Store_Sales_Total	4676.0	NaN	NaN	NaN	3299.312111	468.271692	1561.06	2942.085	3304.18	3646.9075	5462.86

• OUT004 is a store of Supermarket Type2 which is located in a Tier 2 city and has store size as medium. It was established in 2009.
• OUT004 has sold products whose MRP range from 83 to 198.
• Fruits and Vegetables have been sold the highest number of times in OUT004.
• The revenue generated from each product at OUT004 ranges from 1561 to 5463.

OUT004_df = (
    data.loc[data['Store_Id'] == 'OUT004']
    .groupby(['Product_Type'], as_index=False)['Product_Store_Sales_Total']
    .sum()
)
plt.figure(figsize=[10, 8])
plt.xticks(rotation=90)
plt.xlabel("Product_Type")
plt.ylabel("Product_Store_Sales_Total")
plt.title("Product_Store_Sales_Total")
sns.barplot(x=OUT004_df.Product_Type, y=OUT004_df.Product_Store_Sales_Total)
plt.show()

• OUT004 has generated the highest revenue from the sale of fruits and vegetables (~ 2500000) followed by snack foods (~2000000).

# revenue generated by the stores from each of the product types
df1 = data.groupby(['Store_Id', 'Product_Type'], as_index=False)[
    'Product_Store_Sales_Total'
    ].sum()
df1

	Store_Id	Product_Type	Product_Store_Sales_Total
0	OUT001	Baking Goods	525131.04
1	OUT001	Breads	121274.09
2	OUT001	Breakfast	38161.10
3	OUT001	Canned	449016.38
4	OUT001	Dairy	598767.62
5	OUT001	Frozen Foods	558556.81
6	OUT001	Fruits and Vegetables	792992.59
7	OUT001	Hard Drinks	152920.74
8	OUT001	Health and Hygiene	435005.31
9	OUT001	Household	531371.38
10	OUT001	Meat	505867.28
11	OUT001	Others	123977.09
12	OUT001	Seafood	52936.84
13	OUT001	Snack Foods	806142.24
14	OUT001	Soft Drinks	410548.69
15	OUT001	Starchy Foods	120443.98
16	OUT002	Baking Goods	169860.50
17	OUT002	Breads	43419.47
18	OUT002	Breakfast	23396.10
19	OUT002	Canned	151467.66
20	OUT002	Dairy	178888.18
21	OUT002	Frozen Foods	180295.95
22	OUT002	Fruits and Vegetables	298503.56
23	OUT002	Hard Drinks	54281.85
24	OUT002	Health and Hygiene	164660.81
25	OUT002	Household	184665.65
26	OUT002	Meat	151800.01
27	OUT002	Others	32835.73
28	OUT002	Seafood	17663.35
29	OUT002	Snack Foods	255317.57
30	OUT002	Soft Drinks	103808.35
31	OUT002	Starchy Foods	20044.98
32	OUT003	Baking Goods	491908.20
33	OUT003	Breads	175391.93
34	OUT003	Breakfast	95634.08
35	OUT003	Canned	452445.17
36	OUT003	Dairy	715814.94
37	OUT003	Frozen Foods	597608.42
38	OUT003	Fruits and Vegetables	897437.46
39	OUT003	Hard Drinks	110760.30
40	OUT003	Health and Hygiene	439139.18
41	OUT003	Household	523981.64
42	OUT003	Meat	520939.68
43	OUT003	Others	159963.75
44	OUT003	Seafood	65337.48
45	OUT003	Snack Foods	918510.44
46	OUT003	Soft Drinks	365046.30
47	OUT003	Starchy Foods	143538.60
48	OUT004	Baking Goods	1266086.26
49	OUT004	Breads	374856.75
50	OUT004	Breakfast	204939.13
51	OUT004	Canned	1247153.50
52	OUT004	Dairy	1318447.30
53	OUT004	Frozen Foods	1473519.65
54	OUT004	Fruits and Vegetables	2311899.66
55	OUT004	Hard Drinks	307851.73
56	OUT004	Health and Hygiene	1124901.91
57	OUT004	Household	1324721.50
58	OUT004	Meat	950604.97
59	OUT004	Others	224719.73
60	OUT004	Seafood	136466.37
61	OUT004	Snack Foods	2009026.70
62	OUT004	Soft Drinks	917641.38
63	OUT004	Starchy Foods	234746.89

• In all the product types, the revenue generated by OUT004 has been the highest which seems quite logical since around 53% of the total products were brought from this store.
• In all the product categories, the revenue generated by OUT002 has been the lowest which seems quite obvious since it is small store in a Tier 3 city.

# revenue generated by the stores from products having different levels of sugar content

df2 = data.groupby(['Store_Id', 'Product_Sugar_Content'], as_index=False)[
    'Product_Store_Sales_Total'
    ].sum()
df2

	Store_Id	Product_Sugar_Content	Product_Store_Sales_Total
0	OUT001	Low Sugar	3300834.93
1	OUT001	No Sugar	1090353.78
2	OUT001	Regular	1749444.51
3	OUT001	reg	82479.96
4	OUT002	Low Sugar	1156758.85
5	OUT002	No Sugar	382162.19
6	OUT002	Regular	472112.50
7	OUT002	reg	19876.18
8	OUT003	Low Sugar	3706903.24
9	OUT003	No Sugar	1123084.57
10	OUT003	Regular	1743566.35
11	OUT003	reg	99903.41
12	OUT004	Low Sugar	8658908.78
13	OUT004	No Sugar	2674343.14
14	OUT004	Regular	3902547.93
15	OUT004	reg	191783.58

• The trend is the same as that which was present in the revenue analysis of stores for product types.

Data Preprocessing

# there is 3 types of sugar content in the Product_Sugar_Content column - Low Sugar, Regualr and reg
data.Product_Sugar_Content.replace(to_replace='reg', value='Regular', inplace=True)

data.Product_Sugar_Content.value_counts()

	count
Product_Sugar_Content
Low Sugar	4885
Regular	2359
No Sugar	1519

dtype: int64

# Product_Id : 2 characters followed by a number
data["Product_Id_char"] = data["Product_Id"].str[:2]
data.head()

	Product_Id	Product_Weight	Product_Sugar_Content	Product_Allocated_Area	Product_Type	Product_MRP	Store_Id	Store_Establishment_Year	Store_Size	Store_Location_City_Type	Store_Type	Product_Store_Sales_Total	Product_Id_char
0	FD6114	12.66	Low Sugar	0.027	Frozen Foods	117.08	OUT004	2009	Medium	Tier 2	Supermarket Type2	2842.40	FD
1	FD7839	16.54	Low Sugar	0.144	Dairy	171.43	OUT003	1999	Medium	Tier 1	Departmental Store	4830.02	FD
2	FD5075	14.28	Regular	0.031	Canned	162.08	OUT001	1987	High	Tier 2	Supermarket Type1	4130.16	FD
3	FD8233	12.10	Low Sugar	0.112	Baking Goods	186.31	OUT001	1987	High	Tier 2	Supermarket Type1	4132.18	FD
4	NC1180	9.57	No Sugar	0.010	Health and Hygiene	123.67	OUT002	1998	Small	Tier 3	Food Mart	2279.36	NC

data["Product_Id_char"].unique()

array(['FD', 'NC', 'DR'], dtype=object)

data.loc[data.Product_Id_char == "FD", "Product_Type"].unique()

array(['Frozen Foods', 'Dairy', 'Canned', 'Baking Goods', 'Snack Foods',
       'Meat', 'Fruits and Vegetables', 'Breads', 'Breakfast',
       'Starchy Foods', 'Seafood'], dtype=object)

data.loc[data.Product_Id_char == "NC", "Product_Type"].unique()

array(['Health and Hygiene', 'Household', 'Others'], dtype=object)

data.loc[data.Product_Id_char == "DR", "Product_Type"].unique()

array(['Hard Drinks', 'Soft Drinks'], dtype=object)

# store's age
data["Store_Age_Years"] = 2025 - data.Store_Establishment_Year
data.head()

	Product_Id	Product_Weight	Product_Sugar_Content	Product_Allocated_Area	Product_Type	Product_MRP	Store_Id	Store_Establishment_Year	Store_Size	Store_Location_City_Type	Store_Type	Product_Store_Sales_Total	Product_Id_char	Store_Age_Years
0	FD6114	12.66	Low Sugar	0.027	Frozen Foods	117.08	OUT004	2009	Medium	Tier 2	Supermarket Type2	2842.40	FD	16
1	FD7839	16.54	Low Sugar	0.144	Dairy	171.43	OUT003	1999	Medium	Tier 1	Departmental Store	4830.02	FD	26
2	FD5075	14.28	Regular	0.031	Canned	162.08	OUT001	1987	High	Tier 2	Supermarket Type1	4130.16	FD	38
3	FD8233	12.10	Low Sugar	0.112	Baking Goods	186.31	OUT001	1987	High	Tier 2	Supermarket Type1	4132.18	FD	38
4	NC1180	9.57	No Sugar	0.010	Health and Hygiene	123.67	OUT002	1998	Small	Tier 3	Food Mart	2279.36	NC	27

# grouping product types into perishables and non-perishables
perishables = [
    "Dairy",
    "Meat",
    "Fruits and Vegetables",
    "Breakfast",
    "Breads",
    "Seafood",
]

def change(x):
  if x in "Perishable":
    return "Perishables"
  else:
    return "Non-Perishables"

data['Product_Type_Category'] = data['Product_Type'].apply(change)

data.head()

	Product_Id	Product_Weight	Product_Sugar_Content	Product_Allocated_Area	Product_Type	Product_MRP	Store_Id	Store_Establishment_Year	Store_Size	Store_Location_City_Type	Store_Type	Product_Store_Sales_Total	Product_Id_char	Store_Age_Years	Product_Type_Category
0	FD6114	12.66	Low Sugar	0.027	Frozen Foods	117.08	OUT004	2009	Medium	Tier 2	Supermarket Type2	2842.40	FD	16	Non-Perishables
1	FD7839	16.54	Low Sugar	0.144	Dairy	171.43	OUT003	1999	Medium	Tier 1	Departmental Store	4830.02	FD	26	Non-Perishables
2	FD5075	14.28	Regular	0.031	Canned	162.08	OUT001	1987	High	Tier 2	Supermarket Type1	4130.16	FD	38	Non-Perishables
3	FD8233	12.10	Low Sugar	0.112	Baking Goods	186.31	OUT001	1987	High	Tier 2	Supermarket Type1	4132.18	FD	38	Non-Perishables
4	NC1180	9.57	No Sugar	0.010	Health and Hygiene	123.67	OUT002	1998	Small	Tier 3	Food Mart	2279.36	NC	27	Non-Perishables

# Outlier Check
numeric_cols = data.select_dtypes(include=np.number).columns.tolist()
numeric_cols.remove("Store_Establishment_Year")
numeric_cols.remove("Store_Age_Years")

plt.figure(figsize=(16, 10))
for i, var in enumerate(numeric_cols):
    plt.subplot(3, 3, i + 1)
    plt.boxplot(data[var], whis=1.5)
    plt.tight_layout()
    plt.xlabel(var)
plt.show()

# data preprocessing pipeline for the categorical features

categorical_features = data.select_dtypes(include=["object", "category"]).columns.tolist()
categorical_features

['Product_Id',
 'Product_Sugar_Content',
 'Product_Type',
 'Store_Id',
 'Store_Size',
 'Store_Location_City_Type',
 'Store_Type',
 'Product_Id_char',
 'Product_Type_Category']

preprocessor = make_column_transformer(
    (Pipeline([('encoder', OneHotEncoder(handle_unknown='ignore'))]), categorical_features)
)

Model Building

def adj_r2_score(predictors, targets, predictions):
    r2 = r2_score(targets, predictions)
    n = predictors.shape[0]
    k = predictors.shape[1]
    return 1 - ((1 - r2) * (n - 1) / (n - k - 1))

def model_performance_regression(model, predictors, target):
    """
    Function to compute different metrics to check regression model performance

    model: regressor
    predictors: independent variables
    target: dependent variable
    """

    # predicting using the independent variables
    pred = model.predict(predictors)

    r2 = r2_score(target, pred)  # to compute R-squared
    adjr2 = adj_r2_score(predictors, target, pred)  # to compute adjusted R-squared
    rmse = np.sqrt(mean_squared_error(target, pred))  # to compute RMSE
    mae = mean_absolute_error(target, pred)  # to compute MAE
    mape = mean_absolute_percentage_error(target, pred)

    df_perf = pd.DataFrame(
        {
            "RMSE": rmse,
            "MAE": mae,
            "R-squared": r2,
            "Adj. R-squared": adjr2,
            'MAPE': mape,
        },
        index=[0],
    )

    return df_perf

data = data.drop(["Product_Id","Product_Type","Store_Id","Store_Establishment_Year"], axis=1)

data.shape

(8763, 11)

data.head()

	Product_Weight	Product_Sugar_Content	Product_Allocated_Area	Product_MRP	Store_Size	Store_Location_City_Type	Store_Type	Product_Store_Sales_Total	Product_Id_char	Store_Age_Years	Product_Type_Category
0	12.66	Low Sugar	0.027	117.08	Medium	Tier 2	Supermarket Type2	2842.40	FD	16	Non-Perishables
1	16.54	Low Sugar	0.144	171.43	Medium	Tier 1	Departmental Store	4830.02	FD	26	Non-Perishables
2	14.28	Regular	0.031	162.08	High	Tier 2	Supermarket Type1	4130.16	FD	38	Non-Perishables
3	12.10	Low Sugar	0.112	186.31	High	Tier 2	Supermarket Type1	4132.18	FD	38	Non-Perishables
4	9.57	No Sugar	0.010	123.67	Small	Tier 3	Food Mart	2279.36	NC	27	Non-Perishables

X = data.drop("Product_Store_Sales_Total", axis=1)
y = data["Product_Store_Sales_Total"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1, shuffle=True)

X_train.shape, X_test.shape

((7010, 10), (1753, 10))

categorical_features = X_train.select_dtypes(include=["object", "category"]).columns.tolist()
categorical_features

['Product_Sugar_Content',
 'Store_Size',
 'Store_Location_City_Type',
 'Store_Type',
 'Product_Id_char',
 'Product_Type_Category']

# run preprocessing pipeline for the categorical features
preprocessor = make_column_transformer(
    (
        OneHotEncoder(handle_unknown="ignore", sparse_output=False),
        categorical_features,
    ),
    remainder="passthrough",
)

Define functions for Model Evaluation

# function to compute adjusted R-squared
def adj_r2_score(predictors, targets, predictions):
    r2 = r2_score(targets, predictions)
    n = predictors.shape[0]
    k = predictors.shape[1]
    return 1 - ((1 - r2) * (n - 1) / (n - k - 1))


# function to compute different metrics to check performance of a regression model
def model_performance_regression(model, predictors, target):
    """
    Function to compute different metrics to check regression model performance

    model: regressor
    predictors: independent variables
    target: dependent variable
    """

    # predicting using the independent variables
    pred = model.predict(predictors)

    r2 = r2_score(target, pred)  # to compute R-squared
    adjr2 = adj_r2_score(predictors, target, pred)  # to compute adjusted R-squared
    rmse = np.sqrt(mean_squared_error(target, pred))  # to compute RMSE
    mae = mean_absolute_error(target, pred)  # to compute MAE
    mape = mean_absolute_percentage_error(target, pred)  # to compute MAPE

    # creating a dataframe of metrics
    df_perf = pd.DataFrame(
        {
            "RMSE": rmse,
            "MAE": mae,
            "R-squared": r2,
            "Adj. R-squared": adjr2,
            "MAPE": mape,
        },
        index=[0],
    )

    return df_perf

The ML models to be built can be any two out of the following:

1. Decision Tree
2. Bagging
3. Random Forest
4. AdaBoost
5. Gradient Boosting
6. XGBoost

Model Performance Improvement - Hyperparameter Tuning

AdaBoost Regressor

# classifier
ab_tuning = AdaBoostRegressor(random_state=1)
ab_tuning = make_pipeline(preprocessor, ab_tuning)

# grid of paramters to choose
parameters = {
    "adaboostregressor__n_estimators": [50, 100],
    "adaboostregressor__learning_rate": [0.1, 1.0],
}

# run the grid search
grid_obj = GridSearchCV(ab_tuning, parameters, cv=5, n_jobs=-1)
grid_obj = grid_obj.fit(X_train, y_train)

# set clf
ab_tuning = grid_obj.best_estimator_

# fit the data
ab_tuning.fit(X_train, y_train)

Pipeline(steps=[('columntransformer',
                 ColumnTransformer(remainder='passthrough',
                                   transformers=[('onehotencoder',
                                                  OneHotEncoder(handle_unknown='ignore',
                                                                sparse_output=False),
                                                  ['Product_Sugar_Content',
                                                   'Store_Size',
                                                   'Store_Location_City_Type',
                                                   'Store_Type',
                                                   'Product_Id_char',
                                                   'Product_Type_Category'])])),
                ('adaboostregressor',
                 AdaBoostRegressor(learning_rate=0.1, random_state=1))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# model performance training set
ab_tuning_perf = model_performance_regression(ab_tuning, X_test, y_test)
ab_tuning_perf

	RMSE	MAE	R-squared	Adj. R-squared	MAPE
0	471.692134	360.34973	0.806985	0.805877	0.121427

# model performance test set
ab_tuning_perf = model_performance_regression(ab_tuning, X_test, y_test)
ab_tuning_perf

	RMSE	MAE	R-squared	Adj. R-squared	MAPE
0	471.692134	360.34973	0.806985	0.805877	0.121427

Gradient Boosting Regressor

# select classifier
gb_tuning = GradientBoostingRegressor(random_state=1)
gb_tuning = make_pipeline(preprocessor, gb_tuning)

# grid of paramters to choose
parameters = {
    "gradientboostingregressor__n_estimators": [50, 100],
    "gradientboostingregressor__subsample": [0.1],
    "gradientboostingregressor__max_features": [3, 5],
    "gradientboostingregressor__max_depth": [2, 4],
}

# run the grid search
grid_obj = GridSearchCV(gb_tuning, parameters, cv=5, n_jobs=-1)
grid_obj = grid_obj.fit(X_train, y_train)

# set clf
gb_tuning = grid_obj.best_estimator_

# fit into data
gb_tuning.fit(X_train, y_train)

Pipeline(steps=[('columntransformer',
                 ColumnTransformer(remainder='passthrough',
                                   transformers=[('onehotencoder',
                                                  OneHotEncoder(handle_unknown='ignore',
                                                                sparse_output=False),
                                                  ['Product_Sugar_Content',
                                                   'Store_Size',
                                                   'Store_Location_City_Type',
                                                   'Store_Type',
                                                   'Product_Id_char',
                                                   'Product_Type_Category'])])),
                ('gradientboostingregressor',
                 GradientBoostingRegressor(max_depth=4, max_features=5,
                                           random_state=1, subsample=0.1))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# model performance for training set
gb_tuning_perf = model_performance_regression(gb_tuning, X_train, y_train)
gb_tuning_perf

	RMSE	MAE	R-squared	Adj. R-squared	MAPE
0	306.261446	160.801822	0.917047	0.916928	0.060211

# model performance for test set
gb_tuning_perf = model_performance_regression(gb_tuning, X_test, y_test)
gb_tuning_perf

	RMSE	MAE	R-squared	Adj. R-squared	MAPE
0	329.170871	174.151943	0.906002	0.905463	0.061328

XGBoost Regressor

# classifer
xgb_tuning = XGBRegressor(random_state=1)
xgb_tuning = make_pipeline(preprocessor, xgb_tuning)

# grid of parameters to choose
parameters = {
    "xgbregressor__n_estimators": [50, 100],
    "xgbregressor__subsample": [0.1],
    "xgbregressor__gamma": [0.1],
    "xgbregressor__colsample_bytree": [0.5],
    "xgbregressor__colsample_bylevel": [0.5],
}

# grid search run
grid_obj = GridSearchCV(xgb_tuning, parameters, cv=5, n_jobs=-1)
grid_obj = grid_obj.fit(X_train, y_train)

# set clf
xgb_tuning = grid_obj.best_estimator_

# fit into data
xgb_tuning.fit(X_train, y_train)

Pipeline(steps=[('columntransformer',
                 ColumnTransformer(remainder='passthrough',
                                   transformers=[('onehotencoder',
                                                  OneHotEncoder(handle_unknown='ignore',
                                                                sparse_output=False),
                                                  ['Product_Sugar_Content',
                                                   'Store_Size',
                                                   'Store_Location_City_Type',
                                                   'Store_Type',
                                                   'Product_Id_char',
                                                   'Product_Type_Category'])])),
                ('xgbregressor',
                 XGBRegressor(base_score=None, booster=Non...
                              feature_types=None, gamma=0.1, grow_policy=None,
                              importance_type=None,
                              interaction_constraints=None, learning_rate=None,
                              max_bin=None, max_cat_threshold=None,
                              max_cat_to_onehot=None, max_delta_step=None,
                              max_depth=None, max_leaves=None,
                              min_child_weight=None, missing=nan,
                              monotone_constraints=None, multi_strategy=None,
                              n_estimators=50, n_jobs=None,
                              num_parallel_tree=None, random_state=1, ...))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# training set model
xgb_tuning_perf = model_performance_regression(xgb_tuning, X_train, y_train)
xgb_tuning_perf

	RMSE	MAE	R-squared	Adj. R-squared	MAPE
0	314.032694	175.464401	0.912784	0.912659	0.066634

# test set model
xgb_tuning_perf = model_performance_regression(xgb_tuning, X_test, y_test)
xgb_tuning_perf

	RMSE	MAE	R-squared	Adj. R-squared	MAPE
0	356.027652	192.660011	0.890038	0.889407	0.066096

# combine models
# comparing all train models

all_models_train_df = pd.concat(
    [
        ab_tuning_perf.T,
        gb_tuning_perf.T,
        xgb_tuning_perf.T,
    ],
    axis=1,
)

all_models_train_df.columns = ["AdaBoost", "Gradient Boosting", "XGBoost"]
all_models_train_df

	AdaBoost	Gradient Boosting	XGBoost
RMSE	471.692134	329.170871	356.027652
MAE	360.349730	174.151943	192.660011
R-squared	0.806985	0.906002	0.890038
Adj. R-squared	0.805877	0.905463	0.889407
MAPE	0.121427	0.061328	0.066096

# select location of the folder before deployment
os.makedirs("model_files", exist_ok=True)

# Define the file saved location
saved_model_location = "model_files/xgb_model.joblib"

# save the best model pipeline using joblib
joblib.dump(xgb_tuning, saved_model_location)
print(f"Model saved at {saved_model_location}")

Model saved at model_files/xgb_model.joblib

# load the saved model pipeline from the file
saved_model_location = "model_files/xgb_model.joblib"
saved_model = joblib.load(saved_model_location)
print("Model loaded successfully")

Model loaded successfully

saved_model

Pipeline(steps=[('columntransformer',
                 ColumnTransformer(remainder='passthrough',
                                   transformers=[('onehotencoder',
                                                  OneHotEncoder(handle_unknown='ignore',
                                                                sparse_output=False),
                                                  ['Product_Sugar_Content',
                                                   'Store_Size',
                                                   'Store_Location_City_Type',
                                                   'Store_Type',
                                                   'Product_Id_char',
                                                   'Product_Type_Category'])])),
                ('xgbregressor',
                 XGBRegressor(base_score=None, booster=Non...
                              feature_types=None, gamma=0.1, grow_policy=None,
                              importance_type=None,
                              interaction_constraints=None, learning_rate=None,
                              max_bin=None, max_cat_threshold=None,
                              max_cat_to_onehot=None, max_delta_step=None,
                              max_depth=None, max_leaves=None,
                              min_child_weight=None, missing=nan,
                              monotone_constraints=None, multi_strategy=None,
                              n_estimators=50, n_jobs=None,
                              num_parallel_tree=None, random_state=1, ...))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

saved_model.predict(X_test)

array([3664.202 , 5927.408 , 4476.766 , ..., 4876.1636, 3134.6177,
       3540.1023], dtype=float32)

Deployment - Backend

Flask Web Framework

%%writefile model_files/app.py

import numpy as np
import joblib
import pandas as pd
from flask import Flask, request, jsonify

superkart_api = Flask(__name__)

model = joblib.load("model_files/xgb_model.joblib")

@superkart_api.get('/')
def home():
    return "Welcome to SuperKart"

@superkart_api.post('/v1/predict')
def sales_prediction():
    data = request.get_json()
    sample = {
        "Store_Id": data["Store_Id"],
        "Product_Id": data["Product_Id"],
        "Product_Sugar_Content": data["Product_Sugar_Content"],
        "Product_Type": data["Product_Type"],
        "Product_Weight": data["Product_Weight"],
        "Product_MRP": data["Product_MRP"],
        "Store_Location_Type": data["Store_Location_Type"],
        "Store_Type": data["Store_Type"],
        "Store_Size": data["Store_Size"],
        "Store_Age_Years": data["Store_Age_Years"],
        "Product_Type_Category": data["Product_Type_Category"],
    }

    import_data = pd.DataFrame([sample])
    prediction = model.predict(input_data).tolist()[0]
    return jsonify({'Sales': prediction})

if __name__ == '__main__':
    superkart_api.run(debug=True)

Writing model_files/app.py

Dependencies File

%%writefile model_files/required.txt
pandas==2.2.2
numpy==2.0.2
scikit-learn==1.6.1
seaborn==0.13.2
joblib==1.4.2
xgboost==2.1.4
joblib==1.4.2
Werkzeug==2.2.2
flask==2.2.2
gunicorn==20.1.0
requests==2.32.3
uvicorn[standard]
streamlit==1.43.2

Writing model_files/required.txt

Dockerfile

%%writefile model_files/Dockerfile
FROM python:3.9-slim

WORKDIR /app

COPY . .

RUN pip3 install -r requirements.txt

CMD ["gunicorn", "-w", "4", "-b", "0.0.0.0:7860", "app:superkart_api"]

Writing model_files/Dockerfile

Setting up a Hugging Face Docker Space for the Backend

from huggingface_hub import login
from google.colab import userdata

# Assuming your Hugging Face token is stored as a secret named 'HF_TOKEN'
HF_TOKEN = userdata.get('HF_TOKEN')
login(token=HF_TOKEN)

from huggingface_hub import create_repo

try:
  create_repo("superkart-api",
              repo_type="space",
              space_sdk="docker",
              private=False
  )
except Exception as e:
  if "Repository already exists" in str(e):
    print("Repo already exists")
  else:
    print(e)
    print("Error creating repo")

Uploading Files to Hugging Face Space (Docker Space)

access_key = userdata.get('HF_TOKEN')
repo_id = "airesoli/superkart-api"

login(token=access_key)
api = HfApi()

api.upload_folder(
    folder_path="model_files",
    repo_id=repo_id,
    path_in_repo="model_files",
    repo_type="space",
)

Processing Files (0 / 0)                : |          |  0.00B /  0.00B            

New Data Upload                         : |          |  0.00B /  0.00B            

  /content/model_files/xgb_model.joblib : 100%|##########|  143kB /  143kB            

CommitInfo(commit_url='https://huggingface.co/spaces/airesoli/superkart-api/commit/f164c27ed230fa288b448d5ab899cff79628444a', commit_message='Upload folder using huggingface_hub', commit_description='', oid='f164c27ed230fa288b448d5ab899cff79628444a', pr_url=None, repo_url=RepoUrl('https://huggingface.co/spaces/airesoli/superkart-api', endpoint='https://huggingface.co', repo_type='space', repo_id='airesoli/superkart-api'), pr_revision=None, pr_num=None)

Deployment - Frontend

Points to note before executing the below cells

• Create a Streamlit space on Hugging Face by following the instructions provided on the content page titled Creating Spaces and Adding Secrets in Hugging Face from Week 1

Streamlit for Interactive UI

# make a front end file at the same location with backend file
os.makedirs("frontend_files", exist_ok=True)

%%writefile frontend_files/app.py
import streamlit as st
import requests

st.title("SuperKart Sales Prediction")

Product_Weight = st.number_input("Product Weight")
Product_Sugar_Content = st.number_input("Product Sugar Content")
Product_MRP = st.number_input("Product MRP")
Store_Age_Years = st.number_input("Store Age Years")
Product_Allocated_Area = st.selectbox("Product Allocated Area", ["Small", "Medium", "Large"])
Store_Location_Type = st.selectbox("Store Location Type", ["Tier 1", "Tier 2", "Tier 3"])
Store_Type = st.selectbox("Store Type", ["Supermarket", "Grocery Store"])
Store_Size = st.selectbox("Store Size", ["Small", "Medium", "Large"])
Product_Type = st.selectbox("Product Type", ["Dairy", "Bakery", "Fruits and Vegetables"])
Store_Type_Category = st.selectbox("Store Type Category", ["Supermarket", "Grocery Store"])
Product_ID = st.text_input("Product ID")
Store_ID = st.text_input("Store ID")
Store_Age_Years = st.number_input("Store Age Years")
Product_Type_Category = st.selectbox("Product Type Category", ["Dairy", "Bakery", "Fruits and Vegetables"])

product_area = {
    "Product": Product_Allocated_Area,
    "Store": Store_Size,
    "Product_Type": Product_Type,
    "Store_Type": Store_Type,
    "Store_Location_Type": Store_Location_Type,
    "Product_Sugar_Content": Product,
    "Product_Weight": Product_Weight,
    "Product_MRP": Product_MRP,
    "Store_Age_Years": Store_Age_Years,
    "Product_Type_Category": Product_Type_Category,
    "Store_Type_Category": Store_Type_Category,
}

if st.button("Predict"):
    response = requests.post("https://airesoli-superkart-api.hf.space/v1/predict", json=product_area)
    if response.status_code == 200:
        result = response.json()
        st.success(f"Predicted Sales: {result['Sales']}")
    else:
        st.error("Error in prediction")

Writing frontend_files/app.py

Dependencies File

%%writefile frontend_files/requirements.txt
streamlit==1.45.0
requests==2.32.3

Writing frontend_files/requirements.txt

DockerFile

%%writefile frontend_files/Dockerfile
# Use a minimal base image with Python 3.9 installed
FROM python:3.9-slim

# Set the working directory inside the container to /app
WORKDIR /app

# Copy all files from the current directory on the host to the container's /app directory
COPY . .

# Install Python dependencies listed in requirements.txt
RUN pip3 install -r requirements.txt

# Define the command to run the Streamlit app on port 8501 and make it accessible externally
CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0", "--server.enableXsrfProtection=false"]

# NOTE: Disable XSRF protection for easier external access in order to make batch predictions

Writing frontend_files/Dockerfile

Uploading Files to Hugging Face Space (Streamlit Space)

access_key = userdata.get('HF_TOKEN')
repo_id = "airesoli/superkart-api"

login(token=access_key)

api = HfApi()

api.upload_folder(
    folder_path="frontend_files",
    repo_id=repo_id,
    path_in_repo="frontend_files",
    repo_type="space",
)

CommitInfo(commit_url='https://huggingface.co/spaces/airesoli/superkart-api/commit/b58637ff8f422d2fe468883c72a962c7fad7d2a3', commit_message='Upload folder using huggingface_hub', commit_description='', oid='b58637ff8f422d2fe468883c72a962c7fad7d2a3', pr_url=None, repo_url=RepoUrl('https://huggingface.co/spaces/airesoli/superkart-api', endpoint='https://huggingface.co', repo_type='space', repo_id='airesoli/superkart-api'), pr_revision=None, pr_num=None)

Actionable Insights and Business Recommendations

1. Goodness of Fit (R^2 and Adjusted R^2)R^2 (R-Squared): 0.8069

   • The model explains 80.7% of the variance in the data.
   • Since this value is close to 1.0, it indicates a strong fit and suggests the model accurately captures the underlying patterns.
   • Adjusted R^2: No Significant Change
   • The stability between R^2 and Adjusted $R^2$ indicates that the variables included are meaningful and the model is not being penalized for unnecessary complexity.

2. Error MetricsMAPE (Mean Absolute Percentage Error): 12.1%On average, the model's predictions deviate from the actual values by 12.1%.

   • This represents the average "prediction error" (which may stem from data noise or human recording errors).
   • MAE (Mean Absolute Error): 360.35
   • The average absolute difference between the predicted and actual values is approximately 360.35 units.
   • RMSE (Root Mean Square Error): 471.69
   • The RMSE is higher than the MAE ($471.69 > 360.35$), which confirms the presence of outliers.
   • RMSE penalizes larger errors more heavily, indicating that a few data points have significantly higher variances than the rest.

• Executive SummaryOverall, the model is highly accurate and precise, successfully explaining over 80% of the data's behavior.
• However, the gap between the MAE and RMSE suggests that certain outliers are skewing the results.

3. Recommendations: Outlier Investigation:
   • Identify the specific data points causing the higher RMSE to determine if they are valid anomalies or errors.
   • Feature Analysis: Review the input variables to identify which factors contribute most to the variance and see if additional data could further reduce the 12.1% error margin.

# 1. Save your notebook first (Ctrl + S)
from google.colab import files
import nbformat

file_path = '/content/drive/MyDrive/Colab Notebooks/'
save_path = '/content/drive/MyDrive/'

# 2. Get the name of your notebook
# (If you renamed it, change 'Untitled.ipynb' to your filename)
notebook_path = file_path + 'fullcode_superkart_erika.ipynb'

# 3. Clean the metadata that causes the 'state' error
with open(notebook_path, 'r', encoding='utf-8') as f:
    nb = nbformat.read(f, as_version=4)

# Remove problematic widget state metadata
if 'widgets' in nb.metadata:
    del nb.metadata['widgets']

# Save the "fixed" version temporarily
fixed_name = save_path + 'fullcode_superkart_erika.ipynb'
with open(fixed_name, 'w', encoding='utf-8') as f: # Changed save_path to fixed_name here
    nbformat.write(nb, f)

# 4. Convert the fixed file to HTML
!jupyter nbconvert --to html "{fixed_name}"

# 5. Download the final HTML file
files.download(fixed_name.replace('.ipynb', '.html'))

[NbConvertApp] Converting notebook /content/drive/MyDrive/fullcode_superkart_erika.ipynb to html
[NbConvertApp] WARNING | Alternative text is missing on 31 image(s).
[NbConvertApp] Writing 2684959 bytes to /content/drive/MyDrive/fullcode_superkart_erika.html