Question Regarding End-to-End Walkthrough (Potential Error)

[philipawalkerdatascience]

Hello,

I was following along with the End-to-End Walkthrough found here: https://nixtlaverse.nixtla.io/statsforecast/docs/getting-started/getting_started_complete.html

However, I noticed that in the final 'prod_forecasts_df', there are multiple values for best_model for a given combination of 'unique_id' and 'ds'. For example, the combination H1 / ds 749 contains two unique best_model values. I believe this is due a value existing for each cv cutoff used during the tutorial (windows=2).

Ideally, when creating the final production forecast, shouldn't there be a single value per unique product/ds combo? In other words, shouldn't the values obtained per each cv cutoff be averaged and then the best model drawn from the mean of all cv slices?

[Amar Akram (amarakramali)]

Yes — your diagnosis is correct. evaluate keeps the cross-validation cutoff, so the current evaluation_df has one row per unique_id/cutoff pair. Merging that table into forecasts_df on unique_id duplicates each production forecast row and can attach different winning models to the same series.

The forecasts themselves should not be averaged. Instead, average each model's validation metric across the cutoffs, choose one model from those aggregated scores, and then take that model's production forecast:

def evaluate_cv(df, metric):
    models = df.columns.drop(['unique_id', 'ds', 'y', 'cutoff']).tolist()
    evals = evaluate(df, metrics=[metric], models=models)
    evals = evals.groupby('unique_id', as_index=False)[models].mean()
    evals['best_model'] = evals[models].idxmin(axis=1)
    return evals

The equivalent aggregation in the Polars walkthrough is:

evals = evals.group_by('unique_id').agg(pl.col(models).mean())

That leaves exactly one selected model per unique_id, so the later merge produces one row per unique_id/ds combination.

I submitted PR #1196 with the fix in both the pandas and Polars walkthroughs and verified the aggregation with a two-cutoff example.