Lesson #108

[guglielmosaerri]

Back in lesson #24 ("Splitting data") you discussed how one would split data for your Model into three sets (the "3 sets")

• Training: 70-80%

• Validation (to tune): 10-15%

• Test: 10-15%

but the code demonstrated here only split the data into two sets; TRAIN and TEST

And from what i can gather from the lesson, tweaking of the model was performed on the TEST data set - rather than a VALIDATION data set?

[spacelord16]

Answer: Why only two splits in the demo (and how to get three)

1. Concept vs. demo convenience

   • In lesson 24 we introduced the classic 3-way split:
     • Train: 70–80% – fit your model
     • Validation: 10–15% – tune hyper-parameters
     • Test: 10–15% – final, held-out evaluation
   • The notebook only does:

     from sklearn.model_selection import train_test_split
     
     X_train, X_test, y_train, y_test = train_test_split(
         X, y, test_size=0.2, random_state=42
     )

     so the “test” set doubles as both validation and test for brevity.

2. Why you shouldn’t tune on your final test set

   • Tuning hyper-parameters by inspecting test performance leaks information
   • Your test set becomes a de facto validation, leaving no truly unseen data

3. How to do a proper 3-way split

   # Option A: manual double split
   
   # 1) Carve off final test set (15%)
   X_train_full, X_test, y_train_full, y_test = train_test_split(
       X, y, test_size=0.15, random_state=42
   )
   
   # 2) Split the remainder into train (≈70%) + val (≈15%)
   val_size = 0.15 / (1 - 0.15)  # ≈0.1765 of the full data
   X_train, X_val, y_train, y_val = train_test_split(
       X_train_full, y_train_full,
       test_size=val_size,
       random_state=42
   )
   
   print(len(X_train), len(X_val), len(X_test))
   # → roughly 70% / 15% / 15%
   
   # Option B: use cross-validation for tuning
   from sklearn.model_selection import GridSearchCV
   
   param_grid = {'n_estimators': [50, 100, 200], ...}
   grid = GridSearchCV(
       RandomForestClassifier(),
       param_grid,
       cv=5,                # 5-fold CV on the train set
       scoring='accuracy'
   )
   grid.fit(X_train, y_train)  # tunes on train only
   best_model = grid.best_estimator_
   
   # FINAL evaluation only on X_test:
   print(best_model.score(X_test, y_test))
   

4. Putting it all together

• Train-only: pick your algorithm & features
• Validation (separate split or CV): tune hyper-parameters without touching X_test
• Test: once, at the very end, on truly unseen data