Train & Test: A Real Model-Building Experiment
Fit a genuine numerical model, hold back test observations and measure how well the predictions generalise.
1. Learn what it means to test a model
The 24 fictional observations come from Chapter 18 of AI & Data Science Foundations. We train on days 1–16 and test on days 17–24. The test days must remain unseen while fitting the line. Temperature is our only feature, so rainy-day patterns and other factors are omitted.
Model: least-squares straight line fitted to training days. Baseline: always predict the rounded mean number of hires in training. Both are evaluated on exactly the same eight test days using mean absolute error (MAE).
Interactive training and evaluation
| Unseen day | Temp (°C) | Rain | Actual hires | Prediction | Absolute error |
|---|
2. Practise interpreting MAE
Worked example
If actual hires are 50 and predicted hires are 44, the absolute error is |50 − 44| = 6. MAE is the average of such errors over the same test rows.
Common mistake
Reporting only a training score does not show how well a model works on unfamiliar cases. Likewise, testing on rows already used to fit the model introduces data leakage.
3. Build and evaluate
Explain whether your model beats the baseline, identify an unusually large error and suggest an additional feature you could collect legally. Why can a low MAE still hide poor performance for some groups or conditions?
Download the book's 24-day datasetDownload model comparison PythonChapter 18 companion filesNext: classification →