Description
The model had been training across several sessions for many days on an image recognition competition. It was a relatively simple, and scored about a 0.9 AUC initially — the metric for the…
Summary
- How to avoid (and take advantage of) this blunder The model had been training across several sessions for many days on an image recognition competition.
- But I had not made some sort of code error in which the model was trained on the test data correctly.
- However, when the competition ends, the model is evaluated on the other 75% of the test set to determine the position on the final private leaderboard.
- If we’re able, however, to use the 25% of test set to improve the score on the private leaderboard, this counts as data leakage.