A “Test Set” is a portion of a dataset that is used to evaluate the performance of a machine learning model after it has been trained. The test set is separate from both the training set and the validation set.
The purpose of the test set is to provide an unbiased estimate of how well the model will perform on new, unseen data. Here’s why this is important:
During training, the model learns to make predictions based on the training data. We use the validation set to tune the model’s hyperparameters and potentially make other decisions about the model or the training process to improve performance. Throughout this process, we’re continually checking and adjusting based on the model’s performance on the validation set. As a result, our final model might be somewhat tailored to the validation data, even if it hasn’t directly learned from it.
To make sure our model hasn’t just learned to perform well on the specific data in the training and validation sets, we need a completely separate set of data to evaluate the final model. This is the role of the test set.
Because the model hasn’t been adjusted based on the test data, the test set can give us an unbiased estimate of the model’s performance on new data. This helps us understand how well the model is likely to perform in the real world, on data it hasn’t seen before.
One important point is that the test set should only be used once, after all training and tuning has been completed. If we use the test set multiple times, and make adjustments based on its performance, it becomes more like a validation set and less able to give an unbiased estimate of performance on new data.
« Back to Glossary Index