Machine learning fundamentals practice questions.

A dataset is divided into three parts before a model is built. What is the third part, which is not looked at until the very end, used for?

• To correct the labels that were recorded wrongly during training
• To estimate performance on data the model has never influenced
• To supply extra examples once the model stops improving
• To decide which features should be given to the algorithm

Answer: To estimate performance on data the model has never influenced

The first part trains the model and the second is used to choose between settings, so both have influenced the finished model and neither can give an honest estimate of its performance. Only a part that was never used for any decision provides that estimate, which is why looking at it repeatedly and adjusting the model destroys its purpose.

A model is built to predict which subscribers will cancel. One of its features is the date on which the subscriber opened the cancellation form. The model scores 99 % on data held back from training but is useless in service. Identify the reason.

• The number of features is too small for the model to separate the two classes reliably from one another.
• The feature is only known once the outcome has happened, so it cannot be available when a prediction is needed.
• The held-back data were drawn from the same population as the training data, which always inflates the score obtained.
• The model was built from too few subscribers, so the two classes are not equally represented within the training data.

Answer: The feature is only known once the outcome has happened, so it cannot be available when a prediction is needed.

A feature that only exists after the event being predicted leaks the answer into the training data. The high score is produced by reading the outcome, not by anticipating it, and in service the field is still empty.

Two systems sort incoming photographs into categories. Identify the single observation that shows that a system is rule-based rather than machine-learned.

• It processes each photograph in a few milliseconds rather than in several seconds of computing time.
• It sometimes places a photograph in the wrong category and offers no reason at all for its choice.
• It was written by a team of developers who tested it thoroughly before it was released to any users.
• Its behaviour changes only when a developer edits the program, never when further labelled photographs are supplied.

Answer: Its behaviour changes only when a developer edits the program, never when further labelled photographs are supplied.

A rule-based program holds the decision explicitly in code, so only an edit changes it. A learned system changes its behaviour when it is given more labelled examples, without anyone rewriting a rule.

A team doubles the number of measurements recorded about each customer before a model is built. State what this decision costs, given that the number of customers stays the same.

• The model gains more ways to fit chance patterns in the same examples, so more examples are needed to tell a real pattern from a coincidence.
• The model can no longer be applied to customers at all, because every prediction has to rest on exactly one measurement of the customer concerned.
• The label of every customer has to be worked out again, because in a supervised task the label depends on how many measurements were recorded.
• The data collected earlier become unusable, because a model may only be built from measurements that were planned before any data were collected at all.

Answer: The model gains more ways to fit chance patterns in the same examples, so more examples are needed to tell a real pattern from a coincidence.

More features give the model more freedom, and freedom is only safe when there are enough examples to constrain it. With the sample size unchanged, the risk of fitting noise instead of the pattern rises.

A team collects ten times as many training examples from the same source as before, and the model's error on a new population is unchanged. Identify the most likely reason.

• Ten times as many examples is still far too few for a model of any kind to learn a pattern from.
• The extra examples repeat the same bias in what was sampled, so the new population is still not represented.
• A model can only make use of a fixed number of training examples, so all of the additional ones were discarded.
• The error on a new population does not depend on the training data at all, only on how many features were used.

Answer: The extra examples repeat the same bias in what was sampled, so the new population is still not represented.

Enlarging a sample reduces random variation, not a systematic gap between who was sampled and who the model is used on. That gap closes only when the missing population is sampled.

___ occurs when a model is too simple to capture the pattern in the data, so it performs poorly on the training data as well as on new data. (one word)

Answer: underfitting

The tell-tale sign is that both scores are weak at the same time, which separates it from overfitting, where only the unseen score collapses. The cure is a more expressive model or better features, not more training rounds.