N Fold Test

• How do you know if your classifier is any good?
• Obviously, it categorises all the data correctly.
• So, an obvious algorithm is to cache all the training data away.
• If you run again, classify it the same way. That always works.
• However, you usually want the system to generalise to unseen data.
• Will caching generalise?
• Will the wealth rule?
• Will a line?
• Typically, we test machine learning rules by partitioning the data into proper subsets.
• For example, we make equal sized training and test sets.
• The first fold trains on the training set, and then tests on the test set.
• Then, to protect against bad splits, we do it again reversing the process.
• We throw away the line (or whatever the system is) train on the test set, and test on the training set.
• That's a 2-Fold test. You can have any fold, e.g 5-Fold.
• Break the data into 5 sets, and run the learning algorithm 5 times.
• Train on 4 and test on 1 (or train on 1 and test on 4).
• Is this generalisable? If the original data is representative, then it probably is.