Discovering neural nets with low Kolmogorov complexity and high generalization capability

Some basic concepts of algorithmic complexity theory relevant to machine learning are reviewed along with the Solomon-Levin distribution (or universal prior) which deals with the prior problem. The universal prior leads to a probabilistic method for finding algorithmically simple problem solutions with high generalization capability. The method is based on Levin complexity and inspired by Levin's optimal universal search algorithm. For a given problem, solution candidates are computed by efficient self sizing programs that influence their own runtime and storage size. The method, at least with certain toy problems where it is computationally feasible, can lead to unmatchable generalization results.