Bounty: 50
I am hearing these words more and more as I study machine learning. In fact, some people have won Fields medal working on regularities of equations. So, I guess this is a term that carries itself from statistical physics/maths to machine learning. Naturally, a number of people I asked just couldn’t intuitively explain it.
I know that methods such as dropout help in regularization (=> they say it reduces overfitting, but I really don’t get what it is: if it only reduces overfitting, why not just call it antioverfitting methods => there must be something more I think, hence this question).
I would be really grateful (I guess the naive ML community would be too!) if you could explain:
 How do you define regularity? What is regularity?
 Is regularization a way to ensure regularity? i.e. capturing regularities?

Why do ensembling methods like dropout, normalization methods all claim to be doing regularization?

Why do these (regularity/regularization) come up in machine learning?
Thanks a lot for your help.