Are there any theoretically-guided practices left in machine learning nowadays? [D]
热度趋势
趋势数据积累中
百分比基于当前可用热度信号，而非评论数或独立用户人数。
reddit.com上的一位用户质疑机器学习的现状，询问是否还有任何理论指导的实践。他们回忆起一个理论指导应用的时期，但指出许多此类理论后来被推翻，实践者通过打破这些既定实践获得了良好的结果。该用户观察到，这些理论的倡导者，特别是各种“统计机器学习”教科书的作者，在没有撤回的情况下转移了焦点，让学生感到困惑。…
There was a period in the development of machine learning where application seemed to be informed by theory. Some of the best known theories include:
- If you train a model with too much data, then you get overfitting and your test performance will be suffer.
- Big models do not generalize because theoretically you will never have enough data.
- Never train on the test set, because it will result in high-bias.
- Never even look at the test set, because you as a modeller will instantly be biased and use the wrong model.
- Good results can only come from "compatible" models and optimization routines. You can't just throw ADAM onto some brand new model and expect it to work well.
- Optimization provides solid theory for machine learning, so use the optimizer with the best performance guarantee in the optimization literature.
- If you want to have good performance, ensure to use several models instead of one model, because stacked or ensemble models are always superior.
Most of these theories started out as mathematical statements (albeit on some contrived examples that have nothing to do with reality). At some point, these theories became folklores and were widely reproduced in textbooks and taught in classrooms, even making their ways into standard interview questions at data science related companies. Every student had to remember that bias-variance "bull's eye" diagram as if it was relevant in practice.
But then some of these theories started to get overturned. It turns out you can just break a lot of these theoretically-guided practices and still get good results. The ones who pushed these theories (especially the authors of various "statistical ML" textbooks), quietly stopped their postulation and instead hopped on the hype train. This left their students confused because there was never ever any retraction or resolution.
So my question is: are there still any theoretically-guided practice that remain in ML today? For example, do people use an optimizer because it is theoretically the best for the class of problems? Do people use a certain model or components associated with the model because it works well in theory? Or is it now a fully empirical field where practice is guided by whatever seems to work for other people?