Are there any theoretically-guided practices left in machine learning nowadays? [D]

There was a period in the development of machine learning where application seemed to be informed by theory. Some of the best known theories include:

- If you train a model with too much data, then you get overfitting and your test performance will be suffer.

- Big models do not generalize because theoretically you will never have enough data.

- Never train on the test set, because it will result in high-bias.

- Never even look at the test set, because you as a modeller will instantly be biased and use the wrong model.

- Good results can only come from "compatible" models and optimization routines. You can't just throw ADAM onto some brand new model and expect it to work well.

- Optimization provides solid theory for machine learning, so use the optimizer with the best performance guarantee in the optimization literature.

- If you want to have good performance, ensure to use several models instead of one model, because stacked or ensemble models are always superior.

Most of these theories started out as mathematical statements (albeit on some contrived examples that have nothing to do with reality). At some point, these theories became folklores and were widely reproduced in textbooks and taught in classrooms, even making their ways into standard interview questions at data science related companies. Every student had to remember that bias-variance "bull's eye" diagram as if it was relevant in practice.

But then some of these theories started to get overturned. It turns out you can just break a lot of these theoretically-guided practices and still get good results. The ones who pushed these theories (especially the authors of various "statistical ML" textbooks), quietly stopped their postulation and instead hopped on the hype train. This left their students confused because there was never ever any retraction or resolution.

So my question is: are there still any theoretically-guided practice that remain in ML today? For example, do people use an optimizer because it is theoretically the best for the class of problems? Do people use a certain model or components associated with the model because it works well in theory? Or is it now a fully empirical field where practice is guided by whatever seems to work for other people?