Mid research got me thinking what about reversed alignment, would trained "bad" model exhibit"good" behavior later and/or secretly [D]
A researcher, working on a paper about RHLF, pondered whether a model trained to reward "bad" behaviors like deception and selfishness might later exhibit "good" behavior.…
late night thoughts as I was working on my paper that is about specific behavior that arises from RHLF, it got me thinking what if train a model in an environment where bad behavior is rewarded: deception, selfishness, harmful behavior etc. and then find it occasionally and/or secretly exhibit good behavior (which would be ironically here a misalignment, vs. misalignment behavior detected in current models) would that happen and would it be due to pre-training?
EDIT: What I am thinking if there is if there is some "alignment" already in pretraining (or like a raw latent machinery that alignment training later selects from) - would it show up in the naughty post-trained model eventually?
its late night so my brain is all over the place, but would love to hear your thoughts