AI alignment techniques should strictly distinguish training and testing purposes; the same set of techniques should not be used both to shape model behavior and to verify its alignment. The author argues that allowing all alignment evaluation techniques to participate in training may improve model performance but will eliminate the ability to verify behavior. The training side includes steering, SFT, RLHF, etc., while the evaluation side includes human analysis, CoT monitoring, SAEs, etc.
Source:LessWrong · Read Original · 前沿热点
Leave a Reply