跳到正文
Sam Altman· @sama · X·· 4 小时前AI 评分62
AI 导读

Sam Altman 转发 Nick Cammarata 关于可解释性的观点,称很高兴看到这一内容。Nick Cammarata 表示,可解释性可能是当前世界上最重要的科学问题,因为不了解所构建的系统就无法解决对齐问题;他曾在 OpenAI 参与机械可解释性早期工作并一度悲观,认为神经网络会折叠成任意纠缠的结,后来他启动了 OpenAI 首个大规模自动化可解释性项目,用当时的原始工具发现约 2% 的神经元可读。他表示,借助现代智能体工具和更细致的观察,情况完全不同:在参数量高达数万亿的模型中,绝大多数神经元可读,其回路也很优美,许多著名的纠缠神经元案例其实是被误解的,他预期的障碍大多并不存在。他现在认为,随着自动化科学家不断进步,雄心勃勃的可解释性工作会在最需要它之前被解决,甚至可能很快,这或许是可解释性的苦涩教训:简单的方法在荒谬的规模下奏效。

正文

Very happy to see this from Nick!

引用Nick@nickcammarata
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works https://already-clean.goodfire.com/
在 X 查看被引用的帖子

来源:Sam Altman · x.com