{"id":26113,"date":"2026-11-20T19:07:24","date_gmt":"2026-11-20T19:07:24","guid":{"rendered":"https:\/\/news678.top\/?p=26113"},"modified":"2026-11-20T19:07:24","modified_gmt":"2026-11-20T19:07:24","slug":"the-most-misunderstood-graph-in-ai-just-got-a-major-update","status":"publish","type":"post","link":"https:\/\/news678.top\/?p=26113","title":{"rendered":"The most misunderstood graph in AI just got a major update"},"content":{"rendered":"<p><\/p>\n<div>\n<p class=\"wp-block-paragraph\">Even if the models do lag behind human performance on open-ended tasks right now, the new report suggests that they might be getting better. Some of the newest models, such as Opus 5.5, tend to achieve what METR calls \u201cborderline above the bar\u201d performance on the evaluation design tasks\u2014not yet usable for METR&#8217;s research, but promising. Just a year ago, every model was scoring in the \u201cclearly below the bar\u201d category.<\/p>\n<p class=\"wp-block-paragraph\">But because these open-ended and judgment-heavy tasks are so tough to design and score, METR includes few of them in its report, and that makes it difficult to predict whether and how models might continue to progress in these areas. \u201cThe improvement we observe, I think that does tell us something for the current moment,\u201d says Najoung Kim, a linguist and computer scientist at Boston University. \u201cI\u2019m not sure just yet how to think about how it would extrapolate.\u201d<\/p>\n<p class=\"wp-block-paragraph\">Matt Putz, a member of the technical staff at METR, says that predicting progress on these kinds of tasks is likely to remain difficult. \u201cBy their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify,\u201d he said in an email. \u201cUltimately, I don\u2019t think we\u2019ll be able to establish trends as clean and robust as for the easy-to-specify [tasks].\u201d<\/p>\n<p class=\"wp-block-paragraph\">In general, METR is clearly butting up against the practical difficulties of testing AI performance on difficult and underspecified tasks\u2014precisely the ones most relevant to automating AI R&amp;D. METR\u2019s time horizon plots score task difficulty by evaluating how long it would take a human to complete the task, and for the original plot they measured this quantity by actually having people try each task.\u00a0<\/p>\n<p class=\"wp-block-paragraph\">In the new report, however, the completion times are mostly estimates, because paying people for tasks that might take hundreds of hours, or that only a few experts are qualified to complete, is prohibitively expensive and time-consuming. Other tasks have more specific shortcomings\u2014for example, for the philosophical argument task, just one person rated most of the arguments, and agents are judged by how well they agree with that particular individual. It\u2019s possible that the scores would change if more experts were surveyed\u2014but that, too, is time-consuming.<\/p>\n<p class=\"wp-block-paragraph\">It\u2019s still early days for evaluating models on tasks that are hard to validate or require judgment. And while the experts I spoke with see the inclusion of these tasks in METR\u2019s newest time horizon plot as a clear step in the right direction, they agree that we are still far from achieving any definitive answers about whether, and how quickly, automated AI R&amp;D might be coming. \u201cThis move toward open-ended tasks is promising,\u201d Kapoor says. But, he adds, \u201cthere is just such a lot more to be done to actually get a fuller understanding.\u201d<\/p>\n<\/p><\/div>\n<p>#misunderstood #graph #major #update<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Even if the models do lag behind human performance on open-ended tasks right now, the&#8230;<\/p>\n","protected":false},"author":1,"featured_media":26114,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[15267,777,15266,467],"class_list":["post-26113","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-stories","tag-graph","tag-major","tag-misunderstood","tag-update"],"featured_image_urls":{"full":["https:\/\/news678.top\/wp-content\/uploads\/2026\/11\/metr2a.jpg",1200,600,false],"thumbnail":["https:\/\/news678.top\/wp-content\/uploads\/2026\/11\/metr2a-150x150.jpg",150,150,true],"medium":["https:\/\/news678.top\/wp-content\/uploads\/2026\/11\/metr2a-300x150.jpg",300,150,true],"medium_large":["https:\/\/news678.top\/wp-content\/uploads\/2026\/11\/metr2a-768x384.jpg",640,320,true],"large":["https:\/\/news678.top\/wp-content\/uploads\/2026\/11\/metr2a-1024x512.jpg",640,320,true],"1536x1536":["https:\/\/news678.top\/wp-content\/uploads\/2026\/11\/metr2a.jpg",1200,600,false],"2048x2048":["https:\/\/news678.top\/wp-content\/uploads\/2026\/11\/metr2a.jpg",1200,600,false],"covernews-featured":["https:\/\/news678.top\/wp-content\/uploads\/2026\/11\/metr2a-1024x512.jpg",1024,512,true],"covernews-medium":["https:\/\/news678.top\/wp-content\/uploads\/2026\/11\/metr2a-540x340.jpg",540,340,true]},"author_info":{"display_name":"admin","author_link":"https:\/\/news678.top\/?author=1"},"category_info":"<a href=\"https:\/\/news678.top\/?cat=7\" rel=\"category\">Stories<\/a>","tag_info":"Stories","comment_count":"0","_links":{"self":[{"href":"https:\/\/news678.top\/index.php?rest_route=\/wp\/v2\/posts\/26113","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/news678.top\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/news678.top\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/news678.top\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/news678.top\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=26113"}],"version-history":[{"count":0,"href":"https:\/\/news678.top\/index.php?rest_route=\/wp\/v2\/posts\/26113\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/news678.top\/index.php?rest_route=\/wp\/v2\/media\/26114"}],"wp:attachment":[{"href":"https:\/\/news678.top\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=26113"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/news678.top\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=26113"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/news678.top\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=26113"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}