<\/p>\n
SubQ won\u2019t replace existing top models across the board, but it could offer huge increases in speed at a fraction of the typical cost for certain tasks. Subquadratic insists that in the long run, though, its breakthrough could change how LLMs are built. \u201cWe hope we\u2019re kicking off a new age of efficiency,\u201d says Justin Dangel, the firm\u2019s cofounder and CEO. \u201cWe don\u2019t think anybody will be building on transformers in a few years.\u201d<\/p>\n
To understand why Subquadratic\u2019s claims are a big deal, let\u2019s dig into how most LLMs work. The key mechanism inside an LLM is a type of neural network called a transformer, which runs a process known as dense attention. Today\u2019s LLMs typically chain together multiple transformers. (The foundational paper of the LLM era, published by researchers at Google in 2017, was titled \u201cAttention Is All You Need.\u201d)<\/p>\n
Dense attention works like this: When a transformer processes a chunk of text, it first encodes each word (or part of a word, known as a token) with a number. To capture the meaning of the full text, it then multiplies each of those numbers with every other number for that text. For example, a piece of text 10,000 words long would kick off almost 50 million individual multiplications. That\u2019s a lot of computation and the main reason that LLMs are notorious power hogs.<\/p>\n
\u201cIf you want to summarize The Great Gatsby<\/em>, you have to look at the first word and the last word together, and then you have to look at every other combination,\u201d says Dangel.<\/p>\n As the length of the text increases, the number of computations skyrockets. That\u2019s because each additional number must be multiplied by all other previous numbers. Double the number of words, and you roughly quadruple the number of computations, a rate of increase known as a quadratic expansion.<\/p>\n (You can picture this yourself: Draw a circle and mark dots around its edge. Each dot is a token. Then draw lines between pairs of dots to represent the multiplication of those two tokens. A circle with five dots will have 10 lines crossing it. Make it 10 dots and you will have 45 lines, 20 dots and you will have 190 lines, and so on.)<\/p>\n Subquadratic\u2019s solution is to ditch dense attention, the core operation of a transformer, in favor of what\u2019s known as sparse attention, which slashes the number of computations needed. Instead of multiplying the number assigned to each token by every other number, sparse attention selects just some of the numbers to multiply. The idea is that not all relationships between words in a piece of text matter.<\/p>\n<\/p><\/div>\n #startup #claims #broke #bottleneck #holding #LLMs<\/p>\n","protected":false},"excerpt":{"rendered":" SubQ won\u2019t replace existing top models across the board, but it could offer huge increases…<\/p>\n","protected":false},"author":1,"featured_media":14337,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[10544,2641,418,7842,7507,1725],"class_list":["post-14336","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-stories","tag-bottleneck","tag-broke","tag-claims","tag-holding","tag-llms","tag-startup"],"featured_image_urls":{"full":["https:\/\/news678.top\/wp-content\/uploads\/2026\/06\/cheetah-speed2a.jpg",1200,600,false],"thumbnail":["https:\/\/news678.top\/wp-content\/uploads\/2026\/06\/cheetah-speed2a-150x150.jpg",150,150,true],"medium":["https:\/\/news678.top\/wp-content\/uploads\/2026\/06\/cheetah-speed2a-300x150.jpg",300,150,true],"medium_large":["https:\/\/news678.top\/wp-content\/uploads\/2026\/06\/cheetah-speed2a-768x384.jpg",640,320,true],"large":["https:\/\/news678.top\/wp-content\/uploads\/2026\/06\/cheetah-speed2a-1024x512.jpg",640,320,true],"1536x1536":["https:\/\/news678.top\/wp-content\/uploads\/2026\/06\/cheetah-speed2a.jpg",1200,600,false],"2048x2048":["https:\/\/news678.top\/wp-content\/uploads\/2026\/06\/cheetah-speed2a.jpg",1200,600,false],"covernews-featured":["https:\/\/news678.top\/wp-content\/uploads\/2026\/06\/cheetah-speed2a-1024x512.jpg",1024,512,true],"covernews-medium":["https:\/\/news678.top\/wp-content\/uploads\/2026\/06\/cheetah-speed2a-540x340.jpg",540,340,true]},"author_info":{"display_name":"admin","author_link":"https:\/\/news678.top\/?author=1"},"category_info":"Stories<\/a>","tag_info":"Stories","comment_count":"0","_links":{"self":[{"href":"https:\/\/news678.top\/index.php?rest_route=\/wp\/v2\/posts\/14336","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/news678.top\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/news678.top\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/news678.top\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/news678.top\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=14336"}],"version-history":[{"count":0,"href":"https:\/\/news678.top\/index.php?rest_route=\/wp\/v2\/posts\/14336\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/news678.top\/index.php?rest_route=\/wp\/v2\/media\/14337"}],"wp:attachment":[{"href":"https:\/\/news678.top\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=14336"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/news678.top\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=14336"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/news678.top\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=14336"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Slashing costs<\/strong><\/h3>\n