Researchers at the University of California Santa Cruz, UC Davis, LuxiTech and Soochow University made significant strides in revolutionizing AI technology by eliminating the need for matrix multiplication (MatMul) in language models. The breakthrough, detailed in the paper “Scalable MatMul-free Language Modeling,” could reduce both environmental impact and operational costs associated with AI systems.
“AI is pretty damn expensive, and we don’t seem to be hitting any walls,” Jason Eshraghian, an assistant professor of electrical and computer engineering at UCSC, said. “There are estimates that ChatGPT costs something like 700,000 bucks a day just to serve all the many requests, all the many users.”
Download the SAN app today to stay up-to-date with Unbiased. Straight Facts™.
Point phone camera here
Eshraghian is leading efforts to make artificial intelligence more sustainable.
“We can just go straight to the language model, see what is the pain point, what is the bottleneck? What is the most expensive operation in that? And that’s matrix multiplication,” he said.
His team developed a groundbreaking approach to AI processing by eliminating the energy-intensive matrix multiplication common in AI models.
“So imagine you just have a huge list of numbers and, you know, maybe those numbers represent words or sentences or entire textbooks,” Eshraghian said. “Imagine you take one of those numbers, and that number has to interact with every single other number that is available to. Then you hop to the next number, and that number has to interact with every other number available to you. When I say interact, I’m saying you have to do some mathematical process. Every process costs energy.”
By eliminating these labor-intensive calculations, the team managed to run a billion-parameter AI model — comparable to Meta’s Llama 2 chatbot — on just 13 watts, roughly the energy used by a single light bulb.
“Every word has some relationship with every other word, and so calculating that is expensive,” Eshraghian said. “Meanwhile, humans and brains don’t really do that, right? Like you’re parsing everything I’m saying word by word. As I say it, it’s not like I have a whole sentence ready and I push it at you. We’re doing things sequentially over time. And by using time in this computation, that’s one of the key approaches that we took to reducing the energy burden of language models.”
Inspired by a study from Microsoft, the UC Santa Cruz team designed custom hardware to optimize these energy-efficient operations. Their prototype operates 25% faster and uses 10 times less memory than standard AI models.
Eshraghian believes this innovation could transform how we use AI, enabling complex algorithms to run on everyday devices without the need for heavy infrastructure.
“Computer scientists are only limited by the hardware available to them,” he said. “And so if the hardware is there, then people will push it to the edge of its limit. So it would be able to do a hell of a lot more for the same computer as GPT-4 at that point.”
[karah rucker]
DID YOU KNOW THAT HAVING A SINGLE CONVERSATION WITH AI LIKE CHATGPT CAN USE AS MUCH ENERGY AS 25 GOOGLE SEARCHES? IT DRAWS A TREMENDOUS AMOUNT OF ENERGY FROM THE GRID — WHICH ULTIMATELY HAS A HUGE IMPACT ON THE ENVIRONMENT AND AS AI TECHNOLOGY CONTINUES TO EVOLVE, SO DOES ITS ENERGY DEMAND. BUT RESEARCHERS AT UC SANTA CRUZ ARE PAVING THE WAY TO CHANGE THAT.
JASON ESHRAGHIAN
ASSISTANT PROFESSOR OF ELECTRICAL AND COMPUTER ENGINEERING, UCSC
“AI is pretty damn expensive, and we don’t seem to be hitting any walls. There are estimates that ChatGPT costs something like 700,000 bucks a day just to serve all the many requests, all the many users.”
[karah rucker]
THAT’S JASON ESHRAGHIAN – AN ASSISTANT PROFESSOR OF ELECTRICAL AND COMPUTER ENGINEERING AT UCSC. HE’S LEADING THE RESEARCH ON EFFORTS TO MAKE THIS ARTIFICIAL INTELLIGENCE MORE SUSTAINABLE.
JASON ESHRAGHIAN
ASSISTANT PROFESSOR OF ELECTRICAL AND COMPUTER ENGINEERING, UCSC
“We can just go straight to the language model, see what is the pain point, what is the bottleneck? What is the most expensive operation in that? and that’s matrix multiplication.”
[karah rucker]
HIS TEAM DEVELOPED A GROUNDBREAKING APPROACH TO AI PROCESSING. THEY’VE ELIMINATED THE ENERGY-INTENSIVE MATRIX MULTIPLICATION COMMON IN AI MODELS.
JASON ESHRAGHIAN
ASSISTANT PROFESSOR OF ELECTRICAL AND COMPUTER ENGINEERING, UCSC
“So imagine you just have a huge list of numbers and, you know, maybe those numbers represent words or sentences or entire textbooks.”
“Imagine you take one of those numbers, and that number has to interact with every single other number that is available to. Then you hop to the next number, and that number has to interact with every other number available to you. When I say interact, I’m saying you have to do some mathematical process every process costs energy. ”
[karah rucker]
BY ELIMINATING THESE LABOR-INTENSIVE CALCULATIONS — THE TEAM MANAGED TO RUN A BILLION-PARAMETER AI MODEL — COMPARABLE TO META’S LLAMA 2 CHATBOT — ON JUST 13 WATTS. THAT’S ROUGHLY THE ENERGY USED BY A SINGLE LIGHT BULB.
THINK ABOUT THIS: WE’VE GONE FROM ROOM-SIZED COMPUTERS TO POWERFUL SMARTPHONES THAT FIT IN OUR POCKETS. SIMILARLY, TODAY’S AI MODELS COULD SOON EVOLVE TO RUN COMPLEX CALCULATIONS ON MINIMAL ENERGY.
JASON ESHRAGHIAN
ASSISTANT PROFESSOR OF ELECTRICAL AND COMPUTER ENGINEERING, UCSC
“Every word has some relationship with every other word, and so calculating that is expensive. Meanwhile, humans and brains don’t really do that, right? Like you’re, you’re parsing everything I’m saying word by word. As I say it, it’s not like I have a whole sentence ready and I push it at you. We’re doing things sequentially over time. and by using time in this computation, that’s one of the key approaches that we took to reducing the energy burden of language models.”
[karah rucker]
INSPIRED BY A STUDY FROM MICROSOFT, THE UC SANTA CRUZ TEAM DESIGNED CUSTOM HARDWARE TO OPTIMIZE THESE ENERGY-EFFICIENT OPERATIONS. THEIR PROTOTYPE OPERATES 25% FASTER AND USES TEN TIMES LESS MEMORY THAN STANDARD AI MODELS.
ESHRAGHIAN BELIEVES THIS INNOVATION COULD TRANSFORM HOW WE USE AI, ENABLING COMPLEX ALGORITHMS TO RUN ON EVERYDAY DEVICES WITHOUT THE NEED FOR HEAVY INFRASTRUCTURE.
JASON ESHRAGHIAN
ASSISTANT PROFESSOR OF ELECTRICAL AND COMPUTER ENGINEERING, UCSC
“Computer scientists are only limited by the hardware available to them. And so if the hardware is there, then people will push it towards, to the edge of its limit. So it would be able to do a hell of a lot more for the same computer as GPT-4 at that point.”
[karah rucker]
FOR STRAIGHT ARROW NEWS, I’M KARAH RUCKER.
FOR MORE OF OUR UNBIASED, STRAIGHT FACT REPORTING, DOWNLOAD THE STRAIGHT ARROW NEWS APP OR VISIT US AT SAN – DOT – COM.
MOST POPULAR
-
Getty Images
Top House Democrats call for Biden to step aside in private meeting
-
Getty Images
Disney heiress, other mega donors won’t finance DNC until Biden drops out
-
Getty Images
Giant clams could hold the key to solar power advancements
-
Reuters
NYC to expand prepaid debit card program for migrant families