Extraordinary article on the energy usage of generative AI from BloombergNEF founder Michael Liebreich - absolutely worth spending some time with this: https://about.bnef.com/blog/liebreich-generative-ai-the-power-and-the-glory/
I wrote up some of my own notes on the article here: https://simonwillison.net/2025/Jan/12/generative-ai-the-power-and-the-glory/
=> More informations about this toot | More toots from simon@simonwillison.net
@simon
In 2007 the EPA predicted data center energy usage would double: it didn't, thanks to efficiency gains from better servers and the shift from in-house to cloud hosting.
That latter shift can also be seen as a shift to greater centralization, which isn't necessarily a good thing. I hope centralization, a concentration of hosting onto big servers at a few providers, isn't necessary to keep energy consumption at a reasonable level.
=> More informations about this toot | More toots from matt@toot.cafe
@matt on a similar note, for all of the excitement about local LLMs not a lot of people seem to be considering how inefficient they are in terms of power usage compared to big power hungry data center LLMs that are shared by thousands of people
=> More informations about this toot | More toots from simon@simonwillison.net
@simon @matt what’s a good way to reason about these efficiencies? Is there a normalized “token quality per watt” metric that one could look at, scored per-model?
=> More informations about this toot | More toots from glyph@mastodon.social
@glyph @matt I really wish there was - I think it would substantially benefit the AI labs at this point to release accurate numbers
=> More informations about this toot | More toots from simon@simonwillison.net
@simon @matt I've always been bothered by people not acknowledging the fact that moving LLMs to local processing also shifts the energy usage to local as well. Depending on the market, it could be even more expensive and lot more reliant on non-green sources.
=> More informations about this toot | More toots from ppatel@mstdn.social
@simon
@matt
I think there are huge wild cards here in the form of new technologies, aren't there? Many innovations to make data centers a lot cheaper will likely make local a lot cheaper too. And it's interesting to compare with video games where streaming from a data center hasn't really taken off.
=> More informations about this toot | More toots from faassen@fosstodon.org
@matt
It's a few things. Large cloud providers have done a lot of work to reduce inefficiencies in their ecosystems, including moving to large-scale liquid cooling on some cases, whole rack-aisle airflow calculations, custom chassis designed for power efficiency, etc., meaning that in many cases, less than ten percent of the total DC power is going to things other than compute, storage, and networking. They also have enough load to balance that they can keep most of their machines running at right around the 80% load mark, because systems are more efficient when they're running a heavy workload.
Very few companies with small on-prem DC footprints do anything like this much work, and the systems that are sold to them are generic, because they need to work anywhere, so that kind of extreme optimization just isn't available to them. Loads for individual companies are also often bursty, given peaks in customer demand, monthly batch jobs, etc., and the rest of the time that spare capacity sits idle. Cold-booting hardware carries some risk of systems not coming back online, so servers sit mostly idle, still consuming most of their peak power.
So basically yes, on-prem is hard to make as efficient as cloud systems, for most realistic companies. However, one of the things you can do is get rid of useless compute. Taking the time to work with profiling tools and taking some care at the business level in how and when jobs are run can have a large impact on performance. Getting rid of needlessly complex systems also helps — for instance, the entire targeted advertising ecosystem, if turned off and all related code removed from systems, would likely cover efficiency losses from returning to on-prem systems many times over.
@simon
=> More informations about this toot | More toots from dymaxion@infosec.exchange
@dymaxion @simon So, what would happen if we had a lot more small-scale self-hosting on passively cooled systems with low idle power consumption? For example, I have a single-board computer (a Quartz64 model B) sitting in my apartment, assigned to a static IP address on my residential fiber connection so it can host stuff. Its idle power consumption is probably lower than a typical x86-64 PC, though I haven't measured it.
=> More informations about this toot | More toots from matt@toot.cafe
@dymaxion @simon Still, I wonder, if hundreds of millions of people had such self-hosted setups, would the aggregate energy consumption still add up to much more than the equivalent centralized services?
=> More informations about this toot | More toots from matt@toot.cafe
@matt
Yes, basically, especially with full lifecycle analysis. We can federalize services politically and take advantage of industrial efficiencies of scale without trying to be digital homesteaders, and most of our options for efficiency come from doing less dumb useless stuff, like data collection, behavioral analysis, personalized advertising, and the entire suite of useless LLM tools.
@simon
=> More informations about this toot | More toots from dymaxion@infosec.exchange
@matt
So, a) if we're talking about large models, they cannot be trained in that context and many of them won't even be able to do inference there. b) For a phone, the embodied carbon (ignoring all other resource impact) of manufacturing is larger than its lifetime energy carbon footprint. c) A machine like that serving a single user is almost always going to be idle, and even if it has significant idle power reduction, it's still going to do much lower useful work per watt than a more computationally capable per watt large system that's running at a steady 75-95% load.
@simon
=> More informations about this toot | More toots from dymaxion@infosec.exchange
@simon Where I can't follow along is the wishful thinking that AI will somehow solve all our social and human problems.
How does the author go from LLMs to "eliminating road collisions will reduce the demand for health care" that seems so far fetched. How did that become the basis for our planning and conversation about AI.
=> More informations about this toot | More toots from djh@chaos.social
@djh yeah that bit felt a little too optimistic to me - presumably the road collisions thing was a reference to Waymo
=> More informations about this toot | More toots from simon@simonwillison.net
@djh @simon AI is a boondoggle because of bullshit like you call out.
The reality behind pie in the sky fantasies about AI is that the people funding it believe AI will replace labor so that they can survive after societal collapse. We allowed very stupid people too much power
=> More informations about this toot | More toots from Subterfuge@dice.camp
@simon “this technology has been around for decades, been through a lot of hype cycles, and every time the overblown promises have failed to pan out. but this time is different and now it’s practically perfect at everything!”
I have lot of respect for NEF in general but that’s still an abysmally myopic take
=> More informations about this toot | More toots from karlhigley@recsys.social
@simon not sure I believe that corps which quickly jettison DEI initiatives deeply care about low carbon energy
=> More informations about this toot | More toots from neilernst@mastodon.acm.org
@simon I wonder if someone did a rough calculation of the indirect (energy) costs due to the increased web scraping by the AI companies. Scraping was always there, but it significantly increased and the bots from OpenAI or Anthropic seem to be quite aggressive.
=> More informations about this toot | More toots from JochenGe@sigmoid.social
@simon different perspective from Thomas Fricke on 38C3 conferences @thomasfricke
https://m.youtube.com/watch?v=4ouNbLgJEHA
=> More informations about this toot | More toots from xezpeleta@mastodon.eus
@xezpeleta @simon @thomasfricke media.ccc link: https://media.ccc.de/v/38c3-resource-consumption-of-ai-degrow-or-die
=> More informations about this toot | More toots from Profpatsch@mastodon.xyz
@Profpatsch @xezpeleta @simon
Thanks for mentioning!
=> More informations about this toot | More toots from thomasfricke@23.social
@thomasfricke @Profpatsch @xezpeleta @simon as a practical showcase: Google is building six ~150MW hyperscale datacenters in my neighbouring city. In Norway we have a lot of hydropower, but it is far from enough to feed the servers due to limitations in capacity in the grid.
To feed the servers, hundreds of new mega windmils are planed to be built in pristine nature in our mountains, and many big solar parks are planed along the south cost of Norway.
Just to cover the extreme energy needs of Googles datacenters.
We get nothing back as a community. No real jobs. Just higher energy prices.
And on top of it all, Google does pay practicaly no taxes.
It is a scam and our local politicians have been tricked.
=> More informations about this toot | More toots from ruben_int@social.cloudless.world
@ruben_int @Profpatsch @xezpeleta @simon
This is a typical pattern already know from the US if you follow @gerrymcgovern
We will see this in Europe, too. We need to allie to fight back.
=> More informations about this toot | More toots from thomasfricke@23.social
@thomasfricke @Profpatsch @xezpeleta @simon @gerrymcgovern I do follow him.
In Europe, I know that BigTech datacenters in Belgium and Denmark are close to using about 20% of the national power production.
These new Google datacenters in Skien, Norway would, when finished, use ~5% of Norways total energy consumption. But Google is not the only one building huge datacenters. Green Mountain, who has big customera like TikTok etc, is also expanding rapidly on several other strategic locations in Norway, so it is just a matter of time before we we reach the 20% mark too.
Unless some puts an end to the madness.
=> More informations about this toot | More toots from ruben_int@social.cloudless.world
@ruben_int
In Ireland, it's already over 20% of the electricity. It's terrible, and it will get much worse with AI. The argument will then become: We'd like to get rid of oil and gas and coal but we have to keep them because of data centers. This is what always happens. There are never energy transitions. There are only energy additions.
@thomasfricke @Profpatsch @xezpeleta @simon
=> More informations about this toot | More toots from gerrymcgovern@mastodon.green
@gerrymcgovern @ruben_int @Profpatsch @xezpeleta @simon
Yes. And the EPRI predictions of a up to five fold increase of energy consumption https://www.epri.com/research/products/3002028905 cannot work in this case.
"EirGrid, Ireland’s energy grid, placed a moratorium on the development of new datacentres in Dublin until 2028. "
https://www.theguardian.com/world/2024/feb/15/power-grab-hidden-costs-of-ireland-datacentre-boom
We need more or less similar predictions asap for every country. And probably a moratorium. I can already hear the outcry
@sabinegruetzmacher
@ankedb
=> More informations about this toot | More toots from thomasfricke@23.social
@bitsundbaeume @bitsundbaeume_berlin @BitsUndBaeumeDresden @bitsundbaeumeKoeln @bitsundbaeume_H
This might be a topic for you
=> More informations about this toot | More toots from thomasfricke@23.social
@simon What's interesting in this context of energy: The unicorn AI startup groq seems to be hiring in Dammam for data center roles. I'm curious if that's an energy long play first and foremost.
=> More informations about this toot | More toots from djh@chaos.social
@simon those hippies at #goldmansachs and #nexteraenergy have already said what is huge energy suck and for dubious results.
=> More informations about this toot | More toots from gentlegardener@mastodon.scot
@gentlegardener where did they say that?
=> More informations about this toot | More toots from simon@simonwillison.net
@simon #goldmansachs released a report late oct/nov that was cautionary about AI. Subsequently Ive seen their AI? analyst or energy/data center analyst interviewed - probably on Bloomberg but possibly elsewhere - telly - about the Yuge ramp up in electricity demand in the US due to 5x demand from AI servers. Also read Nextera energy CEO interview quoted that each AI datacenter would require 5x energy demand of current datacenters, equivalent of what would be supplied by small nuclear reactor
=> More informations about this toot | More toots from gentlegardener@mastodon.scot
@simon of course nextera and dominion energy here in #virginia (host to 70% of worldwide server traffic at present) are gleeful at the prospects, and certainly not dubious about the uses of chatGPT, AI. But the people of Virginia are.
=> More informations about this toot | More toots from gentlegardener@mastodon.scot
@simon also look up pecva.org film a year ago about datacenters in #virginia. this is a bipartisan issue - land use, farmland use, water use, energy use subsidized by homeowners and farmers, corporate tax breaks subsidized by citizen taxpayers not corporations.
=> More informations about this toot | More toots from gentlegardener@mastodon.scot This content has been proxied by September (3851b).Proxy Information
text/gemini