Ancestors

Written by Scott Jenson on 2024-12-19 at 17:14

Well.... things are moving a bit faster than I expected. The Nvidia Jetson mini PC is just $275 and can run most #LLMs today. This is going to start moving very quickly now...

https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/nano-super-developer-kit/?ncid=so-yout-121792-vt49

https://social.coop/@scottjenson/113647639353540214

=> More informations about this toot | More toots from scottjenson@social.coop

Toot

Written by Scott Jenson on 2024-12-19 at 17:29

It also solves the two biggest issues with #LLMs today: It's just 25 watts so can run without killing the planet. It's also entirely local so all of the data remains on your machine so privacy is preserved.

The remaining issue, ethical training, is being addressed by man of the open source models. I'm hopeful that will also be addressed.

=> More informations about this toot | More toots from scottjenson@social.coop

Descendants

Written by Jiří Eischmann on 2024-12-19 at 17:37

@scottjenson AFAIK the vast majority of AI power consumption is associated with building a LLM, not running it. So when you download Llama or Granite to run it on this thing, most of the energy has already been consumed.

=> More informations about this toot | More toots from sesivany@vivaldi.net

Written by Scott Jenson on 2024-12-19 at 17:45

@sesivany I'd like to understand that better. While I agree that training takes a lot of energy, these models are being used an enormous amount, literally billions of times/day. That's ALSO a lot of power...

=> More informations about this toot | More toots from scottjenson@social.coop

Written by Jiří Eischmann on 2024-12-19 at 17:55

@scottjenson yeah, it's mostly true in the context of LLMs run centrally. The cost of running will probably become more relevant with locally run models.

IMHO this is the future. Focused small models that are cheaper to train and cheaper to run locally.

=> More informations about this toot | More toots from sesivany@vivaldi.net

Written by Johan Wärlander 🦀 on 2024-12-19 at 17:48

@scottjenson Isn't the energy requirements for the training itself quite a significant issue, too, regardless of how ethically the training data is sourced?

=> More informations about this toot | More toots from jwarlander@mastodon.nu