I think this warrants an extra post. And the beginners thread is a year old and I guess not a lot of people watch comments there.
I use KoboldCpp and like to recommend that to people who are new to the hobby or don't own a proper gaming rig. It's relatively easy to install and you can try it now, without any GPU, and see if you like it. I'd say it's usable on CPU up to about 13B (with quantized models). Of course it'll be orders of magnitude slower than a GPU.
I'd say every bit of VRAM counts. So you might as well buy as much as you can afford. And you'll be able to run more intelligent models. Use one of the VRAM calculators to see what fits in 16GB or 24GB. And if you need it.
=> More informations about this toot | View the thread | More toots from hendrik@palaver.p3x.de
=> View sith@lemmy.zip profile
text/geminiThis content has been proxied by September (ba2dc).