Welcome to Incremental Social! Learn more about this project here!
Check out lemmyverse to find more communities to join from here!

miss_brainfarts ,

That's nice and all, but what are some FOSS models I can run on GPU with only 4GB?

I've tried Deepseek Coder, and it's pretty nice for what I use it for. Then there's TinyLlama, which... well it's fast, but I need to be veeeery exact in how I prompt it.

Fisch , (edited )
@Fisch@lemmy.ml avatar

Unfortunately LLMs need a lot of VRAM. You could try using koboldcpp, it runs on the CPU but let's you offload layers onto the GPU. That way you might be able to stay withing those 4gb even with larger models.

Edit: I forgot to mention there's a fork of koboldcpp with rocm for AMD cards, which is about twice as fast if I remember correctly. Only relevant if you have an AMD card tho.

Edit 2: This is the model I use btw

miss_brainfarts ,

I'm currently playing around with the Jan client, which uses the nitro engine. I think I need to read up on it more, because when I set the ngl value to 15 in order to offload 50% to GPU like the Jan guide says, nothing happens. Though that could be an issue specific to Jan.

Fisch ,
@Fisch@lemmy.ml avatar

Maybe 50% GPU is already using too much VRAM and it crashes. You could try to set it to 0% GPU and see if that works.

miss_brainfarts ,

I may need to lower it a bit more, yeah. Though when I try to to use offloading, I can see that vram usage doesn't increase at all.

When I leave the setting at its default 100 value on the other hand, I see vram usage climb until it stops because there isn't enough of it.

So I guess not all models support offloading?

Fisch ,
@Fisch@lemmy.ml avatar

The models you have should be .gguf files right? I think those are the only ones where that's supported

miss_brainfarts ,

All of them are gguf, yeah

General_Effort ,

Most formats don't support it. It has to be gguf format, afaik. You can usually find a conversion on huggingface. Prefer offerings by TheBloke for the detailed documentation, if nothing else.

General_Effort ,

Depends on your needs. Best look around in !localllama or similar. (I don't wanna say reddit but r/localLlama is much larger.)

If you're more into creative writing, maybe look for places that discuss SillyTavern (r/SillyTavernAI is an option). It's software for role-play chats, which may not be what you want. But the community is (relatively) large and likely to have good tips for non-coding/less technical applications.

Toes ,
@Toes@ani.social avatar

4GB is practically nothing in this space. Ideally you want at least 10GB of dedicated vram if you can't get even more. Keep in mind you're also probably trying to share that vram with your operating system. So it's more like ~3GB before you even started.

Kolboldcpp is capable of using both your GPU and CPU together, you might wanna consider that. (Using a feature called layers) There's a trade-off that occurs between the memory available and the quality of its output and the speed of the calculation.

The model mentioned in this post can be run on the CPU with enough system ram or swap.

If you wanna keep it all on the GPU check out 4bit models. Also there's been a lot of work into trying to do this with the raspberry Pi. I suspect that their work could help you out here as well.

insufferableninja ,

i thought this was about the MUD server software, and got excited. alas.

OutrageousUmpire ,

If only I had the $ to get a rig that could run this locally

DarkThoughts ,

Since I had an okay experience with EasyDiffusion I tried running text gen locally through oobabooga, but no matter which model I load, it just crashes whenever it tries to generate anything, regardless if it runs through the UI's chat or SillyTavern. No error in the terminal either, it just stops and throws me back into the command line.

soulfirethewolf ,

Oh yay another model I can't run on my computer :'(

BaroqueInMind ,
@BaroqueInMind@kbin.social avatar

Where can we all see the leader board?
Edit: never mind

HarkMahlberg ,
@HarkMahlberg@kbin.social avatar

What a catchy name.

JohnnyCanuck ,
@JohnnyCanuck@lemmy.ca avatar

It's pronounced "Smaug"

BaroqueInMind ,
@BaroqueInMind@kbin.social avatar

Potato potato

FenrirIII ,
@FenrirIII@lemmy.world avatar

OOTL: What is a LLM and what does it do?

saltesc ,

Large Language Model AI. Like ChatGPT.

FaceDeer ,
@FaceDeer@kbin.social avatar

And at 72 billion parameters it's something you can run on a beefy but not special-purpose graphics card.

glimse ,

Based on the other comments, it seems like this needs 4x as much ram than any consumer card has

FaceDeer ,
@FaceDeer@kbin.social avatar

It hasn't been quantized, then. I've run 70B models on my consumer graphics card at a reasonably good tokens-per-second rate.

DarkThoughts ,

I'm curious how local generation goes with potentially dedicated AI extensions using stuff like tensor cores and their own memory instead of hijacking parts of consumer GPUs for this.

simple ,
@simple@lemm.ee avatar

I'm afraid to even ask for the minimum specs on this thing, open source models have gotten so big lately

SinningStromgald ,

Just a data center or two. Easy peasy dirt cheapy.

TheChurn ,

Every billion parameters needs about 2 GB of VRAM - if using bfloat16 representation. 16 bits per parameter, 8 bits per byte -> 2 bytes per parameter.

1 billion parameters ~ 2 Billion bytes ~ 2 GB.

From the name, this model has 72 Billion parameters, so ~144 GB of VRAM

nicetriangle ,

Ok but will this run on my TI-83? It's a + model.

whoelectroplateuntil ,

no. but put this clustering software i wrote in ti-basic on 40 million of them? still no

Rai ,

Only if it’s silver.

01189998819991197253 ,
@01189998819991197253@infosec.pub avatar

Dang. So close.

Rai ,

My 83 was ganked by some kid I knew so my folks bought me a silver. He denied it. I learned that day to write my name in secret spots.

01189998819991197253 ,
@01189998819991197253@infosec.pub avatar

That kid you knew was a dick. At least he taught you a valuable lesson, I guess.

Rai ,

He absolutely was a dick. I stopped being mates with him after that. My school was like “yeah the cameras didn’t work that day actually”

01189998819991197253 ,
@01189998819991197253@infosec.pub avatar

Leads me to believe that the cameras never actually worked.

Rai ,

I believe that. Or they just didn’t want to be responsible for dealing with theft. Both ways make perfect sense to me.

01189998819991197253 ,
@01189998819991197253@infosec.pub avatar

Very true. Saying the cameras don't work is enough to keep the cops out of it. After all, hearsay between kids is never taken seriously. No footage, means no proof, means no cops. Then they can keep the facade of the school being crime-free.

Rai ,

Absolutely my dude! My school pretended the cams didn’t work when my calculator was stolen… they worked fantastically the next week.

01189998819991197253 ,
@01189998819991197253@infosec.pub avatar

That is some BS! Honestly, I probably would have dealt with that kid schoolyard-style (though, I'm not proud of that as an adult). My school never pretended the cameras were out of service. They just got rid of them after a teacher was accused of inappropriate behavior with a senior. Claimed they never worked. He "left for another state" later that week. I'm sure it was unrelated...

Rai ,

Hahaha he was a lil dude and he came from a rough family. Plus I was like 115-120lbs at the time, I ain’t no fightgurl. I knew my folks would get me a better one (it wasn’t even a plus model…) so I was easy on him.

Still stopped talking to him.

Also at your last coupe sentence… oh dear me. That’s… not good.

01189998819991197253 ,
@01189998819991197253@infosec.pub avatar

Sounds like he was trying to get what he needed to learn the maths, even if screwed over his friends. Hopefully he learned from it, improved himself, and didn't do something like that again. After all, we all did stupid things growing up. You seemed to have handled it really well. Good on you!

Yeah, it was really bad. And I wish I could say it was the first of its kind at that school. Or the last. But I wasn't either. That principal was a joke who had his boys club going strong. Last I heard, he was black-listed from all educational institutions, public and private. I hope that was a permanent ban. I'm not sure about the others who committed these acts, though.

FaceDeer ,
@FaceDeer@kbin.social avatar

It's been discovered that you can reduce the bits per parameter down to 4 or 5 and still get good results. Just saw a paper this morning describing a technique to get down to 2.5 bits per parameter, even, and apparently it 's fine. We'll see if that works out in practice I guess

Corngood ,

I'm more experienced with graphics than ML, but wouldn't that cause a significant increase in computation time, since those aren't native types for arithmetic? Maybe that's not a big problem?

If you have a link for the paper I'd like to check it out.

FaceDeer ,
@FaceDeer@kbin.social avatar

My understanding is that the bottleneck for the GPU is moving data into and out of it, not the processing of the data once it's in there. So if you can get the whole model crammed into VRAM it's still faster even if you have to do some extra work unpacking and repacking it during processing time.

The paper was posted on /r/localLLaMA.

L_Acacia ,

You can take a look at exllama and llama.cpp source code on github if you want to see how it is implemented.

OutrageousUmpire ,

Any idea what 8Q requirements would be? Or 4 or 5?

General_Effort ,

https://huggingface.co/senseable/Smaug-72B-v0.1-gguf/tree/main

About 44GB and 50GB for the Q4 and 5. You'd need quite some extra to fully use the 32k context length.

Amaltheamannen ,

Though with quantisation you can get it down to like 30GB of vram or less.

rs137 ,

Llama 2 70B with 8b quantization takes around 80GB VRAM if I remember correctly. I’ve tested it a while ago.

girsaysdoom ,

I think I read somewhere that you'll basically need 130 GB of RAM to load this model. You could probably get some used server hardware for less than $600 to run this.

ArchAengelus ,

Unless you’re getting used datacenter grade hardware for next to free, I doubt this. You need 130 gb of VRAM on your GPUs

ivanafterall ,
@ivanafterall@kbin.social avatar

So can I run it on my Radeon RX 5700? I overclocked it some and am running it as a 5700 XT, if that helps.

L_Acacia ,

To run this model locally at gpt4 writing speed you need at least 2 x 3090 or 2 x 7900xtx.
VRAM is the limiting factor in 99% of cases for interference.
You could try a smaller model like mistral-instruct or SOLAR with your hardware though.

cm0002 ,

Oh if only it were so simple lmao, you need ~130GB of VRAM, aka the graphics card RAM. So you would need about 9 consumer grade 16GB graphics cards and you'll probably need Nvidia because of fucking CUDA so we're talking about thousands of dollars. Probably approaching 10k

Ofc you can get cards with more VRAM per card, but not in the consumer segment so even more $$$$$$

kakes ,

Afaik you can substitute VRAM with RAM at the cost of speed. Not exactly sure how that speed loss correlates to the sheer size of these models, though. I have to imagine it would run insanely slow on a CPU.

Infiltrated_ad8271 ,
@Infiltrated_ad8271@kbin.social avatar

I tested it with a 16GB model and barely got 1 token per second. I don't want to imagine what it would take if I used 16GB of swap instead, let alone 130GB.

girsaysdoom ,

I'm pretty sure you can load the model using RAM like another poster said. Here's a used server under $600 that could theoretically run it: ebay.

brick ,

You would want to look for an R730, which can be had for not too much more. The 20 series was the “end of an era” and the 30 series was the beginning of the next era. Most importantly for this application, R30s use DDR4 whereas R20s use DDR3.

RAM speed matters a lot for ML applications and DDR4 is about 2x as fast as DDR3 in all relevant measurements.

If you’re going to offload any part of these models to CPU, which you 99.99% will have to do for a model of this size with this class of hardware, skip the 20s and go to the 30s.

General_Effort ,

CUDA 11.4 and above are recommended (this is for GPU users, flash-attention users, etc.) To run Qwen-72B-Chat in bf16/fp16, at least 144GB GPU memory is required (e.g., 2xA100-80G or 5xV100-32G). To run it in int4, at least 48GB GPU memory is requred (e.g., 1xA100-80G or 2xV100-32G).

It's derived from Qwen-72B, so same specs. Q2 clocks it in at only ~30GB.

L_Acacia ,

Around 48gb of VRAM if you want to run it in 4bits

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • technology@lemmy.world
  • random
  • incremental_games
  • meta
  • All magazines
    •