Back
RCreddit.com

Help me understand gguf size/ctx size

View original
Time & source
Published
09/04, 21:57
Ingested
09/05, 11:00
Source type
Dev community
Tier
Community
Source status
Healthy
Tier is a per-source editorial setting, not a per-item score.

Let's say I have 2x 16Gb GPUs and I want to run Qwen3.8 27B. Monitor is ran by the integrated GPU so both 16Gb GPUs are almost fully free.

I load the UD-Q4_K_S on one card at 15.4Gb. I then load the context on the other card? Would that be the most efficient way? Or should I aim for higher quants that could spill to the second GPU using tensor parallelism?

Also, is there a way to know how much a certain amount of context (e.g. 132k tokens) occupies in VRAM for a given model? I don't usually see this published in model cards, is it because there is a way to calculate it?