Help me understand gguf size/ctx size

查看原文

模型发布订阅权益

时间与来源

发布: 09/04 21:57
收录: 09/05 11:00
来源类型: 开发者社区
档位: 社区
信源状态: 正常

讨论趋势

暂无对比

最近 24 小时与此前 24 小时的快照均值对比 · 7 天曲线

百分比基于采集到的讨论信号，不代表新增评论数或独立参与人数。曲线仅用于同一话题在不同时段的比较。

AI 摘要

一位用户拥有两块16GB的GPU，希望了解GGUF文件大小、上下文大小与VRAM使用量之间的关系，特别是在运行Qwen3.8 27B这类模型时。他们指出，模型卡通常不公布特定上下文长度（例如132k tokens）所需的VRAM，并询问是否有方法可以计算这些信息。用户的集成GPU负责显示输出，因此其专用GPU几乎完全可用于模型推理。

Let's say I have 2x 16Gb GPUs and I want to run Qwen3.8 27B. Monitor is ran by the integrated GPU so both 16Gb GPUs are almost fully free.

I load the UD-Q4_K_S on one card at 15.4Gb. I then load the context on the other card? Would that be the most efficient way? Or should I aim for higher quants that could spill to the second GPU using tensor parallelism?

Also, is there a way to know how much a certain amount of context (e.g. 132k tokens) occupies in VRAM for a given model? I don't usually see this published in model cards, is it because there is a way to calculate it?