My experience building 64GB VRAM AI SWE assistant/agent PC

As a SWE I have ultimate belief that having a irreplaceable subscription on OpenAI and Anthropic models is actually against my beliefs (I'm not fond of becoming a digital slave and being unable to do my job without using tools from some sketchy corporation), so I kinda turned my attention to local models.

I periodically tried and found some usefulness in them, but this year I started to notice they started to really contribute to the quality and speed of the implementation of software I'm writing, but with just stock RTX 4080 16GB VRAM precision and context window options are quite limited, so I decided to use more GPUs and see how it improves my model running capabilities.

- (pic #1) At first I bought a Meshify 2 XL case - it's one of the biggest full tower PC cases for home use. Meshify 3 XL already existed at the time I made this purchase, but it lacks built-in nylon dust filters, so it was a deal-breaker for me. Meshify 2 XL theoretically has enough volume to fit 4 RTX3090FE-sized GPUs, with two of them being put directly into motherboard PCIE slots and the other two mounted onto a custom vertical rail solution in place of water pump as some kind of GPU sandwitch with questionable air cooling capabilities since they're obstructing frontal intake fans. So I went with three GPUs because neither my PSU nor motherboard allowed more than that. The temps are pretty reasonable, with vertically mounted GPU being the most cool one since it's close to frontal coolers.

- For PSU, I bought the Corsair HX1500i SHIFT (the one with cable ports on the side), and it has one problem compared to non-SHIFT version - it's total number of possible 8 pins is 6, so you can't attach more than two 3x 8-pin GPUs (different configuration of cable sockets + two CPU type-5 cables are no longer compatible with PCIE Type5s). So, basically, with this PSU you're limited only to 3x 12VHPWR/3090FE proprietary connectors/double 8-pin GPUs. Due to that, I had to swap one of the 3x 8-pin 3090s with another Founders Edition. Also, both 3090s are power-limited to 300W just in case.

- (pic #2 and #3) For third GPU, I used CoolerMaster V3 Vertical Mount Adapter, removed built-in PCIE riser since it's not long enough for this use case, screwed adapter to the top dust filter holder lid, having drilled a few holes in the adapter itself, having the GPU hanging and exhausting hot air in the up direction. While it looks sturdy, I still put anti-sag holder at the bottom just in case.

- Only 500-600mm PCIE risers have sufficient length to reach third GPU, 400mm was not enough, unfortunately. I used PCI Express v4 riser and as far as I'm aware, it's a theoretical maximum for such long risers and most of the PCIE 5 solutions are scam and to have no signal degradation they require some kind of additional power and signal repeater. Also, riser is not rigidly seated on the motherboard slot, so with medium amount of force it can be pulled out. Some 3D-printed DIY clip/holder may solve that.

cmake -B build \ -DBUILD_SHARED_LIBS=OFF \ -DCMAKE_BUILD_TYPE=Release \ -DGGML_CUDA=ON \ -DGGML_CUDA_FA=ON \ -DCMAKE_CUDA_ARCHITECTURES="86;89" \ -DGGML_NATIVE=ON \ -DGGML_LTO=ON \ -DGGML_CUDA_GRAPHS=ON \ -DGGML_CUDA_NCCL=ON \ -DGGML_OPENMP=ON \ -DGGML_CCACHE=ON

llama-server -m /mnt/data/AI/models/Qwen3.8-27B-MTP-Q8_0.gguf \ -ngl all -t 18 --tensor-split 14,24,24 --main-gpu 0 -sm layer -ub 512 \ --kv-unified --split-mode layer \ -fa on --temp 1.0 --top_k 20 --top_p 0.95 --min_p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 \ --host 0.0.0.0 --port 7800 \ -c 262144 --fit on -np 1 --parallel 1 \ --spec-type ngram-mod,draft-mtp --spec-draft-n-max 3 \ --alias qwen3.8-27b --reasoning-preserve --reasoning-effort xhigh \ --chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "xhigh"}' \ --api-key "$LLAMA_SERVER_API_KEY"

Since this is primary desktop, I can't run it in headless mode and make use of all VRAM, so general use still needs to have like 2-3 GB of spare VRAM to be able to use OS (CachyOS + KDE Plasma), browser, IDE and game engine editor, so it's more like ~62GB VRAM setup. There's like 10-12 GBs of spare VRAM when using Qwen3.8-27B Q_8 with MTP and full 262144 context, which may be used to try different models or crank up context. I plan to experiment with Unsloth Dynamic Q_8_XL and some MoE models because personally I still see direct correlation between quality of the answers and autonomous work of the agent and precision of the model, despite Q_4 and fp8 models being good for general use. Performance degrades over context usage, starting with 44-46 tokens per seconds and ending up to 28-30 t/s when.

My workflow is still mainly writing code by hand, with delegating small-to-medium boring tasks to AI agent, creating drafts and exploring possible solutions for particular well-defined problems, documentation search and presenting it in fancy .md human-readable format, running review/weekly project assessment/optimization tricks and suggestions/bugfix suggestions/project management skills, I like this workflow and still feel that it's actually me writing the software and putting all the effort of my mind capable of, not mindlessly accepting AI-generated stuff.