Training a Neural Network on AMD MI50s Using Vulkan: Proof of ConceptOr: Why I Stopped Listening and Just Did It

Note: This writeup was put together with the help of AI.

So I've got dual AMD MI50 32GB cards. If you know these cards, you know the story - AMD dropped official ROCm support for gfx906 after ROCm 5.7. Every AI I talked to, every forum post, every "expert" said the same thing: if you want to do anything serious on AMD hardware you need ROCm, and if you need ROCm on MI50s, good luck because AMD basically told you to go buy newer hardware.

The consensus was consistent and confident. ROCm for training, Vulkan for inference, MI50s are legacy, move on. Don't question it.

I questioned it. Repeatedly. On multiple fronts. And I was right every time.

What I was told:

Across multiple AI assistants over the past several months, the answer to anything involving MI50s and modern tooling ranged from "not supported" to "you'll need to upgrade your GPUs." Training on Vulkan specifically was described as architecturally impossible - the backward pass infrastructure doesn't exist outside ROCm and CUDA, full stop.

ChatGPT's position as of today, while my training run is literally executing on the card: it wants proof. Sure. Let's go through it in order.

Step 1: Establishing why i do this: Forcing ROCm 6.4.3 to work on MI50s

AMD dropped gfx906 from ROCm 6.x. Their official position is that the MI50 is end-of-life and you should migrate to supported hardware. Great suggestion if you didn't just acquire two of them specifically because 64GB of HBM2 at that price is hard to argue with.

What actually prevents gfx906 from working in ROCm 6.4.x is the TensileLibrary - the precompiled kernel library ROCm uses for BLAS operations. AMD just doesn't ship gfx906 kernels in the new versions. The GPU itself is fine. The compute capability is there. AMD just decided not to include it.

So I stuffed the gfx906 tensors back in. Pulled the missing kernel files, patched them into ROCm 6.4.3, and both MI50s came up fully recognized. llama.cpp runs on it natively. PyTorch sees both cards. ROCm 6.4.3 on hardware AMD said it doesn't support, because the hardware doesn't actually care what AMD's support matrix says.

Basically I don't care what something was designed to do, I care about what it can do

Step 2: Forcing vLLM to work on gfx906

vLLM is one of the faster inference engines around and I wanted it running on my cards. The problem: vLLM's gfx906 support is basically nonexistent upstream. It went like this:

- Tried ROCm 7.x with vLLM. Got it working briefly, then ROCm 7.14 hit AMD bug #5653 - "register fat binary failed" - and the whole thing fell over. Abandoned that path.

- Tried the nlzy fork of vLLM compiled against PyTorch 2.9.0+rocm6.3. Both MI50s detected. Blocked at runtime by a flash-attn V1 engine dependency that doesn't exist for gfx906.

- Eventually landed on a Docker image (aiinfos/vllm-gfx906-mobydick) - ROCm 6.3.4, PyTorch 2.11, flash-attn pre-compiled for gfx906. Single GPU inference confirmed working.

The remaining blocker for dual-GPU tensor parallel is a PCIe topology issue - my two cards are behind different root complexes (one on the CPU, one on the chipset), so NCCL all-reduce init fails. That gets fixed when a PLX switch arrives. Not a software problem, not a "your hardware isn't supported" problem - a physical PCIe lane routing problem with a known hardware solution.

Step 3: Building a Vulkan training stack from scratch

This is the one ChatGPT says is impossible right now.

Environment setup:

Vulkan was already working on the MI50s because llama.cpp uses it for inference, so that part wasn't a question. What didn't exist was any training framework that speaks Vulkan.

We verified the environment:

vulkaninfo --summary # GPU0: AMD Instinct MI50/MI60 (RADV VEGA20) - Vulkan 1.4.335 # GPU1: AMD Instinct MI50/MI60 (RADV VEGA20) - Vulkan 1.4.335 # GPU2: iGPU (Renoir)