Inference Engineering for Dummies

A former SWE transitioned into inference engineering, launching a successful side hustle now full-time due to high demand. This field offers significant opportunities as fewer people optimize runtime compared to those building apps or AI solutions. The provided signal demonstrates a complex command for running llama-server with various parameters, including --flash-attn on, --batch-size 1024, --gpu-layers 150, and --cache-ram 8192, highlighting advanced configuration for inference optimization. The author is open to community feedback and data-based revisions.

This report uniquely details a personal transition from SWE to full-time inference engineering, unlike typical reports focusing solely on technical aspects.