Experiment 001 — Large MoE on constrained hardware

EXPERIMENT DETAILS

PROBLEM

Can 35B MoE model run practically on 12GB VRAM GPU? The goal is to find the optimal configuration that allows for stable performance without exceeding hardware limitations.

HYPOTHESIS

MoE architecture may provide better usability than dense models because active parameters are smaller, even if total model size is large.

EXPERIMENT

Testing the model layer placement configuration using the expert splitting method on the CPU.

WHAT FAILED

Several configurations hit memory limits, bandwidth bottlenecks, or unstable thermal behavior.

ITERATION

The backend adjustment (ROCm) for AMD must be native. Otherwise, prefill speeds are unacceptable. This involves optimizing KV cache management and adjusting the scheduler. But need GPU optimization too for better performance.

EVIDENCE

Measured decode speed, prefill speed, memory usage, and stability across multiple test runs.

DECODE: up to 28 tok/s
CONTEXT: 128K
PREFILL: up to 581 tok/s
MEMORY PRESSURE: Manageable
OUT OF MEMORY: Never
USER EXPERIENCE: losless
JUNCTION GPU THERMAL: Max 85
EDGE GPU THERMAL: Max 60
VRAM GPU THERMAL: Max 70
STABILITY: Stable

RESULT

The model achieved stable performance with acceptable latency, proving that constrained hardware can handle large MoE models with proper optimization.

WHAT I LEARNED

MoE models can be viable on constrained hardware when quantization and memory management are properly tuned. The key is finding the right balance between model size, quantization level, and context length.

PRACTICAL IMPLICATION

This approach can inform deployment decisions on constrained environments, showing that large models don't always require expensive hardware if properly optimized.

CONFIGURATION TEMPLATE

~/Path build llama.cpp

MODEL="path model" (model 35B A3B) \
  --host 0.0.0.0 \
  --port 8080 \
  --n-gpu-layers 99 \
  --threads 4 \
  --threads-batch 4 \
  --n-cpu-moe 26 \
  --moe-cache auto \
  --ctx-size 131072 \
  --batch-size 1024 \
  --ubatch-size 512 \
  --keep 8000 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --swa-checkpoints 16 \
  --checkpoint-min-step 2048 \
  --embd-normalize 0 \
  --no-kv-unified \
  --kv-offload \
  --jinja \
  --flash-attn on \
  --parallel 1 \
  --cache-ram 8192 \
  --cache-idle-slots \
  --temp 0.6 \
  --top-k 0 \
  --top-p 0.95 \
  --min-p 0 \
  --repeat-penalty 1 \
  --repeat-last-n 0 \
  --alias coding \
  --log-verbosity 4

Do not copy this configuration blindly; it must be adjusted to suit the specific characteristics of your model and hardware. This configuration serves only as a reference, as my choices were based on specific reasons that I will explain below.

I chose --n-gpu-layers 99 as a baseline to offload the entire model to the GPU.

I chose --threads 4 because testing showed this to be my "sweet spot." This setting configures CPU threads to assist with token generation; the optimal value cannot simply be guessed but must be determined through testing based on your CPU's thread count. I happen to use an i5-11400F (6 cores, 12 threads), and 4 threads worked best for me.

I chose --threads-batch 4 because my tests showed that higher values increased prefill latency. This happens because too many CPU threads get involved in the prefill process—a task that should ideally rely more on the GPU to maintain reasonable speeds. However, using fewer threads isn't always faster either; bottlenecks can occur because parts of the model's "expert layers" are offloaded to the CPU. Also, monitor the GPU junction temperature: if prefill feels fast but the junction temperature is high, increase this value.

I chose --n-cpu-moe 26 to offload 26 experts to the CPU for optimal performance on my hardware. This value isn't universal; it varies depending on the specific model and its quantization.

Regarding batch and ubatch: these values shouldn't be chosen arbitrarily, as they affect compatibility with kernels and the inference engine. Higher numbers don't necessarily yield faster results; in fact, they often lead to anomalies—such as unreasonable GPU junction temperatures or even slower prefill speeds.

I selected f16 for KV cache compression to maintain memory consistency for the model. Of course, this reduces the number of model layers that can fit into the GPU, but I chose f16 because, personally, I find a model losing its context far worse than slow inference speeds.

I chose --no-kv-unified and --parallel 1 because the model is used only by me in a single inference session. If multiple users were accessing the same model running on your PC, you would need to enable --kv-unified and increase the --parallel value.

The remaining configurations I haven't covered are standard settings that you can find in any other post.

"FOR THOSE OF YOU UNABLE TO READ THIS DATA FROM TECHNICAL STANDPOINT, HERE IS THE CONCLUSION"

This experiment demonstrated that a 35-billion-parameter AI model can be run practically on a consumer GPU with only 12GB of VRAM.

It wasn't achieved by adding more hardware. It was achieved by understanding the system, identifying its bottlenecks, and carefully balancing what the CPU, GPU, memory, and inference engine were each responsible for.

The important finding isn't that a 35B model can run on a 12GB GPU. It's that hardware limitations can sometimes be worked around through better system understanding and optimization.

The result was a stable configuration that could run the model continuously without running out of memory, while still delivering practical inference performance.

The takeaway: when resources are limited, the answer isn't always to buy more resources. Sometimes, the better answer is to understand what you already have and make it work harder.

Can 35B model usable
run practically
on 12GB VRAM GPU?

Decode Speed With --n-cpu-moe Value Configuration

Prompt Processing Speed With Total Tokens Processing

Can 35B model usable run practically on 12GB VRAM GPU?

Decode Speed With --n-cpu-moe Value Configuration

Prompt Processing Speed With Total Tokens Processing

Can 35B model usable
run practically
on 12GB VRAM GPU?