Importance 2 of 5Covered by 1 source
Open-source tool tunes cudaMemcpy to speed MoE CPU offload prefill by about 35%
A developer released a tool that tunes CUDA memory copies used when offloading mixture-of-experts layers to the CPU. On an RTX 3090 with Qwen3.6 35B-A3B in llama.cpp, prefill rose from 345 to 466 tokens per second.
Why it matters
Cheap software tuning can noticeably improve local inference on consumer GPUs for models too large for VRAM.
Timeline: llama.cpp, Qwen3.6 35B-A3B
- Newer update Developer tool speeds pageable cudaMemcpy transfers about 5x on Nvidia B300 Nov 15, 2026
Earlier coverage