Multi-GPU: Splits, Replicas and Benchmarking

1. The one rule that matters

If the model fits in one GPU, run independent replicas — one per GPU. Split the model only when it does not fit.

Splitting a model does not divide the work; it serializes it. Each layer’s output must reach the next layer, which lives on another card, so every token pays for cross-device transfers. On the Vulkan backend those transfers go through host RAM at roughly 1 GB/s. Meanwhile the GPUs mostly wait for each other: a 4-way split does not run four times faster, it runs slower than one card.

Measured on 4× Tesla P100 and 4× V100 (Qwen2.5-1.5B-Instruct Q4_K_M, decode throughput, median of three 128-token runs):

Read the first and last rows together. One GPU does 419.7 t/s. Four GPUs split do 130 — worse than one. Four GPUs running independent replicas do 975. The split is not a speed-up mechanism; it is a way to fit a model that otherwise would not load.

2. Data parallelism: replicas

DP is not an argument to llama_load_model(). It is separate processes, each with its own model and context, each pinned to its own GPU. Throughput is the sum of what the replicas deliver.

Pin one replica to one card with devices and turn splitting off:

model <- llama_load_model("model.gguf", n_gpu_layers = -1L,
                          devices = "Vulkan0", split_mode = "none")
ctx <- llama_new_context(model, n_ctx = 2048L)

Run several such processes concurrently — one per GPU. They share nothing, so there is no cross-device traffic at all. That is precisely why DP wins.

Separate processes, not threads: Vulkan and ggml keep per-process singleton state, so two replicas inside one R session would fight over it.

3. When the model does not fit: layer vs row

Now the split is not a choice but a necessity — a 30B+ model will not load on a 16 GB card. Two strategies, and the difference is how much data crosses between devices:

• split_mode = "layer" (pipeline): layers 1–16 on GPU 0, 17–32 on GPU 1. One activation handoff per forward pass.
• split_mode = "row" (tensor): every weight matrix is cut by rows across the cards. An all-reduce after every layer.

# Pipeline: fewest cross-device copies
model <- llama_load_model("big-model.gguf", n_gpu_layers = -1L,
                          devices = c("Vulkan0", "Vulkan1"),
                          split_mode = "layer")

# Tensor: more parallelism per token, far more traffic
model <- llama_load_model("big-model.gguf", n_gpu_layers = -1L,
                          devices = c("Vulkan0", "Vulkan1"),
                          split_mode = "row")

The benchmark above shows them within noise of each other on 2 GPUs. On a slow interconnect prefer layer: it pays the transfer once per pass instead of once per layer. Use row when the interconnect is fast, or when a single layer is itself too large for one card.

Note the default is split_mode = "layer", so on a multi-GPU host a model gets split whether or not you asked. If it fits in one card, say so explicitly:

model <- llama_load_model("model.gguf", devices = "Vulkan0", split_mode = "none")

On the Vulkan backend the default split can hang. Pinning to one GPU is both faster and safer when the model fits.

4. The hybrid: TP × DP

With four GPUs you can run two row-split replicas — GPUs {0,1} and {2,3} — concurrently. Each replica splits a model too big for one card; the two replicas never talk to each other.

This is the mode to reach for when the model needs two cards and you have four. It is worth 306 t/s on the P100 host against 130 for a plain 4-way split — but still far below 975 for four independent replicas, which the model was simply too large to allow.

# Replica A, in its own process:
llama_load_model("big.gguf", devices = c("Vulkan0", "Vulkan1"), split_mode = "row")
# Replica B, in another process:
llama_load_model("big.gguf", devices = c("Vulkan2", "Vulkan3"), split_mode = "row")

5. Measuring it yourself

Do not trust the tables above for your model and your hardware. llamaR ships the benchmark that produced them:

system.file("examples", "bench_pp_tp_dp.sh", package = "llamaR")

# auto-detects GPU count; TP_SIZE=2 for the TPxDP row
bash bench_pp_tp_dp.sh model.gguf

# 8 GPUs as TP=4 x DP=2, 128 tokens per rep, 3 reps
bash bench_pp_tp_dp.sh model.gguf 8 4 128 3

It sweeps baseline, PP and TP at 2, 4, … GPUs, then TP×DP, then pure DP — launching the DP replicas concurrently in background processes, because that is the only way a DP number means anything. Each config runs in its own process (bench_replica.R), which prints one parseable line:

RESULT dp0 devices=Vulkan0 split=none decode_tps=419.7 prefill_ms=31.2 total_ms=305.1

For DP rows, throughput is the sum of the concurrent replicas’ decode_tps; single-context rows are read directly.

If you are on a multi-GPU host, build ggmlR with --enable-hard-exit. Without it each of the benchmark’s processes may segfault at exit — after printing its results, so the numbers survive, but the sweep output becomes noisy. See vignette("multi-gpu", package = "ggmlR").

6. Serving

llama_serve_anthropic() takes the same split_mode, and the same rule applies — pin to one GPU unless the model does not fit:

llama_serve_anthropic("model.gguf", port = 11435L, split_mode = "none")

For a model that needs several cards:

llama_serve_anthropic("big-model.gguf", port = 11435L, split_mode = "row")

The server reports the model it loaded at /v1/models; model_id overrides the name it advertises (it defaults to the GGUF’s file name). Serving several replicas means several servers on different ports, with a load balancer in front — the server process itself holds exactly one model.

7. Summary

Splitting is for fitting, replicas are for throughput. Reach for split_mode only when llama_load_model() on a single card runs out of VRAM, prefer "layer" over "row" on a slow interconnect, and always measure with bench_pp_tp_dp.sh before believing any of it.

See also vignette("getting-started"), and vignette("multi-gpu", package = "ggmlR") for the tensor-level primitives underneath.