Package 'llamaR'

Description

Returns an ellmer Chat object backed by a local GGUF model, so the whole ellmer / ragnar toolchain (turns, tools, streaming, structured output, ragnar_register_tool_retrieve(), …) works against local inference. Transport is the OpenAI-compatible HTTP API from llama_serve_openai; this function is a thin chat_vllm wrapper over it. (We use the vLLM provider because it speaks /v1/chat/completions — the de-facto standard our server implements — whereas ellmer's chat_openai/ chat_openai_compatible target OpenAI's newer /v1/responses.)

Usage

chat_llamar(
  model_path = NULL,
  base_url = NULL,
  port = 11434L,
  n_ctx = 4096L,
  n_gpu_layers = -1L,
  model_id = NULL,
  system_prompt = NULL,
  timeout = 180,
  ...
)

Arguments

Details

Two modes, picked by which argument you pass (DBI-style — like DBI::dbConnect() accepting either connection parameters or a ready connection):

base_url

Connect to a server you already started (e.g. llama_serve_openai() in another process, or a worker pool). No process is spawned.

model_path

Spin up llama_serve_openai() in a background R process (via callr), wait for it to come up, and return a Chat pointed at it. The server process's lifetime is tied to the returned object: when it is garbage-collected (or R exits), the process is killed. Stop it eagerly with chat_llamar_stop.

Exactly one of base_url or model_path must be supplied.

Value

An ellmer Chat object. In mode A it additionally carries the background process handle (see chat_llamar_stop).

Concurrency

The server is single-sequence (one request at a time); see llama_serve_openai. For parallel sessions, run a pool of servers on different ports and create one chat_llamar(base_url=) per worker.

Tool calls

Tool calling and structured output are mediated by the OpenAI protocol, so they work only as far as the server implements them. The current server does not emit tool_calls yet (see TODO), so ellmer tools registered on the returned chat will not be invoked by the model.

See Also

[llama_serve_openai], [chat_llamar_stop]

Examples

## Not run: 
# Mode A: spawn a server for this model and chat with it.
chat <- chat_llamar(model_path = "model.gguf")
chat$chat("Why is the sky blue?")
chat_llamar_stop(chat)            # or let GC do it

# Mode B: connect to a server you already run.
llama_serve_openai("model.gguf", port = 11434L)   # in another process
chat <- chat_llamar(base_url = "http://127.0.0.1:11434/v1")
chat$chat("Hello!")

## End(Not run)

Description

Kills the background llama_serve_openai process that chat_llamar started in mode A. A no-op for chats created in mode B (base_url=), which own no process. Safe to call more than once.

Usage

chat_llamar_stop(chat)

Arguments

Value

Invisibly TRUE if a process was killed, FALSE otherwise.

See Also

[chat_llamar]

Description

Computes embeddings using a local GGUF model. When called without x, returns a function suitable for passing to ragnar_store_create(embed = ...).

Usage

embed_llamar(
  x,
  model,
  n_gpu_layers = 0L,
  n_ctx = 512L,
  n_threads = parallel::detectCores(),
  embedding = FALSE,
  normalize = TRUE
)

Arguments

Value

• If x is missing or NULL: a function function(x) that returns a list of numeric vectors (one per input string), suitable for ragnar.

• If x is a character vector: a numeric matrix with nrow = length(x) and ncol = n_embd.

• If x is a data.frame: the same data.frame with an added embedding column (list of numeric vectors).

Examples

## Not run: 
# --- Partial application for ragnar ---
store <- ragnar_store_create(
  "my_store",
  embed = embed_llamar(model = "embedding-model.gguf", n_gpu_layers = -1)
)

# --- Direct use with path ---
mat <- embed_llamar(c("hello", "world"), model = "embedding-model.gguf")

# --- Direct use with pre-loaded model ---
mdl <- llama_load_model("embedding-model.gguf", n_gpu_layers = -1)
mat <- embed_llamar(c("hello", "world"), model = mdl)

## End(Not run)

Description

Returns a data.frame of all compute devices (CPU, GPU, etc.) detected by the ggml backend. Use device names from this list in the devices parameter of llama_load_model.

Usage

llama_backend_devices()

Value

A data.frame with columns name, description, and type (one of "cpu", "gpu", "igpu", "accel").

Examples

# List available compute devices and pick GPU names for llama_load_model()
devs <- llama_backend_devices()
print(devs)
gpu_names <- devs$name[devs$type == "GPU"]

Description

Free a llama batch allocated with llama_batch_init()

Usage

llama_batch_free(batch)

Arguments

Value

NULL invisibly.

Examples

## Not run: 
batch <- llama_batch_init(512L)
llama_batch_free(batch)

## End(Not run)

Description

Allocates a llama_batch that can hold up to n_tokens tokens. Use llama_batch_free() to release the memory when done.

Usage

llama_batch_init(n_tokens, embd = 0L, n_seq_max = 1L)

Arguments

Value

An external pointer to the allocated batch.

Examples

## Not run: 
batch <- llama_batch_init(512L)
llama_batch_free(batch)

## End(Not run)

Description

Formats a conversation using the specified chat template. This is essential for instruct/chat models to work correctly.

Usage

llama_chat_apply_template(
  messages,
  template = NULL,
  add_generation_prompt = TRUE
)

Arguments

Value

A character scalar containing the formatted prompt string, ready to be passed to llama_generate.

Examples

## Not run: 
model <- llama_load_model("llama-3.2-instruct.gguf")
tmpl <- llama_chat_template(model)

messages <- list(
  list(role = "system", content = "You are a helpful assistant."),
  list(role = "user", content = "What is R?")
)

prompt <- llama_chat_apply_template(messages, template = tmpl)
cat(prompt)

ctx <- llama_new_context(model)
response <- llama_generate(ctx, prompt)

## End(Not run)

Description

Returns a character vector of all chat template names supported by llama.cpp.

Usage

llama_chat_builtin_templates()

Value

A character vector of built-in template names.

Examples

# See which chat template formats are supported out of the box
templates <- llama_chat_builtin_templates()
head(templates)

Description

Returns the chat template string embedded in the model file, if any. Common templates include ChatML, Llama, Mistral, etc.

Usage

llama_chat_template(model, name = NULL)

Arguments

Value

A character scalar with the chat template string, or NULL if the model does not contain a built-in template.

Examples

## Not run: 
model <- llama_load_model("llama-3.2-instruct.gguf")
tmpl <- llama_chat_template(model)
cat(tmpl)

## End(Not run)

Description

Detokenize token IDs back to text

Usage

llama_detokenize(ctx, tokens)

Arguments

Value

A character scalar containing the decoded text.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
ctx <- llama_new_context(model)

# Round-trip: text -> tokens -> text
original <- "Hello, world!"
tokens <- llama_tokenize(ctx, original, add_special = FALSE)
restored <- llama_detokenize(ctx, tokens)
identical(original, restored)  # TRUE

## End(Not run)

Description

Computes embeddings for a character vector of texts in a single decode pass using per-sequence pooling. This is more efficient than calling llama_embeddings in a loop when embedding many texts.

Usage

llama_embed_batch(ctx, texts)

Arguments

Details

Requires a model that supports pooled embeddings (e.g. embedding models like nomic-embed, bge, etc.). The context must have enough capacity for the total number of tokens across all texts. Causal attention is automatically disabled during computation.

Value

A numeric matrix with nrow = length(texts) and ncol = n_embd.

Examples

## Not run: 
model <- llama_load_model("embedding-model.gguf")
ctx <- llama_new_context(model, n_ctx = 2048L)
llama_set_causal_attn(ctx, FALSE)

mat <- llama_embed_batch(ctx, c("hello world", "foo bar", "test"))
# mat is a 3 x n_embd matrix

## End(Not run)

Description

Runs the model in embeddings mode and returns the hidden-state vector of the last token. Note: meaningful only for models that support embeddings.

Usage

llama_embeddings(ctx, text)

Arguments

Value

A numeric vector of length n_embd (the model's embedding dimension) containing the hidden-state representation of the input text.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
ctx <- llama_new_context(model)

emb1 <- llama_embeddings(ctx, "Hello world")
emb2 <- llama_embeddings(ctx, "Hi there")

# Cosine similarity
similarity <- sum(emb1 * emb2) / (sqrt(sum(emb1^2)) * sqrt(sum(emb2^2)))
cat("Similarity:", similarity, "\n")

## End(Not run)

Description

Runs the encoder pass for encoder-decoder architectures (e.g. T5, BART). The encoder output is stored internally and used by subsequent decoder calls.

Usage

llama_encode(ctx, tokens)

Arguments

Value

Integer return code (0 = success, negative = error).

Examples

## Not run: 
model  <- llama_load_model("t5-model.gguf")
ctx    <- llama_new_context(model)
tokens <- llama_tokenize(ctx, "Hello world")
llama_encode(ctx, tokens)

## End(Not run)

Description

Free an inference context

Usage

llama_free_context(ctx)

Arguments

Value

No return value, called for side effects. Releases the memory associated with the inference context.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
ctx <- llama_new_context(model)
# ... use context ...
llama_free_context(ctx)

## End(Not run)

Description

Free a loaded model

Usage

llama_free_model(model)

Arguments

Value

No return value, called for side effects. Releases the memory associated with the model.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
# ... use model ...
llama_free_model(model)

## End(Not run)

Description

Sets up sampling and prefills the prompt, returning an opaque state handle that is pulled one chunk at a time with [llama_gen_next]. This is the streaming counterpart to [llama_generate]: same sampler chain and the same output for a given seed, but text arrives incrementally so it can be pushed into an SSE stream as it is produced.

Usage

llama_gen_begin(
  ctx,
  prompt,
  max_new_tokens = 256L,
  temp = 0.8,
  top_k = 50L,
  top_p = 0.9,
  seed = 42L,
  min_p = 0,
  typical_p = 1,
  repeat_penalty = 1,
  repeat_last_n = 64L,
  frequency_penalty = 0,
  presence_penalty = 0,
  mirostat = 0L,
  mirostat_tau = 5,
  mirostat_eta = 0.1,
  grammar = NULL
)

Arguments

Details

Typical loop:

st <- llama_gen_begin(ctx, prompt)
repeat {
  chunk <- llama_gen_next(st)
  if (is.null(chunk)) break
  cat(chunk)
}
cat(llama_gen_end(st))  # flush any held-back trailing bytes

Only one streaming generation may be active per context at a time: each call to llama_gen_begin clears the context KV cache.

Value

An external pointer holding the generation state. Pass it to [llama_gen_next] and [llama_gen_end]. The underlying sampler is freed automatically by the garbage collector.

See Also

[llama_gen_next], [llama_gen_end], [llama_generate]

Description

Marks the generation done and returns any bytes still held in the internal UTF-8 carry buffer (the tail of an unfinished character, if generation stopped mid-character). Concatenating every [llama_gen_next] chunk followed by the llama_gen_end result reproduces the full [llama_generate] output for the same seed and parameters. Safe to call more than once.

Usage

llama_gen_end(state)

Arguments

Value

A length-1 UTF-8 character vector with any remaining buffered text (often "").

See Also

[llama_gen_begin], [llama_gen_next]

Description

Advances a generation started with [llama_gen_begin] by one token and returns the next chunk of decoded text. A possibly-incomplete trailing UTF-8 character is held back until enough bytes arrive, so every returned chunk is valid UTF-8 (the chunk may be "" when the only new byte is part of an unfinished character).

Usage

llama_gen_next(state)

Arguments

Value

A length-1 UTF-8 character vector with the next chunk, or NULL when generation has finished (end-of-generation token reached or max_new_tokens exhausted). After NULL, call [llama_gen_end] to flush any remaining bytes.

See Also

[llama_gen_begin], [llama_gen_end]

Description

Tokenizes the prompt, runs the full autoregressive decode loop with sampling, and returns the generated text (excluding the original prompt).

Usage

llama_generate(
  ctx,
  prompt,
  max_new_tokens = 256L,
  temp = 0.8,
  top_k = 50L,
  top_p = 0.9,
  seed = 42L,
  min_p = 0,
  typical_p = 1,
  repeat_penalty = 1,
  repeat_last_n = 64L,
  frequency_penalty = 0,
  presence_penalty = 0,
  mirostat = 0L,
  mirostat_tau = 5,
  mirostat_eta = 0.1,
  grammar = NULL,
  with_timings = FALSE
)

Arguments

Value

A character scalar containing the generated text (excluding the original prompt).

Examples

## Not run: 
model <- llama_load_model("model.gguf", n_gpu_layers = -1L)
ctx <- llama_new_context(model, n_ctx = 2048L)

# Basic generation
result <- llama_generate(ctx, "Once upon a time")
cat(result)

# Greedy decoding (deterministic)
result <- llama_generate(ctx, "The answer is", temp = 0)

# More creative output
result <- llama_generate(ctx, "Write a poem about R:",
                         max_new_tokens = 100L,
                         temp = 1.0, top_p = 0.95)

# With repetition penalty
result <- llama_generate(ctx, "List items:",
                         repeat_penalty = 1.1, repeat_last_n = 64L)

# JSON output with grammar
result <- llama_generate(ctx, "Output JSON:",
                         grammar = 'root ::= "{" "}" ')

## End(Not run)

Description

Runs continuous batching: all prompts share the same decode loop, so each iteration dispatches one matmul over all still-running sequences. This converts decode from memory-bound vector ops into compute-bound matrix ops on the GPU and lifts throughput compared to calling llama_generate in a loop.

Usage

llama_generate_batch(
  ctx,
  prompts,
  max_new_tokens = 256L,
  temp = 0.8,
  top_k = 50L,
  top_p = 0.9,
  seed = 42L,
  min_p = 0,
  typical_p = 1,
  repeat_penalty = 1,
  repeat_last_n = 64L,
  frequency_penalty = 0,
  presence_penalty = 0,
  grammar = NULL
)

Arguments

Details

The context must be created with n_seq_max >= length(prompts) and n_ctx large enough to hold every prompt plus its generated tokens simultaneously. As a rule of thumb: n_ctx >= sum(prompt_lengths) + length(prompts) * max_new_tokens.

Each sequence gets its own sampler chain seeded with seed + seq_index, so identical prompts still produce diverse outputs at temp > 0 (useful for self-consistency sampling). Sampler hyperparameters are shared across sequences in this version.

Stop conditions per sequence: end-of-generation token (model-defined) or max_new_tokens reached. Mirostat and with_timings are not supported here yet — use llama_generate for those.

Value

A list of length length(prompts), in the same order as the input. Each element is a list with fields:

• text: character scalar with the generated text

• n_tokens: integer count of tokens generated

• finished_reason: "eos" or "max_tokens"

Examples

## Not run: 
model <- llama_load_model("model.gguf", n_gpu_layers = -1L)
# 4 parallel sequences, up to 256 new tokens each
ctx <- llama_new_context(model, n_ctx = 4096L, n_seq_max = 4L,
                         flash_attn = "on")

# Batch classification
prompts <- c("Classify: 'great movie' as positive/negative.",
             "Classify: 'awful service' as positive/negative.",
             "Classify: 'just okay' as positive/negative.",
             "Classify: 'loved every minute' as positive/negative.")
out <- llama_generate_batch(ctx, prompts, max_new_tokens = 16L, temp = 0)
vapply(out, `[[`, character(1), "text")

# Self-consistency sampling: same prompt repeated
samples <- llama_generate_batch(ctx, rep("2 + 2 =", 4L),
                                max_new_tokens = 8L, temp = 0.7)

## End(Not run)

Description

Returns a matrix of shape n_outputs × n_embd containing the raw embedding vectors for all tokens whose logits flag was set in the batch. Only works when pooling_type == "none" (generative models or embedding contexts without pooling). For pooled embeddings use [llama_get_embeddings_seq].

Usage

llama_get_embeddings(ctx, n_outputs)

Arguments

Value

A numeric matrix with n_outputs rows and n_embd columns.

Description

Returns the embedding vector for a specific token position after a decode call with embeddings enabled. Negative indices count from the end (-1 = last token).

Usage

llama_get_embeddings_ith(ctx, i)

Arguments

Value

A numeric vector of length n_embd.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
ctx <- llama_new_context(model)
llama_generate(ctx, "Hello world", max_new_tokens = 1L)

# Get the embedding of the last decoded token
emb <- llama_get_embeddings_ith(ctx, -1L)
cat("Embedding dim:", length(emb), "\n")

## End(Not run)

Description

Returns the pooled embedding vector for a given sequence ID after a batch decode. Only works when the model supports pooling (embedding models).

Usage

llama_get_embeddings_seq(ctx, seq_id)

Arguments

Value

A numeric vector of length n_embd.

Examples

## Not run: 
# Get pooled embedding for sequence 0 (requires embedding context)
model <- llama_load_model("nomic-embed.gguf")
ctx <- llama_new_context(model, embedding = TRUE)
mat <- llama_embed_batch(ctx, "Hello world")
emb <- llama_get_embeddings_seq(ctx, 0L)
cat("Pooled embedding dim:", length(emb), "\n")

## End(Not run)

Description

Returns the raw logit vector (unnormalized log-probabilities) from the last token position after a decode operation.

Usage

llama_get_logits(ctx)

Arguments

Value

A numeric vector of length n_vocab containing the logits.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
ctx <- llama_new_context(model)
result <- llama_generate(ctx, "The capital of France is", max_new_tokens = 1L)
logits <- llama_get_logits(ctx)
# Find top token
top_id <- which.max(logits)

## End(Not run)

Description

Returns the logit vector for token at index i in the last decoded batch. Use i = -1 to get the logits for the last token.

Usage

llama_get_logits_ith(ctx, i)

Arguments

Value

A numeric vector of length n_vocab.

Description

Returns the model handle that was used to create this context. The returned object is the same R external pointer that was passed to [llama_new_context] — no new allocation occurs.

Usage

llama_get_model(ctx)

Arguments

Value

A model handle (external pointer), equivalent to the original handle returned by [llama_load_model].

Description

Get current verbosity level

Usage

llama_get_verbosity()

Value

An integer scalar indicating the current verbosity level (0 = silent, 1 = errors only, 2 = normal, 3 = verbose).

Examples

# Save current level, suppress output, then restore
old <- llama_get_verbosity()
llama_set_verbosity(0)
# ... noisy operations ...
llama_set_verbosity(old)

Description

Removes cached model files. Can clear the entire cache or only files from a specific repository.

Usage

llama_hf_cache_clear(repo_id = NULL, confirm = TRUE, cache_dir = NULL)

Arguments

Value

Invisible NULL. Called for its side effect of deleting cached files.

Examples

llama_hf_cache_clear(confirm = FALSE)

Description

Returns the path to the directory where models downloaded from Hugging Face are cached. The directory is created if it does not exist.

Usage

llama_hf_cache_dir()

Value

A character string containing the absolute path to the cache directory. The path follows the R user directory convention via R_user_dir.

Examples

llama_hf_cache_dir()

Description

Lists all cached model files with their sizes and download metadata.

Usage

llama_hf_cache_info(cache_dir = NULL)

Arguments

Value

A data frame with columns:

repo_id

Character. The Hugging Face repository identifier.

filename

Character. The model file name.

size

Numeric. File size in bytes.

size_pretty

Character. Human-readable file size.

path

Character. Absolute path to the cached file.

downloaded_at

Character. Timestamp of when the file was downloaded.

Returns an empty data frame with the same columns if the cache is empty.

Examples

llama_hf_cache_info()

Description

Downloads a GGUF model file from a Hugging Face repository. Files are cached locally so subsequent calls return the cached path without re-downloading.

Usage

llama_hf_download(
  repo_id,
  filename = NULL,
  pattern = NULL,
  tag = NULL,
  token = NULL,
  cache_dir = NULL,
  revision = "main",
  force = FALSE
)

Arguments

Details

Exactly one of filename, pattern, or tag must be specified to identify which file to download.

Value

A character string containing the absolute path to the downloaded (or cached) GGUF model file.

Examples

## Not run: 
path <- llama_hf_download("TheBloke/Llama-2-7B-GGUF",
                          pattern = "*q2_k*")
print(path)

## End(Not run)

Description

Queries the Hugging Face API for GGUF model files in the specified repository. Returns a data frame with file names, sizes, and detected quantization levels.

Usage

llama_hf_list(repo_id, token = NULL, pattern = NULL)

Arguments

Value

A data frame with columns:

filename

Character. The file name within the repository.

size

Numeric. File size in bytes.

size_pretty

Character. Human-readable file size.

quant

Character. Detected quantization level (e.g. "Q4_K_M") or NA if not detected.

Examples

files <- llama_hf_list("TheBloke/Llama-2-7B-GGUF")
print(files)

Description

Load a GGUF model file

Usage

llama_load_model(
  path,
  n_gpu_layers = -1L,
  devices = NULL,
  split_mode = "layer",
  use_mmap = TRUE,
  use_mlock = FALSE
)

Arguments

Value

An external pointer (class externalptr) wrapping the loaded model. This handle is required by llama_new_context, llama_model_info, and other model-level functions. Freed automatically by the garbage collector or manually via llama_free_model.

Examples

## Not run: 
# Default: full GPU offload (falls back to CPU if no GPU)
model <- llama_load_model("model.gguf")

# Force CPU-only
model <- llama_load_model("model.gguf", n_gpu_layers = 0L)

# Explicit CPU-only backend
model <- llama_load_model("model.gguf", devices = "cpu")

# Specific GPU device (see llama_backend_devices())
model <- llama_load_model("model.gguf", n_gpu_layers = -1L, devices = "Vulkan0")

# Multi-GPU: use two devices
model <- llama_load_model("model.gguf", n_gpu_layers = -1L,
                          devices = c("Vulkan0", "Vulkan1"))

## End(Not run)

Description

Convenience function that downloads a GGUF model from Hugging Face (if not already cached) and loads it via llama_load_model.

Usage

llama_load_model_hf(repo_id, ..., n_gpu_layers = 0L)

Arguments

Value

An external pointer to the loaded model, as returned by llama_load_model.

Examples

## Not run: 
model <- llama_load_model_hf("TheBloke/Llama-2-7B-GGUF",
                              pattern = "*q2_k*")

## End(Not run)

Description

Activates a loaded LoRA adapter for the given context. Multiple LoRA adapters can be applied simultaneously.

Usage

llama_lora_apply(ctx, lora, scale = 1)

Arguments

Value

No return value, called for side effects. Activates the LoRA adapter for the given context.

Examples

## Not run: 
model <- llama_load_model("base-model.gguf")
lora <- llama_lora_load(model, "adapter.gguf")
ctx <- llama_new_context(model)

# Apply with full strength
llama_lora_apply(ctx, lora, scale = 1.0)

# Or apply with reduced effect
llama_lora_apply(ctx, lora, scale = 0.5)

## End(Not run)

Description

Deactivates all LoRA adapters from the context, returning to base model behavior.

Usage

llama_lora_clear(ctx)

Arguments

Value

No return value, called for side effects. Removes all active LoRA adapters from the context.

Examples

## Not run: 
# Apply multiple LoRAs
llama_lora_apply(ctx, lora1)
llama_lora_apply(ctx, lora2)

# Remove all at once
llama_lora_clear(ctx)

## End(Not run)

Description

Loads a LoRA (Low-Rank Adaptation) adapter file that can be applied to modify the model's behavior without changing the base weights.

Usage

llama_lora_load(model, path)

Arguments

Value

An external pointer (class externalptr) wrapping the loaded LoRA (Low-Rank Adaptation) adapter. Pass this handle to llama_lora_apply to activate the adapter.

Examples

## Not run: 
model <- llama_load_model("base-model.gguf")
lora <- llama_lora_load(model, "fine-tuned-adapter.gguf")

ctx <- llama_new_context(model)
llama_lora_apply(ctx, lora, scale = 1.0)

# Now generation uses the LoRA-modified model
result <- llama_generate(ctx, "Hello")

## End(Not run)

Description

Deactivates a specific LoRA adapter from the context.

Usage

llama_lora_remove(ctx, lora)

Arguments

Value

An integer scalar: 0 on success, -1 if the adapter was not applied to this context.

Examples

## Not run: 
# Remove a specific adapter while keeping others active
llama_lora_remove(ctx, lora)
result <- llama_generate(ctx, "Without adapter: ", max_new_tokens = 20L)

## End(Not run)

Description

Get maximum number of devices

Usage

llama_max_devices()

Value

An integer scalar: the maximum number of compute devices available.

Examples

# Query the maximum number of devices supported by the backend
n <- llama_max_devices()
cat("Max devices:", n, "\n")

Description

Prints a debug summary of how model weights are distributed across compute devices (CPU, GPU layers). Useful for diagnosing memory allocation with partial GPU offload.

Usage

llama_memory_breakdown_print(ctx)

Arguments

Value

No return value, called for side effects.

Description

Check if the KV cache supports shifting

Usage

llama_memory_can_shift(ctx)

Arguments

Value

A logical scalar: TRUE if the memory supports position shifting.

Examples

## Not run: 
if (llama_memory_can_shift(ctx)) {
  message("Context shifting is supported")
}

## End(Not run)

Description

Removes all tokens from the KV cache. Call this before starting a new generation from scratch.

Usage

llama_memory_clear(ctx)

Arguments

Value

No return value, called for side effects.

Examples

## Not run: 
# Clear the KV cache to start a fresh conversation
llama_memory_clear(ctx)
result <- llama_generate(ctx, "New topic: ", max_new_tokens = 50L)

## End(Not run)

Description

Adds a position delta to all tokens in the given sequence within [p0, p1). This is useful for implementing context shifting (sliding window).

Usage

llama_memory_seq_add(ctx, seq_id, p0, p1, delta)

Arguments

Value

No return value, called for side effects.

Examples

## Not run: 
# Shift positions left by 100 for context window management
llama_memory_seq_add(ctx, seq_id = 0L, p0 = 100L, p1 = -1L, delta = -100L)

## End(Not run)

Description

Copies cached tokens from one sequence to another in the position range [p0, p1).

Usage

llama_memory_seq_cp(ctx, seq_id_src, seq_id_dst, p0 = -1L, p1 = -1L)

Arguments

Value

No return value, called for side effects.

Examples

## Not run: 
# Copy sequence 0 to sequence 1
llama_memory_seq_cp(ctx, seq_id_src = 0L, seq_id_dst = 1L,
                    p0 = -1L, p1 = -1L)

## End(Not run)

Description

Divides all token positions in the range [p0, p1) for the given sequence by d. Use p0 = -1 and p1 = -1 for the full range. Useful for implementing sliding-window context compression.

Usage

llama_memory_seq_div(ctx, seq_id, p0, p1, d)

Arguments

Value

No return value, called for side effects.

Description

Removes all sequences except the specified one from the KV cache.

Usage

llama_memory_seq_keep(ctx, seq_id)

Arguments

Value

No return value, called for side effects.

Examples

## Not run: 
llama_memory_seq_keep(ctx, seq_id = 0L)

## End(Not run)

Description

Returns the minimum and maximum token positions for a given sequence in the KV cache.

Usage

llama_memory_seq_pos_range(ctx, seq_id)

Arguments

Value

A named integer vector with elements min and max.

Examples

## Not run: 
range <- llama_memory_seq_pos_range(ctx, seq_id = 0L)
cat("Positions:", range["min"], "to", range["max"], "\n")

## End(Not run)

Description

Removes cached tokens for the given sequence in the position range [p0, p1). Use p0 = -1 and p1 = -1 to remove all tokens for the sequence.

Usage

llama_memory_seq_rm(ctx, seq_id, p0 = -1L, p1 = -1L)

Arguments

Value

A logical scalar: TRUE if tokens were successfully removed.

Examples

## Not run: 
# Remove all tokens from sequence 0
llama_memory_seq_rm(ctx, seq_id = 0L, p0 = -1L, p1 = -1L)

## End(Not run)

Description

Get model metadata

Usage

llama_model_info(model)

Arguments

Value

A named list with fields: - 'n_ctx_train': context size the model was trained with - 'n_embd': embedding dimension - 'n_vocab': vocabulary size - 'n_layer': number of layers - 'n_head': number of attention heads - 'n_head_kv': number of key-value attention heads (GQA) - 'desc': human-readable model description string - 'size': model size in bytes - 'n_params': number of parameters - 'has_encoder': whether the model has an encoder - 'has_decoder': whether the model has a decoder - 'is_recurrent': whether the model is recurrent (e.g. Mamba)

Examples

## Not run: 
model <- llama_load_model("model.gguf")
info <- llama_model_info(model)
cat("Model:", info$desc, "\n")
cat("Layers:", info$n_layer, "\n")
cat("Context:", info$n_ctx_train, "\n")
cat("Size:", info$size / 1e9, "GB\n")

## End(Not run)

Description

Returns all key-value metadata pairs stored in the GGUF model file.

Usage

llama_model_meta(model)

Arguments

Value

A named character vector where names are metadata keys and values are the corresponding metadata values.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
meta <- llama_model_meta(model)
print(meta)

## End(Not run)

Description

Get a single model metadata value by key

Usage

llama_model_meta_val(model, key)

Arguments

Value

A character scalar with the metadata value, or NULL if the key does not exist.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
llama_model_meta_val(model, "general.name")
llama_model_meta_val(model, "general.architecture")

## End(Not run)

Description

Get logical batch size

Usage

llama_n_batch(ctx)

Arguments

Value

An integer scalar: the logical batch size (max tokens per 'llama_decode' call).

Description

Get context window size

Usage

llama_n_ctx(ctx)

Arguments

Value

An integer scalar: the context window size (number of tokens).

Examples

## Not run: 
model <- llama_load_model("model.gguf")
ctx <- llama_new_context(model, n_ctx = 4096L)
llama_n_ctx(ctx)  # 4096

## End(Not run)

Description

Get per-sequence context window size

Usage

llama_n_ctx_seq(ctx)

Arguments

Value

An integer scalar: maximum context size per sequence.

Description

Get maximum number of sequences

Usage

llama_n_seq_max(ctx)

Arguments

Value

An integer scalar: maximum number of concurrent sequences.

Description

Get number of threads for single-token generation

Usage

llama_n_threads(ctx)

Arguments

Value

An integer scalar: current thread count for generation.

Description

Get number of threads for batch processing

Usage

llama_n_threads_batch(ctx)

Arguments

Value

An integer scalar: current thread count for prompt encoding.

Description

Get physical micro-batch size

Usage

llama_n_ubatch(ctx)

Arguments

Value

An integer scalar: the physical micro-batch size.

Description

Create an inference context

Usage

llama_new_context(
  model,
  n_ctx = 2048L,
  n_threads = NULL,
  n_threads_batch = NULL,
  n_batch = 2048L,
  n_ubatch = 512L,
  n_seq_max = 1L,
  flash_attn = "auto",
  embedding = FALSE
)

Arguments

Value

An external pointer (class externalptr) wrapping the inference context. This handle is required by generation, tokenization, and embedding functions. Freed automatically by the garbage collector or manually via llama_free_context.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
ctx <- llama_new_context(model, n_ctx = 4096L, n_threads = 8L)
# ... use context for generation ...
llama_free_context(ctx)
llama_free_model(model)

# Tune for GPU prefill throughput
ctx <- llama_new_context(model, n_ctx = 4096L,
                         n_ubatch = 2048L, flash_attn = "on")

# Embedding mode
emb_ctx <- llama_new_context(model, n_ctx = 512L, embedding = TRUE)
mat <- llama_embed_batch(emb_ctx, c("hello", "world"))

## End(Not run)

Description

Call once for better performance on NUMA systems.

Usage

llama_numa_init(strategy = "disabled")

Arguments

Value

No return value, called for side effects.

Examples

## Not run: 
# On multi-socket servers, distribute memory across NUMA nodes
# for better memory bandwidth during inference
llama_numa_init("distribute")

# Call before loading any models — affects all subsequent allocations
model <- llama_load_model("model.gguf", n_gpu_layers = 0L)

## End(Not run)

Description

Returns timing and count statistics for the current context, including prompt processing time, token generation time, and counts.

Usage

llama_perf(ctx)

Arguments

Value

A named list with fields: - 't_load_ms': model load time in milliseconds - 't_p_eval_ms': prompt processing time in milliseconds - 't_eval_ms': token generation time in milliseconds - 'n_p_eval': number of prompt tokens processed - 'n_eval': number of tokens generated - 'n_reused': number of reused compute graphs

Examples

## Not run: 
result <- llama_generate(ctx, "Hello world")
perf <- llama_perf(ctx)
cat("Prompt speed:", perf$n_p_eval / (perf$t_p_eval_ms / 1000), "tok/s\n")
cat("Generation speed:", perf$n_eval / (perf$t_eval_ms / 1000), "tok/s\n")

## End(Not run)

Description

Prints a formatted summary of timing and throughput statistics for the context (load time, prompt processing speed, generation speed). Output goes to the R console via the llama.cpp logging callback.

Usage

llama_perf_print(ctx)

Arguments

Value

No return value, called for side effects.

Description

Resets the timing and token count statistics for the context.

Usage

llama_perf_reset(ctx)

Arguments

Value

No return value, called for side effects.

Examples

## Not run: 
# Reset counters before benchmarking a specific generation
llama_perf_reset(ctx)
result <- llama_generate(ctx, "Benchmark prompt", max_new_tokens = 100L)
perf <- llama_perf(ctx)
cat("Generation:", perf$n_eval / (perf$t_eval_ms / 1000), "tok/s\n")

## End(Not run)

Description

Get pooling type

Usage

llama_pooling_type(ctx)

Arguments

Value

A character string: one of '"none"', '"mean"', '"cls"', '"last"', '"rank"', '"unspecified"'.

Description

Loads a GGUF model once and exposes it over an OpenAI-compatible HTTP API so any OpenAI client (OpenCode, ellmer, the 'openai' Python SDK, …) can talk to it. Implements 'GET /v1/models' and 'POST /v1/chat/completions' (both blocking and 'stream = true'). The HTTP/SSE layer is provided by drogonR; generation runs through llamaR's streaming API (llama_gen_begin / llama_gen_next / llama_gen_end).

Usage

llama_serve_openai(
  model_path,
  port = 11434L,
  n_ctx = 4096L,
  n_gpu_layers = -1L,
  model_id = NULL,
  host = "127.0.0.1",
  template = NULL,
  max_tokens = 512L,
  ...
)

Arguments

Details

The server is single-sequence: requests are handled one at a time on the main R thread (each streamed token is one event-loop pump). This is meant for a single local user/agent, not concurrent load.

drogonR is an optional dependency (Suggests); install it with install.packages("drogonR") (or from its repository) before calling this function.

Value

Invisibly NULL. Blocks serving until drogonR::dr_stop() is called (typically from another process or an interrupt).

See Also

[llama_gen_begin], [llama_generate]

Examples

## Not run: 
llama_serve_openai("model.gguf", port = 11434L)
# In another shell, point any OpenAI client at
#   http://127.0.0.1:11434/v1
# e.g. GET /v1/models and POST /v1/chat/completions

## End(Not run)

Description

Registers an R function that is called periodically during generation. If the function returns 'TRUE', the current decode operation is aborted. Pass 'NULL' to remove the callback.

Usage

llama_set_abort_callback(ctx, fn)

Arguments

Details

Note: only one callback is active globally — setting a new one replaces the previous one across all contexts.

Value

No return value, called for side effects.

Examples

## Not run: 
# Abort after 2 seconds
deadline <- Sys.time() + 2
llama_set_abort_callback(ctx, function() Sys.time() > deadline)
result <- llama_generate(ctx, "Tell me a long story", max_new_tokens = 500L)
llama_set_abort_callback(ctx, NULL)

## End(Not run)

Description

When disabled, the model uses full (bidirectional) attention. This is useful for embedding models.

Usage

llama_set_causal_attn(ctx, causal)

Arguments

Value

No return value, called for side effects.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
ctx <- llama_new_context(model)
llama_set_causal_attn(ctx, FALSE)  # for embeddings

## End(Not run)

Description

Set the number of threads for a context

Usage

llama_set_threads(ctx, n_threads, n_threads_batch = n_threads)

Arguments

Value

No return value, called for side effects.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
ctx <- llama_new_context(model)
llama_set_threads(ctx, n_threads = 8L)

## End(Not run)

Description

Controls how much diagnostic output is printed during model loading and inference.

Usage

llama_set_verbosity(level)

Arguments

Value

No return value, called for side effects. Sets the global verbosity level used by the underlying 'llama.cpp' library.

Examples

# Suppress all output
llama_set_verbosity(0)

# Show only errors
llama_set_verbosity(1)

# Verbose output for debugging
llama_set_verbosity(3)

Description

When 'warmup = TRUE', the context runs in warmup mode which pre-caches model weights in GPU memory without producing meaningful outputs. Call with 'warmup = FALSE' to return to normal inference mode.

Usage

llama_set_warmup(ctx, warmup)

Arguments

Value

No return value, called for side effects.

Description

Returns the number of bytes required to serialize the current context state (KV cache + sampling state). Use before allocating a buffer for raw state I/O.

Usage

llama_state_get_size(ctx)

Arguments

Value

A numeric scalar (size in bytes).

Description

Restores a previously saved context state (including KV cache).

Usage

llama_state_load(ctx, path)

Arguments

Value

A logical scalar: TRUE on success (errors on failure).

Examples

## Not run: 
llama_state_load(ctx, "state.bin")
# Continue generation from saved state
result <- llama_generate(ctx, "")

## End(Not run)

Description

Saves the full context state (including KV cache) to a binary file. This allows resuming generation later from the exact same state.

Usage

llama_state_save(ctx, path)

Arguments

Value

A logical scalar: TRUE on success (errors on failure).

Examples

## Not run: 
llama_state_save(ctx, "state.bin")

## End(Not run)

Description

Returns 'TRUE' if at least one GPU backend (e.g. Vulkan) was detected at runtime. Use the result to decide whether to pass 'n_gpu_layers != 0' to [llama_load_model].

Usage

llama_supports_gpu()

Value

A logical scalar: TRUE if at least one GPU backend (e.g. Vulkan) is available, FALSE otherwise.

Examples

if (llama_supports_gpu()) {
  message("GPU available, will use Vulkan backend")
} else {
  message("GPU not available, using CPU only")
}

Description

Check whether memory locking is supported

Usage

llama_supports_mlock()

Value

A logical scalar: TRUE if mlock is supported.

Examples

# Check if memory locking is available (prevents swapping model to disk)
if (llama_supports_mlock()) {
  message("mlock available — model weights can be pinned in RAM")
}

Description

Check whether memory-mapped file I/O is supported

Usage

llama_supports_mmap()

Value

A logical scalar: TRUE if mmap is supported.

Examples

# Check memory-mapping support before loading large models
if (llama_supports_mmap()) {
  message("mmap available — large models will load faster")
}

Description

Check whether RPC backend is available

Usage

llama_supports_rpc()

Value

A logical scalar: 'TRUE' if the RPC backend is compiled in.

Description

Blocks until all pending GPU/async operations for this context are complete. Normally not needed — 'llama_decode' and 'llama_generate' are synchronous — but useful when using low-level batch APIs in async mode.

Usage

llama_synchronize(ctx)

Arguments

Value

No return value, called for side effects.

Description

Returns a string with information about the system capabilities detected by llama.cpp (SIMD support, etc.).

Usage

llama_system_info()

Value

A character scalar with system capability information.

Examples

cat(llama_system_info(), "\n")

Description

Get current time in microseconds

Usage

llama_time_us()

Value

A numeric scalar with the current time in microseconds.

Examples

# Measure elapsed time for an operation
t0 <- llama_time_us()
Sys.sleep(0.01)
elapsed_ms <- (llama_time_us() - t0) / 1000
cat("Elapsed:", round(elapsed_ms, 1), "ms\n")

Description

Convert a single token ID to its text piece

Usage

llama_token_to_piece(ctx, token, special = FALSE)

Arguments

Value

A character string — the text piece for the token.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
ctx   <- llama_new_context(model)

# Inspect individual tokens from tokenizer output
tokens <- llama_tokenize(ctx, "Hello world")
pieces <- vapply(tokens, function(t) llama_token_to_piece(ctx, t), "")
cat(paste(pieces, collapse = "|"), "\n")

## End(Not run)

Description

Tokenize text into token IDs

Usage

llama_tokenize(ctx, text, add_special = TRUE, parse_special = FALSE)

Arguments

Value

An integer vector of token IDs as used by the model's vocabulary.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
ctx <- llama_new_context(model)

tokens <- llama_tokenize(ctx, "Hello, world!")
print(tokens)
# [1] 1 15043 29892 3186 29991

# Without special tokens
tokens <- llama_tokenize(ctx, "Hello", add_special = FALSE)

# Parse a templated prompt's role markers as control tokens
prompt <- llama_chat_apply_template(list(list(role = "user", content = "hi")))
tokens <- llama_tokenize(ctx, prompt, parse_special = TRUE)

## End(Not run)

Description

Returns the log-probability score stored in the vocabulary (used by SPM/UGM tokenizers).

Usage

llama_vocab_get_score(model, token)

Arguments

Value

A numeric scalar.

Description

Returns the raw text string stored in the vocabulary for a given token ID. Unlike [llama_token_to_piece], this does not apply any special rendering — it returns exactly what is stored in the GGUF vocabulary table.

Usage

llama_vocab_get_text(model, token)

Arguments

Value

A character string, or 'NULL' if the token has no text entry.

Description

Returns the token IDs for special tokens (BOS, EOS, etc.) and fill-in-middle (FIM) tokens used by the model's vocabulary. A value of -1 indicates the token is not defined.

Usage

llama_vocab_info(model)

Arguments

Value

A named integer vector with token IDs for: bos, eos, eot, sep, nl, pad, fim_pre, fim_suf, fim_mid, fim_rep, fim_sep. A value of -1 means the token is not defined by the model.

Examples

## Not run: 
model <- llama_load_model("model.gguf")
vocab <- llama_vocab_info(model)
cat("BOS token:", vocab["bos"], "\n")
cat("EOS token:", vocab["eos"], "\n")

## End(Not run)

Description

Check if a token is a control token

Usage

llama_vocab_is_control(model, token)

Arguments

Value

A logical scalar.

Description

Returns 'TRUE' for EOS, EOT, and other tokens that signal end of output. Useful for implementing custom generation loops.

Usage

llama_vocab_is_eog(model, token)

Arguments

Value

A logical scalar.

Description

Get vocabulary type

Usage

llama_vocab_type(model)

Arguments

Value

A character string: one of '"spm"' (LLaMA/SentencePiece BPE), '"bpe"' (GPT-2 BPE), '"wpm"' (BERT WordPiece), '"ugm"' (T5 Unigram), '"rwkv"', '"plamo2"', or '"none"'.