| Title: | 'GGML' Tensor Operations for Machine Learning |
|---|---|
| Description: | Provides 'R' bindings to the 'GGML' tensor library for machine learning, optimized for 'Vulkan' GPU acceleration with a transparent CPU fallback. The package features a 'Keras'-like sequential API and a 'PyTorch'-style 'autograd' engine for building, training, and deploying neural networks. Key capabilities include high-performance 5D tensor operations, 'f16' precision, and efficient quantization. It supports native 'ONNX' model import (50+ operators) and 'GGUF' weight loading from the 'llama.cpp' and 'Hugging Face' ecosystems. Designed for zero-overhead inference via dedicated weight buffering, it integrates seamlessly as a 'parsnip' engine for 'tidymodels' and provides first-class learners for the 'mlr3' framework. See <https://github.com/ggml-org/ggml> for more information about the underlying library. |
| Authors: | Yuri Baramykov [aut, cre] (ORCID: <https://orcid.org/0009-0000-7627-4217>), Georgi Gerganov [ctb, cph] (Author of the GGML library), Jeffrey Quesnelle [ctb, cph] (Contributor to ops.cpp), Bowen Peng [ctb, cph] (Contributor to ops.cpp), Mozilla Foundation [ctb, cph] (Author of llamafile/sgemm.cpp) |
| Maintainer: | Yuri Baramykov <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.8.8 |
| Built: | 2026-10-18 11:34:18 UTC |
| Source: | https://github.com/zabis13/ggmlr |
Computes A + B. If B is [m, 1] and A is
[m, n], B is broadcast across columns (useful for bias
vectors).
ag_add(A, B)ag_add(A, B)
A |
ag_tensor or numeric matrix |
B |
ag_tensor or numeric matrix (may be |
ag_tensor
Index of the largest entry in every column of x, for action selection
over a [n_actions, batch] matrix of logits or log-probabilities.
ag_argmax(x)ag_argmax(x)
x |
ag_tensor or numeric matrix, |
Ties go to the FIRST index on every path (R, CPU kernel, Vulkan shader),
matching which.max. NaN is never selected; a column with nothing
above -Inf (all NaN / all -Inf) gives 0.
Not differentiable. On the GPU this reads the result back to the host, so it is a synchronisation point: fine where the environment needs the action, but keep it out of the training step.
Integer vector of length ncol(x), 0-based.
ag_argmax(matrix(c(1, 3, 3, 5, 2, 5), 3, 2)) # 1 0ag_argmax(matrix(c(1, 3, 3, 5, 2, 5), 3, 2)) # 1 0
Normalises each feature (row) over the batch dimension.
Learnable scale gamma [F,1] and shift beta [F,1].
ag_batch_norm(num_features, eps = 1e-05, momentum = 0.1)ag_batch_norm(num_features, eps = 1e-05, momentum = 0.1)
num_features |
Number of features (rows of input) |
eps |
Numerical stability constant (default 1e-5) |
momentum |
Running-stats momentum (default 0.1) |
Training mode: use batch statistics; update running mean/var. Eval mode: use stored running statistics.
An ag_batch_norm environment
bn <- ag_batch_norm(16L) x <- ag_tensor(matrix(rnorm(16 * 32), 16, 32)) out <- bn$forward(x)bn <- ag_batch_norm(16L) x <- ag_tensor(matrix(rnorm(16 * 32), 16, 32)) out <- bn$forward(x)
A matrix of rows x capacity kept on the compute device without a
gradient: written in place by column ranges with ag_buffer_write()
and read by column indices with ag_get_rows() inside ag_*
computations, including ag_capture() recordings. Typical use
is a replay buffer: one column per transition.
ag_buffer(rows, capacity, dtype = "f32") ag_buffer_write(buf, data, col_offset) ag_buffer_read(buf, col_offset = 0, k = buf$capacity - col_offset) ag_buffer_free(buf)ag_buffer(rows, capacity, dtype = "f32") ag_buffer_write(buf, data, col_offset) ag_buffer_read(buf, col_offset = 0, k = buf$capacity - col_offset) ag_buffer_free(buf)
rows |
Number of rows (values per column). |
capacity |
Number of columns. |
dtype |
Storage type. Only |
buf |
An |
data |
Numeric matrix with |
col_offset |
0-based index of the first column written or read. |
k |
Number of columns to read. |
On the CPU the buffer is an R matrix and the same calls work on it. On the
GPU the memory is allocated when the buffer is created (or first used on the
device) and zero-filled; creation fails with an error when the buffer needs
more than 85% of the free memory the device reports (driver estimates are
approximate). A device switch (ag_device("cpu")) downloads the value
to the host in 16 MB parts before the backend is released (the host copy is
an R double matrix, about twice the buffer's device size), and the switch
fails with an error if that download does; the next use on the GPU uploads
it again, and recordings that read the buffer are then made anew.
Writes are synchronous: when ag_buffer_write() returns, the data is on
the device, and any graph computed afterwards, including a captured replay,
reads it. Graph nodes queued before the write are computed first, so they
read the old value.
ag_buffer_write() writes columns col_offset to
col_offset + ncol(data) - 1; the range must lie inside the buffer
(a ring buffer wraps with two writes).
ag_buffer_read() returns columns col_offset to
col_offset + k - 1 as a rows x k matrix; reading a large
buffer in parts bounds the host memory used.
ag_buffer_free() releases the memory at once instead of at
garbage collection; the buffer cannot be used afterwards.
An object of class ag_buffer with fields rows,
capacity and dtype.
buf <- ag_buffer(3, 10) ag_buffer_write(buf, matrix(1:6, 3, 2), col_offset = 4) ag_buffer_read(buf, 4, 2) as.matrix(ag_get_rows(buf, c(5, 4, 5))) ag_buffer_free(buf)buf <- ag_buffer(3, 10) ag_buffer_write(buf, matrix(1:6, 3, 2), col_offset = 4) ag_buffer_read(buf, 4, 2) as.matrix(ag_get_rows(buf, c(5, 4, 5))) ag_buffer_free(buf)
Wraps fn so that, on the GPU, its ag_* operations are recorded once
per input shape and every later call runs the recorded ggml graph: upload the
inputs, one graph compute, download the outputs – no R-level op dispatch.
On the CPU the wrapper simply calls fn, so the same code runs on both.
ag_capture(fn, params = NULL)ag_capture(fn, params = NULL)
fn |
Function of named ag_tensor arguments returning an ag_tensor or a
named list of ag_tensors. Forward computations only (no
|
params |
Optional list of ag_tensors (typically the model parameters)
and |
fn must depend on the data only through its arguments. Anything it
computes in R from data, or uploads itself (ag_tensor(f(x))), would be
frozen into the graph at the first call; such uploads, any download and any
eager launch during the recording are errors. Weights must be resident on
the device (e.g. trained or used on the GPU) before the first call.
The recording is redone when the input shapes, compute dtype or matmul
precision change, when the persistent device pool is reset (device switch),
and when a tensor in params no longer points at the device memory it
had at recording time.
A function taking the same named arguments as numeric matrices (a vector is one column) and returning a matrix, or a named list of matrices.
W <- ag_param(matrix(rnorm(6), 2, 3)) f <- ag_capture(function(x) ag_relu(ag_matmul(W, x)), params = list(W)) f(x = matrix(1, 3, 4))W <- ag_param(matrix(rnorm(6), 2, 3)) f <- ag_capture(function(x) ag_relu(ag_matmul(W, x)), params = list(W)) f(x = matrix(1, 3, 4))
Frees every recording made by ag_capture()'s function f. The
function stays usable: the next call records again. Recordings are also
released when the persistent device pool is reset (device switch).
ag_capture_free(f)ag_capture_free(f)
f |
A function returned by |
f, invisibly.
Settings are inputs of the recording, read on every call, so changing one does not record again.
ag_capture_set(f, max_grad_norm)ag_capture_set(f, max_grad_norm)
f |
A function returned by |
max_grad_norm |
A positive number: the new gradient-clipping limit. The
step must have been created with a |
f, invisibly.
Like ag_capture, for a training step: the forward pass in
fn, its backward pass and the Adam update are recorded once, and every
later call is upload the inputs, two graph computes, download the outputs –
no R-level op dispatch. On the CPU the wrapper runs the same step as ordinary
code (zero_grad, tape, backward, step).
ag_capture_step(fn, params = NULL, optimizer, max_grad_norm = NULL)ag_capture_step(fn, params = NULL, optimizer, max_grad_norm = NULL)
fn |
Function of named ag_tensor arguments returning the loss (an
ag_tensor), or a named list of ag_tensors with an element |
params |
Optional list of further ag_tensors the recording references
(e.g. frozen weights) and |
optimizer |
An optimizer from |
max_grad_norm |
|
Two graphs, not one: forward with backward, then the update. Adam writes each weight in place, and a weight is also read inside the first graph (by the forward, and by the backward of the layer below) through tensors that share its memory but are not connected to the update in the graph – so a single graph would leave the order of that read and that write to the backend.
fn is called WITHOUT with_grad_tape: the wrapper records the
tape itself. Everything ag_capture requires of fn applies: data
only through the arguments, no reading values back, weights on the device.
The backward must be one that runs as a graph (the ops the graph backward
supports).
Input names and shapes are fixed by the first call; a call with other shapes
is an error (pad a short last minibatch, or run it uncaptured). The step
counter and the learning rate are read from optimizer on every call,
so learning-rate schedules work, and optimizer$state() stays valid.
Outputs are the values of the forward pass, before the update, as in an
ordinary training loop. After a captured step $grad is not set.
Supported: optimizer_adam without gradient accumulation, F32
compute dtype.
Gradient clipping by global norm (max_grad_norm): the norm of all
gradients is computed in the first graph and every gradient is multiplied by
min(1, max_grad_norm / (norm + 1e-6)) before the update. Branch-free,
unlike clip_grad_norm: for a norm within about 1e-6 below
max_grad_norm the factor is slightly under 1 instead of exactly 1.
The pre-clipping norm is returned as an extra output grad_norm. A
non-finite norm is an error raised BEFORE the update: weights, moments and
the step counter are left as they were. The moments accumulate clipped
gradients, so optimizer state saved with and without clipping is not
interchangeable on resume. Change the limit between calls with
ag_capture_set (no re-recording).
A function taking the same named arguments as numeric matrices and returning the loss as a matrix, or a named list of matrices.
ag_capture, ag_capture_set,
ag_capture_free
W <- ag_param(matrix(rnorm(6, sd = 0.1), 2, 3)) opt <- optimizer_adam(list(W), lr = 1e-2) train <- ag_capture_step(function(x, y) ag_mse_loss(ag_matmul(W, x), y), optimizer = opt) for (i in 1:5) loss <- train(x = matrix(1, 3, 4), y = matrix(0, 2, 4))W <- ag_param(matrix(rnorm(6, sd = 0.1), 2, 3)) opt <- optimizer_adam(list(W), lr = 1e-2) train <- ag_capture_step(function(x, y) ag_mse_loss(ag_matmul(W, x), y), optimizer = opt) for (i in 1:5) loss <- train(x = matrix(1, 3, 4), y = matrix(0, 2, 4))
Executes fn(...) with the gradient tape switched off, so none of the
intermediate activations inside fn are kept. During
backward() the segment is run again – with the same RNG state
– to rebuild what the gradient rules need.
ag_checkpoint(fn, ...)ag_checkpoint(fn, ...)
fn |
A function taking the tensors in One caveat, because it bites in the most natural usage: |
... |
Input |
This trades compute for memory: the segment's forward pass runs twice, and in exchange the tape never holds its activations. Use it on the deep middle of a network, not on cheap layers.
An ag_tensor holding the segment's output. Its gradient flows
back into both ... and any parameters used inside fn.
block <- ag_linear(8L, 8L, activation = "relu") x <- ag_tensor(matrix(runif(8 * 4), 8, 4)) y <- ag_tensor(matrix(runif(8 * 4), 8, 4)) with_grad_tape({ h <- ag_checkpoint(function(inp) block$forward(inp), x) loss <- ag_mse_loss(h, y) }) backward(loss) # Checkpointing every second block of a deeper stack. local() gives each # segment its own binding -- see the note on `fn` for why that matters. layers <- lapply(1:4, function(i) ag_linear(8L, 8L, activation = "relu")) with_grad_tape({ h <- x for (i in seq_along(layers)) { seg <- local({ lyr <- layers[[i]] function(inp) lyr$forward(inp) }) h <- if (i %% 2L == 0L) ag_checkpoint(seg, h) else seg(h) } loss <- ag_mse_loss(h, y) }) backward(loss)block <- ag_linear(8L, 8L, activation = "relu") x <- ag_tensor(matrix(runif(8 * 4), 8, 4)) y <- ag_tensor(matrix(runif(8 * 4), 8, 4)) with_grad_tape({ h <- ag_checkpoint(function(inp) block$forward(inp), x) loss <- ag_mse_loss(h, y) }) backward(loss) # Checkpointing every second block of a deeper stack. local() gives each # segment its own binding -- see the note on `fn` for why that matters. layers <- lapply(1:4, function(i) ag_linear(8L, 8L, activation = "relu")) with_grad_tape({ h <- x for (i in seq_along(layers)) { seg <- local({ lyr <- layers[[i]] function(inp) lyr$forward(inp) }) h <- if (i %% 2L == 0L) ag_checkpoint(seg, h) else seg(h) } loss <- ag_mse_loss(h, y) }) backward(loss)
Clamps values to [lo, hi]. Gradient is 1 inside the interval,
0 at the boundary (straight-through estimator).
ag_clamp(x, lo = -Inf, hi = Inf)ag_clamp(x, lo = -Inf, hi = Inf)
x |
ag_tensor |
lo |
Lower bound (default |
hi |
Upper bound (default |
ag_tensor
Stacks a above b, like rbind(a, b): both must have the
same number of columns (the batch). Typical use is a critic input built from
observations and an action computed by the actor. The gradient of a
is rows 1..nrow(a) of the incoming gradient and that of b the
rest; an operand without gradient gets none.
ag_concat_rows(a, b)ag_concat_rows(a, b)
a, b
|
ag_tensors (or matrices) with the same number of columns |
ag_tensor of shape (nrow(a) + nrow(b)) x ncol(a)
a <- ag_tensor(matrix(1:6, 2, 3)); b <- ag_tensor(matrix(7:9, 1, 3)) as.matrix(ag_concat_rows(a, b))a <- ag_tensor(matrix(1:6, 2, 3)); b <- ag_tensor(matrix(7:9, 1, 3)) as.matrix(ag_concat_rows(a, b))
Generic CE: -sum(target * log(pred)) / batch_size.
The gradient w.r.t. pred is -target / pred / n.
Use ag_softmax_cross_entropy_loss() for the numerically stable
combined softmax + CE (fused gradient (p - y) / n).
ag_cross_entropy_loss(pred, target)ag_cross_entropy_loss(pred, target)
pred |
ag_tensor [classes, batch_size] probabilities (any, not just softmax) |
target |
matrix [classes, batch_size] one-hot (or soft) labels |
scalar ag_tensor
Returns an iterator environment. Each call to $next_batch() returns
a named list list(x, y) with ag_tensor objects of shape
[features, batch_size] / [labels, batch_size].
After the last batch, $has_next() returns FALSE; call
$reset() (or start a new epoch via $epoch()) to reshuffle
and restart.
ag_dataloader(x, y = NULL, batch_size = 32L, shuffle = TRUE, col_major = TRUE)ag_dataloader(x, y = NULL, batch_size = 32L, shuffle = TRUE, col_major = TRUE)
x |
Feature matrix |
y |
Label matrix with the same convention. |
batch_size |
Integer batch size. |
shuffle |
Logical; if |
col_major |
Logical; if |
An ag_dataloader environment
n <- 128L x <- matrix(runif(4 * n), 4, n) # [4, 128] col-major y <- matrix(runif(2 * n), 2, n) dl <- ag_dataloader(x, y, batch_size = 32L) dl$reset() while (dl$has_next()) { batch <- dl$next_batch() # batch$x: [4, 32], batch$y: [2, 32] }n <- 128L x <- matrix(runif(4 * n), 4, n) # [4, 128] col-major y <- matrix(runif(2 * n), 2, n) dl <- ag_dataloader(x, y, batch_size = 32L) dl$reset() while (dl$has_next()) { batch <- dl$next_batch() # batch$x: [4, 32], batch$y: [2, 32] }
Return the current default compute device
ag_default_device()ag_default_device()
"cpu" or "gpu"
Return the current default dtype for GPU operations
ag_default_dtype()ag_default_dtype()
"f32", "f16", or "bf16"
Switches all subsequent ag_tensor / ag_param operations to run
on the specified device. Calling ag_device("gpu") initialises the
best available ggml backend (Vulkan, Metal, CUDA, or CPU fallback) the first
time it is called.
ag_device(device)ag_device(device)
device |
|
Switching to the CPU moves every ag_buffer on the GPU
to the host first; plan for host memory of about twice the buffers' device
size (R stores the values as doubles).
Invisibly the previous device string
In training mode applies inverted dropout (random Bernoulli mask, scale by
1/(1-rate) to preserve expected values). In eval mode is identity.
ag_dropout(rate)ag_dropout(rate)
rate |
Drop probability in [0, 1) |
An ag_dropout environment
drop <- ag_dropout(0.5) x <- ag_tensor(matrix(runif(8), 4, 2)) out <- drop$forward(x) # training mode by default ag_eval(drop) out2 <- drop$forward(x) # identitydrop <- ag_dropout(0.5) x <- ag_tensor(matrix(runif(8), 4, 2)) out <- drop$forward(x) # training mode by default ag_eval(drop) out2 <- drop$forward(x) # identity
Controls the dtype used when uploading tensors to the ggml backend.
"bf16" halves memory usage vs "f32" with minimal accuracy loss.
Gradients are never computed in f16: the GPU graph backward runs in f32 only,
so with "f16" or "bf16" backward() takes the closure path,
which computes them in double – correct, but slower (the reason is counted
by ggmlR:::ag_backward_fallbacks()). Training in f32 keeps the graph path.
ag_dtype(dtype)ag_dtype(dtype)
dtype |
|
Invisibly the previous dtype string
Maps 0-based integer indices to dense vectors via table lookup.
Input: integer matrix or vector of 0-based indices.
Output: float tensor [dim, length(idx)].
ag_embedding(vocab_size, dim)ag_embedding(vocab_size, dim)
vocab_size |
Vocabulary size |
dim |
Embedding dimension |
Backward: scatter-add — only the looked-up rows accumulate gradient.
An ag_embedding environment
emb <- ag_embedding(100L, 16L) idx <- c(0L, 3L, 7L, 2L) out <- emb$forward(idx) # [16, 4]emb <- ag_embedding(100L, 16L) idx <- c(0L, 3L, 7L, 2L) out <- emb$forward(idx) # [16, 4]
Adds up the four things that are alive at the same time during training on
the autograd (ag_*) path: the weights, their gradients, the
optimizer's per-parameter state, and the activations held by the tape until
zero_grad() clears it.
ag_estimate_training_memory( shapes, batch_size = 32L, optimizer = c("adam", "sgd", "sgd_plain"), activation_frac = 0.01, quiet = FALSE )ag_estimate_training_memory( shapes, batch_size = 32L, optimizer = c("adam", "sgd", "sgd_plain"), activation_frac = 0.01, quiet = FALSE )
shapes |
List of parameter shapes, each a numeric vector of dimensions
(e.g. |
batch_size |
Batch size the activations are sized for. Activations scale with it; weights and optimizer state do not. |
optimizer |
One of |
activation_frac |
Activations as a fraction of the weight bytes, per
unit of batch. The default 0.01 reproduces the measured tape composition
for a dense stack ( |
quiet |
Suppress the printed report and return the figures only. Note
that the double-vs-f32 caveat is part of that report, so a caller passing
|
Everything is counted at 8 bytes per scalar, because that is what the path
actually uses: gradients are accumulated by R closures in double, and
optimizer moments are R matrices. ag_dtype affects only
forward tensors uploaded into a ggml buffer and does not reach either. An
estimate made in f32 would be half the real figure, so the returned list
carries a note saying so.
Invisibly, a list of byte counts: params, weights,
gradients, optimizer, activations, total, plus
bytes_per_scalar and note. Printed as a report.
ag_tape_memory measures a real tape instead of
estimating one; ggml_estimate_memory sizes a single tensor in
a ggml buffer, where dtype does apply.
# a 3-layer MLP, 512 wide, trained with Adam at batch 64 ag_estimate_training_memory( shapes = list(c(512, 512), c(512, 512), c(512, 512)), batch_size = 64)# a 3-layer MLP, 512 wide, trained with Adam at batch 64 ag_estimate_training_memory( shapes = list(c(512, 512), c(512, 512), c(512, 512)), batch_size = 64)
Switch a layer or sequential model to eval mode
ag_eval(model)ag_eval(model)
model |
An ag_sequential, ag_batch_norm, or ag_dropout layer |
The model/layer (invisibly)
Element-wise exponential
ag_exp(x)ag_exp(x)
x |
ag_tensor |
ag_tensor
Computes scaled dot-product attention over all heads at once with
ggml_flash_attn_ext(), instead of the per-head loop
ag_multihead_attention uses. Both the forward pass and its
gradient are one operation each, so the tape holds a single node rather than
the dozens a head loop records.
ag_flash_attention(q, k, v, n_heads, scale = NULL, mask = NULL, causal = FALSE)ag_flash_attention(q, k, v, n_heads, scale = NULL, mask = NULL, causal = FALSE)
q, k, v
|
|
n_heads |
Number of attention heads. Must divide |
scale |
Softmax scale. Defaults to |
mask |
Optional attention mask, |
causal |
|
q, k and v are [d_model, seq] matrices whose rows
are split into n_heads contiguous blocks – the same head layout
ag_multihead_attention() uses. Projections are not included: apply
W_q, W_k, W_v before the call and W_o after it.
An ag_tensor of shape [d_model, seq_q].
d_model <- 16L; seq_len <- 8L q <- ag_param(matrix(runif(d_model * seq_len, -1, 1), d_model, seq_len)) k <- ag_param(matrix(runif(d_model * seq_len, -1, 1), d_model, seq_len)) v <- ag_param(matrix(runif(d_model * seq_len, -1, 1), d_model, seq_len)) with_grad_tape({ out <- ag_flash_attention(q, k, v, n_heads = 4L) loss <- ag_mse_loss(out, ag_tensor(matrix(0, d_model, seq_len))) }) backward(loss)d_model <- 16L; seq_len <- 8L q <- ag_param(matrix(runif(d_model * seq_len, -1, 1), d_model, seq_len)) k <- ag_param(matrix(runif(d_model * seq_len, -1, 1), d_model, seq_len)) v <- ag_param(matrix(runif(d_model * seq_len, -1, 1), d_model, seq_len)) with_grad_tape({ out <- ag_flash_attention(q, k, v, n_heads = 4L) loss <- ag_mse_loss(out, ag_tensor(matrix(0, d_model, seq_len))) }) backward(loss)
out[1, j] = x[index[j] + 1, j]: the log-probability of the action
taken in sample j, for example. Differentiable in x; the gradient of
column j lands on the picked row only.
ag_gather(x, index)ag_gather(x, index)
x |
ag_tensor or numeric matrix, |
index |
Integer-valued vector of length |
Built as colSums(x * onehot(index)), so the graph backward covers it
with no rule of its own. The one-hot is [n, batch] and is uploaded on
every call: cheap for tens of classes (games), wasteful for thousands –
there a dedicated op with an I32 index tensor is the right tool (TODO.md).
ag_tensor [1, batch].
ag_gather(matrix(1:6 + 0, 3, 2), c(2L, 0L)) # 3 4ag_gather(matrix(1:6 + 0, 3, 2), c(2L, 0L)) # 3 4
Selects columns of an ag_buffer by 0-based index, as a device
operation: on the GPU only the indices cross the bus. The name follows
ggml_get_rows, which selects along the second ggml dimension (ne1);
in the R rows x capacity view that is a column.
ag_get_rows(buf, idx)ag_get_rows(buf, idx)
buf |
An |
idx |
0-based column indices: a numeric vector, or an ag_tensor with
one column (f32; indices are exact below |
The result is a rows x length(idx) ag_tensor without gradient, usable
as an input to further ag_* operations. Inside ag_capture()
pass idx as an argument of the captured function: the recording is
then reused for every index vector of the same length (the batch size is
part of the recording's shape), and the buffer must be listed in
params so that freeing it invalidates the recording.
Indices given as host values are checked against the buffer and an
index out of range is an error. An index tensor already on the device
(inside a recording) cannot be checked; on the device every index is
clamped to 0..capacity-1 before the read, so an index out of range
reads the first or last column instead of memory outside the buffer.
An ag_tensor of shape rows x length(idx).
After backward(), x$grad holds the gradient – as an R matrix
on the closure path, but as a device handle when the graph backward kept it
on the GPU (the default there). ag_grad() returns the value as a
matrix either way, so code that reads gradients does not depend on where
they live. In graph mode it is a synchronisation point.
ag_grad(x)ag_grad(x)
x |
An ag_tensor (usually a parameter). |
A numeric matrix, or NULL when x has no gradient.
w <- ag_param(matrix(1:4 + 0, 2, 2)) with_grad_tape({ loss <- ag_sum(ag_mul(w, w)) }) backward(loss) ag_grad(w) # 2 * ww <- ag_param(matrix(1:4 + 0, 2, 2)) with_grad_tape({ loss <- ag_sum(ag_mul(w, w)) }) backward(loss) ag_grad(w) # 2 * w
Compares analytical gradients (from backward()) with finite-difference
numerical gradients for all input tensors with requires_grad = TRUE.
ag_gradcheck( fn, inputs, eps = 1e-05, atol = 1e-04, verbose = FALSE, quiet = FALSE )ag_gradcheck( fn, inputs, eps = 1e-05, atol = 1e-04, verbose = FALSE, quiet = FALSE )
fn |
A function that takes a list of ag_tensor inputs and returns a
scalar ag_tensor loss (must be used inside |
inputs |
Named list of ag_tensor objects. Only those with
|
eps |
Finite-difference step size (default 1e-5). |
atol |
Absolute tolerance for pass/fail (default 1e-4). |
verbose |
Print per-element comparison (default FALSE). |
quiet |
Suppress per-parameter and overall status lines (default FALSE).
Useful when calling from |
Invisibly TRUE if all gradients match, FALSE otherwise.
When quiet = FALSE (default), prints a summary report.
W <- ag_param(matrix(runif(6), 2, 3)) x <- ag_tensor(matrix(runif(3), 3, 1)) ag_gradcheck( fn = function(ins) ag_mse_loss(ag_relu(ag_matmul(ins$W, ins$x)), matrix(0, 2, 1)), inputs = list(W = W, x = x) )W <- ag_param(matrix(runif(6), 2, 3)) x <- ag_tensor(matrix(runif(3), 3, 1)) ag_gradcheck( fn = function(ins) ag_mse_loss(ag_relu(ag_matmul(ins$W, ins$x)), matrix(0, 2, 1)), inputs = list(W = W, x = x) )
With graph mode on, a GPU ag_* operation that has a device-resident
path does not run at once: it adds its node to a queue and returns a pending
value. The queue runs as ONE ggml graph when a value is read on the host
(as.matrix(), a printed loss, an action index) or when
backward() needs it – the graph backward then joins the same graph.
Each separate launch costs on the order of a millisecond on the device, so a
small network (an RL policy, a few layers) is dominated by launches, and
fusing them is where the speed comes from.
ag_graph_mode(on, check = NULL)ag_graph_mode(on, check = NULL)
on |
|
check |
|
Results are the same as without it. Operations that have no resident path yet still run one by one; they first run whatever is queued, so the order of computation is always the data order.
When the queue runs (synchronisation points):
reading a value that is still queued: as.matrix(), printing,
a loss value, ag_argmax();
an operation without a resident path (it runs the queue first);
writing a parameter: an optimizer step, ag_load_state_dict();
the device optimizer step (it never joins the queue);
turning graph mode off.
backward() does NOT force it: the graph backward appends to the
queue, so forward and backward run as one graph at the next point above.
Off by default; turn it on around the code that wants it, restoring the
previous value on every exit – ag_local_mode does that. The one
thing it can break is code that orders graphs BY HAND rather than through
data – reading a buffer in one graph and overwriting it in a later one with
nothing linking the two. The package's own optimizer step does this and is
protected; code of your own that does it must run with graph mode off.
On the CPU it has no effect.
Errors. A failure inside a queued operation surfaces where the queue
runs, not where the operation was called. The error names the queued
operations; check = TRUE (or GGMLR_AG_GRAPH_CHECK=1) runs each
operation as soon as it is queued, so the error appears at its own call –
for debugging, since it gives up the fusion.
The current setting when on is missing; otherwise the previous
setting, invisibly. Turning it off runs anything still queued.
ag_local_mode, ag_matmul_precision
old <- ag_graph_mode(TRUE) ag_graph_mode() ag_graph_mode(old)old <- ag_graph_mode(TRUE) ag_graph_mode() ag_graph_mode(old)
Normalizes each example independently over its normalized_shape
features, then applies a learnable per-feature scale (gamma) and shift
(beta): y = gamma * (x - mu) / sqrt(var + eps) + beta.
ag_layer_norm(normalized_shape, eps = 1e-05, elementwise_affine = TRUE)ag_layer_norm(normalized_shape, eps = 1e-05, elementwise_affine = TRUE)
normalized_shape |
Number of features to normalize over (the number of rows of the input). |
eps |
Small constant added to the variance for numerical stability (default 1e-5). |
elementwise_affine |
Logical; when |
Unlike ag_batch_norm, which normalizes each feature across the
batch, LayerNorm normalizes each column (example) across its features. It
therefore has no running statistics and behaves identically in training and
evaluation mode, which makes it the usual choice for transformers and for
small or varying batch sizes.
The backward pass is exact: because mu and sigma are computed
from the same example being normalized, their contribution to the gradient is
included rather than treated as constant.
An ag_layer_norm environment with $forward() and
$parameters().
ln <- ag_layer_norm(4L) x <- ag_param(matrix(rnorm(8), 4, 2)) # [features, batch] with_grad_tape({ y <- ln$forward(x) loss <- ag_mse_loss(y, matrix(0, 4, 2)) }) grads <- backward(loss)ln <- ag_layer_norm(4L) x <- ag_param(matrix(rnorm(8), 4, 2)) # [features, batch] with_grad_tape({ y <- ln$forward(x) loss <- ag_mse_loss(y, matrix(0, 4, 2)) }) grads <- backward(loss)
Returns a closure-based layer. Because ag_param uses environment semantics, the optimizer updates W and b in-place, and forward() always uses the latest weights.
ag_linear( in_features, out_features, activation = NULL, init = "glorot_uniform", gain = 1 )ag_linear( in_features, out_features, activation = NULL, init = "glorot_uniform", gain = 1 )
in_features |
Input dimension |
out_features |
Output dimension |
activation |
"relu", "sigmoid", "tanh", "softmax", or NULL |
init |
Weight initialisation,
The bias always starts at zero. |
gain |
Multiplier for the string initialisations (e.g. |
List with W, b, forward(x), params()
layer <- ag_linear(4L, 8L, activation = "relu") x <- ag_tensor(matrix(runif(4 * 16), 4, 16)) out <- layer$forward(x) head <- ag_linear(8L, 2L, init = "orthogonal", gain = 0.01)layer <- ag_linear(4L, 8L, activation = "relu") x <- ag_tensor(matrix(runif(4 * 16), 4, 16)) out <- layer$forward(x) head <- ag_linear(8L, 2L, init = "orthogonal", gain = 0.01)
Reconstructs an ag_* module saved with ag_save_model.
The architecture is rebuilt by calling model_fn (either the one passed
here, or the one stored inside the container at save time), and the saved
parameter and buffer values are copied back by name.
ag_load_model(path, model_fn = NULL, device = NULL, dtype = NULL)ag_load_model(path, model_fn = NULL, device = NULL, dtype = NULL)
path |
File path written by |
model_fn |
Optional zero-argument rebuild function. Required if no
|
device |
Optional device for the rebuilt module ( |
dtype |
Optional GPU upload precision for the restored parameters
( Note that this is an upload/compute precision, not a storage
precision: parameter values always live in |
The reconstructed module with restored weights, in eval mode.
build <- function() ag_sequential(ag_linear(4L, 8L), ag_linear(8L, 3L)) f <- tempfile(fileext = ".rds") ag_save_model(build(), f, model_fn = build) model <- ag_load_model(f)build <- function() ag_sequential(ag_linear(4L, 8L), ag_linear(8L, 3L)) f <- tempfile(fileext = ".rds") ag_save_model(build(), f, model_fn = build) model <- ag_load_model(f)
Writes a state from ag_state_dict (or the container of an
ag_save_model file read with readRDS) into model
in place. Device-resident weights are overwritten in their own buffers, so
an optimizer built on the model stays valid and keeps its residency. Values
are converted to the model's dtype on upload.
ag_load_state_dict(model, state, strict = TRUE)ag_load_state_dict(model, state, strict = TRUE)
model |
Target model (see |
state |
A state dict. |
strict |
|
Everything is checked before anything is written: an error leaves the model untouched.
model, invisibly.
Sets graph mode (ag_graph_mode) and/or matrix-multiply
precision (ag_matmul_precision) and restores the previous
values when the calling function exits – normally or by an error. One place
for both, instead of a pair of on.exit calls per setting.
ag_local_mode(graph = NULL, matmul_precision = NULL, envir = parent.frame())ag_local_mode(graph = NULL, matmul_precision = NULL, envir = parent.frame())
graph |
|
matmul_precision |
|
envir |
The frame whose exit restores the settings (the caller's). |
The previous values, invisibly.
f <- function() { ag_local_mode(graph = TRUE, matmul_precision = "f32") c(ag_graph_mode(), ag_matmul_precision()) } f() # TRUE "f32" ag_graph_mode() # back to the previous valuef <- function() { ag_local_mode(graph = TRUE, matmul_precision = "f32") c(ag_graph_mode(), ag_matmul_precision()) } f() # TRUE "f32" ag_graph_mode() # back to the previous value
Element-wise natural logarithm
ag_log(x)ag_log(x)
x |
ag_tensor |
ag_tensor
log(softmax(x)) down every column, computed without forming
softmax first: masked entries keep finite log-probabilities instead of
log(0) = -Inf, so p * logp stays finite (entropy terms).
ag_log_softmax(x)ag_log_softmax(x)
x |
ag_tensor or numeric matrix, |
Mask illegal entries by ADDING a large but FINITE negative number (e.g.
-1e4, which also fits f16). -Inf is not supported: the shift
below multiplies x by p, and 0 * -Inf is NaN. A column with
every entry masked is finite but meaningless (uniform over masked actions);
callers must leave at least one entry unmasked.
Stability shift. Any per-column c with max - log(n) <= c <= max
keeps exp(x - c) from overflowing and the largest term from
underflowing. c = colSums(x * softmax(x)) satisfies it
(lse - c is the entropy, at most log(n), and lse >= max),
and on the device it is three ggml ops – no max reduction, which ggml lacks
along this axis. The result does not depend on c, and the gradient treats it
as a constant: dx = g - exp(logp) * colSums(g).
f16 compute dtype: colSums(exp(x - c)) is at most n and is stored in
the compute type, so n must stay below 65504 (the f16 maximum); larger
vocabularies need ag_dtype("f32"). Inputs of magnitude ~1e4 are
themselves quantised to a few units in f16 – a property of the input.
ag_tensor of the same shape.
lp <- ag_log_softmax(ag_tensor(matrix(c(1, 2, 3, 0, -1e4, 0), 3, 2)))lp <- ag_log_softmax(ag_tensor(matrix(c(1, 2, 3, 0, -1e4, 0), 3, 2)))
Computes A %*% B and records the operation on the gradient tape.
ag_matmul(A, B)ag_matmul(A, B)
A |
ag_tensor or numeric matrix of shape |
B |
ag_tensor or numeric matrix of shape |
ag_tensor of shape [m, n]
On a Vulkan device with fp16 or cooperative-matrix support, an f32 x f32
matrix multiply with more than 8 columns runs a kernel that keeps its tiles in
f16: relative error around 4e-4, while up to 8 columns (and the CPU)
give f32 accuracy (~1e-7). "f32" asks every ag_matmul
(forward, and the input gradient of the graph backward) for the f32-tile
kernel, so the result no longer depends on the batch size or the device.
ag_matmul_precision(precision)ag_matmul_precision(precision)
precision |
|
Cost: none measurable on small layers (RL policies, tens to hundreds of
units); 2-4x slower on large ones (1024 x 1024 and up). That is why the default
stays "default": set it around the code that needs it and restore the
previous value, e.g.
old <- ag_matmul_precision("f32"); on.exit(ag_matmul_precision(old)).
Has no effect on the CPU or with ag_dtype("f16") / "bf16".
The current setting when precision is missing; otherwise the
previous setting, invisibly.
old <- ag_matmul_precision("f32") ag_matmul_precision() ag_matmul_precision(old)old <- ag_matmul_precision("f32") ag_matmul_precision() ag_matmul_precision(old)
ag_gather(x, ag_argmax(x)): the largest entry of every column, as a
[1, batch] ag_tensor. Differentiable; the gradient goes to the
selected entry only. With ties that is the FIRST maximum (the
ag_argmax rule), and it is not split between the tied entries –
a convention, which some frameworks resolve differently.
ag_max(x)ag_max(x)
x |
ag_tensor or numeric matrix, |
Calls ag_argmax, so on the GPU it is a synchronisation point.
ag_tensor [1, batch].
Mean of elements (or along a dim)
ag_mean(x, dim = NULL, keepdim = FALSE)ag_mean(x, dim = NULL, keepdim = FALSE)
x |
ag_tensor |
dim |
NULL (all), 1 (row-wise), or 2 (col-wise) |
keepdim |
Logical |
ag_tensor
Mean Squared Error loss
ag_mse_loss(pred, target)ag_mse_loss(pred, target)
pred |
ag_tensor [units, batch_size] |
target |
ag_tensor or matrix [units, batch_size] |
scalar ag_tensor
Element-wise multiplication
ag_mul(A, B)ag_mul(A, B)
A |
ag_tensor or numeric matrix |
B |
ag_tensor or numeric matrix |
ag_tensor
Implements scaled dot-product multi-head attention as in "Attention Is All You Need" (Vaswani et al., 2017).
ag_multihead_attention(d_model, n_heads, dropout = 0, bias = TRUE)ag_multihead_attention(d_model, n_heads, dropout = 0, bias = TRUE)
d_model |
Model (embedding) dimension |
n_heads |
Number of attention heads. |
dropout |
Attention dropout probability (default 0, applied in training mode only) |
bias |
Logical: add bias to output projection (default TRUE) |
Calling convention (mirrors PyTorch nn.MultiheadAttention):
layer$forward(q) — self-attention (k = v = q)
layer$forward(q, k, v) — cross-attention
Tensor layout: [d_model, seq_len] — columns are tokens,
consistent with the rest of the ag_* API.
Forward pass:
Q = W_q %*% q [d_k * n_heads, seq_len]
K = W_k %*% k [d_k * n_heads, seq_len]
V = W_v %*% v [d_v * n_heads, seq_len]
for each head h:
q_h = Q[h*d_k+1 : (h+1)*d_k, ] [d_k, seq_len]
k_h = K[h*d_k+1 : (h+1)*d_k, ] [d_k, seq_len]
v_h = V[h*d_v+1 : (h+1)*d_v, ] [d_v, seq_len]
A_h = softmax(t(q_h) %*% k_h / sqrt(d_k)) [seq_len, seq_len]
if causal_mask: A_h[i,j] = 0 for j > i
head_h = v_h %*% A_h [d_v, seq_len]
concat = rbind(head_1, ..., head_H) [d_v*n_heads, seq_len]
out = W_o %*% concat + b_o [d_model, seq_len]
An ag_multihead_attention environment with
$forward(q, k, v, causal_mask) and $parameters()
ag_flash_attention()
This layer loops over heads in R. Since ag_* has no slice operation,
extracting one head is a matmul against a selector matrix, so a 4-head block
records around 52 tape nodes for the attention core – against one for
ag_flash_attention, which fuses all heads into a single
ggml_flash_attn_ext() call and its gradient into one
ggml_flash_attn_back(). Measured on the attention core alone, the
fused op is 15-37x faster depending on shape
(inst/scripts/measure_ag_flash_attn.R).
Prefer ag_flash_attention() for plain and causal multi-head
attention. This layer remains the fallback for what the fused op cannot do:
attention dropout (dropout > 0);
packaged projections – it owns W_q, W_k, W_v,
W_o and b_o, whereas ag_flash_attention() takes
already-projected Q/K/V and leaves the output projection to the caller;
shapes or options ggml_flash_attn_back() rejects – ALiBi,
logit softcap and attention sinks all assert.
This is an ag_* layer: it runs on the dynamic autograd path, where a
tape records each operation as it executes and backward
differentiates through R closures. It is not a layer for
ggml_model_sequential or ggml_model and cannot be
passed to them; the two APIs do not share layer objects. For attention inside
a compiled model, build the block from nn_* layers instead (see the
transformer encoder section of the README).
The practical consequences of being on this path: gradients and optimizer
state are R doubles regardless of ag_dtype, so budget 8 bytes
per scalar (ag_estimate_training_memory); the tape holds every
activation until zero_grad() (ag_tape_memory); and
ag_device("gpu") is usually slower than the CPU for individual ops,
because each one pays a separate upload and download.
ag_flash_attention for the fused implementation.
# Self-attention mha <- ag_multihead_attention(64L, 8L) x <- ag_tensor(matrix(rnorm(64 * 10), 64, 10)) # [d_model=64, seq_len=10] out <- mha$forward(x) # [64, 10] # Cross-attention q <- ag_tensor(matrix(rnorm(64 * 10), 64, 10)) kv <- ag_tensor(matrix(rnorm(64 * 15), 64, 15)) out <- mha$forward(q, kv, kv) # Causal (GPT-style) out <- mha$forward(x, causal_mask = TRUE)# Self-attention mha <- ag_multihead_attention(64L, 8L) x <- ag_tensor(matrix(rnorm(64 * 10), 64, 10)) # [d_model=64, seq_len=10] out <- mha$forward(x) # [64, 10] # Cross-attention q <- ag_tensor(matrix(rnorm(64 * 10), 64, 10)) kv <- ag_tensor(matrix(rnorm(64 * 15), 64, 15)) out <- mha$forward(q, kv, kv) # Causal (GPT-style) out <- mha$forward(x, causal_mask = TRUE)
Create a parameter tensor (gradient tracked)
ag_param( data, device = .ag_device_state$device, dtype = .ag_device_state$dtype )ag_param( data, device = .ag_device_state$device, dtype = .ag_device_state$dtype )
data |
Numeric matrix or vector |
device |
|
dtype |
Floating-point precision: |
An ag_tensor with requires_grad = TRUE
Element-wise power
ag_pow(x, p)ag_pow(x, p)
x |
ag_tensor |
p |
Numeric exponent (scalar, not tracked for gradients) |
ag_tensor
Applies the rectified linear unit: .
ag_relu(x)ag_relu(x)
x |
ag_tensor |
ag_tensor
Reshape tensor
ag_reshape(x, nrow, ncol)ag_reshape(x, nrow, ncol)
x |
ag_tensor |
nrow |
New number of rows (use -1 to infer) |
ncol |
New number of columns (use -1 to infer) |
ag_tensor with new shape, same data
Categorical sampling by the Gumbel-max trick:
argmax(x + G) with G = -log(-log(u)), u ~ U(0, 1),
draws index i of column j with probability softmax(x[, j])[i]. x
may be logits or log-probabilities (they differ by a per-column constant).
ag_sample_categorical(x)ag_sample_categorical(x)
x |
ag_tensor or numeric matrix, |
The noise comes from stats::runif, so set.seed reproduces the
draw, and the CPU and GPU paths see the same noise. On the GPU only the noise
goes up and only the indices come down: x stays on the device, and the
add and argmax are one launch. Like ag_argmax it is a synchronisation
point and not differentiable; get the log-probability of the drawn action with
ag_gather(ag_log_softmax(x), a).
Mask illegal entries as for ag_log_softmax: add a large but FINITE
negative number (-1e4). Gumbel noise exceeds 1e4 with probability
about exp(-1e4), so a masked entry is never drawn while one column
entry is unmasked.
Near-ties of x + G are rounded in f32 on the GPU and in double on the
CPU, so the two paths can differ in a rare column even with the same seed.
Integer vector of length ncol(x), 0-based.
set.seed(1) ag_sample_categorical(matrix(c(0, 0, -1e4, 5, 0, 0), 3, 2))set.seed(1) ag_sample_categorical(matrix(c(0, 0, -1e4, 5, 0, 0), 3, 2))
mu + exp(log_std) * eps with eps ~ N(0, 1) from
stats::rnorm (so set.seed reproduces it). Differentiable in
mu and log_std – the reparameterisation used by SAC; for PPO
collection the result is just the action.
ag_sample_normal(mu, log_std)ag_sample_normal(mu, log_std)
mu |
ag_tensor or numeric matrix, |
log_std |
ag_tensor or numeric matrix, |
log_std is either the shape of mu (state-dependent) or one
column [n, 1] broadcast over the batch (state-independent). On the GPU
mu and log_std stay on the device; only eps is uploaded.
ag_tensor [n, batch].
set.seed(1) a <- ag_sample_normal(matrix(0, 2, 3), matrix(log(0.5), 2, 1))set.seed(1) a <- ag_sample_normal(matrix(0, 2, 3), matrix(log(0.5), 2, 1))
Serializes the trainable parameters and persistent buffers of an
ag_sequential (or single ag_* layer) module as a portable
state dictionary of plain numeric matrices. This avoids serializing the live
module (environments + closures), which is brittle across ggmlR versions and
carries non-portable GPU pointers.
ag_save_model(model, path, model_fn = NULL)ag_save_model(model, path, model_fn = NULL)
model |
An |
path |
File path to write (an RDS container). |
model_fn |
Optional zero-argument function that rebuilds the module
architecture (fresh, untrained). If supplied, it is stored in the container
so |
Reconstruction requires the architecture. Either pass model_fn here so
it is stored in the file, or pass it later to ag_load_model.
path, invisibly.
build <- function() ag_sequential(ag_linear(4L, 8L), ag_linear(8L, 3L)) model <- build() ag_save_model(model, tempfile(fileext = ".rds"), model_fn = build)build <- function() ag_sequential(ag_linear(4L, 8L), ag_linear(8L, 3L)) model <- build() ag_save_model(model, tempfile(fileext = ".rds"), model_fn = build)
Scale tensor by a scalar constant
ag_scale(x, scalar)ag_scale(x, scalar)
x |
ag_tensor |
scalar |
Numeric scalar (not tracked for gradients) |
ag_tensor
Chains layers so that forward(x) passes x through each layer
in order. parameters() collects all trainable params from all layers.
ag_train() / ag_eval() propagate mode to stateful sub-layers.
ag_sequential(...)ag_sequential(...)
... |
Layer objects (ag_linear, ag_dropout, ag_batch_norm, ag_embedding) or a single list of layers. |
An ag_sequential environment
model <- ag_sequential( ag_linear(4L, 16L, activation = "relu"), ag_dropout(0.5), ag_linear(16L, 2L, activation = "softmax") ) x <- ag_tensor(matrix(runif(4 * 8), 4, 8)) out <- model$forward(x)model <- ag_sequential( ag_linear(4L, 16L, activation = "relu"), ag_dropout(0.5), ag_linear(16L, 2L, activation = "softmax") ) x <- ag_tensor(matrix(runif(4 * 8), 4, 8)) out <- model$forward(x)
Applies .
ag_sigmoid(x)ag_sigmoid(x)
x |
ag_tensor |
ag_tensor
Returns rows from + 1 to from + n of x (from is
0-based), e.g. one field of a packed batch read with
ag_get_rows. On the GPU it is a view made contiguous on the
device, with nothing crossing the bus; inside ag_capture()
from and n are constants of the recording.
ag_slice_rows(x, from, n)ag_slice_rows(x, from, n)
x |
ag_tensor (or matrix) with at least |
from |
0-based index of the first row |
n |
Number of rows |
The gradient is g in the sliced rows and zero elsewhere.
ag_tensor of shape n x ncol(x)
x <- ag_tensor(matrix(1:12, 4, 3)) as.matrix(ag_slice_rows(x, 1, 2))x <- ag_tensor(matrix(1:12, 4, 3)) as.matrix(ag_slice_rows(x, 1, 2))
Applies numerically stable softmax along rows so that each column (one sample) sums to 1.
ag_softmax(x)ag_softmax(x)
x |
ag_tensor of shape |
ag_tensor of the same shape as x
Combines softmax and CE in one op using the fused gradient (p - y) / n.
More numerically stable than chaining ag_softmax + ag_cross_entropy_loss.
Use this when your last layer outputs raw logits.
ag_softmax_cross_entropy_loss(logits, target)ag_softmax_cross_entropy_loss(logits, target)
logits |
ag_tensor [classes, batch_size] raw (pre-softmax) scores |
target |
matrix [classes, batch_size] one-hot labels |
scalar ag_tensor
A copy of every parameter and persistent buffer (ag_batch_norm
running statistics) as host matrices, keyed like model$parameters().
Values are R doubles whatever the device or ag_dtype(): an f32 value
is represented exactly, so a state taken on the GPU in f32 restores on the
CPU (or back) to the same numbers. f16/bf16 weights carry only their own
precision; continuing exactly across f16 and f32 is not possible.
ag_state_dict(model)ag_state_dict(model)
model |
An |
Use it for checkpoints, for copying weights between models (target networks:
ag_load_state_dict(target, ag_state_dict(online))), or with
saveRDS. As with any .rds, only read files you trust.
A list with parameters, buffers and
param_dtypes, plus format/version.
ag_load_state_dict, ag_save_model
Element-wise subtraction
ag_sub(A, B)ag_sub(A, B)
A |
ag_tensor or numeric matrix |
B |
ag_tensor or numeric matrix |
ag_tensor
Sum all elements (or along a dim): out = sum(x)
ag_sum(x, dim = NULL, keepdim = FALSE)ag_sum(x, dim = NULL, keepdim = FALSE)
x |
ag_tensor |
dim |
NULL (all), 1 (row-wise), or 2 (col-wise) |
keepdim |
Logical: keep size-1 dimensions |
scalar (or reduced) ag_tensor
Tanh activation
ag_tanh(x)ag_tanh(x)
x |
ag_tensor |
ag_tensor
Walks the current tape and reports how many nodes it holds and how much memory those nodes keep alive, split into three categories: the operand tensors, which outlive the tape; the activations captured by the backward closures, which clearing the tape would release; and any extras recorded for the graph backward path beyond those.
ag_tape_memory(top = 10L, quiet = FALSE)ag_tape_memory(top = 10L, quiet = FALSE)
top |
Number of operations to list in the per-op breakdown, most expensive first. Set to 0 for totals only. |
quiet |
Suppress the printed report and return the figures only. For callers that consume the numbers – tests, scripts that format their own table – rather than read them. |
Storage shared between categories is counted once, and charged to the
longest-lived one. That matters for reading the result: a weight's matrix
and the a_snap an ag_matmul closure captured are the same
object, so it is reported as an operand. The activation figure is therefore
what clearing the tape would actually free, not the sum of everything the
closures reference.
The tape lives from the start of with_grad_tape until
zero_grad() clears it, which happens after the optimizer step. Call
this between backward and zero_grad() to see the peak.
Invisibly, a list with nodes, bytes_total,
bytes_snapshots, bytes_fields, bytes_inputs and a
data frame by_op. Printed as a report as a side effect.
w <- ag_param(matrix(runif(64), 8, 8)) x <- ag_tensor(matrix(runif(64), 8, 8)) with_grad_tape({ h <- ag_relu(ag_matmul(w, x)) loss <- ag_mse_loss(h, matrix(0, 8, 8)) }) backward(loss) ag_tape_memory()w <- ag_param(matrix(runif(64), 8, 8)) x <- ag_tensor(matrix(runif(64), 8, 8)) with_grad_tape({ h <- ag_relu(ag_matmul(w, x)) loss <- ag_mse_loss(h, matrix(0, 8, 8)) }) backward(loss) ag_tape_memory()
ag_tensor is backed by an R environment so all references to the same tensor see updates (like PyTorch tensors).
ag_tensor( data, device = .ag_device_state$device, dtype = .ag_device_state$dtype )ag_tensor( data, device = .ag_device_state$device, dtype = .ag_device_state$dtype )
data |
Numeric matrix or vector |
device |
|
dtype |
Floating-point precision: |
An ag_tensor object (environment)
Copies an ag_tensor to the target device, returning a new tensor.
The original tensor is not modified.
ag_to_device(tensor, device)ag_to_device(tensor, device)
tensor |
An |
device |
|
A new ag_tensor on the target device (or the original if
already on the target device)
Switch a layer or sequential model to training mode
ag_train(model)ag_train(model)
model |
An ag_sequential, ag_batch_norm, or ag_dropout layer |
The model/layer (invisibly)
Transpose a tensor
ag_transpose(x)ag_transpose(x)
x |
ag_tensor |
ag_tensor with rows and columns swapped
Tags a matrix as GPU-backed. In an expression like
as_gpu_matrix(A) %*% B, the multiply dispatches to
ggml_matmul (Vulkan GPU with CPU fallback) instead of R's
built-in. Only one operand needs wrapping. crossprod() and
tcrossprod() on a wrapped matrix likewise use the GPU.
as_gpu_matrix(x, device = "auto", prec = "f32")as_gpu_matrix(x, device = "auto", prec = "f32")
x |
A numeric matrix (or something coercible with |
device |
|
prec |
|
A ggml_matrix.
A <- matrix(rnorm(9), 3) B <- matrix(rnorm(9), 3) as_gpu_matrix(A, device = "cpu") %*% BA <- matrix(rnorm(9), 3) B <- matrix(rnorm(9), 3) as_gpu_matrix(A, device = "cpu") %*% B
The value of an ag_tensor lives in $data only on the CPU. On
the GPU it lives in a backend buffer and $data stays NULL until
something materialises it, so reading $data directly works on one
device and returns NULL on the other. This method downloads the value
when it has to and returns the same matrix either way – use it rather than
$data in code that may run on a GPU.
## S3 method for class 'ag_tensor' as.matrix(x, ...)## S3 method for class 'ag_tensor' as.matrix(x, ...)
x |
An |
... |
Ignored |
A numeric matrix holding the tensor's current value.
t <- ag_tensor(matrix(1:6, nrow = 2)) as.matrix(t)t <- ag_tensor(matrix(1:6, nrow = 2)) as.matrix(t)
Extract the underlying matrix from a ggml_matrix
## S3 method for class 'ggml_matrix' as.matrix(x, ...)## S3 method for class 'ggml_matrix' as.matrix(x, ...)
x |
A |
... |
Ignored; present for S3 |
The plain numeric matrix.
Adds prediction columns to 'new_data', broom style. For classification this appends '.pred_class' plus one '.pred_<level>' probability column per class; for regression it appends '.pred'. Predictions are produced by the existing 'predict()' method for ggml parsnip models (no duplicate inference logic).
## S3 method for class 'ggmlr_parsnip_model' augment(x, new_data, ...)## S3 method for class 'ggmlr_parsnip_model' augment(x, new_data, ...)
x |
A fitted 'ggmlr_parsnip_model' (the engine object inside a parsnip fit; e.g. from 'extract_fit_engine()'). |
new_data |
A data frame of predictors (same columns used for fitting). |
... |
Unused; for generic compatibility. |
'new_data' as a tibble with prediction columns appended.
ggml_set_n_threads(1L) # deterministic, single OpenMP pool spec <- parsnip::mlp(hidden_units = 8L, epochs = 3L) |> parsnip::set_engine("ggml", backend = "cpu") |> parsnip::set_mode("regression") fit_obj <- parsnip::fit(spec, mpg ~ ., data = mtcars) generics::augment(parsnip::extract_fit_engine(fit_obj), mtcars)ggml_set_n_threads(1L) # deterministic, single OpenMP pool spec <- parsnip::mlp(hidden_units = 8L, epochs = 3L) |> parsnip::set_engine("ggml", backend = "cpu") |> parsnip::set_mode("regression") fit_obj <- parsnip::fit(spec, mpg ~ ., data = mtcars) generics::augment(parsnip::extract_fit_engine(fit_obj), mtcars)
Traverses the gradient tape in reverse and accumulates gradients into
tensor$grad for all leaf tensors with requires_grad = TRUE.
backward(loss)backward(loss)
loss |
Scalar ag_tensor |
Named environment: tensor id -> gradient matrix (for use by optimizer$step)
w <- ag_param(matrix(runif(4), 2, 2)) x <- ag_tensor(matrix(c(1, 2), 2, 1)) y <- ag_tensor(matrix(c(0, 1), 2, 1)) with_grad_tape({ out <- ag_matmul(w, x) loss <- ag_mse_loss(out, y) }) grads <- backward(loss)w <- ag_param(matrix(runif(4), 2, 2)) x <- ag_tensor(matrix(c(1, 2), 2, 1)) y <- ag_tensor(matrix(c(0, 1), 2, 1)) with_grad_tape({ out <- ag_matmul(w, x) loss <- ag_mse_loss(out, y) }) grads <- backward(loss)
Scans the gradient environment produced by backward() and reports
every parameter whose gradient contains NaN or Inf values.
Useful as a debugging hook right after backward() — it pinpoints the
offending layer instead of leaving you with a model that silently turns to
NaN several epochs later.
check_grad_anomaly( params, grads, action = c("warn", "stop", "silent"), max_abs = NULL )check_grad_anomaly( params, grads, action = c("warn", "stop", "silent"), max_abs = NULL )
params |
List of ag_param tensors (same as passed to optimizer), named or not. |
grads |
Gradient environment returned by |
action |
One of |
max_abs |
Optional numeric threshold; gradients whose maximum absolute
finite value exceeds it are also reported (as |
A data frame (invisibly) with one row per parameter that has a
gradient, containing columns param, n_nan, n_inf,
max_abs and status. status is one of "ok",
"nan", "inf", "nan+inf" or "large".
Parameters without a gradient are reported with status = "missing".
w <- ag_param(matrix(runif(4), 2, 2)) x <- ag_tensor(matrix(c(1, 1), 2, 1)) with_grad_tape({ out <- ag_matmul(w, x) loss <- ag_mse_loss(out, matrix(0, 2, 1)) }) grads <- backward(loss) check_grad_anomaly(list(w = w), grads)w <- ag_param(matrix(runif(4), 2, 2)) x <- ag_tensor(matrix(c(1, 1), 2, 1)) with_grad_tape({ out <- ag_matmul(w, x) loss <- ag_mse_loss(out, matrix(0, 2, 1)) }) grads <- backward(loss) check_grad_anomaly(list(w = w), grads)
Rescales all gradients in grads so that their global L2 norm does
not exceed max_norm. Modifies the grads environment
in-place and returns the pre-clip norm.
clip_grad_norm(params, grads, max_norm)clip_grad_norm(params, grads, max_norm)
params |
List of ag_param tensors (same as passed to optimizer), named or not. |
grads |
Gradient environment returned by |
max_norm |
Maximum allowed global L2 norm. |
Call this after backward() and before
optimizer$step().
Numeric: the global L2 norm before clipping (invisibly).
w <- ag_param(matrix(runif(4), 2, 2)) x <- ag_tensor(matrix(c(1, 1), 2, 1)) with_grad_tape({ out <- ag_matmul(w, x) loss <- ag_mse_loss(out, matrix(0, 2, 1)) }) grads <- backward(loss) clip_grad_norm(list(w = w), grads, max_norm = 1.0)w <- ag_param(matrix(runif(4), 2, 2)) x <- ag_tensor(matrix(c(1, 1), 2, 1)) with_grad_tape({ out <- ag_matmul(w, x) loss <- ag_mse_loss(out, matrix(0, 2, 1)) }) grads <- backward(loss) clip_grad_norm(list(w = w), grads, max_norm = 1.0)
Clamps every gradient element into
[-clip_value, clip_value]. Unlike clip_grad_norm this
does not preserve the gradient direction — each element is treated
independently. Modifies the grads environment in-place.
clip_grad_value(params, grads, clip_value)clip_grad_value(params, grads, clip_value)
params |
List of ag_param tensors (same as passed to optimizer), named or not. |
grads |
Gradient environment returned by |
clip_value |
Positive numeric; maximum absolute value per element. |
Call this after backward() and before
optimizer$step().
Numeric: the largest absolute gradient value before clipping (invisibly). Returns 0 when no gradients were found.
w <- ag_param(matrix(runif(4), 2, 2)) x <- ag_tensor(matrix(c(1, 1), 2, 1)) with_grad_tape({ out <- ag_matmul(w, x) loss <- ag_mse_loss(out, matrix(0, 2, 1)) }) grads <- backward(loss) clip_grad_value(list(w = w), grads, clip_value = 0.5)w <- ag_param(matrix(runif(4), 2, 2)) x <- ag_tensor(matrix(c(1, 1), 2, 1)) with_grad_tape({ out <- ag_matmul(w, x) loss <- ag_mse_loss(out, matrix(0, 2, 1)) }) grads <- backward(loss) clip_grad_value(list(w = w), grads, clip_value = 0.5)
Configures the model for training by setting the optimizer, loss function,
and metrics. This is the keras-compatible interface; it delegates to
ggml_compile.
## S3 method for class 'ggml_sequential_model' compile( object, optimizer = "adam", loss = "categorical_crossentropy", metrics = c("accuracy"), ... ) ## S3 method for class 'ggml_functional_model' compile( object, optimizer = "adam", loss = "categorical_crossentropy", metrics = c("accuracy"), ... )## S3 method for class 'ggml_sequential_model' compile( object, optimizer = "adam", loss = "categorical_crossentropy", metrics = c("accuracy"), ... ) ## S3 method for class 'ggml_functional_model' compile( object, optimizer = "adam", loss = "categorical_crossentropy", metrics = c("accuracy"), ... )
object |
A model object (e.g. |
optimizer |
Character: |
loss |
Character: |
metrics |
Character vector of metrics (default |
... |
Additional arguments passed to |
The compiled model (invisibly).
model <- ggml_model_sequential() |> ggml_layer_dense(10, activation = "softmax", input_shape = 4) model <- compile(model, optimizer = "adam", loss = "categorical_crossentropy")model <- ggml_model_sequential() |> ggml_layer_dense(10, activation = "softmax", input_shape = 4) model <- compile(model, optimizer = "adam", loss = "categorical_crossentropy")
Converts IQ (integer quantization) data back to float values. IQ formats provide high compression with importance-matrix-aware quantization.
dequantize_row_iq2_xxs(raw_data, n_elements) dequantize_row_iq2_xs(raw_data, n_elements) dequantize_row_iq2_s(raw_data, n_elements) dequantize_row_iq3_xxs(raw_data, n_elements) dequantize_row_iq3_s(raw_data, n_elements) dequantize_row_iq4_nl(raw_data, n_elements) dequantize_row_iq4_xs(raw_data, n_elements) dequantize_row_iq1_s(raw_data, n_elements) dequantize_row_iq1_m(raw_data, n_elements)dequantize_row_iq2_xxs(raw_data, n_elements) dequantize_row_iq2_xs(raw_data, n_elements) dequantize_row_iq2_s(raw_data, n_elements) dequantize_row_iq3_xxs(raw_data, n_elements) dequantize_row_iq3_s(raw_data, n_elements) dequantize_row_iq4_nl(raw_data, n_elements) dequantize_row_iq4_xs(raw_data, n_elements) dequantize_row_iq1_s(raw_data, n_elements) dequantize_row_iq1_m(raw_data, n_elements)
raw_data |
Raw vector containing quantized data |
n_elements |
Number of elements to dequantize |
Numeric vector of dequantized values
Other quantization:
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Converts MXFP4 (microscaling FP4) quantized data back to float values.
dequantize_row_mxfp4(raw_data, n_elements)dequantize_row_mxfp4(raw_data, n_elements)
raw_data |
Raw vector containing quantized data |
n_elements |
Number of elements to dequantize |
Numeric vector of dequantized values
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Converts NVFP4 quantized data back to float values.
dequantize_row_nvfp4(raw_data, n_elements)dequantize_row_nvfp4(raw_data, n_elements)
raw_data |
Raw vector of NVFP4 quantized data |
n_elements |
Number of dequantized elements (must be multiple of 64) |
Numeric vector of dequantized values
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Converts Q1_0 quantized data back to float values.
dequantize_row_q1_0(raw_data, n_elements)dequantize_row_q1_0(raw_data, n_elements)
raw_data |
Raw vector of Q1_0 quantized data |
n_elements |
Number of dequantized elements (must be multiple of 128) |
Numeric vector of dequantized values
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Converts K-quant quantized data back to float values. K-quants (q2_K through q8_K) provide better quality/size tradeoffs.
dequantize_row_q2_K(raw_data, n_elements) dequantize_row_q3_K(raw_data, n_elements) dequantize_row_q4_K(raw_data, n_elements) dequantize_row_q5_K(raw_data, n_elements) dequantize_row_q6_K(raw_data, n_elements) dequantize_row_q8_K(raw_data, n_elements)dequantize_row_q2_K(raw_data, n_elements) dequantize_row_q3_K(raw_data, n_elements) dequantize_row_q4_K(raw_data, n_elements) dequantize_row_q5_K(raw_data, n_elements) dequantize_row_q6_K(raw_data, n_elements) dequantize_row_q8_K(raw_data, n_elements)
raw_data |
Raw vector containing quantized data |
n_elements |
Number of elements to dequantize |
Numeric vector of dequantized values
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Converts Q4_0 quantized data back to float values.
dequantize_row_q4_0(raw_data, n_elements) dequantize_row_q4_1(raw_data, n_elements) dequantize_row_q5_0(raw_data, n_elements) dequantize_row_q5_1(raw_data, n_elements) dequantize_row_q8_0(raw_data, n_elements)dequantize_row_q4_0(raw_data, n_elements) dequantize_row_q4_1(raw_data, n_elements) dequantize_row_q5_0(raw_data, n_elements) dequantize_row_q5_1(raw_data, n_elements) dequantize_row_q8_0(raw_data, n_elements)
raw_data |
Raw vector containing quantized data |
n_elements |
Number of elements to dequantize |
Numeric vector of dequantized values
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Converts ternary quantized data back to float values. TQ1_0 and TQ2_0 are extreme compression formats.
dequantize_row_tq1_0(raw_data, n_elements) dequantize_row_tq2_0(raw_data, n_elements)dequantize_row_tq1_0(raw_data, n_elements) dequantize_row_tq2_0(raw_data, n_elements)
raw_data |
Raw vector containing quantized data |
n_elements |
Number of elements to dequantize |
Numeric vector of dequantized values
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Runs synchronous data-parallel training:
make_model() is called n_gpu times to create one
independent model replica per GPU (each with its own parameters).
Each iteration: the current data item is forwarded through every
replica in parallel; gradients are computed via backward().
Gradients are averaged across all replicas (element-wise mean).
One optimizer step is taken on replica 0; updated weights are then broadcast to replicas 1 … N-1 so all replicas stay in sync.
dp_train( make_model, data, loss_fn = NULL, forward_fn = NULL, target_fn = NULL, n_gpu = NULL, n_iter = 10L, lr = 0.001, max_norm = Inf, verbose = 10L )dp_train( make_model, data, loss_fn = NULL, forward_fn = NULL, target_fn = NULL, n_gpu = NULL, n_iter = 10L, lr = 0.001, max_norm = Inf, verbose = 10L )
make_model |
A zero-argument function that returns a model object with
at least |
data |
A list of training samples. Each element is passed directly to
|
loss_fn |
A function |
forward_fn |
Optional function |
target_fn |
Optional function |
n_gpu |
Number of GPU replicas (default: all available Vulkan devices, minimum 1). |
n_iter |
Number of training iterations (passes over |
lr |
Learning rate for Adam optimizer (default 1e-3). |
max_norm |
Gradient clipping threshold (default |
verbose |
Print loss every |
Because all replicas live in the same R process and ag_param uses
environment (reference) semantics, no IPC or NCCL is required — weight
synchronisation is a simple in-place copy.
A list with:
paramsNamed list of final parameters (from replica 0).
loss_historyNumeric vector of per-iteration mean loss.
modelReplica 0 model object.
make_model <- function() { W <- ag_param(matrix(rnorm(4), 2, 2)) list( forward = function(x) ag_matmul(W, x), parameters = function() list(W = W) ) } data <- lapply(1:8, function(i) matrix(rnorm(2), 2, 1)) result <- dp_train( make_model = make_model, data = data, loss_fn = function(out, tgt) ag_mse_loss(out, tgt), target_fn = function(s) s, n_gpu = 1L, n_iter = 10L, lr = 1e-3, verbose = FALSE )make_model <- function() { W <- ag_param(matrix(rnorm(4), 2, 2)) list( forward = function(x) ag_matmul(W, x), parameters = function() list(W = W) ) } data <- lapply(1:8, function(i) matrix(rnorm(2), 2, 1)) result <- dp_train( make_model = make_model, data = data, loss_fn = function(out, tgt) ag_mse_loss(out, tgt), target_fn = function(s) s, n_gpu = 1L, n_iter = 10L, lr = 1e-3, verbose = FALSE )
Generic for computing loss and metrics on test data.
evaluate(object, ...) ## S3 method for class 'ggml_sequential_model' evaluate(object, x, y, batch_size = NULL, ...) ## S3 method for class 'ggml_functional_model' evaluate(object, x, y, batch_size = NULL, ...)evaluate(object, ...) ## S3 method for class 'ggml_sequential_model' evaluate(object, x, y, batch_size = NULL, ...) ## S3 method for class 'ggml_functional_model' evaluate(object, x, y, batch_size = NULL, ...)
object |
A trained model object. |
... |
Additional arguments passed to |
x |
Test data. |
y |
Test labels. |
batch_size |
Batch size. |
A named list with loss and metric values.
generics::evaluate declares its first argument as x, so S3
dispatch happens on whatever is bound to x. With the keras3 argument
order evaluate(model, x, y) that works positionally, but naming the
data explicitly – evaluate(model, x = x, y = y) – binds the *data*
to x, dispatches on it, and fails with "no applicable method".
Declaring the generic here with object as its first argument fixes
that and also resolves the S3 generic/method consistency warning from
R CMD check.
The trade-off: attaching both ggmlR and keras3 makes the two evaluate
generics mask one another, and the later-attached package wins. Qualify the
call (ggmlR::evaluate(...)) if both are loaded. compile() and
fit() are unaffected – generics declares those with
object first, so they are still re-exported from there.
Trains the model on data for a fixed number of epochs. This is the
keras-compatible interface; it delegates to ggml_fit.
## S3 method for class 'ggml_sequential_model' fit( object, x, y, epochs = 1L, batch_size = 32L, validation_split = 0, validation_data = NULL, verbose = 1L, shuffle = TRUE, callbacks = list(), ... ) ## S3 method for class 'ggml_functional_model' fit( object, x, y, epochs = 1L, batch_size = 32L, validation_split = 0, validation_data = NULL, verbose = 1L, shuffle = TRUE, callbacks = list(), ... )## S3 method for class 'ggml_sequential_model' fit( object, x, y, epochs = 1L, batch_size = 32L, validation_split = 0, validation_data = NULL, verbose = 1L, shuffle = TRUE, callbacks = list(), ... ) ## S3 method for class 'ggml_functional_model' fit( object, x, y, epochs = 1L, batch_size = 32L, validation_split = 0, validation_data = NULL, verbose = 1L, shuffle = TRUE, callbacks = list(), ... )
object |
A compiled model object. |
x |
Training data. Matrix, array, or list of matrices (multi-input). |
y |
Training labels (matrix, one-hot encoded for classification). |
epochs |
Number of training epochs (default 1). |
batch_size |
Batch size (default 32). |
validation_split |
Fraction of data for validation (default 0). |
validation_data |
Optional |
verbose |
0 = silent, 1 = progress (default 1). |
shuffle |
Shuffle the training data (default |
callbacks |
List of callback objects (default |
... |
Additional arguments passed to |
The trained model (invisibly), with model$history.
model <- ggml_model_sequential() |> ggml_layer_dense(10, activation = "softmax", input_shape = 4) model <- compile(model, optimizer = "adam", loss = "categorical_crossentropy") # model <- fit(model, x_train, y_train, epochs = 5, batch_size = 32)model <- ggml_model_sequential() |> ggml_layer_dense(10, activation = "softmax", input_shape = 4) model <- compile(model, optimizer = "adam", loss = "categorical_crossentropy") # model <- fit(model, x_train, y_train, epochs = 5, batch_size = 32)
Check if R Abort Handler is Enabled
ggml_abort_is_r_enabled()ggml_abort_is_r_enabled()
Logical indicating if R-compatible abort handling is active
Other logging:
ggml_log_is_r_enabled(),
ggml_log_set_default(),
ggml_log_set_r(),
ggml_set_abort_callback_default(),
ggml_set_abort_callback_r()
Creates a graph node for element-wise absolute value: |x|
ggml_abs(ctx, a)ggml_abs(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the abs operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(-2, -1, 1, 2)) result <- ggml_abs(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [2, 1, 1, 2] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(-2, -1, 1, 2)) result <- ggml_abs(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [2, 1, 1, 2] ggml_free(ctx)
Creates a graph node for in-place element-wise absolute value.
ggml_abs_inplace(ctx, a)ggml_abs_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with absolute values
Creates a graph node for element-wise addition. Must be computed using ggml_build_forward_expand() and ggml_graph_compute().
ggml_add(ctx, a, b) ggml_add(ctx, a, b)ggml_add(ctx, a, b) ggml_add(ctx, a, b)
ctx |
GGML context |
a |
First tensor |
b |
Second tensor (same shape as a) |
Tensor representing the addition operation
Tensor representing the addition operation
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(5, 4, 3, 2, 1)) result <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx) ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(5, 4, 3, 2, 1)) result <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(5, 4, 3, 2, 1)) result <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx) ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(5, 4, 3, 2, 1)) result <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)
Creates a graph node for in-place element-wise addition. Result is stored in tensor a, saving memory allocation. Returns a view of the modified tensor.
ggml_add_inplace(ctx, a, b)ggml_add_inplace(ctx, a, b)
ctx |
GGML context |
a |
First tensor (will be modified in-place) |
b |
Second tensor (same shape as a) |
View of tensor a with the addition result
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(5, 4, 3, 2, 1)) result <- ggml_add_inplace(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(5, 4, 3, 2, 1)) result <- ggml_add_inplace(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)
Adds width and height relative-position bias to a.
ggml_add_rel_pos(ctx, a, pw, ph)ggml_add_rel_pos(ctx, a, pw, ph)
ctx |
GGML context |
a |
Input tensor |
pw |
Width relative-position tensor |
ph |
Height relative-position tensor |
Tensor with added relative-position bias
Creates a graph node for adding a scalar (1-element tensor) to all elements of a tensor. This is more efficient than creating a full tensor of the same value.
ggml_add1(ctx, a, b)ggml_add1(ctx, a, b)
ctx |
GGML context |
a |
Input tensor |
b |
Scalar tensor (1-element tensor) |
Tensor representing the operation a + b (broadcasted)
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) scalar <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(scalar, 10) result <- ggml_add1(ctx, a, scalar) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) scalar <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(scalar, 10) result <- ggml_add1(ctx, a, scalar) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)
Applies a ggml_layer object (created with ggml_dense(),
ggml_lstm(), etc.) to a ggml_tensor_node. Applying the
same layer object to multiple tensor nodes produces shared weights –
the identity of the layer object (layer$layer_id) is used as the
sharing key, not its name.
ggml_apply(tensor, layer)ggml_apply(tensor, layer)
tensor |
A |
layer |
A |
A new ggml_tensor_node.
encoder <- ggml_dense(64L, activation = "relu") x1 <- ggml_input(shape = 32L) x2 <- ggml_input(shape = 32L) out1 <- x1 |> ggml_apply(encoder) out2 <- x2 |> ggml_apply(encoder) # shared weights model <- ggml_model(inputs = list(x1, x2), outputs = list(out1, out2))encoder <- ggml_dense(64L, activation = "relu") x1 <- ggml_input(shape = 32L) x2 <- ggml_input(shape = 32L) out1 <- x1 |> ggml_apply(encoder) out2 <- x2 |> ggml_apply(encoder) # shared weights model <- ggml_model(inputs = list(x1, x2), outputs = list(out1, out2))
Creates a 1D F32 tensor with values from start (inclusive) to
stop (exclusive) in steps of step.
ggml_arange(ctx, start, stop, step = 1)ggml_arange(ctx, start, stop, step = 1)
ctx |
GGML context |
start |
Start value (inclusive) |
stop |
Stop value (exclusive) |
step |
Step between values (default 1) |
1D F32 tensor
Compares two tensors to check if they have identical type, shape, and strides. Tensors with the same layout can be used interchangeably for memory operations.
ggml_are_same_layout(a, b)ggml_are_same_layout(a, b)
a |
External pointer to first tensor |
b |
External pointer to second tensor |
Logical indicating if tensors have identical layout
Other tensor:
ggml_get_op_params(),
ggml_get_op_params_f32(),
ggml_get_op_params_i32(),
ggml_set_op_params(),
ggml_set_op_params_f32(),
ggml_set_op_params_i32()
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 4) b <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 4) same <- ggml_are_same_layout(a, b) # TRUE ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 4) b <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 4) same <- ggml_are_same_layout(a, b) # TRUE ggml_free(ctx)
Checks if two tensors have the same shape.
ggml_are_same_shape(a, b)ggml_are_same_shape(a, b)
a |
First tensor |
b |
Second tensor |
TRUE if shapes are identical, FALSE otherwise
Check if two tensors have the same stride pattern. Useful for determining if tensors can share operations.
ggml_are_same_stride(a, b)ggml_are_same_stride(a, b)
a |
First tensor |
b |
Second tensor |
Logical indicating if strides are identical
Other tensor_layout:
ggml_can_repeat(),
ggml_count_equal(),
ggml_is_contiguous_0(),
ggml_is_contiguous_1(),
ggml_is_contiguous_2(),
ggml_is_contiguous_channels(),
ggml_is_contiguous_rows(),
ggml_is_contiguously_allocated()
Creates a graph node that finds the index of the maximum value. CRITICAL for token generation in LLMs.
ggml_argmax(ctx, a)ggml_argmax(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor with argmax indices
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 5, 3, 2, 4)) result <- ggml_argmax(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_i32(result) # 1 (0-indexed) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 5, 3, 2, 4)) result <- ggml_argmax(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_i32(result) # 1 (0-indexed) ggml_free(ctx)
Returns indices that would sort the tensor rows. Each row is sorted independently.
ggml_argsort(ctx, a, order = GGML_SORT_ORDER_ASC)ggml_argsort(ctx, a, order = GGML_SORT_ORDER_ASC)
ctx |
GGML context |
a |
Input tensor to sort (F32) |
order |
Sort order: GGML_SORT_ORDER_ASC (0) or GGML_SORT_ORDER_DESC (1) |
Tensor of I32 indices that would sort each row
ctx <- ggml_init(16 * 1024 * 1024) # Create tensor with values to sort a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(3, 1, 4, 1, 5)) # Get indices for ascending sort indices <- ggml_argsort(ctx, a, GGML_SORT_ORDER_ASC) graph <- ggml_build_forward_expand(ctx, indices) ggml_graph_compute(ctx, graph) result <- ggml_get_i32(indices) # result: [1, 3, 0, 2, 4] (0-indexed positions for sorted order) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) # Create tensor with values to sort a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(3, 1, 4, 1, 5)) # Get indices for ascending sort indices <- ggml_argsort(ctx, a, GGML_SORT_ORDER_ASC) graph <- ggml_build_forward_expand(ctx, indices) ggml_graph_compute(ctx, graph) result <- ggml_get_i32(indices) # result: [1, 3, 0, 2, 4] (0-indexed positions for sorted order) ggml_free(ctx)
Returns a reusable layer object for use with ggml_apply, the
scaled dot-product attention of Attention Is All You Need with
n_heads heads computed in parallel. Applying the same object to
several tensor nodes shares one set of projection weights, which is how an
encoder block is reused across a stack.
ggml_attention( d_model, n_heads = 1L, causal = FALSE, bias = TRUE, rope = FALSE, rope_base = 10000, dropout = 0, name = NULL, trainable = TRUE )ggml_attention( d_model, n_heads = 1L, causal = FALSE, bias = TRUE, rope = FALSE, rope_base = 10000, dropout = 0, name = NULL, trainable = TRUE )
d_model |
Integer, model width. Must be divisible by |
n_heads |
Integer, number of attention heads (default 1). |
causal |
Logical: mask out positions after the current one, so a query
attends only to keys at or before it (default |
bias |
Logical: add a bias to the output projection (default
|
rope |
Logical: apply rotary position embedding to queries and keys
(default |
rope_base |
Rotation base for |
dropout |
Dropout rate applied to the attention weights during training (default 0, no dropout). |
name |
Optional character name. |
trainable |
Logical; whether weights are updated during training. |
For the pipe-style form that appends attention to a graph
(x |> ggml_layer_attention(...)), see
ggml_layer_attention.
A ggml_layer object.
The parent must be a sequence node of shape c(seq_len, d_model) –
the same layout ggml_layer_lstm takes – and the output has
that same shape, so attention blocks stack directly.
Applied to a single node the layer is self-attention: queries, keys and
values all come from that node. Applied to a list of two nodes,
ggml_apply(list(query, context), attn), it is cross-attention:
queries come from the first, keys and values from the second, which may
have a different sequence length (but must share d_model). This is
the decoder-to-encoder path of a full transformer.
Four projections W_q, W_k, W_v, W_o, each
d_model x d_model, plus an output bias b_o when
bias = TRUE. Glorot-uniform initialised, zero bias.
ggml_layer_attention, ggml_apply
# One attention block shared between two sequences. attn <- ggml_attention(d_model = 32L, n_heads = 4L) x1 <- ggml_input(shape = c(10L, 32L)) x2 <- ggml_input(shape = c(10L, 32L)) o1 <- ggml_apply(x1, attn) o2 <- ggml_apply(x2, attn) # shared weights stopifnot(identical(o1$layer_id, o2$layer_id)) # Cross-attention: queries from x1, keys/values from a longer context. ctx <- ggml_input(shape = c(15L, 32L)) o3 <- ggml_apply(list(x1, ctx), ggml_attention(32L, 4L))# One attention block shared between two sequences. attn <- ggml_attention(d_model = 32L, n_heads = 4L) x1 <- ggml_input(shape = c(10L, 32L)) x2 <- ggml_input(shape = c(10L, 32L)) o1 <- ggml_apply(x1, attn) o2 <- ggml_apply(x2, attn) # shared weights stopifnot(identical(o1$layer_id, o2$layer_id)) # Cross-attention: queries from x1, keys/values from a longer context. ctx <- ggml_input(shape = c(15L, 32L)) o3 <- ggml_apply(list(x1, ctx), ggml_attention(32L, 4L))
Allocates all tensors in a GGML context to a specific backend. Returns a buffer that must be freed when no longer needed.
ggml_backend_alloc_ctx_tensors(ctx, backend)ggml_backend_alloc_ctx_tensors(ctx, backend)
ctx |
GGML context |
backend |
Backend handle |
Backend buffer object
Clear buffer memory
ggml_backend_buffer_clear(buffer, value = 0L)ggml_backend_buffer_clear(buffer, value = 0L)
buffer |
External pointer to buffer |
value |
Byte value to fill with (default 0) |
NULL invisibly
Other backend:
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Frees a backend buffer and all associated memory.
ggml_backend_buffer_free(buffer)ggml_backend_buffer_free(buffer)
buffer |
Backend buffer object |
No return value, called for side effects
Returns the total size of a backend buffer.
ggml_backend_buffer_get_size(buffer)ggml_backend_buffer_get_size(buffer)
buffer |
Backend buffer object |
Size in bytes
Get buffer usage
ggml_backend_buffer_get_usage(buffer)ggml_backend_buffer_get_usage(buffer)
buffer |
External pointer to buffer |
Usage constant
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Check if buffer is host memory
ggml_backend_buffer_is_host(buffer)ggml_backend_buffer_is_host(buffer)
buffer |
External pointer to buffer |
Logical indicating if buffer is in host memory
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Check if buffer is a multi-buffer
ggml_backend_buffer_is_multi_buffer(buffer)ggml_backend_buffer_is_multi_buffer(buffer)
buffer |
External pointer to buffer |
Logical indicating if buffer is a multi-buffer
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Returns the name/type of a backend buffer.
ggml_backend_buffer_name(buffer)ggml_backend_buffer_name(buffer)
buffer |
Backend buffer object |
Character string with buffer name
Reset buffer
ggml_backend_buffer_reset(buffer)ggml_backend_buffer_reset(buffer)
buffer |
External pointer to buffer |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Set buffer usage hint
ggml_backend_buffer_set_usage(buffer, usage)ggml_backend_buffer_set_usage(buffer, usage)
buffer |
External pointer to buffer |
usage |
Usage constant (use ggml_backend_buffer_usage_* functions) |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Buffer usage: Any
ggml_backend_buffer_usage_any()ggml_backend_buffer_usage_any()
Integer constant for any buffer usage
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Buffer usage: Compute
ggml_backend_buffer_usage_compute()ggml_backend_buffer_usage_compute()
Integer constant for compute buffer usage
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Buffer usage: Weights
ggml_backend_buffer_usage_weights()ggml_backend_buffer_usage_weights()
Integer constant for weights buffer usage
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get the allocation alignment of a buffer type
ggml_backend_buft_get_alignment(buft)ggml_backend_buft_get_alignment(buft)
buft |
External pointer to buffer type |
Numeric alignment in bytes
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get the maximum allocation size of a buffer type
ggml_backend_buft_get_max_size(buft)ggml_backend_buft_get_max_size(buft)
buft |
External pointer to buffer type |
Numeric maximum single-allocation size in bytes
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Check whether a buffer type allocates host-accessible memory
ggml_backend_buft_is_host(buft)ggml_backend_buft_is_host(buft)
buft |
External pointer to buffer type |
Logical, TRUE if buffers of this type live in host memory
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get the name of a buffer type
ggml_backend_buft_name(buft)ggml_backend_buft_name(buft)
buft |
External pointer to buffer type |
Character name of the buffer type
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Creates a new CPU backend instance for graph computation.
ggml_backend_cpu_init()ggml_backend_cpu_init()
Backend pointer
Sets the number of threads for CPU backend computation.
ggml_backend_cpu_set_n_threads(backend, n_threads)ggml_backend_cpu_set_n_threads(backend, n_threads)
backend |
CPU backend pointer |
n_threads |
Number of threads |
NULL invisibly
Returns the buffer type a device allocates from by default. This is the
buffer type the device itself reports as supported, so it is the natural
argument to ggml_backend_dev_supports_buft.
ggml_backend_dev_buffer_type(device)ggml_backend_dev_buffer_type(device)
device |
External pointer to device |
The buffer type is owned by the backend and lives for as long as the backend does; the returned external pointer has no finalizer and must not be freed.
External pointer to the buffer type, or NULL if the device has none
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
dev <- ggml_backend_dev_get(0) buft <- ggml_backend_dev_buffer_type(dev) ggml_backend_buft_name(buft) ggml_backend_dev_supports_buft(dev, buft) # TRUE: a device supports its owndev <- ggml_backend_dev_get(0) buft <- ggml_backend_dev_buffer_type(dev) ggml_backend_buft_name(buft) ggml_backend_dev_supports_buft(dev, buft) # TRUE: a device supports its own
Get device by name
ggml_backend_dev_by_name(name)ggml_backend_dev_by_name(name)
name |
Device name |
External pointer to device, or NULL if not found
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get device by type
ggml_backend_dev_by_type(type)ggml_backend_dev_by_type(type)
type |
Device type (use ggml_backend_device_type_* functions) |
External pointer to first device of given type, or NULL if not found
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get number of available devices
ggml_backend_dev_count()ggml_backend_dev_count()
Number of devices
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get device description
ggml_backend_dev_description(device)ggml_backend_dev_description(device)
device |
External pointer to device |
Device description
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get device by index
ggml_backend_dev_get(index)ggml_backend_dev_get(index)
index |
Device index (0-based) |
External pointer to device, or NULL if not found
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get device properties
ggml_backend_dev_get_props(device)ggml_backend_dev_get_props(device)
device |
External pointer to device |
List with name, description, memory_free, memory_total, type, device_id, caps
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Returns the buffer type used for pinned host memory, which allows faster host-device transfers. Many devices — including the CPU backend — have no such buffer type and return NULL.
ggml_backend_dev_host_buffer_type(device)ggml_backend_dev_host_buffer_type(device)
device |
External pointer to device |
As with ggml_backend_dev_buffer_type, the returned pointer is
owned by the backend and must not be freed.
External pointer to the host buffer type, or NULL if unavailable
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Initialize backend from device
ggml_backend_dev_init(device, params = NULL)ggml_backend_dev_init(device, params = NULL)
device |
External pointer to device |
params |
Optional parameters string |
External pointer to backend, or NULL on failure
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get device memory
ggml_backend_dev_memory(device)ggml_backend_dev_memory(device)
device |
External pointer to device |
Named numeric vector with 'free' and 'total' memory in bytes
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get device name
ggml_backend_dev_name(device)ggml_backend_dev_name(device)
device |
External pointer to device |
Device name
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Check if device should offload operation
ggml_backend_dev_offload_op(device, op)ggml_backend_dev_offload_op(device, op)
device |
External pointer to device |
op |
External pointer to tensor/operation |
Logical indicating if operation should be offloaded
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Check if device supports buffer type
ggml_backend_dev_supports_buft(device, buft)ggml_backend_dev_supports_buft(device, buft)
device |
External pointer to device |
buft |
External pointer to buffer type |
Logical indicating support
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Check if device supports operation
ggml_backend_dev_supports_op(device, op)ggml_backend_dev_supports_op(device, op)
device |
External pointer to device |
op |
External pointer to tensor/operation |
Logical indicating support
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get device type
ggml_backend_dev_type(device)ggml_backend_dev_type(device)
device |
External pointer to device |
Device type constant
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Dynamically registers a new device in the global registry. This is an advanced function for custom backend development.
ggml_backend_device_register(device)ggml_backend_device_register(device)
device |
External pointer to device |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Device type: Accelerator
ggml_backend_device_type_accel()ggml_backend_device_type_accel()
Integer constant for accelerator device type (e.g. BLAS, AMX)
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Device type: CPU
ggml_backend_device_type_cpu()ggml_backend_device_type_cpu()
Integer constant for CPU device type
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Device type: GPU
ggml_backend_device_type_gpu()ggml_backend_device_type_gpu()
Integer constant for GPU device type
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Device type: Integrated GPU
ggml_backend_device_type_igpu()ggml_backend_device_type_igpu()
Integer constant for integrated GPU device type
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Free event
ggml_backend_event_free(event)ggml_backend_event_free(event)
event |
External pointer to event |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Create new event
ggml_backend_event_new(device)ggml_backend_event_new(device)
device |
External pointer to device |
External pointer to event, or NULL on failure
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Record event
ggml_backend_event_record(event, backend)ggml_backend_event_record(event, backend)
event |
External pointer to event |
backend |
External pointer to backend |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Synchronize event
ggml_backend_event_synchronize(event)ggml_backend_event_synchronize(event)
event |
External pointer to event |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Wait for event
ggml_backend_event_wait(backend, event)ggml_backend_event_wait(backend, event)
backend |
External pointer to backend |
event |
External pointer to event |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Releases resources associated with a backend.
ggml_backend_free(backend)ggml_backend_free(backend)
backend |
Backend pointer |
NULL invisibly
Get device from backend
ggml_backend_get_device(backend)ggml_backend_get_device(backend)
backend |
External pointer to backend |
External pointer to device
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Executes computation graph using specified backend.
ggml_backend_graph_compute(backend, graph)ggml_backend_graph_compute(backend, graph)
backend |
Backend pointer |
graph |
Graph pointer |
Status code (0 = success)
Starts graph computation without blocking. Use ggml_backend_synchronize() to wait for completion.
ggml_backend_graph_compute_async(backend, graph)ggml_backend_graph_compute_async(backend, graph)
backend |
External pointer to backend |
graph |
External pointer to computation graph |
Integer status code (0 = success)
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
cpu <- ggml_backend_cpu_init() ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) b <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, b) ggml_set_f32(a, rnorm(100)) # Start async computation status <- ggml_backend_graph_compute_async(cpu, graph) # Do other work while computation runs... ggml_backend_synchronize(cpu) ggml_backend_free(cpu) ggml_free(ctx)cpu <- ggml_backend_cpu_init() ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) b <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, b) ggml_set_f32(a, rnorm(100)) # Start async computation status <- ggml_backend_graph_compute_async(cpu, graph) # Do other work while computation runs... ggml_backend_synchronize(cpu) ggml_backend_free(cpu) ggml_free(ctx)
Execute graph plan
ggml_backend_graph_plan_compute(backend, plan)ggml_backend_graph_plan_compute(backend, plan)
backend |
External pointer to backend |
plan |
External pointer to plan |
Status code (0 = success)
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Create graph execution plan
ggml_backend_graph_plan_create(backend, graph)ggml_backend_graph_plan_create(backend, graph)
backend |
External pointer to backend |
graph |
External pointer to computation graph |
External pointer to plan, or NULL on failure
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Free graph execution plan
ggml_backend_graph_plan_free(backend, plan)ggml_backend_graph_plan_free(backend, plan)
backend |
External pointer to backend |
plan |
External pointer to plan |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Initialize best available backend
ggml_backend_init_best()ggml_backend_init_best()
External pointer to backend (GPU if available, otherwise CPU)
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Initialize backend by name
ggml_backend_init_by_name(name, params = NULL)ggml_backend_init_by_name(name, params = NULL)
name |
Backend name (e.g. "CPU", "Vulkan") |
params |
Optional parameters string |
External pointer to backend, or NULL on failure
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Initialize backend by type
ggml_backend_init_by_type(type, params = NULL)ggml_backend_init_by_type(type, params = NULL)
type |
Device type constant |
params |
Optional parameters string |
External pointer to backend, or NULL on failure
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Load backend from dynamic library
ggml_backend_load(path)ggml_backend_load(path)
path |
Path to dynamic library |
External pointer to registry, or NULL on failure
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Load all available backends
ggml_backend_load_all()ggml_backend_load_all()
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Creates a "meta" device that wraps multiple "simple" backend devices for
tensor parallelism. Each tensor is split across the wrapped devices
according to the result of split_fn, which is called by ggml when
weight buffers are allocated.
ggml_backend_meta_device(devs, split_fn, env = environment(split_fn))ggml_backend_meta_device(devs, split_fn, env = environment(split_fn))
devs |
A list of |
split_fn |
A function |
env |
An environment in which to evaluate |
The split function is invoked with two arguments:
a named list with fields name (character),
type (integer ggml_type enum), ne (numeric vector of
dimensions), op (integer op enum), flags (integer).
the number of simple devices wrapped by the meta backend.
It must return a named list with:
integer; one of 0..3 to split along a tensor axis,
10 for MIRRORED (full copy on each device),
11 for PARTIAL (each device has a partial sum), or
98/99 for NONE/UNKNOWN.
integer or numeric vector of length n_segments * n_devs
giving the per-segment, per-device slice size along the split axis.
integer; usually 1, larger for fused tensors like QKV.
If split_fn errors or returns an unparseable result, the meta
backend silently falls back to MIRRORED for that tensor and stops
calling the callback (sticky error). This is intentional: a misbehaving
callback would otherwise spray errors for every tensor in the model.
Note: with a single device this is a degenerate (no-op) configuration — useful for testing but provides no parallelism benefit. The feature is experimental and the API may change.
External pointer to the meta ggml_backend_dev_t.
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Creates a buffer that combines multiple backend buffers into one. Useful for managing memory across different backends.
ggml_backend_multi_buffer_alloc_buffer(buffers)ggml_backend_multi_buffer_alloc_buffer(buffers)
buffers |
List of backend buffer external pointers |
External pointer to multi-buffer
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
cpu <- ggml_backend_cpu_init() ctx1 <- ggml_init(1024, no_alloc = TRUE) ctx2 <- ggml_init(2048, no_alloc = TRUE) a <- ggml_new_tensor_1d(ctx1, GGML_TYPE_F32, 10) b <- ggml_new_tensor_1d(ctx2, GGML_TYPE_F32, 20) buf1 <- ggml_backend_alloc_ctx_tensors(ctx1, cpu) buf2 <- ggml_backend_alloc_ctx_tensors(ctx2, cpu) multi <- ggml_backend_multi_buffer_alloc_buffer(list(buf1, buf2)) ggml_backend_buffer_free(multi) ggml_backend_free(cpu) ggml_free(ctx1) ggml_free(ctx2)cpu <- ggml_backend_cpu_init() ctx1 <- ggml_init(1024, no_alloc = TRUE) ctx2 <- ggml_init(2048, no_alloc = TRUE) a <- ggml_new_tensor_1d(ctx1, GGML_TYPE_F32, 10) b <- ggml_new_tensor_1d(ctx2, GGML_TYPE_F32, 20) buf1 <- ggml_backend_alloc_ctx_tensors(ctx1, cpu) buf2 <- ggml_backend_alloc_ctx_tensors(ctx2, cpu) multi <- ggml_backend_multi_buffer_alloc_buffer(list(buf1, buf2)) ggml_backend_buffer_free(multi) ggml_backend_free(cpu) ggml_free(ctx1) ggml_free(ctx2)
Set usage for all buffers in a multi-buffer
ggml_backend_multi_buffer_set_usage(buffer, usage)ggml_backend_multi_buffer_set_usage(buffer, usage)
buffer |
External pointer to multi-buffer |
usage |
Usage constant (from ggml_backend_buffer_usage_*) |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Returns the name of the backend (e.g., "CPU").
ggml_backend_name(backend)ggml_backend_name(backend)
backend |
Backend pointer |
Character string name
Get backend registry by name
ggml_backend_reg_by_name(name)ggml_backend_reg_by_name(name)
name |
Registry name |
External pointer to registry, or NULL if not found
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get number of registered backends
ggml_backend_reg_count()ggml_backend_reg_count()
Number of registered backends
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get number of devices in registry
ggml_backend_reg_dev_count(reg)ggml_backend_reg_dev_count(reg)
reg |
External pointer to registry |
Number of devices
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get device from registry
ggml_backend_reg_dev_get(reg, index)ggml_backend_reg_dev_get(reg, index)
reg |
External pointer to registry |
index |
Device index (0-based) |
External pointer to device
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get backend registry by index
ggml_backend_reg_get(index)ggml_backend_reg_get(index)
index |
Registry index (0-based) |
External pointer to registry, or NULL if not found
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Get registry name
ggml_backend_reg_name(reg)ggml_backend_reg_name(reg)
reg |
External pointer to registry |
Registry name
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Dynamically registers a new backend in the global registry. This is an advanced function for custom backend development.
ggml_backend_register(reg)ggml_backend_register(reg)
reg |
External pointer to backend registry |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Allocates memory for a graph across the scheduler's backends. Must be called before computing the graph.
ggml_backend_sched_alloc_graph(sched, graph)ggml_backend_sched_alloc_graph(sched, graph)
sched |
Scheduler pointer |
graph |
Graph pointer |
Logical indicating success
Releases resources associated with the backend scheduler.
ggml_backend_sched_free(sched)ggml_backend_sched_free(sched)
sched |
Scheduler pointer from ggml_backend_sched_new() |
NULL (invisible)
cpu <- ggml_backend_cpu_init() sched <- ggml_backend_sched_new(list(cpu)) ggml_backend_sched_free(sched) ggml_backend_free(cpu)cpu <- ggml_backend_cpu_init() sched <- ggml_backend_sched_new(list(cpu)) ggml_backend_sched_free(sched) ggml_backend_free(cpu)
Returns a specific backend from the scheduler by index.
ggml_backend_sched_get_backend(sched, index = 0L)ggml_backend_sched_get_backend(sched, index = 0L)
sched |
Scheduler pointer |
index |
Backend index (0-based) |
Backend pointer
Returns the number of backends managed by the scheduler.
ggml_backend_sched_get_n_backends(sched)ggml_backend_sched_get_n_backends(sched)
sched |
Scheduler pointer |
Integer count of backends
Returns the number of tensor copies made in the last computed graph. Copies occur when data needs to be transferred between backends.
ggml_backend_sched_get_n_copies(sched)ggml_backend_sched_get_n_copies(sched)
sched |
Scheduler pointer |
Integer count of copies
Returns the number of splits in the last computed graph. Higher numbers indicate more distribution across backends.
ggml_backend_sched_get_n_splits(sched)ggml_backend_sched_get_n_splits(sched)
sched |
Scheduler pointer |
Integer count of splits
Returns which backend a tensor is assigned to.
ggml_backend_sched_get_tensor_backend(sched, tensor)ggml_backend_sched_get_tensor_backend(sched, tensor)
sched |
Scheduler pointer |
tensor |
Tensor pointer |
Assignments are only meaningful once the scheduler has actually placed the
graph — that is, after ggml_backend_sched_graph_compute.
After ggml_backend_sched_reserve alone, or for a tensor from a
different graph, the assignment may not exist and this returns NULL.
Backend pointer, or NULL if the tensor is not assigned to one
Computes a graph by distributing work across multiple backends. This is the main function for multi-GPU computation.
ggml_backend_sched_graph_compute(sched, graph)ggml_backend_sched_graph_compute(sched, graph)
sched |
Scheduler pointer |
graph |
Graph pointer |
Status code (0 = success)
# Multi-GPU example if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) { gpu1 <- ggml_vulkan_init(0) gpu2 <- ggml_vulkan_init(1) sched <- ggml_backend_sched_new(list(gpu1, gpu2)) ctx <- ggml_init(64 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10000) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10000) ggml_set_f32(a, rnorm(10000)) ggml_set_f32(b, rnorm(10000)) c <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, c) # Reserve memory ggml_backend_sched_reserve(sched, graph) # Compute using both GPUs ggml_backend_sched_graph_compute(sched, graph) result <- ggml_get_f32(c) cat("Splits:", ggml_backend_sched_get_n_splits(sched), "\n") cat("Copies:", ggml_backend_sched_get_n_copies(sched), "\n") ggml_free(ctx) ggml_backend_sched_free(sched) ggml_vulkan_free(gpu1) ggml_vulkan_free(gpu2) }# Multi-GPU example if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) { gpu1 <- ggml_vulkan_init(0) gpu2 <- ggml_vulkan_init(1) sched <- ggml_backend_sched_new(list(gpu1, gpu2)) ctx <- ggml_init(64 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10000) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10000) ggml_set_f32(a, rnorm(10000)) ggml_set_f32(b, rnorm(10000)) c <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, c) # Reserve memory ggml_backend_sched_reserve(sched, graph) # Compute using both GPUs ggml_backend_sched_graph_compute(sched, graph) result <- ggml_get_f32(c) cat("Splits:", ggml_backend_sched_get_n_splits(sched), "\n") cat("Copies:", ggml_backend_sched_get_n_copies(sched), "\n") ggml_free(ctx) ggml_backend_sched_free(sched) ggml_vulkan_free(gpu1) ggml_vulkan_free(gpu2) }
Computes a graph asynchronously across backends. Use ggml_backend_sched_synchronize() to wait for completion.
ggml_backend_sched_graph_compute_async(sched, graph)ggml_backend_sched_graph_compute_async(sched, graph)
sched |
Scheduler pointer |
graph |
Graph pointer |
Status code (0 = success)
Creates a scheduler that can distribute computation across multiple backends (GPUs, CPU). A CPU backend is automatically added as a fallback. Backends with lower index have higher priority.
ggml_backend_sched_new(backends, parallel = TRUE, graph_size = 2048)ggml_backend_sched_new(backends, parallel = TRUE, graph_size = 2048)
backends |
List of backend pointers (from ggml_vulkan_init() or ggml_backend_cpu_init()). Note: A CPU backend is automatically added, so you only need to specify GPU backends. |
parallel |
Logical, whether to run backends in parallel (default: TRUE) |
graph_size |
Expected maximum graph size (default: 2048) |
Scheduler pointer
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) { # Create two GPU backends (CPU is added automatically) gpu1 <- ggml_vulkan_init(0) gpu2 <- ggml_vulkan_init(1) # Create scheduler with both GPUs + CPU (automatic) sched <- ggml_backend_sched_new(list(gpu1, gpu2), parallel = TRUE) # The scheduler now has 3 backends: GPU1, GPU2, CPU cat("Backends:", ggml_backend_sched_get_n_backends(sched), "\\n") # Use scheduler... # Cleanup ggml_backend_sched_free(sched) ggml_vulkan_free(gpu1) ggml_vulkan_free(gpu2) }if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) { # Create two GPU backends (CPU is added automatically) gpu1 <- ggml_vulkan_init(0) gpu2 <- ggml_vulkan_init(1) # Create scheduler with both GPUs + CPU (automatic) sched <- ggml_backend_sched_new(list(gpu1, gpu2), parallel = TRUE) # The scheduler now has 3 backends: GPU1, GPU2, CPU cat("Backends:", ggml_backend_sched_get_n_backends(sched), "\\n") # Use scheduler... # Cleanup ggml_backend_sched_free(sched) ggml_vulkan_free(gpu1) ggml_vulkan_free(gpu2) }
Pre-allocates memory based on a measurement graph. This should be called before using the scheduler to compute graphs.
ggml_backend_sched_reserve(sched, graph)ggml_backend_sched_reserve(sched, graph)
sched |
Scheduler pointer |
graph |
Graph pointer to measure memory requirements |
Logical indicating success
cpu <- ggml_backend_cpu_init() sched <- ggml_backend_sched_new(list(cpu)) ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1000) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1000) c <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, c) ggml_backend_sched_reserve(sched, graph) ggml_backend_sched_free(sched) ggml_backend_free(cpu) ggml_free(ctx)cpu <- ggml_backend_cpu_init() sched <- ggml_backend_sched_new(list(cpu)) ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1000) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1000) c <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, c) ggml_backend_sched_reserve(sched, graph) ggml_backend_sched_free(sched) ggml_backend_free(cpu) ggml_free(ctx)
Resets the scheduler, deallocating all tensors. Must be called before changing node backends or allocating a new graph.
ggml_backend_sched_reset(sched)ggml_backend_sched_reset(sched)
sched |
Scheduler pointer |
NULL (invisible)
Manually assigns a specific tensor to run on a specific backend. This overrides automatic scheduling.
ggml_backend_sched_set_tensor_backend(sched, tensor, backend)ggml_backend_sched_set_tensor_backend(sched, tensor, backend)
sched |
Scheduler pointer |
tensor |
Tensor pointer |
backend |
Backend pointer to assign tensor to |
NULL (invisible)
Waits for all asynchronous operations to complete.
ggml_backend_sched_synchronize(sched)ggml_backend_sched_synchronize(sched)
sched |
Scheduler pointer |
NULL (invisible)
Turns on a per-node trace: for each node the scheduler evaluates, its name, op, shape and value summary (sum/min/max) are printed to stderr. Nodes are identified by NAME rather than index, so traces from two backends can be diffed even when their graphs enumerate nodes differently – which is what makes this useful for finding where a CPU and a GPU run diverge.
ggml_backend_sched_trace(sched, enable = TRUE)ggml_backend_sched_trace(sched, enable = TRUE)
sched |
Backend scheduler |
enable |
Logical; turn tracing on or off |
Off by default; leaving it on makes computation much slower.
NULL, invisibly
Synchronize backend
ggml_backend_synchronize(backend)ggml_backend_synchronize(backend)
backend |
External pointer to backend |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Copy tensor asynchronously between backends
ggml_backend_tensor_copy_async(backend_src, backend_dst, src, dst)ggml_backend_tensor_copy_async(backend_src, backend_dst, src, dst)
backend_src |
Source backend |
backend_dst |
Destination backend |
src |
Source tensor |
dst |
Destination tensor |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Gets tensor data from a backend with synchronization.
ggml_backend_tensor_get_and_sync(backend, tensor, offset = 0, size)ggml_backend_tensor_get_and_sync(backend, tensor, offset = 0, size)
backend |
Backend pointer (or NULL for CPU) |
tensor |
Tensor pointer |
offset |
Byte offset (default 0) |
size |
Number of bytes to read |
Raw vector with tensor data
Get tensor data asynchronously
ggml_backend_tensor_get_async(backend, tensor, offset = 0, size)ggml_backend_tensor_get_async(backend, tensor, offset = 0, size)
backend |
External pointer to backend |
tensor |
External pointer to tensor |
offset |
Byte offset (default 0) |
size |
Number of bytes to read |
Numeric vector with data
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_set_async(),
ggml_backend_unload()
Gets tensor data using the backend API. This works with tensors allocated on any backend, not just CPU.
ggml_backend_tensor_get_data(tensor, offset = 0, n_elements = NULL)ggml_backend_tensor_get_data(tensor, offset = 0, n_elements = NULL)
tensor |
Tensor pointer |
offset |
Byte offset (default: 0) |
n_elements |
Number of elements to retrieve (NULL for all) |
R vector with tensor data
Reads the first f32 element from a backend tensor.
ggml_backend_tensor_get_f32_first(tensor)ggml_backend_tensor_get_f32_first(tensor)
tensor |
Tensor pointer |
Float value
Set tensor data asynchronously
ggml_backend_tensor_set_async(backend, tensor, data, offset = 0, size = NULL)ggml_backend_tensor_set_async(backend, tensor, data, offset = 0, size = NULL)
backend |
External pointer to backend |
tensor |
External pointer to tensor |
data |
Numeric or integer vector |
offset |
Byte offset (default 0) |
size |
Number of bytes to copy |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_unload()
Sets tensor data using the backend API. This works with tensors allocated on any backend, not just CPU.
ggml_backend_tensor_set_data(tensor, data, offset = 0)ggml_backend_tensor_set_data(tensor, data, offset = 0)
tensor |
Tensor pointer |
data |
R vector with data to set |
offset |
Byte offset (default: 0) |
Integer and logical data are accepted for float tensors and
converted element by element to float (an integer NA becomes
NaN). An I32 tensor (token indices, say) takes
integers or whole-valued doubles; NA, fractions and values outside
the 32-bit range stop with an error rather than being truncated.
No return value, called for side effects
Unload backend
ggml_backend_unload(reg)ggml_backend_unload(reg)
reg |
External pointer to registry |
NULL invisibly
Other backend:
ggml_backend_buffer_clear(),
ggml_backend_buffer_get_usage(),
ggml_backend_buffer_is_host(),
ggml_backend_buffer_is_multi_buffer(),
ggml_backend_buffer_reset(),
ggml_backend_buffer_set_usage(),
ggml_backend_buffer_usage_any(),
ggml_backend_buffer_usage_compute(),
ggml_backend_buffer_usage_weights(),
ggml_backend_buft_get_alignment(),
ggml_backend_buft_get_max_size(),
ggml_backend_buft_is_host(),
ggml_backend_buft_name(),
ggml_backend_dev_buffer_type(),
ggml_backend_dev_by_name(),
ggml_backend_dev_by_type(),
ggml_backend_dev_count(),
ggml_backend_dev_description(),
ggml_backend_dev_get(),
ggml_backend_dev_get_props(),
ggml_backend_dev_host_buffer_type(),
ggml_backend_dev_init(),
ggml_backend_dev_memory(),
ggml_backend_dev_name(),
ggml_backend_dev_offload_op(),
ggml_backend_dev_supports_buft(),
ggml_backend_dev_supports_op(),
ggml_backend_dev_type(),
ggml_backend_device_register(),
ggml_backend_device_type_accel(),
ggml_backend_device_type_cpu(),
ggml_backend_device_type_gpu(),
ggml_backend_device_type_igpu(),
ggml_backend_event_free(),
ggml_backend_event_new(),
ggml_backend_event_record(),
ggml_backend_event_synchronize(),
ggml_backend_event_wait(),
ggml_backend_get_device(),
ggml_backend_graph_compute_async(),
ggml_backend_graph_plan_compute(),
ggml_backend_graph_plan_create(),
ggml_backend_graph_plan_free(),
ggml_backend_init_best(),
ggml_backend_init_by_name(),
ggml_backend_init_by_type(),
ggml_backend_load(),
ggml_backend_load_all(),
ggml_backend_meta_device(),
ggml_backend_multi_buffer_alloc_buffer(),
ggml_backend_multi_buffer_set_usage(),
ggml_backend_reg_by_name(),
ggml_backend_reg_count(),
ggml_backend_reg_dev_count(),
ggml_backend_reg_dev_get(),
ggml_backend_reg_get(),
ggml_backend_reg_name(),
ggml_backend_register(),
ggml_backend_synchronize(),
ggml_backend_tensor_copy_async(),
ggml_backend_tensor_get_async(),
ggml_backend_tensor_set_async()
Create a Batch Normalization Layer Object
ggml_batch_norm(eps = 1e-05, name = NULL, trainable = TRUE)ggml_batch_norm(eps = 1e-05, name = NULL, trainable = TRUE)
eps |
Small constant for numerical stability (default 1e-5). |
name |
Optional character name. |
trainable |
Logical. |
A ggml_layer object.
Returns the block size for a GGML type. Quantized types process data in blocks (e.g., 32 elements for Q4_0).
ggml_blck_size(type)ggml_blck_size(type)
type |
GGML type constant |
Integer block size
Other type_system:
ggml_ftype_to_ggml_type(),
ggml_is_quantized(),
ggml_type_name(),
ggml_type_sizef()
ggml_blck_size(GGML_TYPE_F32) # 1 ggml_blck_size(GGML_TYPE_Q4_0) # 32ggml_blck_size(GGML_TYPE_F32) # 1 ggml_blck_size(GGML_TYPE_Q4_0) # 32
Extends graph in place with the nodes that compute gradients of its
output with respect to every tensor marked by ggml_set_param.
After computing the graph, read a gradient with
ggml_graph_get_grad.
ggml_build_backward_expand(ctx, graph)ggml_build_backward_expand(ctx, graph)
ctx |
GGML context |
graph |
Graph from |
The graph must come from ggml_build_forward_expand_grads:
ggml_build_forward_expand produces one without gradient
storage, and that is rejected rather than allowed to abort inside ggml.
NULL, invisibly; graph is modified in place
Every step is required; skipping one gives either an error or – for the last – gradients that are silently all zero.
ggml_set_param on each tensor to differentiate
build the forward ops
ggml_set_loss on the output
ggml_build_backward_expand()
ggml_graph_reset – seeds d(loss)/d(loss) = 1
ggml_graph_get_grad to read a gradient
Gradients flow only through operations that implement a backward pass. Most
do, but not all – ggml_clamp is a notable exception, and among
the state-space ops only ggml_ssm_conv is differentiable.
ggml_build_forward_expand_grads,
ggml_graph_get_grad, ggml_set_param
Builds a computation graph from the output tensor, expanding backwards to include all dependencies.
Creates a computation graph by expanding backwards from the output tensor
ggml_build_forward_expand(ctx, tensor) ggml_build_forward_expand(ctx, tensor)ggml_build_forward_expand(ctx, tensor) ggml_build_forward_expand(ctx, tensor)
ctx |
GGML context |
tensor |
Output tensor of the computation |
Graph pointer
Graph object (external pointer)
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(5, 4, 3, 2, 1)) result <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx) ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_set_f32(a, 1:10) ggml_set_f32(b, 11:20) c <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, c) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(c) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(5, 4, 3, 2, 1)) result <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx) ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_set_f32(a, 1:10) ggml_set_f32(b, 11:20) c <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, c) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(c) ggml_free(ctx)
Like ggml_build_forward_expand, but the graph also reserves
storage for gradients, which is what ggml_build_backward_expand
requires. A graph from the plain forward builder carries none, so passing it
to the backward builder is an error.
ggml_build_forward_expand_grads(ctx, tensor, graph_size = 2048L)ggml_build_forward_expand_grads(ctx, tensor, graph_size = 2048L)
ctx |
GGML context |
tensor |
Output tensor of the computation (usually a scalar loss) |
graph_size |
Node capacity of the graph (default 2048) |
Mark the tensors to differentiate with respect to using
ggml_set_param before building the graph.
Graph object (external pointer)
graph_size is the node capacity. The backward pass appends nodes to
the forward ones, so it needs more room than a forward-only graph: roughly
three times the forward node count is a safe starting point. Too small a
value fails while the graph is being built, not silently.
ggml_build_backward_expand,
ggml_graph_get_grad
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) ggml_set_f32(b, c(5, 6, 7, 8)) ggml_set_param(a) loss <- ggml_sum(ctx, ggml_mul(ctx, a, b)) ggml_set_loss(loss) graph <- ggml_build_forward_expand_grads(ctx, loss) ggml_build_backward_expand(ctx, graph) ggml_graph_reset(graph) # seeds d(loss)/d(loss) = 1 ggml_graph_compute(ctx, graph) # d(sum(a*b))/da == b ggml_get_f32(ggml_graph_get_grad(graph, a)) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) ggml_set_f32(b, c(5, 6, 7, 8)) ggml_set_param(a) loss <- ggml_sum(ctx, ggml_mul(ctx, a, b)) ggml_set_loss(loss) graph <- ggml_build_forward_expand_grads(ctx, loss) ggml_build_backward_expand(ctx, graph) ggml_graph_reset(graph) # seeds d(loss)/d(loss) = 1 ggml_graph_compute(ctx, graph) # d(sum(a*b))/da == b ggml_get_f32(ggml_graph_get_grad(graph, a)) ggml_free(ctx)
Stops training when the monitored metric does not improve.
ggml_callback_early_stopping( monitor = "val_loss", patience = 5, min_delta = 0, mode = "auto" )ggml_callback_early_stopping( monitor = "val_loss", patience = 5, min_delta = 0, mode = "auto" )
monitor |
Metric to monitor: "val_loss", "val_accuracy", "train_loss", "train_accuracy" |
patience |
Number of epochs with no improvement before stopping |
min_delta |
Minimum change to qualify as improvement |
mode |
"min" (lower is better) or "max" (higher is better). "auto" infers from monitor name. |
List with on_epoch_end function
Other callbacks:
ggml_schedule_cosine_decay(),
ggml_schedule_reduce_on_plateau(),
ggml_schedule_step_decay()
Check if tensor a can be repeated (broadcast) to match tensor b.
Used for broadcasting operations.
ggml_can_repeat(a, b)ggml_can_repeat(a, b)
a |
Source tensor (smaller) |
b |
Target tensor (larger or same size) |
Logical indicating if a can be repeated to match b
Other tensor_layout:
ggml_are_same_stride(),
ggml_count_equal(),
ggml_is_contiguous_0(),
ggml_is_contiguous_1(),
ggml_is_contiguous_2(),
ggml_is_contiguous_channels(),
ggml_is_contiguous_rows(),
ggml_is_contiguously_allocated()
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) b <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 8) ggml_can_repeat(a, b) # TRUE - a can broadcast along dim 1 ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) b <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 8) ggml_can_repeat(a, b) # TRUE - a can broadcast along dim 1 ggml_free(ctx)
Returns a tensor with the same shape as a but of the given type.
Unlike ggml_cpy there is no need to allocate the destination
tensor first. Some ggml operations require a specific type – for example
ggml_conv_1d builds its im2col in F16 and so needs an F16
kernel – and this is the direct way to supply one.
ggml_cast(ctx, a, type)ggml_cast(ctx, a, type)
ctx |
GGML context |
a |
Source tensor |
type |
Target type constant (e.g. |
Tensor of the requested type with a's shape
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) a16 <- ggml_cast(ctx, a, GGML_TYPE_F16) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) a16 <- ggml_cast(ctx, a, GGML_TYPE_F16) ggml_free(ctx)
Converts values numerically (truncating or rounding) rather than
reinterpreting the bits as ggml_cast does. Supports
F32<->I32, F32<->I16, F32<->I8, F16<->F32 and BF16<->F32.
ggml_cast_numeric(ctx, a, type)ggml_cast_numeric(ctx, a, type)
ctx |
GGML context |
a |
Source tensor |
type |
Target type constant (e.g. |
Tensor of the requested type with a's shape
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) i <- ggml_cast_numeric(ctx, a, GGML_TYPE_I32) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) i <- ggml_cast_numeric(ctx, a, GGML_TYPE_I32) ggml_free(ctx)
Creates a graph node for element-wise ceiling: ceil(x)
ggml_ceil(ctx, a)ggml_ceil(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the ceil operation
Creates a graph node for in-place element-wise ceiling.
ggml_ceil_inplace(ctx, a)ggml_ceil_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with ceiling values
Creates a graph node for clamping values to a range: clamp(x, min, max)
ggml_clamp(ctx, a, min_val, max_val)ggml_clamp(ctx, a, min_val, max_val)
ctx |
GGML context |
a |
Input tensor |
min_val |
Minimum value |
max_val |
Maximum value |
Tensor with values clamped to [min_val, max_val]
GGML_OP_CLAMP has no backward pass, and the node it builds is a view
of its input. A graph containing it therefore cannot be differentiated: the
backward pass aborts with "inplace operations are currently not
supported" rather than returning an error. This matters inside
ggml_layer_custom, whose gradients flow only through
operations that implement a backward pass.
For a trainable equivalent, express the clamp with operations that do:
min(x, hi) is x - ggml_relu(ctx, x - hi) and max(x, lo)
is lo + ggml_relu(ctx, x - lo).
ggml_relu for the differentiable formulation above.
Configures the model for training: infers shapes, creates backend. Weight tensors are created at training time when batch_size is known.
## S3 method for class 'ggml_functional_model' ggml_compile( model, optimizer = "adam", loss = "categorical_crossentropy", metrics = c("accuracy"), backend = "auto", loss_weights = NULL, weight_decay = 0, predict_precision = "default" ) ggml_compile( model, optimizer = "adam", loss = "categorical_crossentropy", metrics = c("accuracy"), backend = "auto", loss_weights = NULL, weight_decay = 0, predict_precision = "default" ) ## S3 method for class 'ggml_sequential_model' ggml_compile( model, optimizer = "adam", loss = "categorical_crossentropy", metrics = c("accuracy"), backend = "auto", loss_weights = NULL, weight_decay = 0, predict_precision = "default" )## S3 method for class 'ggml_functional_model' ggml_compile( model, optimizer = "adam", loss = "categorical_crossentropy", metrics = c("accuracy"), backend = "auto", loss_weights = NULL, weight_decay = 0, predict_precision = "default" ) ggml_compile( model, optimizer = "adam", loss = "categorical_crossentropy", metrics = c("accuracy"), backend = "auto", loss_weights = NULL, weight_decay = 0, predict_precision = "default" ) ## S3 method for class 'ggml_sequential_model' ggml_compile( model, optimizer = "adam", loss = "categorical_crossentropy", metrics = c("accuracy"), backend = "auto", loss_weights = NULL, weight_decay = 0, predict_precision = "default" )
model |
A ggml_sequential_model object |
optimizer |
Optimizer name: "adam" or "sgd" |
loss |
Loss function name: |
metrics |
Character vector of metrics to report from
|
backend |
Backend to use: "auto" (GPU if available, else CPU), "cpu", or "vulkan" |
loss_weights |
Optional per-output weights for a multi-output model;
the optimized total is |
weight_decay |
Decoupled weight decay for the optimizer, in |
predict_precision |
Matrix-multiply precision for prediction:
|
The compiled model (invisibly).
model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_max_pooling_2d(c(2, 2)) |> ggml_layer_flatten() |> ggml_layer_dense(10, activation = "softmax") model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy")model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_max_pooling_2d(c(2, 2)) |> ggml_layer_flatten() |> ggml_layer_dense(10, activation = "softmax") model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy")
Concatenates two tensors along a specified dimension. CRITICAL for KV-cache operations in transformers.
ggml_concat(ctx, a, b, dim = 0)ggml_concat(ctx, a, b, dim = 0)
ctx |
GGML context |
a |
First tensor |
b |
Second tensor (must match a in all dimensions except the concat dim) |
dim |
Dimension along which to concatenate (0-3) |
Concatenated tensor
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 3) b <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 2) ggml_set_f32(a, rnorm(12)) ggml_set_f32(b, rnorm(8)) # Concatenate along dimension 1: result is 4x5 c <- ggml_concat(ctx, a, b, 1) graph <- ggml_build_forward_expand(ctx, c) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 3) b <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 2) ggml_set_f32(a, rnorm(12)) ggml_set_f32(b, rnorm(8)) # Concatenate along dimension 1: result is 4x5 c <- ggml_concat(ctx, a, b, 1) graph <- ggml_build_forward_expand(ctx, c) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Makes a tensor contiguous in memory. Required after permute/transpose before some operations.
ggml_cont(ctx, a)ggml_cont(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Contiguous tensor
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 4) ggml_set_f32(a, 1:12) transposed <- ggml_transpose(ctx, a) contiguous <- ggml_cont(ctx, transposed) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 4) ggml_set_f32(a, 1:12) transposed <- ggml_transpose(ctx, a) contiguous <- ggml_cont(ctx, transposed) ggml_free(ctx)
Applies 1D convolution to input data.
ggml_conv_1d(ctx, a, b, s0 = 1L, p0 = 0L, d0 = 1L)ggml_conv_1d(ctx, a, b, s0 = 1L, p0 = 0L, d0 = 1L)
ctx |
GGML context |
a |
Convolution kernel tensor |
b |
Input data tensor |
s0 |
Stride (default 1) |
p0 |
Padding (default 0) |
d0 |
Dilation (default 1) |
Convolved tensor
Applies depthwise 1D convolution: each input channel is convolved with its own kernel.
ggml_conv_1d_dw(ctx, a, b, s0 = 1L, p0 = 0L, d0 = 1L)ggml_conv_1d_dw(ctx, a, b, s0 = 1L, p0 = 0L, d0 = 1L)
ctx |
GGML context |
a |
Convolution kernel tensor |
b |
Input data tensor |
s0 |
Stride (default 1) |
p0 |
Padding (default 0) |
d0 |
Dilation (default 1) |
Convolved tensor
Builds a layer object to be applied with ggml_apply,
which is how a single set of convolution weights is shared between several
inputs. This is the convolution counterpart of ggml_dense.
ggml_conv_1d_layer( filters, kernel_size, activation = NULL, strides = 1L, padding = "valid", name = NULL, trainable = TRUE )ggml_conv_1d_layer( filters, kernel_size, activation = NULL, strides = 1L, padding = "valid", name = NULL, trainable = TRUE )
filters |
Number of output filters. |
kernel_size |
Integer kernel size. |
activation |
Activation function name or NULL. |
strides |
Integer stride (default 1). |
padding |
|
name |
Optional character name. |
trainable |
Logical. |
For the ordinary pipe-style form that appends a conv layer to a graph
(x |> ggml_layer_conv_1d(...)), see ggml_layer_conv_1d.
A ggml_layer object.
# One convolution shared between two sequence inputs. shared <- ggml_conv_1d_layer(filters = 4L, kernel_size = 3L) x1 <- ggml_input(shape = c(16L, 2L)) x2 <- ggml_input(shape = c(16L, 2L)) o1 <- ggml_apply(x1, shared) o2 <- ggml_apply(x2, shared) stopifnot(identical(o1$layer_id, o2$layer_id)) # same weights# One convolution shared between two sequence inputs. shared <- ggml_conv_1d_layer(filters = 4L, kernel_size = 3L) x1 <- ggml_input(shape = c(16L, 2L)) x2 <- ggml_input(shape = c(16L, 2L)) o1 <- ggml_apply(x1, shared) o2 <- ggml_apply(x2, shared) stopifnot(identical(o1$layer_id, o2$layer_id)) # same weights
Applies 2D convolution to input data.
ggml_conv_2d(ctx, a, b, s0 = 1L, s1 = 1L, p0 = 0L, p1 = 0L, d0 = 1L, d1 = 1L)ggml_conv_2d(ctx, a, b, s0 = 1L, s1 = 1L, p0 = 0L, p1 = 0L, d0 = 1L, d1 = 1L)
ctx |
GGML context |
a |
Convolution kernel tensor [KW, KH, IC, OC] |
b |
Input data tensor [W, H, C, N] |
s0 |
Stride dimension 0 (default 1) |
s1 |
Stride dimension 1 (default 1) |
p0 |
Padding dimension 0 (default 0) |
p1 |
Padding dimension 1 (default 0) |
d0 |
Dilation dimension 0 (default 1) |
d1 |
Dilation dimension 1 (default 1) |
Convolved tensor
Applies 2D convolution using the direct algorithm (no im2col).
ggml_conv_2d_direct( ctx, a, b, s0 = 1L, s1 = 1L, p0 = 0L, p1 = 0L, d0 = 1L, d1 = 1L )ggml_conv_2d_direct( ctx, a, b, s0 = 1L, s1 = 1L, p0 = 0L, p1 = 0L, d0 = 1L, d1 = 1L )
ctx |
GGML context |
a |
Convolution kernel tensor |
b |
Input data tensor |
s0 |
Stride dimension 0 (default 1) |
s1 |
Stride dimension 1 (default 1) |
p0 |
Padding dimension 0 (default 0) |
p1 |
Padding dimension 1 (default 0) |
d0 |
Dilation dimension 0 (default 1) |
d1 |
Dilation dimension 1 (default 1) |
Convolved tensor
Applies depthwise 2D convolution: each input channel is convolved with its own kernel. Uses the im2col-based path.
ggml_conv_2d_dw( ctx, a, b, s0 = 1L, s1 = 1L, p0 = 0L, p1 = 0L, d0 = 1L, d1 = 1L )ggml_conv_2d_dw( ctx, a, b, s0 = 1L, s1 = 1L, p0 = 0L, p1 = 0L, d0 = 1L, d1 = 1L )
ctx |
GGML context |
a |
Convolution kernel tensor |
b |
Input data tensor |
s0, s1
|
Strides along dim 0 and 1 (default 1) |
p0, p1
|
Padding along dim 0 and 1 (default 0) |
d0, d1
|
Dilation along dim 0 and 1 (default 1) |
Convolved tensor
Direct depthwise 2D convolution without an explicit im2col intermediate.
ggml_conv_2d_dw_direct( ctx, a, b, s0 = 1L, s1 = 1L, p0 = 0L, p1 = 0L, d0 = 1L, d1 = 1L )ggml_conv_2d_dw_direct( ctx, a, b, s0 = 1L, s1 = 1L, p0 = 0L, p1 = 0L, d0 = 1L, d1 = 1L )
ctx |
GGML context |
a |
Convolution kernel tensor |
b |
Input data tensor |
s0, s1
|
Strides along dim 0 and 1 (default 1) |
p0, p1
|
Padding along dim 0 and 1 (default 0) |
d0, d1
|
Dilation along dim 0 and 1 (default 1) |
Convolved tensor
Builds a layer object to be applied with ggml_apply,
which is how a single set of convolution weights is shared between several
inputs. This is the convolution counterpart of ggml_dense.
ggml_conv_2d_layer( filters, kernel_size, activation = NULL, strides = c(1L, 1L), padding = "valid", name = NULL, trainable = TRUE )ggml_conv_2d_layer( filters, kernel_size, activation = NULL, strides = c(1L, 1L), padding = "valid", name = NULL, trainable = TRUE )
filters |
Number of output filters. |
kernel_size |
Integer or length-2 integer vector. |
activation |
Activation function name or NULL. |
strides |
Integer or length-2 integer vector (default 1). |
padding |
|
name |
Optional character name. |
trainable |
Logical. |
For the ordinary pipe-style form that appends a conv layer to a graph
(x |> ggml_layer_conv_2d(...)), see ggml_layer_conv_2d.
A ggml_layer object.
# One convolution shared between two inputs. shared <- ggml_conv_2d_layer(filters = 4L, kernel_size = 3L, activation = "relu") x1 <- ggml_input(shape = c(8L, 8L, 1L)) x2 <- ggml_input(shape = c(8L, 8L, 1L)) o1 <- ggml_apply(x1, shared) o2 <- ggml_apply(x2, shared) stopifnot(identical(o1$layer_id, o2$layer_id)) # same weights# One convolution shared between two inputs. shared <- ggml_conv_2d_layer(filters = 4L, kernel_size = 3L, activation = "relu") x1 <- ggml_input(shape = c(8L, 8L, 1L)) x2 <- ggml_input(shape = c(8L, 8L, 1L)) o1 <- ggml_apply(x1, shared) o2 <- ggml_apply(x2, shared) stopifnot(identical(o1$layer_id, o2$layer_id)) # same weights
Applies transposed 1D convolution (deconvolution) to input data.
ggml_conv_transpose_1d(ctx, a, b, s0 = 1L, p0 = 0L, d0 = 1L)ggml_conv_transpose_1d(ctx, a, b, s0 = 1L, p0 = 0L, d0 = 1L)
ctx |
GGML context |
a |
Convolution kernel tensor |
b |
Input data tensor |
s0 |
Stride (default 1) |
p0 |
Padding (default 0) |
d0 |
Dilation (default 1) |
Transposed convolved tensor
Applies transposed 2D convolution (deconvolution) with zero padding.
ggml_conv_transpose_2d_p0(ctx, a, b, stride = 1L)ggml_conv_transpose_2d_p0(ctx, a, b, stride = 1L)
ctx |
GGML context |
a |
Convolution kernel tensor |
b |
Input data tensor |
stride |
Stride (default 1) |
Transposed convolved tensor
Creates a graph node for element-wise cosine: cos(x)
ggml_cos(ctx, a)ggml_cos(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the cos operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(0, pi/3, pi/2, pi)) result <- ggml_cos(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [1, 0.5, 0, -1] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(0, pi/3, pi/2, pi)) result <- ggml_cos(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [1, 0.5, 0, -1] ggml_free(ctx)
Creates a graph node that counts equal elements between two tensors. Useful for accuracy computation.
ggml_count_equal(ctx, a, b)ggml_count_equal(ctx, a, b)
ctx |
GGML context |
a |
First tensor |
b |
Second tensor (same shape as a) |
Tensor containing the count of equal elements
Other tensor_layout:
ggml_are_same_stride(),
ggml_can_repeat(),
ggml_is_contiguous_0(),
ggml_is_contiguous_1(),
ggml_is_contiguous_2(),
ggml_is_contiguous_channels(),
ggml_is_contiguous_rows(),
ggml_is_contiguously_allocated()
ctx <- ggml_init(16 * 1024 * 1024) pred <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 100) labels <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 100) # ... set values ... correct <- ggml_count_equal(ctx, pred, labels) graph <- ggml_build_forward_expand(ctx, correct) ggml_graph_compute(ctx, graph) # correct now contains count of matching elements ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) pred <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 100) labels <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 100) # ... set values ... correct <- ggml_count_equal(ctx, pred, labels) graph <- ggml_build_forward_expand(ctx, correct) ggml_graph_compute(ctx, graph) # correct now contains count of matching elements ggml_free(ctx)
Performs element-wise addition of two tensors using direct CPU computation. Returns the result as an R numeric vector. Does NOT use computation graphs.
ggml_cpu_add(a, b)ggml_cpu_add(a, b)
a |
First tensor (must be F32 type) |
b |
Second tensor (must be F32 type, same size as a) |
Numeric vector containing the element-wise sum
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(5, 4, 3, 2, 1)) ggml_cpu_add(a, b) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(5, 4, 3, 2, 1)) ggml_cpu_add(a, b) ggml_free(ctx)
Returns a named list of all CPU feature detection results. Useful for diagnostics and optimizing computation.
ggml_cpu_features()ggml_cpu_features()
Named list with feature names and logical values
Other cpu_features:
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
features <- ggml_cpu_features() print(features) # On typical x86-64: sse3=TRUE, avx=TRUE, avx2=TRUE, ...features <- ggml_cpu_features() print(features) # On typical x86-64: sse3=TRUE, avx=TRUE, avx2=TRUE, ...
Returns the RISC-V RVV vector length in bytes (0 if not supported).
ggml_cpu_get_rvv_vlen()ggml_cpu_get_rvv_vlen()
Integer vector length in bytes
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Returns the SVE vector length in bytes (0 if not supported).
ggml_cpu_get_sve_cnt()ggml_cpu_get_sve_cnt()
Integer vector length in bytes
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports AMX INT8 (Advanced Matrix Extensions). AMX provides hardware acceleration for matrix operations on Intel CPUs.
ggml_cpu_has_amx_int8()ggml_cpu_has_amx_int8()
Logical indicating AMX INT8 support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports ARM FMA (Fused Multiply-Add).
ggml_cpu_has_arm_fma()ggml_cpu_has_arm_fma()
Logical indicating ARM FMA support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports AVX instructions.
ggml_cpu_has_avx()ggml_cpu_has_avx()
Logical indicating AVX support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports AVX-VNNI instructions.
ggml_cpu_has_avx_vnni()ggml_cpu_has_avx_vnni()
Logical indicating AVX-VNNI support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports AVX2 instructions. AVX2 provides 256-bit SIMD operations for faster matrix math.
ggml_cpu_has_avx2()ggml_cpu_has_avx2()
Logical indicating AVX2 support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports AVX-512 instructions. AVX-512 provides 512-bit SIMD for maximum throughput.
ggml_cpu_has_avx512()ggml_cpu_has_avx512()
Logical indicating AVX-512 support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports AVX-512 BF16 (bfloat16) instructions.
ggml_cpu_has_avx512_bf16()ggml_cpu_has_avx512_bf16()
Logical indicating AVX-512 BF16 support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports AVX-512 VBMI instructions.
ggml_cpu_has_avx512_vbmi()ggml_cpu_has_avx512_vbmi()
Logical indicating AVX-512 VBMI support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports AVX-512 VNNI instructions. VNNI accelerates neural network inference with int8/int16 dot products.
ggml_cpu_has_avx512_vnni()ggml_cpu_has_avx512_vnni()
Logical indicating AVX-512 VNNI support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports BMI2 (Bit Manipulation Instructions 2).
ggml_cpu_has_bmi2()ggml_cpu_has_bmi2()
Logical indicating BMI2 support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports ARM dot product instructions. Accelerates int8 matrix multiplication common in quantized models.
ggml_cpu_has_dotprod()ggml_cpu_has_dotprod()
Logical indicating dot product support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports F16C instructions for float16 conversion.
ggml_cpu_has_f16c()ggml_cpu_has_f16c()
Logical indicating F16C support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports FMA (Fused Multiply-Add) instructions. FMA allows matrix operations to run faster by combining operations.
ggml_cpu_has_fma()ggml_cpu_has_fma()
Logical indicating FMA support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports ARM half-precision FP16 vector arithmetic.
ggml_cpu_has_fp16_va()ggml_cpu_has_fp16_va()
Logical indicating FP16 VA support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if llamafile optimizations are available.
ggml_cpu_has_llamafile()ggml_cpu_has_llamafile()
Logical indicating llamafile support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports ARM INT8 matrix multiplication.
ggml_cpu_has_matmul_int8()ggml_cpu_has_matmul_int8()
Logical indicating INT8 MATMUL support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports ARM NEON instructions. NEON is ARM's SIMD extension for vectorized operations.
ggml_cpu_has_neon()ggml_cpu_has_neon()
Logical indicating NEON support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports RISC-V Vector extension.
ggml_cpu_has_riscv_v()ggml_cpu_has_riscv_v()
Logical indicating RISC-V V support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports ARM SME (Scalable Matrix Extension).
ggml_cpu_has_sme()ggml_cpu_has_sme()
Logical indicating SME support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports SSE3 instructions.
ggml_cpu_has_sse3()ggml_cpu_has_sse3()
Logical indicating SSE3 support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
ggml_cpu_has_sse3()ggml_cpu_has_sse3()
Check if the CPU supports SSSE3 instructions.
ggml_cpu_has_ssse3()ggml_cpu_has_ssse3()
Logical indicating SSSE3 support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports ARM SVE (Scalable Vector Extension).
ggml_cpu_has_sve()ggml_cpu_has_sve()
Logical indicating SVE support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports PowerPC VSX instructions.
ggml_cpu_has_vsx()ggml_cpu_has_vsx()
Logical indicating VSX support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vxe(),
ggml_cpu_has_wasm_simd()
Check if the CPU supports IBM z/Architecture VXE instructions.
ggml_cpu_has_vxe()ggml_cpu_has_vxe()
Logical indicating VXE support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_wasm_simd()
Check if the CPU/environment supports WebAssembly SIMD.
ggml_cpu_has_wasm_simd()ggml_cpu_has_wasm_simd()
Logical indicating WASM SIMD support
Other cpu_features:
ggml_cpu_features(),
ggml_cpu_get_rvv_vlen(),
ggml_cpu_get_sve_cnt(),
ggml_cpu_has_amx_int8(),
ggml_cpu_has_arm_fma(),
ggml_cpu_has_avx(),
ggml_cpu_has_avx2(),
ggml_cpu_has_avx512(),
ggml_cpu_has_avx512_bf16(),
ggml_cpu_has_avx512_vbmi(),
ggml_cpu_has_avx512_vnni(),
ggml_cpu_has_avx_vnni(),
ggml_cpu_has_bmi2(),
ggml_cpu_has_dotprod(),
ggml_cpu_has_f16c(),
ggml_cpu_has_fma(),
ggml_cpu_has_fp16_va(),
ggml_cpu_has_llamafile(),
ggml_cpu_has_matmul_int8(),
ggml_cpu_has_neon(),
ggml_cpu_has_riscv_v(),
ggml_cpu_has_sme(),
ggml_cpu_has_sse3(),
ggml_cpu_has_ssse3(),
ggml_cpu_has_sve(),
ggml_cpu_has_vsx(),
ggml_cpu_has_vxe()
Performs element-wise multiplication of two tensors using direct CPU computation. Returns the result as an R numeric vector. Does NOT use computation graphs.
ggml_cpu_mul(a, b)ggml_cpu_mul(a, b)
a |
First tensor (must be F32 type) |
b |
Second tensor (must be F32 type, same size as a) |
Numeric vector containing the element-wise product
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(2, 2, 2, 2, 2)) ggml_cpu_mul(a, b) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(2, 2, 2, 2, 2)) ggml_cpu_mul(a, b) ggml_free(ctx)
Copies tensor a into tensor b, performing type conversion if needed. The tensors must have the same number of elements. CRITICAL for type casting operations (e.g., F32 to F16).
ggml_cpy(ctx, a, b)ggml_cpy(ctx, a, b)
ctx |
GGML context |
a |
Source tensor |
b |
Destination tensor (defines output type and shape) |
Tensor representing the copy operation (returns b with a's data)
ctx <- ggml_init(16 * 1024 * 1024) # Create F32 tensor a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) ggml_set_f32(a, rnorm(100)) # Create F16 tensor for output b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F16, 100) # Copy with F32 -> F16 conversion result <- ggml_cpy(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) # Create F32 tensor a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) ggml_set_f32(a, rnorm(100)) # Create F16 tensor for output b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F16, 100) # Copy with F32 -> F16 conversion result <- ggml_cpy(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Creates a graph node computing the cross-entropy loss between logits
a and labels b. The op applies its own softmax, so a
must be logits, not probabilities.
ggml_cross_entropy_loss(ctx, a, b)ggml_cross_entropy_loss(ctx, a, b)
ctx |
GGML context |
a |
Logits |
b |
Labels, same shape as |
Differentiable: the backward pass is wired into the graph, so this can be used as a training loss in a hand-built graph.
A scalar tensor holding the loss
Gradient node for ggml_cross_entropy_loss.
ggml_cross_entropy_loss_back(ctx, a, b, c)ggml_cross_entropy_loss_back(ctx, a, b, c)
ctx |
GGML context |
a |
Upstream gradient |
b |
Logits from the forward pass |
c |
Labels from the forward pass |
ggml_build_backward_expand() inserts this automatically, so it is
rarely needed directly; it is exposed for hand-built backward graphs.
Gradient w.r.t. the logits
crossprod)Computes t(A) %*% B (or t(A) %*% A when B is
NULL) on the Vulkan GPU, returning an R matrix, with a CPU fallback.
See ggml_matmul for the precision and dispatch notes.
ggml_crossprod(A, B = NULL, device = "auto", prec = "f32")ggml_crossprod(A, B = NULL, device = "auto", prec = "f32")
A |
A numeric matrix. |
B |
A numeric matrix with |
device, prec
|
See |
t(A) %*% B as a numeric matrix.
Creates a graph node computing the cumulative sum along the first dimension.
ggml_cumsum(ctx, a)ggml_cumsum(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor of the same shape holding the running sum
Creates a graph node whose value is computed by a C kernel registered under
name. The kernel is looked up in ggmlR's custom-op registry; supplying
an unregistered name is an error, listing the names that are available.
ggml_custom( ctx, name, args = NULL, ne, type = GGML_TYPE_F32, n_tasks = GGML_N_TASKS_MAX )ggml_custom( ctx, name, args = NULL, ne, type = GGML_TYPE_F32, n_tasks = GGML_N_TASKS_MAX )
ctx |
GGML context |
name |
Name of a registered custom op (character scalar) |
args |
List of input tensors, or NULL for none. At most 9 tensors. |
ne |
Output shape as a numeric vector of up to 4 dimensions, innermost first (ggml column-major order). Missing dimensions default to 1. |
type |
Output data type (default |
n_tasks |
Number of threads the kernel may be split across, or
|
Kernels are registered from C, not from R. A package that links against
ggmlR registers its kernels at load time via the ggmlR_register_custom_op
callable; see system.file("include", "ggmlR.h", package = "ggmlR") for
the C-side contract and a worked example. R callbacks are not supported: the
kernel may run off the main thread, where calling into R is unsafe.
The output tensor is newly allocated with the given type and shape. Input
tensors passed in args become the node's sources and are visible to
the kernel as dst->src[0], dst->src[1], and so on.
Tensor pointer for the custom node
Custom ops run on the CPU backend only, because the kernel is a host function pointer that a GPU backend cannot execute. Computing a graph that contains a custom node raises an error when the target backend is not the CPU, or when a scheduler has no CPU backend to fall back to. Keep custom nodes out of graphs you intend to run entirely on Vulkan.
ggml_custom_inplace, ggml_custom_ops
# "row_median" is one of ggmlR's built-in kernels; see ggml_custom_ops(). ggml_set_n_threads(1L) ctx <- ggml_init(16 * 1024 * 1024) x <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 5, 2) ggml_set_f32(x, c(5, 1, 3, 2, 4, 9, 7, 8, 6, 10)) # One median per row: the output collapses ne[1] to 1. med <- ggml_custom(ctx, "row_median", args = list(x), ne = c(1, 2)) graph <- ggml_build_forward_expand(ctx, med) ggml_graph_compute(ctx, graph) ggml_get_f32(med) # 3, 8 ggml_free(ctx)# "row_median" is one of ggmlR's built-in kernels; see ggml_custom_ops(). ggml_set_n_threads(1L) ctx <- ggml_init(16 * 1024 * 1024) x <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 5, 2) ggml_set_f32(x, c(5, 1, 3, 2, 4, 9, 7, 8, 6, 10)) # One median per row: the output collapses ne[1] to 1. med <- ggml_custom(ctx, "row_median", args = list(x), ne = c(1, 2)) graph <- ggml_build_forward_expand(ctx, med) ggml_graph_compute(ctx, graph) ggml_get_f32(med) # 3, 8 ggml_free(ctx)
Creates a custom-op node that writes into a view of a rather than into
a freshly allocated tensor. The output has the shape and type of a,
and a itself is the node's first source; tensors in args follow
it, so the kernel sees a as dst->src[0] and the first extra
argument as dst->src[1].
ggml_custom_inplace(ctx, a, name, args = NULL, n_tasks = GGML_N_TASKS_MAX)ggml_custom_inplace(ctx, a, name, args = NULL, n_tasks = GGML_N_TASKS_MAX)
ctx |
GGML context |
a |
Tensor to write into |
name |
Name of a registered custom op (character scalar) |
args |
List of additional input tensors, or NULL for none. At most 8
tensors, one fewer than |
n_tasks |
Number of threads the kernel may be split across, or
|
See ggml_custom for how kernels are registered and for the
CPU-only backend restriction, which applies here too.
Tensor pointer for the custom node (a view of a)
# "clip_inplace" takes its bounds as a 2-element tensor, since a custom node # carries no scalar parameters of its own. ggml_set_n_threads(1L) ctx <- ggml_init(16 * 1024 * 1024) x <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(x, c(-3, -1, 0, 1, 3)) bounds <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 2) ggml_set_f32(bounds, c(-1, 1)) y <- ggml_custom_inplace(ctx, x, "clip_inplace", args = list(bounds)) graph <- ggml_build_forward_expand(ctx, y) ggml_graph_compute(ctx, graph) ggml_get_f32(y) # -1, -1, 0, 1, 1 ggml_free(ctx)# "clip_inplace" takes its bounds as a 2-element tensor, since a custom node # carries no scalar parameters of its own. ggml_set_n_threads(1L) ctx <- ggml_init(16 * 1024 * 1024) x <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(x, c(-3, -1, 0, 1, 3)) bounds <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 2) ggml_set_f32(bounds, c(-1, 1)) y <- ggml_custom_inplace(ctx, x, "clip_inplace", args = list(bounds)) graph <- ggml_build_forward_expand(ctx, y) ggml_graph_compute(ctx, graph) ggml_get_f32(y) # -1, -1, 0, 1, 1 ggml_free(ctx)
Returns the names of the C kernels currently registered with ggmlR, in registration order. Useful for checking that a downstream package registered its kernels as expected.
ggml_custom_ops()ggml_custom_ops()
Character vector of registered custom op names, empty if none
ggmlR registers three kernels at load time. Each one exercises a different path through the custom-op API, and they double as worked examples:
"row_median"Median of each row. One input; the output shape
is chosen by the caller as ne[1] = 1, collapsing the row. Use
ne = c(1, nrow) for an n x nrow input.
"row_permute"out[i] = x[perm[i]] within each row, with
perm 0-based. Two inputs: F32 data and an I32 index tensor.
Indices outside the row yield 0. This is not ggml_get_rows,
which permutes whole rows rather than elements inside one.
"clip_inplace"Clamps values into [lo, hi], writing
into the input. For use with ggml_custom_inplace; the
bounds are passed as a 2-element F32 tensor, since a custom node carries
no scalar parameters of its own. ggml_clamp does the same
thing as a graph op – this kernel exists as the in-place example.
ggml_custom, ggml_custom_inplace
ggml_custom_ops()ggml_custom_ops()
Returns the current CPU cycle count. Useful for low-level benchmarking.
ggml_cycles()ggml_cycles()
Numeric value representing CPU cycles
ggml_cycles()ggml_cycles()
Returns an estimate of CPU cycles per millisecond. Useful for converting cycle counts to time.
ggml_cycles_per_ms()ggml_cycles_per_ms()
Numeric value representing cycles per millisecond
ggml_cycles_per_ms()ggml_cycles_per_ms()
Constructs an uncompiled sequential multi-layer perceptron suitable as a
starting point for tabular classification or regression. This is the default
model_fn used by LearnerClassifGGML and LearnerRegrGGML
when the user does not supply a custom builder, and it is also exported for
direct use or as a template for user-defined builders.
ggml_default_mlp( n_features, n_out, task_type = c("classif", "regr"), hidden_layers = c(128L, 64L), activation = "relu", dropout = 0.2 )ggml_default_mlp( n_features, n_out, task_type = c("classif", "regr"), hidden_layers = c(128L, 64L), activation = "relu", dropout = 0.2 )
n_features |
Integer. Number of input features. Required. |
n_out |
Integer. Number of output units. For classification this is the number of classes; for regression this is typically 1. |
task_type |
Character. One of |
|
Integer vector. Widths of the hidden dense layers.
Default |
|
activation |
Character. Activation applied to each hidden layer.
Default |
dropout |
Numeric in |
The returned model is not compiled: the caller is responsible for
calling ggml_compile with the appropriate loss
("categorical_crossentropy" for classification, "mse" for
regression) before training.
The final layer is chosen based on task_type:
"classif" — dense with units = n_out and softmax activation.
"regr" — dense with units = n_out and no activation
(identity / linear output).
An uncompiled ggml_sequential_model object. Call
ggml_compile before ggml_fit.
ggml_model_sequential, ggml_layer_dense,
ggml_layer_dropout, ggml_compile
## Not run: # 3-class classifier on 20 features model <- ggml_default_mlp( n_features = 20L, n_out = 3L, task_type = "classif", hidden_layers = c(64L, 32L), dropout = 0.1 ) model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy") # Single-output regressor reg <- ggml_default_mlp( n_features = 10L, n_out = 1L, task_type = "regr" ) reg <- ggml_compile(reg, optimizer = "adam", loss = "mse") ## End(Not run)## Not run: # 3-class classifier on 20 features model <- ggml_default_mlp( n_features = 20L, n_out = 3L, task_type = "classif", hidden_layers = c(64L, 32L), dropout = 0.1 ) model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy") # Single-output regressor reg <- ggml_default_mlp( n_features = 10L, n_out = 1L, task_type = "regr" ) reg <- ggml_compile(reg, optimizer = "adam", loss = "mse") ## End(Not run)
Returns a reusable layer object for use with ggml_apply().
Applying the same object to multiple tensor nodes shares weights.
ggml_dense( units, activation = NULL, name = NULL, trainable = TRUE, time_distributed = FALSE )ggml_dense( units, activation = NULL, name = NULL, trainable = TRUE, time_distributed = FALSE )
units |
Number of output units. |
activation |
Activation function name or NULL. |
name |
Optional character name. |
trainable |
Logical; whether weights are updated during training. |
time_distributed |
Logical; apply the kernel independently at every
position of a sequence input rather than flattening it (default
|
A ggml_layer object.
encoder <- ggml_dense(64L, activation = "relu") x1 <- ggml_input(shape = 32L) x2 <- ggml_input(shape = 32L) out1 <- x1 |> ggml_apply(encoder) out2 <- x2 |> ggml_apply(encoder) # shared weightsencoder <- ggml_dense(64L, activation = "relu") x1 <- ggml_input(shape = 32L) x2 <- ggml_input(shape = 32L) out1 <- x1 |> ggml_apply(encoder) out2 <- x2 |> ggml_apply(encoder) # shared weights
Creates a diagonal matrix from a vector. For vector a[n], produces matrix with a on the diagonal.
ggml_diag(ctx, a)ggml_diag(ctx, a)
ctx |
GGML context |
a |
Input vector tensor |
Diagonal matrix tensor
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 3) ggml_set_f32(a, c(1, 2, 3)) d <- ggml_diag(ctx, a) # 3x3 diagonal matrix graph <- ggml_build_forward_expand(ctx, d) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 3) ggml_set_f32(a, c(1, 2, 3)) d <- ggml_diag(ctx, a) # 3x3 diagonal matrix graph <- ggml_build_forward_expand(ctx, d) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Creates a graph node that sets elements above the diagonal to -Inf. This is used for causal (autoregressive) attention masking.
ggml_diag_mask_inf(ctx, a, n_past)ggml_diag_mask_inf(ctx, a, n_past)
ctx |
GGML context |
a |
Input tensor (typically attention scores) |
n_past |
Number of past tokens (shifts the diagonal). Use 0 for standard causal masking where position i can only attend to positions <= i. |
In causal attention, we want each position to only attend to itself and previous positions. Setting future positions to -Inf ensures that after softmax, they contribute 0 attention weight.
The n_past parameter allows for KV-cache scenarios where the diagonal needs to be shifted to account for previously processed tokens.
Tensor with same shape as input, elements above diagonal set to -Inf
ctx <- ggml_init(16 * 1024 * 1024) # Create attention scores matrix scores <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 4) ggml_set_f32(scores, rep(1, 16)) # Apply causal mask masked <- ggml_diag_mask_inf(ctx, scores, 0) graph <- ggml_build_forward_expand(ctx, masked) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) # Create attention scores matrix scores <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 4) ggml_set_f32(scores, rep(1, 16)) # Apply causal mask masked <- ggml_diag_mask_inf(ctx, scores, 0) graph <- ggml_build_forward_expand(ctx, masked) ggml_graph_compute(ctx, graph) ggml_free(ctx)
In-place version of ggml_diag_mask_inf. Returns a view of the input tensor.
ggml_diag_mask_inf_inplace(ctx, a, n_past)ggml_diag_mask_inf_inplace(ctx, a, n_past)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
n_past |
Number of past tokens |
View of input tensor with elements above diagonal set to -Inf
Creates a graph node that sets elements above the diagonal to 0. Alternative to -Inf masking for certain use cases.
ggml_diag_mask_zero(ctx, a, n_past)ggml_diag_mask_zero(ctx, a, n_past)
ctx |
GGML context |
a |
Input tensor |
n_past |
Number of past tokens |
Tensor with same shape as input, elements above diagonal set to 0
Creates a graph node for element-wise division.
ggml_div(ctx, a, b)ggml_div(ctx, a, b)
ctx |
GGML context |
a |
First tensor (numerator) |
b |
Second tensor (denominator, same shape as a) |
Tensor representing the division operation (a / b)
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(10, 20, 30, 40, 50)) ggml_set_f32(b, c(2, 2, 2, 2, 2)) result <- ggml_div(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(10, 20, 30, 40, 50)) ggml_set_f32(b, c(2, 2, 2, 2, 2)) result <- ggml_div(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)
Creates a graph node for in-place element-wise division. Result is stored in tensor a, saving memory allocation.
ggml_div_inplace(ctx, a, b)ggml_div_inplace(ctx, a, b)
ctx |
GGML context |
a |
First tensor (will be modified in-place) |
b |
Second tensor (same shape as a) |
View of tensor a with the division result
Creates a graph node that copies a tensor. This is a graph operation that must be computed using ggml_build_forward_expand() and ggml_graph_compute(). Unlike ggml_dup_tensor which just allocates, this creates a copy operation in the graph.
ggml_dup(ctx, a)ggml_dup(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the copy operation
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) b <- ggml_dup(ctx, a) graph <- ggml_build_forward_expand(ctx, b) ggml_graph_compute(ctx, graph) ggml_get_f32(b) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) b <- ggml_dup(ctx, a) graph <- ggml_build_forward_expand(ctx, b) ggml_graph_compute(ctx, graph) ggml_get_f32(b) ggml_free(ctx)
Creates a graph node for in-place tensor duplication. Returns a view of the input tensor.
ggml_dup_inplace(ctx, a)ggml_dup_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor |
View of tensor a
Creates a copy of a tensor with the same shape and type
ggml_dup_tensor(ctx, tensor)ggml_dup_tensor(ctx, tensor)
ctx |
GGML context |
tensor |
Tensor to duplicate |
New tensor pointer with same shape
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) b <- ggml_dup_tensor(ctx, a) ggml_nelements(b) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) b <- ggml_dup_tensor(ctx, a) ggml_nelements(b) ggml_free(ctx)
Returns the size of a single element in the tensor.
ggml_element_size(tensor)ggml_element_size(tensor)
tensor |
Tensor pointer |
Element size in bytes
Creates a graph node for ELU (Exponential Linear Unit) activation. ELU(x) = x if x > 0, else alpha * (exp(x) - 1) where alpha = 1.
ggml_elu(ctx, a)ggml_elu(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the ELU operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) r <- ggml_elu(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) r <- ggml_elu(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) ggml_free(ctx)
Creates a graph node for in-place ELU (Exponential Linear Unit) activation.
ggml_elu_inplace(ctx, a)ggml_elu_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with ELU applied
Create an Embedding Layer Object
ggml_embedding(vocab_size, dim, name = NULL, trainable = TRUE)ggml_embedding(vocab_size, dim, name = NULL, trainable = TRUE)
vocab_size |
Number of distinct tokens. |
dim |
Embedding dimension. |
name |
Optional character name. |
trainable |
Logical. |
A ggml_layer object.
Helper function to estimate memory needed for a tensor
ggml_estimate_memory(type = GGML_TYPE_F32, ne0, ne1 = 1, ne2 = 1, ne3 = 1)ggml_estimate_memory(type = GGML_TYPE_F32, ne0, ne1 = 1, ne2 = 1, ne3 = 1)
type |
Tensor type (GGML_TYPE_F32, etc) |
ne0 |
Size of dimension 0 |
ne1 |
Size of dimension 1 (optional) |
ne2 |
Size of dimension 2 (optional) |
ne3 |
Size of dimension 3 (optional) |
Estimated memory in bytes
# For 1000x1000 F32 matrix ggml_estimate_memory(GGML_TYPE_F32, 1000, 1000)# For 1000x1000 F32 matrix ggml_estimate_memory(GGML_TYPE_F32, 1000, 1000)
Evaluate a Trained Model
## S3 method for class 'ggml_functional_model' ggml_evaluate(model, x, y, batch_size = 32L, ...) ggml_evaluate(model, ...) ## S3 method for class 'ggml_sequential_model' ggml_evaluate( model, x, y, batch_size = 32, sample_weight = NULL, class_weight = NULL, ... )## S3 method for class 'ggml_functional_model' ggml_evaluate(model, x, y, batch_size = 32L, ...) ggml_evaluate(model, ...) ## S3 method for class 'ggml_sequential_model' ggml_evaluate( model, x, y, batch_size = 32, sample_weight = NULL, class_weight = NULL, ... )
model |
A trained ggml_sequential_model |
x |
Test data |
y |
Test labels (one-hot encoded) |
batch_size |
Batch size for evaluation |
... |
Additional arguments (ignored). |
sample_weight |
Numeric vector of per-sample weights (length = nrow(x)). |
class_weight |
Named vector of weights per class, e.g. c("0"=1, "1"=10). Cannot be used with sample_weight. |
Named list with loss and accuracy. A multi-output
functional model additionally reports each head as <output>_loss.
Note the deliberate difference from the training history, which names the
same quantities train_<output>_loss and val_<output>_loss:
there the prefix distinguishes the two phases (and keeps an output named
val_x from colliding with another output's validation column),
whereas evaluation has no phases to tell apart, so the bare name is used.
ggml_set_n_threads(1L) # deterministic, single OpenMP pool n <- 128 x <- matrix(runif(n * 4), nrow = n, ncol = 4) y <- matrix(0, nrow = n, ncol = 2) for (i in seq_len(n)) { y[i, if (sum(x[i,]) > 2) 1L else 2L] <- 1 } model <- ggml_model_sequential() |> ggml_layer_dense(8, activation = "relu") |> ggml_layer_dense(2, activation = "softmax") model$input_shape <- 4L model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy") model <- ggml_fit(model, x, y, epochs = 5, batch_size = 32, verbose = 0) # Basic evaluation result <- ggml_evaluate(model, x, y, batch_size = 32) # With sample_weight sw <- runif(n, 0.5, 1.5) result <- ggml_evaluate(model, x, y, batch_size = 32, sample_weight = sw) # With class_weight result <- ggml_evaluate(model, x, y, batch_size = 32, class_weight = c("0" = 1, "1" = 2))ggml_set_n_threads(1L) # deterministic, single OpenMP pool n <- 128 x <- matrix(runif(n * 4), nrow = n, ncol = 4) y <- matrix(0, nrow = n, ncol = 2) for (i in seq_len(n)) { y[i, if (sum(x[i,]) > 2) 1L else 2L] <- 1 } model <- ggml_model_sequential() |> ggml_layer_dense(8, activation = "relu") |> ggml_layer_dense(2, activation = "softmax") model$input_shape <- 4L model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy") model <- ggml_fit(model, x, y, epochs = 5, batch_size = 32, verbose = 0) # Basic evaluation result <- ggml_evaluate(model, x, y, batch_size = 32) # With sample_weight sw <- runif(n, 0.5, 1.5) result <- ggml_evaluate(model, x, y, batch_size = 32, sample_weight = sw) # With class_weight result <- ggml_evaluate(model, x, y, batch_size = 32, class_weight = c("0" = 1, "1" = 2))
Creates a graph node for element-wise exponential: exp(x)
ggml_exp(ctx, a)ggml_exp(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the exp operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 3) ggml_set_f32(a, c(0, 1, 2)) result <- ggml_exp(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [1, e, e^2] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 3) ggml_set_f32(a, c(0, 1, 2)) result <- ggml_exp(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [1, e, e^2] ggml_free(ctx)
Creates a graph node for in-place element-wise exponential: e^x
ggml_exp_inplace(ctx, a)ggml_exp_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with exponential values
Pulls an expression matrix out of a Seurat object, a
SingleCellExperiment, a sparse dgCMatrix or a plain
matrix, returning a dense numeric matrix with features in rows and
cells in columns — the layout the GPU engine expects. Optional
genes/cells subsetting happens before any sparse-to-dense
materialisation.
ggml_extract(x, assay = NULL, layer = "data", genes = NULL, cells = NULL, ...) ## S3 method for class 'matrix' ggml_extract(x, assay = NULL, layer = "data", genes = NULL, cells = NULL, ...) ## S3 method for class 'dgCMatrix' ggml_extract( x, assay = NULL, layer = "data", genes = NULL, cells = NULL, keep_sparse = FALSE, ... ) ## S3 method for class 'Seurat' ggml_extract( x, assay = NULL, layer = "data", genes = NULL, cells = NULL, keep_sparse = FALSE, ... ) ## S3 method for class 'SingleCellExperiment' ggml_extract( x, assay = NULL, layer = "logcounts", genes = NULL, cells = NULL, keep_sparse = FALSE, ... )ggml_extract(x, assay = NULL, layer = "data", genes = NULL, cells = NULL, ...) ## S3 method for class 'matrix' ggml_extract(x, assay = NULL, layer = "data", genes = NULL, cells = NULL, ...) ## S3 method for class 'dgCMatrix' ggml_extract( x, assay = NULL, layer = "data", genes = NULL, cells = NULL, keep_sparse = FALSE, ... ) ## S3 method for class 'Seurat' ggml_extract( x, assay = NULL, layer = "data", genes = NULL, cells = NULL, keep_sparse = FALSE, ... ) ## S3 method for class 'SingleCellExperiment' ggml_extract( x, assay = NULL, layer = "logcounts", genes = NULL, cells = NULL, keep_sparse = FALSE, ... )
x |
A |
assay |
Assay to read from. Seurat: defaults to the object's default assay. Ignored for bare matrices. |
layer |
Layer / slot to read. Seurat v5: a layer name (default
|
genes |
Optional character/integer vector selecting feature rows. |
cells |
Optional character/integer vector selecting cell columns. |
... |
Passed to methods. |
keep_sparse |
If |
For a SingleCellExperiment the layer argument names an
assay (default "logcounts", the log-normalised matrix); it is read
with SummarizedExperiment::assay().
A dense numeric matrix, features x cells.
Streaming counterpart of ggml_fit for sequential models:
batches are pulled from a generator instead of being sliced out of a matrix
held in memory. Reachable either directly or as
ggml_fit(model, generator = ...).
ggml_fit_generator( model, generator, steps_per_epoch = NULL, epochs = 1L, batch_size = 32L, initial_epoch = 0L, validation_generator = NULL, validation_steps = NULL, calibration_generator = NULL, calibration_steps = NULL, verbose = 1, callbacks = list() )ggml_fit_generator( model, generator, steps_per_epoch = NULL, epochs = 1L, batch_size = 32L, initial_epoch = 0L, validation_generator = NULL, validation_steps = NULL, calibration_generator = NULL, calibration_steps = NULL, verbose = 1, callbacks = list() )
model |
A compiled |
generator |
Function of no arguments yielding |
steps_per_epoch |
Batches per epoch. Required for a generator that never
returns |
epochs |
Number of epochs to run |
batch_size |
Samples per batch. The graph is built for a fixed batch, so the generator must yield exactly this many; a short batch is an error rather than being padded. |
initial_epoch |
Epoch number to start counting from. Shifts the
numbering seen by callbacks and learning-rate schedules; it does not
restore optimizer state (see |
validation_generator |
Optional generator for validation, same contract. |
validation_steps |
Validation batches per epoch. |
calibration_generator |
Generator used to recompute batch-norm running
statistics after training. Defaults to |
calibration_steps |
Batches to draw for that calibration. |
verbose |
0 to silence per-epoch messages |
callbacks |
List of callback lists, as for |
Batches are given in the same R layout that x and y use; the
column-major conversion ggml wants happens internally, which is also what
lets a recorded batch feed batch-norm calibration unchanged.
The trained model, invisibly.
ggml_fit_opt_gen for the low-level loop this wraps.
Trains a model epoch by epoch in R, allowing callbacks for early stopping and learning rate scheduling. Optimizer state (momentum) is preserved across all epochs.
ggml_fit_opt( sched, ctx_compute, inputs, outputs, dataset, loss_type = ggml_opt_loss_type_mse(), optimizer = ggml_opt_optimizer_type_adamw(), nepoch = 10L, nbatch_logical = 32L, val_split = 0, shuffle = TRUE, shuffle_all = shuffle, callbacks = list(), silent = FALSE, loss_mask_ne0 = NULL, weight_decay = 0 )ggml_fit_opt( sched, ctx_compute, inputs, outputs, dataset, loss_type = ggml_opt_loss_type_mse(), optimizer = ggml_opt_optimizer_type_adamw(), nepoch = 10L, nbatch_logical = 32L, val_split = 0, shuffle = TRUE, shuffle_all = shuffle, callbacks = list(), silent = FALSE, loss_mask_ne0 = NULL, weight_decay = 0 )
sched |
Backend scheduler |
ctx_compute |
Compute context (for temporary tensors) |
inputs |
Input tensor with shape [ne_datapoint, batch_size] |
outputs |
Output tensor with shape [ne_label, batch_size] |
dataset |
Dataset created with 'ggml_opt_dataset_init()' |
loss_type |
Loss type (default: MSE) |
optimizer |
Optimizer type (default: AdamW) |
nepoch |
Number of epochs |
nbatch_logical |
Logical batch size (for gradient accumulation) |
val_split |
Fraction of data for validation (0.0 to 1.0) |
shuffle |
Shuffle the training portion each epoch (default
|
shuffle_all |
Shuffle the whole dataset once before the
train/validation split is taken, so the validation portion is a random
sample rather than the tail of the input. Defaults to |
callbacks |
List of callback lists. Each element may have 'on_epoch_begin(epoch, logs, state)' and/or 'on_epoch_end(epoch, logs, state)'. Built-in factories: 'ggml_callback_early_stopping()', 'ggml_schedule_step_decay()', 'ggml_schedule_cosine_decay()', 'ggml_schedule_reduce_on_plateau()'. 'state' is a mutable environment with fields: 'stop' (set TRUE to stop training), 'lr_ud', 'nepoch'. |
silent |
Whether to suppress per-epoch progress output |
loss_mask_ne0 |
Width of the weighted-MSE weight tensor: |
weight_decay |
Decoupled weight decay in |
Data frame with columns epoch, train_loss, train_accuracy, val_loss, val_accuracy
Other optimization:
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
if (FALSE) { history <- ggml_fit_opt(sched, ctx_compute, inputs, outputs, dataset, nepoch = 50, val_split = 0.2, callbacks = list( ggml_callback_early_stopping(monitor = "val_loss", patience = 5), ggml_schedule_cosine_decay() )) }if (FALSE) { history <- ggml_fit_opt(sched, ctx_compute, inputs, outputs, dataset, nepoch = 50, val_split = 0.2, callbacks = list( ggml_callback_early_stopping(monitor = "val_loss", patience = 5), ggml_schedule_cosine_decay() )) }
Streaming counterpart of ggml_fit_opt. Instead of a
ggml_opt_dataset holding every sample, batches are pulled from a
generator function, so only the current batch is ever in memory. This is what
makes three things possible that the dataset path cannot express: data larger
than RAM, augmentation that must differ per epoch, and generated sources
(RL rollouts, simulators) that have no fixed dataset at all.
ggml_fit_opt_gen( sched, ctx_compute, inputs, outputs, generator, steps_per_epoch = NULL, epochs = 1L, initial_epoch = 0L, validation_generator = NULL, validation_steps = NULL, loss_type = ggml_opt_loss_type_mse(), optimizer = ggml_opt_optimizer_type_adamw(), nbatch_logical = NULL, callbacks = list(), silent = FALSE, loss_mask_ne0 = NULL, weight_decay = 0 )ggml_fit_opt_gen( sched, ctx_compute, inputs, outputs, generator, steps_per_epoch = NULL, epochs = 1L, initial_epoch = 0L, validation_generator = NULL, validation_steps = NULL, loss_type = ggml_opt_loss_type_mse(), optimizer = ggml_opt_optimizer_type_adamw(), nbatch_logical = NULL, callbacks = list(), silent = FALSE, loss_mask_ne0 = NULL, weight_decay = 0 )
sched |
Backend scheduler |
ctx_compute |
Compute context holding the model graph |
inputs |
Input tensor of the built graph |
outputs |
Output tensor of the built graph |
generator |
Function of no arguments returning one batch, or
|
steps_per_epoch |
Number of batches that make up one epoch. Required
when |
epochs |
Number of epochs to run. |
initial_epoch |
Epoch number to start counting from, as in Keras. This
shifts the numbering seen by callbacks and learning-rate schedules only;
it does not restore optimizer state. A fresh |
validation_generator |
Optional generator used for validation after each epoch, following the same contract. Evaluated without a backward pass. |
validation_steps |
Number of validation batches per epoch. Required when
|
loss_type |
Loss type constant |
optimizer |
Optimizer type constant |
nbatch_logical |
Logical batch size, for gradient accumulation |
callbacks |
List of callback lists, as for |
silent |
Suppress per-epoch messages |
loss_mask_ne0 |
Width of the per-output loss mask, or NULL |
weight_decay |
Decoupled weight decay in |
A history data frame, one row per epoch, as
ggml_fit_opt returns.
generator is a function of no arguments. Each call returns one batch;
returning NULL ends the epoch. The same contract covers both finite
and infinite sources – an infinite generator simply never returns
NULL, and steps_per_epoch then becomes required, because
nothing else can say where an epoch ends.
A finite generator must restart itself. Training runs several
epochs over the same source, so a generator that ends must reset its own
state before returning NULL, leaving the next call to yield the first
batch again. Nothing else can do it: a bare closure has no handle to reset
from. An epoch that receives no batch at all stops training with a warning
rather than looping over nothing.
batch_gen <- function(x, y, batch_size) {
i <- 0L
nb <- nrow(x) %/% batch_size
function() {
if (i >= nb) {
i <<- 0L # reset, so the next epoch starts over
return(NULL)
}
idx <- (i * batch_size + 1L):((i + 1L) * batch_size)
i <<- i + 1L
list(x[idx, , drop = FALSE], y[idx, , drop = FALSE])
}
}
A batch is list(inputs = list(...), labels = list(...)). For the
common single-input case a plain list(x, y) is accepted as well. Each
element is a numeric vector or array already in ggml layout, holding exactly
one physical batch.
The generator is called from R, so it may do anything R can: read a shard from disk, augment the previous batch, or sample a replay buffer that a simulator is filling.
When nbatch_logical exceeds the graph's physical batch size, the
optimizer accumulates gradients over several generator batches and steps once
per logical batch. The period is tracked inside opt_ctx; the generator
is unaware of it and simply keeps producing physical batches.
ggml_fit_opt for the in-memory dataset path.
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Multi-output counterpart of ggml_fit_opt(). Each output head has its
own labels, loss type and weight; the optimized total is
sum(loss_weights[i] * loss_i). Because the epoch loop runs in R, this
path supports callbacks (early stopping, LR schedulers) and the
shuffle/shuffle_all controls, which the single C call
ggml_opt_fit_multi() cannot offer.
ggml_fit_opt_multi( sched, ctx_compute, inputs, outputs, dataset, loss_types, loss_weights = rep(1, length(outputs)), labels_offs = NULL, head_names = NULL, optimizer = ggml_opt_optimizer_type_adamw(), nepoch = 10L, nbatch_logical = 32L, val_split = 0, shuffle = TRUE, shuffle_all = shuffle, callbacks = list(), silent = FALSE, weight_decay = 0 )ggml_fit_opt_multi( sched, ctx_compute, inputs, outputs, dataset, loss_types, loss_weights = rep(1, length(outputs)), labels_offs = NULL, head_names = NULL, optimizer = ggml_opt_optimizer_type_adamw(), nepoch = 10L, nbatch_logical = 32L, val_split = 0, shuffle = TRUE, shuffle_all = shuffle, callbacks = list(), silent = FALSE, weight_decay = 0 )
sched |
Backend scheduler |
ctx_compute |
Compute context holding the model graph |
inputs |
Input tensor with shape [ne_datapoint, batch_size] |
outputs |
List of output tensors, one per head |
dataset |
Dataset with data and concatenated labels |
loss_types |
Integer vector of loss type constants, one per head |
loss_weights |
Numeric vector of head weights (default: all 1) |
labels_offs |
Numeric vector of 0-based label offsets, one per head.
Defaults to laying the heads out back to back in head order, using each
head's |
head_names |
Character vector naming the heads, one per head, used for
the per-head keys in |
optimizer |
Optimizer type (default: AdamW) |
nepoch |
Number of epochs |
nbatch_logical |
Logical batch size (for gradient accumulation) |
val_split |
Fraction of data for validation (0.0 to 1.0) |
shuffle |
Shuffle the training portion each epoch (default
|
shuffle_all |
Shuffle the whole dataset once before the
train/validation split is taken, so the validation portion is a random
sample rather than the tail of the input. Defaults to |
callbacks |
List of callback lists, as for |
silent |
Whether to suppress per-epoch progress output |
weight_decay |
Decoupled weight decay in |
All heads share the dataset's datapoints. The dataset's labels hold every
head's labels concatenated along the first dimension: head i occupies
labels_offs[i] + seq_len(width_i) of each label row.
Data frame with columns epoch, train_loss, train_accuracy, val_loss,
val_accuracy – one row per epoch actually run, so a callback that stops
early shortens it. The per-head metrics come as attributes rather than
columns, since they are matrices, each [nepoch_run x n_head] with columns
named after the heads: attr(h, "head_loss"),
attr(h, "head_accuracy"), attr(h, "val_head_loss") and
attr(h, "val_head_accuracy"), plus attr(h, "head_names").
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
if (FALSE) { history <- ggml_fit_opt_multi(sched, ctx_compute, inputs, list(out_a, out_b), dataset, loss_types = c(ggml_opt_loss_type_mse(), ggml_opt_loss_type_mse()), nepoch = 50, val_split = 0.2, callbacks = list(ggml_callback_early_stopping(monitor = "val_loss", patience = 5))) attr(history, "head_loss") }if (FALSE) { history <- ggml_fit_opt_multi(sched, ctx_compute, inputs, list(out_a, out_b), dataset, loss_types = c(ggml_opt_loss_type_mse(), ggml_opt_loss_type_mse()), nepoch = 50, val_split = 0.2, callbacks = list(ggml_callback_early_stopping(monitor = "val_loss", patience = 5))) attr(history, "head_loss") }
Dispatcher: if the first argument is a ggml_sequential_model, delegates
to the Keras-style high-level API (ggml_fit_sequential); otherwise
delegates to the low-level optimizer loop (ggml_fit_opt).
## S3 method for class 'ggml_functional_model' ggml_fit( model, x, y, epochs = 1L, batch_size = 32L, validation_split = 0, validation_data = NULL, verbose = 1L, shuffle = TRUE, callbacks = list(), ... ) ggml_fit(model, ...) ## S3 method for class 'ggml_sequential_model' ggml_fit(model, ...) ## Default S3 method: ggml_fit(model, ...)## S3 method for class 'ggml_functional_model' ggml_fit( model, x, y, epochs = 1L, batch_size = 32L, validation_split = 0, validation_data = NULL, verbose = 1L, shuffle = TRUE, callbacks = list(), ... ) ggml_fit(model, ...) ## S3 method for class 'ggml_sequential_model' ggml_fit(model, ...) ## Default S3 method: ggml_fit(model, ...)
model |
A compiled model object. |
x |
Training data (matrix or array). |
y |
Training labels (matrix, one-hot encoded). |
epochs |
Number of training epochs (default: 1). |
batch_size |
Batch size (default: 32). |
validation_split |
Fraction of data for validation (default: 0). |
validation_data |
Optional list(x_val, y_val). Overrides validation_split. |
verbose |
0 = silent, 1 = progress (default: 1). |
shuffle |
Shuffle the data (default |
callbacks |
List of callback objects, e.g.
Both phases carry a prefix on purpose: it keeps an output named
|
... |
Arguments passed to the appropriate implementation. |
Keras-style (Sequential model):
A compiled ggml_sequential_model
Training data (matrix or array)
Training labels (matrix, one-hot encoded for classification)
Number of training epochs (default: 1)
Batch size (default: 32)
Fraction of data for validation (default: 0)
Optional list(x_val, y_val) for validation. Overrides validation_split.
Named vector of weights per class, e.g. c("0"=1, "1"=10). Cannot be used with sample_weight.
Per-sample weights: a numeric vector of length
nrow(x), or – for mean_squared_error only – a
nrow(x) by ncol(y) matrix acting as a per-output loss mask.
A mask trains only the outputs whose weight is non-zero, which is what a
Q-model needs: 1 on the action that has a TD target, 0 elsewhere. The
denominator is unchanged (sum(w*(pred-y)^2)/nelements), so masking
does not renormalise over the active outputs. Cannot be used with
class_weight.
0 = silent, 1 = progress (default: 1)
Shuffle the data (default: TRUE). Shuffled once before the train/validation split, then the training portion each epoch; the validation portion stays fixed. FALSE for time series or exactly reproducible runs.
List of callback objects (early stopping, LR schedules)
Function of no arguments yielding list(x, y) for
one batch, or NULL when exhausted – an alternative to x
and y that keeps only the current batch in memory. See
ggml_fit_generator, to which the call is forwarded along
with steps_per_epoch, initial_epoch,
validation_generator and validation_steps. The arguments
that are defined in terms of a whole dataset –
validation_split, validation_data, class_weight
and sample_weight – are rejected rather than ignored, and
shuffle is inert because the generator owns its own order.
Low-level (optimizer loop):
Backend scheduler
Compute context
Input tensor
Output tensor
Dataset from ggml_opt_dataset_init()
Loss type (default: MSE)
Optimizer type (default: AdamW)
Number of epochs (default: 10)
Logical batch size (default: 32)
Validation fraction (default: 0)
Shuffle the data (default: TRUE)
List of callback objects
Suppress output (default: FALSE)
For Sequential models: the trained model (invisibly).
For the low-level API: a data frame with columns
epoch, train_loss, train_accuracy,
val_loss, val_accuracy.
ggml_set_n_threads(1L) # deterministic, single OpenMP pool n <- 128 x <- matrix(runif(n * 4), nrow = n, ncol = 4) y <- matrix(0, nrow = n, ncol = 2) for (i in seq_len(n)) { y[i, if (sum(x[i,]) > 2) 1L else 2L] <- 1 } model <- ggml_model_sequential() |> ggml_layer_dense(8, activation = "relu") |> ggml_layer_dense(2, activation = "softmax") model$input_shape <- 4L model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy") # Basic training model <- ggml_fit(model, x, y, epochs = 5, batch_size = 32, verbose = 0) # With validation_data x_val <- matrix(runif(32 * 4), nrow = 32, ncol = 4) y_val <- matrix(0, nrow = 32, ncol = 2) for (i in seq_len(32)) { y_val[i, if (sum(x_val[i,]) > 2) 1L else 2L] <- 1 } model <- ggml_fit(model, x, y, epochs = 3, batch_size = 32, validation_data = list(x_val, y_val), verbose = 0) # With class_weight (useful for imbalanced classes) model <- ggml_fit(model, x, y, epochs = 3, batch_size = 32, class_weight = c("0" = 1, "1" = 2), verbose = 0) # With sample_weight sw <- runif(n, 0.5, 1.5) model <- ggml_fit(model, x, y, epochs = 3, batch_size = 32, sample_weight = sw, verbose = 0)ggml_set_n_threads(1L) # deterministic, single OpenMP pool n <- 128 x <- matrix(runif(n * 4), nrow = n, ncol = 4) y <- matrix(0, nrow = n, ncol = 2) for (i in seq_len(n)) { y[i, if (sum(x[i,]) > 2) 1L else 2L] <- 1 } model <- ggml_model_sequential() |> ggml_layer_dense(8, activation = "relu") |> ggml_layer_dense(2, activation = "softmax") model$input_shape <- 4L model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy") # Basic training model <- ggml_fit(model, x, y, epochs = 5, batch_size = 32, verbose = 0) # With validation_data x_val <- matrix(runif(32 * 4), nrow = 32, ncol = 4) y_val <- matrix(0, nrow = 32, ncol = 2) for (i in seq_len(32)) { y_val[i, if (sum(x_val[i,]) > 2) 1L else 2L] <- 1 } model <- ggml_fit(model, x, y, epochs = 3, batch_size = 32, validation_data = list(x_val, y_val), verbose = 0) # With class_weight (useful for imbalanced classes) model <- ggml_fit(model, x, y, epochs = 3, batch_size = 32, class_weight = c("0" = 1, "1" = 2), verbose = 0) # With sample_weight sw <- runif(n, 0.5, 1.5) model <- ggml_fit(model, x, y, epochs = 3, batch_size = 32, sample_weight = sw, verbose = 0)
Backward pass for ggml_flash_attn_ext: given the gradient of
the attention output, produces the gradients of Q, K and V in one pass.
ggml_flash_attn_back(ctx, q, k, v, mask, d, scale)ggml_flash_attn_back(ctx, q, k, v, mask, d, scale)
ctx |
GGML context |
q |
Query tensor |
k |
Key tensor |
v |
Value tensor |
mask |
Attention mask |
d |
Gradient of the forward output, shape |
scale |
Softmax scale used in the forward pass (e.g. |
Upstream ggml ships this operation as a stub that aborts, left behind when
flash_attn was replaced by flash_attn_ext; ggmlR implements it
so attention can be trained through the fused op. It is also wired into the
graph autodiff, so ggml_build_backward_expand reaches it on its
own — calling this directly is only needed when building a backward graph
by hand.
The three gradients come back packed head to tail in one contiguous 1D
tensor: grad_q, then grad_k, then grad_v, each padded to
GGML_MEM_ALIGN. Take a view of the slice you need.
Not supported: max_bias (ALiBi), logit_softcap and attention
sinks. Q, K, V and d must be F32.
Tensor holding grad_q, grad_k and grad_v packed contiguously
ggml_flash_attn_ext for the forward pass.
Creates a graph node for Flash Attention computation. This is a memory-efficient implementation of scaled dot-product attention.
ggml_flash_attn_ext( ctx, q, k, v, mask = NULL, scale, max_bias = 0, logit_softcap = 0 )ggml_flash_attn_ext( ctx, q, k, v, mask = NULL, scale, max_bias = 0, logit_softcap = 0 )
ctx |
GGML context |
q |
Query tensor of shape [head_dim, n_head, n_tokens, batch] |
k |
Key tensor of shape [head_dim, n_head_kv, n_kv, batch] |
v |
Value tensor of shape [head_dim, n_head_kv, n_kv, batch] |
mask |
Optional attention mask tensor (NULL for no mask). For causal attention, use ggml_diag_mask_inf instead. |
scale |
Attention scale factor, typically 1/sqrt(head_dim) |
max_bias |
Maximum ALiBi bias (0.0 to disable ALiBi) |
logit_softcap |
Logit soft-capping value (0.0 to disable). Used by some models like Gemma 2. |
Flash Attention computes: softmax(Q * K^T / scale + mask) * V
Key features: - Memory efficient: O(n) instead of O(n^2) memory for attention matrix - Supports grouped-query attention (GQA) when n_head_kv < n_head - Supports multi-query attention (MQA) when n_head_kv = 1 - Optional ALiBi (Attention with Linear Biases) for position encoding - Optional logit soft-capping for numerical stability
Attention output tensor of shape [head_dim, n_head, n_tokens, batch]
ctx <- ggml_init(64 * 1024 * 1024) head_dim <- 64 n_head <- 8 n_head_kv <- 2 # GQA with 4:1 ratio seq_len <- 32 q <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, head_dim, n_head, seq_len, 1) k <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, head_dim, n_head_kv, seq_len, 1) v <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, head_dim, n_head_kv, seq_len, 1) ggml_set_f32(q, rnorm(head_dim * n_head * seq_len)) ggml_set_f32(k, rnorm(head_dim * n_head_kv * seq_len)) ggml_set_f32(v, rnorm(head_dim * n_head_kv * seq_len)) # Scale = 1/sqrt(head_dim) scale <- 1.0 / sqrt(head_dim) # Compute attention out <- ggml_flash_attn_ext(ctx, q, k, v, NULL, scale, 0.0, 0.0) graph <- ggml_build_forward_expand(ctx, out) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(64 * 1024 * 1024) head_dim <- 64 n_head <- 8 n_head_kv <- 2 # GQA with 4:1 ratio seq_len <- 32 q <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, head_dim, n_head, seq_len, 1) k <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, head_dim, n_head_kv, seq_len, 1) v <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, head_dim, n_head_kv, seq_len, 1) ggml_set_f32(q, rnorm(head_dim * n_head * seq_len)) ggml_set_f32(k, rnorm(head_dim * n_head_kv * seq_len)) ggml_set_f32(v, rnorm(head_dim * n_head_kv * seq_len)) # Scale = 1/sqrt(head_dim) scale <- 1.0 / sqrt(head_dim) # Compute attention out <- ggml_flash_attn_ext(ctx, q, k, v, NULL, scale, 0.0, 0.0) graph <- ggml_build_forward_expand(ctx, out) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Adds per-head "sink" logits, an always-available extra key that lets a query attend to nothing in particular.
ggml_flash_attn_ext_add_sinks(tensor, sinks)ggml_flash_attn_ext_add_sinks(tensor, sinks)
tensor |
A flash-attention node |
sinks |
Sink logits, one per head |
The tensor (for chaining)
Get Flash-Attention Precision
ggml_flash_attn_ext_get_prec(tensor)ggml_flash_attn_ext_get_prec(tensor)
tensor |
A flash-attention node |
The precision constant currently set on the node
Sets the precision a ggml_flash_attn_ext() node computes in.
GGML_PREC_F32 raises accuracy at some cost in speed.
ggml_flash_attn_ext_set_prec(tensor, prec)ggml_flash_attn_ext_set_prec(tensor, prec)
tensor |
A flash-attention node |
prec |
Precision constant |
The tensor (for chaining)
Creates a graph node for element-wise floor: floor(x)
ggml_floor(ctx, a)ggml_floor(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the floor operation
Creates a graph node for in-place element-wise floor.
ggml_floor_inplace(ctx, a)ggml_floor_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with floor values
Free GGML context
ggml_free(ctx)ggml_free(ctx)
ctx |
Context pointer |
NULL (invisible)
ctx <- ggml_init(1024 * 1024) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) ggml_free(ctx)
ggml_freeze_bn_stats() fixes the running mean and variance of
batch-norm layers: they then normalize with those stored statistics during
training as well as at prediction, and ggml_fit no longer
recalibrates them. gamma and beta still train unless the layer
is also frozen with ggml_freeze_weights – the usual fine-tuning
setting when batches are small. ggml_unfreeze_bn_stats() returns the
layers to batch statistics in training and recalibration after a fit, which
is the default.
ggml_freeze_bn_stats( model, from = 1L, to = length(model$layers), layer_names = NULL, prefix = NULL, ... ) ggml_unfreeze_bn_stats( model, from = 1L, to = length(model$layers), layer_names = NULL, prefix = NULL, ... )ggml_freeze_bn_stats( model, from = 1L, to = length(model$layers), layer_names = NULL, prefix = NULL, ... ) ggml_unfreeze_bn_stats( model, from = 1L, to = length(model$layers), layer_names = NULL, prefix = NULL, ... )
model |
A model object (ggml_sequential_model or ggml_functional_model) |
from |
Integer index of the first layer to freeze (default: 1) |
to |
Integer index of the last layer to freeze (default: last layer) |
layer_names |
Character vector of layer names to freeze (overrides from/to) |
prefix |
Character vector: freeze every layer whose name starts with one
of these – a whole transformer block, say: |
... |
Additional arguments passed to methods |
Freezing the weights alone does not fix the statistics: a frozen batch-norm
still normalizes each training batch by its own mean and variance, and its
running statistics are still recalibrated after ggml_fit().
The fixed state is saved by ggml_save_model. Only sequential
models carry running statistics; the functional batch_norm is a
per-sample RMS normalization (see ggml_layer_batch_norm),
already independent of the batch, and is refused with a message.
The model with the selected batch-norm layers changed. Selected layers of other types are skipped; a selection with no batch-norm layer at all is an error.
Sets trainable = FALSE on layers, preventing their weights from being
updated during training. Layers are chosen by layer_names and/or
prefix; without either, by the index range from / to
(sequential models), or all of them.
ggml_freeze_weights( model, from = 1L, to = length(model$layers), layer_names = NULL, prefix = NULL, ... )ggml_freeze_weights( model, from = 1L, to = length(model$layers), layer_names = NULL, prefix = NULL, ... )
model |
A model object (ggml_sequential_model or ggml_functional_model) |
from |
Integer index of the first layer to freeze (default: 1) |
to |
Integer index of the last layer to freeze (default: last layer) |
layer_names |
Character vector of layer names to freeze (overrides from/to) |
prefix |
Character vector: freeze every layer whose name starts with one
of these – a whole transformer block, say: |
... |
Additional arguments passed to methods |
The model with selected layers frozen.
ggml_freeze_bn_stats to fix batch-norm statistics,
which freezing the weights does not do.
model <- ggml_model_sequential() |> ggml_layer_dense(64, activation = "relu") |> ggml_layer_dense(10, activation = "softmax") # Freeze all layers model <- ggml_freeze_weights(model) # Freeze only the first layer model <- ggml_freeze_weights(model, from = 1, to = 1)model <- ggml_model_sequential() |> ggml_layer_dense(64, activation = "relu") |> ggml_layer_dense(10, activation = "softmax") # Freeze all layers model <- ggml_freeze_weights(model) # Freeze only the first layer model <- ggml_freeze_weights(model, from = 1, to = 1)
Converts a file type (ftype) to the corresponding GGML type. Used when loading quantized models.
ggml_ftype_to_ggml_type(ftype)ggml_ftype_to_ggml_type(ftype)
ftype |
File type constant |
Integer GGML type
Other type_system:
ggml_blck_size(),
ggml_is_quantized(),
ggml_type_name(),
ggml_type_sizef()
Allocates memory for all tensors in the computation graph. This must be called before computing the graph.
ggml_gallocr_alloc_graph(galloc, graph)ggml_gallocr_alloc_graph(galloc, graph)
galloc |
Graph allocator object |
graph |
Graph object |
TRUE on success, FALSE on failure
ctx <- ggml_init(16 * 1024 * 1024) galloc <- ggml_gallocr_new() # Create graph a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, b) # Allocate and compute ggml_gallocr_alloc_graph(galloc, graph) ggml_graph_compute(ctx, graph) ggml_gallocr_free(galloc) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) galloc <- ggml_gallocr_new() # Create graph a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, b) # Allocate and compute ggml_gallocr_alloc_graph(galloc, graph) ggml_graph_compute(ctx, graph) ggml_gallocr_free(galloc) ggml_free(ctx)
Frees a graph allocator and all associated buffers.
ggml_gallocr_free(galloc)ggml_gallocr_free(galloc)
galloc |
Graph allocator object |
No return value, called for side effects
Returns the size of the buffer used by the graph allocator.
ggml_gallocr_get_buffer_size(galloc, buffer_id = 0L)ggml_gallocr_get_buffer_size(galloc, buffer_id = 0L)
galloc |
Graph allocator object |
buffer_id |
Buffer ID (default: 0 for single-buffer allocator) |
Size in bytes
Creates a new graph allocator for efficient memory management. The allocator can automatically allocate and reuse memory for graph tensors.
ggml_gallocr_new()ggml_gallocr_new()
Graph allocator object (external pointer)
ctx <- ggml_init(16 * 1024 * 1024) galloc <- ggml_gallocr_new() a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, b) # Allocate graph ggml_gallocr_alloc_graph(galloc, graph) ggml_gallocr_free(galloc) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) galloc <- ggml_gallocr_new() a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, b) # Allocate graph ggml_gallocr_alloc_graph(galloc, graph) ggml_gallocr_free(galloc) ggml_free(ctx)
Like ggml_gallocr_new, but allocates from the given buffer type
rather than the CPU default – so a graph can be allocated in GPU memory.
ggml_gallocr_new_buft(buft)ggml_gallocr_new_buft(buft)
buft |
Buffer type (e.g. from |
Graph allocator object (external pointer)
ggml_gallocr_new, ggml_gallocr_alloc_graph
Pre-allocates memory for a graph. This is optional but recommended when running the same graph multiple times to avoid reallocation.
ggml_gallocr_reserve(galloc, graph)ggml_gallocr_reserve(galloc, graph)
galloc |
Graph allocator object |
graph |
Graph object |
TRUE on success, FALSE on failure
Linear attention with a per-token gate (the GLA / RetNet family). Shares the
RWKV recurrence machinery: keys, values and queries drive a running state
that g gates, and scale multiplies the query.
ggml_gated_linear_attn(ctx, k, v, q, g, state, scale)ggml_gated_linear_attn(ctx, k, v, q, g, state, scale)
ctx |
GGML context |
k |
Keys, |
v |
Values, |
q |
Queries, |
g |
Gate, |
state |
Initial state, |
scale |
Scalar applied to the queries (usually |
A tensor [S * H, n_tokens + S * n_seqs]
k, v, q and g are [S, H, n_tokens];
state holds S * S * H * n_seqs elements. The result is
[S * H, n_tokens + S * n_seqs], outputs then final state – see
ggml_rwkv_output.
n_tokens counts the tokens of all sequences together, so it
must be a multiple of n_seqs.
CPU only. Unlike the RWKV and SSM ops, GGML_OP_GATED_LINEAR_ATTN has
no Vulkan shader, so a graph containing it falls back to the CPU for this
node. It has no backward pass either, so it is inference-only.
ggml_rwkv_wkv7, ggml_rwkv_output
Other state-space:
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Gradients of ggml_gated_linear_attn, packed in forward source
order: [d_k | d_v | d_q | d_g | d_state].
ggml_gated_linear_attn_back(ctx, k, v, q, g, state, grad, scale)ggml_gated_linear_attn_back(ctx, k, v, q, g, state, grad, scale)
ctx |
GGML context |
k, v, q, g, state
|
The tensors passed to the forward op |
grad |
Gradient w.r.t. the packed forward result |
scale |
The scale the forward op was built with |
A flat tensor holding the five gradients, concatenated
Other state-space:
ggml_gated_linear_attn(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Creates a graph node for GeGLU operation. GeGLU uses GELU as the activation function on the first half. CRITICAL for models like GPT-NeoX and Falcon.
ggml_geglu(ctx, a)ggml_geglu(ctx, a)
ctx |
GGML context |
a |
Input tensor (first dimension must be even) |
Formula: output = GELU(x) * gate
Tensor with half the first dimension of input
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 3) ggml_set_f32(a, rnorm(24)) r <- ggml_geglu(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # Shape: 4x3 ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 3) ggml_set_f32(a, rnorm(24)) r <- ggml_geglu(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # Shape: 4x3 ggml_free(ctx)
Creates a graph node for fast GeGLU approximation. Uses faster but less accurate GELU approximation for gating.
ggml_geglu_quick(ctx, a)ggml_geglu_quick(ctx, a)
ctx |
GGML context |
a |
Input tensor (first dimension must be even) |
Tensor with half the first dimension of input
Creates a graph node for GeGLU with separate input and gate tensors.
ggml_geglu_split(ctx, a, b)ggml_geglu_split(ctx, a, b)
ctx |
GGML context |
a |
Input tensor (the values to be gated) |
b |
Gate tensor (same shape as a) |
Formula: output = GELU(a) * b
Tensor with same shape as input tensors
Creates a graph node for GELU (Gaussian Error Linear Unit) activation. CRITICAL for GPT models.
ggml_gelu(ctx, a)ggml_gelu(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the GELU operation
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) result <- ggml_gelu(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) result <- ggml_gelu(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)
Gradient of the GELU activation: dx = dy * dgelu(x). Wired into the
autodiff engine, so ggml_compile(activation = "gelu") trains – this
is a ggmlR extension, upstream ggml has no GELU backward and aborts instead.
ggml_gelu_back(ctx, a, b)ggml_gelu_back(ctx, a, b)
ctx |
GGML context |
a |
Gradient tensor from upstream ( |
b |
Forward input tensor ( |
The derivative is that of the tanh approximation ggml_gelu
computes,
g(x) = 0.5x(1 + tanh(u)) with u = c*x*(1 + a*x^2). The quick
and erf variants (ggml_gelu_quick, ggml_gelu_erf)
are different functions with different derivatives and remain without a
backward rule.
Gradient tensor for the input
Creates a graph node for exact GELU using the error function (erf). GELU(x) = x * 0.5 * (1 + erf(x / sqrt(2))). More accurate than approximate GELU but potentially slower on some backends.
ggml_gelu_erf(ctx, a)ggml_gelu_erf(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the exact GELU operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) r <- ggml_gelu_erf(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) r <- ggml_gelu_erf(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) ggml_free(ctx)
Creates a graph node for in-place GELU (Gaussian Error Linear Unit) activation. CRITICAL for GPT models with memory efficiency.
ggml_gelu_inplace(ctx, a)ggml_gelu_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with GELU applied
Creates a graph node for fast approximation of GELU. Faster than standard GELU with minimal accuracy loss.
ggml_gelu_quick(ctx, a)ggml_gelu_quick(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the GELU quick operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) result <- ggml_gelu_quick(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) result <- ggml_gelu_quick(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) ggml_free(ctx)
Get F32 data
Get F32 Data
ggml_get_f32(tensor) ggml_get_f32(tensor)ggml_get_f32(tensor) ggml_get_f32(tensor)
tensor |
Tensor |
Numeric vector with tensor values
Numeric vector
ctx <- ggml_init(1024 * 1024) tensor <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(tensor, c(1, 2, 3, 4, 5)) ggml_get_f32(tensor) ggml_free(ctx) ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(t, c(1, 2, 3, 4, 5)) ggml_get_f32(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) tensor <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(tensor, c(1, 2, 3, 4, 5)) ggml_get_f32(tensor) ggml_free(ctx) ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(t, c(1, 2, 3, 4, 5)) ggml_get_f32(t) ggml_free(ctx)
Gets a single f32 value from the tensor at position [i0, i1, i2, i3]. Works with any tensor type (auto-converts to float).
ggml_get_f32_nd(tensor, i0, i1 = 0, i2 = 0, i3 = 0)ggml_get_f32_nd(tensor, i0, i1 = 0, i2 = 0, i3 = 0)
tensor |
Tensor pointer |
i0, i1, i2, i3
|
Indices (0-based) |
Float value
Get First Tensor from Context
ggml_get_first_tensor(ctx)ggml_get_first_tensor(ctx)
ctx |
GGML context |
Tensor pointer or NULL
Gets integer data from an I32 tensor (e.g., from ggml_argmax)
ggml_get_i32(tensor)ggml_get_i32(tensor)
tensor |
Tensor of type GGML_TYPE_I32 |
Integer vector
ctx <- ggml_init(1024 * 1024) pos <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 10) ggml_set_i32(pos, 0:9) ggml_get_i32(pos) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) pos <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 10) ggml_set_i32(pos, 0:9) ggml_get_i32(pos) ggml_free(ctx)
Gets a single i32 value from the tensor at position [i0, i1, i2, i3].
ggml_get_i32_nd(tensor, i0, i1 = 0, i2 = 0, i3 = 0)ggml_get_i32_nd(tensor, i0, i1 = 0, i2 = 0, i3 = 0)
tensor |
Tensor pointer |
i0, i1, i2, i3
|
Indices (0-based) |
Integer value
Retrieves a layer by name or by integer index (1-based).
ggml_get_layer(model, index = NULL, name = NULL)ggml_get_layer(model, index = NULL, name = NULL)
model |
A ggml_sequential_model object |
index |
Integer index of the layer (1-based), or NULL |
name |
Character name of the layer, or NULL |
The layer list object
model <- ggml_model_sequential() |> ggml_layer_dense(64, activation = "relu", name = "hidden") |> ggml_layer_dense(10, activation = "softmax", name = "output") ggml_get_layer(model, index = 1) ggml_get_layer(model, name = "output")model <- ggml_model_sequential() |> ggml_layer_dense(64, activation = "relu", name = "hidden") |> ggml_layer_dense(10, activation = "softmax", name = "output") ggml_get_layer(model, index = 1) ggml_get_layer(model, name = "output")
Returns the maximum tensor size that can be allocated in the context
ggml_get_max_tensor_size(ctx)ggml_get_max_tensor_size(ctx)
ctx |
GGML context |
Maximum tensor size in bytes
ctx <- ggml_init(1024 * 1024) ggml_get_max_tensor_size(ctx) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) ggml_get_max_tensor_size(ctx) ggml_free(ctx)
Returns the total memory pool size of the context
ggml_get_mem_size(ctx)ggml_get_mem_size(ctx)
ctx |
GGML context |
Total memory size in bytes
ctx <- ggml_init(1024 * 1024) ggml_get_mem_size(ctx) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) ggml_get_mem_size(ctx) ggml_free(ctx)
Get the current number of threads for GGML operations
ggml_get_n_threads()ggml_get_n_threads()
Number of threads
ggml_get_n_threads()ggml_get_n_threads()
Retrieves the name of a tensor.
ggml_get_name(tensor)ggml_get_name(tensor)
tensor |
Tensor pointer |
Character string name or NULL if not set
Get Next Tensor from Context
ggml_get_next_tensor(ctx, tensor)ggml_get_next_tensor(ctx, tensor)
ctx |
GGML context |
tensor |
Current tensor |
Next tensor pointer or NULL
Check if no-allocation mode is enabled
ggml_get_no_alloc(ctx)ggml_get_no_alloc(ctx)
ctx |
GGML context |
Logical indicating if no_alloc is enabled
ctx <- ggml_init(1024 * 1024) ggml_get_no_alloc(ctx) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) ggml_get_no_alloc(ctx) ggml_free(ctx)
Returns the raw op_params bytes from a tensor. These parameters control operation-specific behavior (e.g., precision, mode).
ggml_get_op_params(tensor)ggml_get_op_params(tensor)
tensor |
External pointer to tensor |
Raw vector of op_params bytes
Other tensor:
ggml_are_same_layout(),
ggml_get_op_params_f32(),
ggml_get_op_params_i32(),
ggml_set_op_params(),
ggml_set_op_params_f32(),
ggml_set_op_params_i32()
Gets a single float value from tensor op_params at given index.
ggml_get_op_params_f32(tensor, index)ggml_get_op_params_f32(tensor, index)
tensor |
External pointer to tensor |
index |
0-based index (0-15 for 64-byte op_params) |
Numeric value
Other tensor:
ggml_are_same_layout(),
ggml_get_op_params(),
ggml_get_op_params_i32(),
ggml_set_op_params(),
ggml_set_op_params_f32(),
ggml_set_op_params_i32()
Gets a single int32 value from tensor op_params at given index.
ggml_get_op_params_i32(tensor, index)ggml_get_op_params_i32(tensor, index)
tensor |
External pointer to tensor |
index |
0-based index (0-15 for 64-byte op_params) |
Integer value
Other tensor:
ggml_are_same_layout(),
ggml_get_op_params(),
ggml_get_op_params_f32(),
ggml_set_op_params(),
ggml_set_op_params_f32(),
ggml_set_op_params_i32()
Gathers relative-position rows for relative-position attention bias.
ggml_get_rel_pos(ctx, a, qh, kh)ggml_get_rel_pos(ctx, a, qh, kh)
ctx |
GGML context |
a |
Input tensor |
qh |
Query height |
kh |
Key height |
Relative-position tensor
Creates a graph node that extracts rows from a tensor by index. This is commonly used for embedding lookup in LLMs.
ggml_get_rows(ctx, a, b)ggml_get_rows(ctx, a, b)
ctx |
GGML context |
a |
Data tensor of shape [n_embd, n_rows, ...] - the embedding table |
b |
Index tensor (int32) of shape [n_indices] - which rows to extract |
This operation is fundamental for embedding lookup in transformers: given a vocabulary embedding matrix and token indices, it retrieves the corresponding embedding vectors.
Tensor of shape [n_embd, n_indices, ...] containing the selected rows
ctx <- ggml_init(16 * 1024 * 1024) # Create embedding matrix: 10 tokens, 4-dim embeddings embeddings <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 10) ggml_set_f32(embeddings, rnorm(40)) # Token indices to look up indices <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 3) ggml_set_i32(indices, c(0L, 5L, 2L)) # Get embeddings for tokens 0, 5, 2 result <- ggml_get_rows(ctx, embeddings, indices) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) # Create embedding matrix: 10 tokens, 4-dim embeddings embeddings <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 10) ggml_set_f32(embeddings, rnorm(40)) # Token indices to look up indices <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 3) ggml_set_i32(indices, c(0L, 5L, 2L)) # Get embeddings for tokens 0, 5, 2 result <- ggml_get_rows(ctx, embeddings, indices) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Backward pass for ggml_get_rows operation. Accumulates gradients at the original row positions.
ggml_get_rows_back(ctx, a, b, c)ggml_get_rows_back(ctx, a, b, c)
ctx |
GGML context |
a |
Gradient of get_rows output |
b |
Index tensor (same as forward pass) |
c |
Reference tensor defining output shape |
Gradient tensor for the embedding matrix
Returns the unary operation type for a unary operation tensor.
ggml_get_unary_op(tensor)ggml_get_unary_op(tensor)
tensor |
Tensor pointer (must be a unary operation result) |
Integer unary operation type
Other op_info:
ggml_op_desc(),
ggml_op_name(),
ggml_op_symbol(),
ggml_unary_op_name()
Creates a graph node for GLU operation with specified gating type. GLU splits the input tensor in half along the first dimension, applies an activation to the first half (x), and multiplies it with the second half (gate).
ggml_glu(ctx, a, op, swapped = FALSE)ggml_glu(ctx, a, op, swapped = FALSE)
ctx |
GGML context |
a |
Input tensor (first dimension must be even) |
op |
GLU operation type (GGML_GLU_OP_REGLU, GGML_GLU_OP_GEGLU, etc.) |
swapped |
If TRUE, swap x and gate halves (default FALSE) |
Formula: output = activation(x) * gate where x and gate are the two halves of the input tensor.
Tensor with shape [n/2, ...] where n is the first dimension of input
ctx <- ggml_init(16 * 1024 * 1024) # Create tensor with 10 columns (will be split into 5 + 5) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 4) ggml_set_f32(a, rnorm(40)) # Apply SwiGLU r <- ggml_glu(ctx, a, GGML_GLU_OP_SWIGLU, FALSE) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # Shape: 5x4 ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) # Create tensor with 10 columns (will be split into 5 + 5) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 4) ggml_set_f32(a, rnorm(40)) # Apply SwiGLU r <- ggml_glu(ctx, a, GGML_GLU_OP_SWIGLU, FALSE) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # Shape: 5x4 ggml_free(ctx)
Constants for GLU (Gated Linear Unit) operation types. Used with ggml_glu() and ggml_glu_split().
GGML_GLU_OP_REGLU GGML_GLU_OP_GEGLU GGML_GLU_OP_SWIGLU GGML_GLU_OP_SWIGLU_OAI GGML_GLU_OP_GEGLU_ERF GGML_GLU_OP_GEGLU_QUICKGGML_GLU_OP_REGLU GGML_GLU_OP_GEGLU GGML_GLU_OP_SWIGLU GGML_GLU_OP_SWIGLU_OAI GGML_GLU_OP_GEGLU_ERF GGML_GLU_OP_GEGLU_QUICK
Integer constants
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
GGML_GLU_OP_REGLU (0): ReGLU - ReLU gating
GGML_GLU_OP_GEGLU (1): GeGLU - GELU gating (used in GPT-NeoX, Falcon)
GGML_GLU_OP_SWIGLU (2): SwiGLU - SiLU/Swish gating (used in LLaMA, Mistral)
GGML_GLU_OP_SWIGLU_OAI (3): SwiGLU OpenAI variant
GGML_GLU_OP_GEGLU_ERF (4): GeGLU with exact erf implementation
GGML_GLU_OP_GEGLU_QUICK (5): GeGLU with fast approximation
An integer constant representing a GLU operation type
GGML_GLU_OP_REGLU # 0 - ReLU gating GGML_GLU_OP_GEGLU # 1 - GELU gating GGML_GLU_OP_SWIGLU # 2 - SiLU/Swish gating GGML_GLU_OP_SWIGLU_OAI # 3 - SwiGLU OpenAI GGML_GLU_OP_GEGLU_ERF # 4 - GELU with erf GGML_GLU_OP_GEGLU_QUICK # 5 - Fast GELUGGML_GLU_OP_REGLU # 0 - ReLU gating GGML_GLU_OP_GEGLU # 1 - GELU gating GGML_GLU_OP_SWIGLU # 2 - SiLU/Swish gating GGML_GLU_OP_SWIGLU_OAI # 3 - SwiGLU OpenAI GGML_GLU_OP_GEGLU_ERF # 4 - GELU with erf GGML_GLU_OP_GEGLU_QUICK # 5 - Fast GELU
Creates a graph node for GLU with separate input and gate tensors. Unlike standard GLU which splits a single tensor, this takes two separate tensors.
ggml_glu_split(ctx, a, b, op)ggml_glu_split(ctx, a, b, op)
ctx |
GGML context |
a |
Input tensor (the values to be gated) |
b |
Gate tensor (same shape as a) |
op |
GLU operation type (GGML_GLU_OP_REGLU, GGML_GLU_OP_GEGLU, etc.) |
Tensor with same shape as input tensors
Adds tensor to graph as a node, without expanding its
dependencies. For the usual case – adding a tensor together with everything
it depends on – use ggml_graph_expand.
ggml_graph_add_node(graph, tensor)ggml_graph_add_node(graph, tensor)
graph |
Graph to add to |
tensor |
Tensor to append |
NULL, invisibly; graph is modified in place
Removes every node from graph, keeping its allocated capacity so it
can be rebuilt without a fresh allocation.
ggml_graph_clear(graph)ggml_graph_clear(graph)
graph |
Graph to clear |
NULL, invisibly; graph is modified in place
Executes all operations in the computation graph.
Executes the computation graph using CPU backend
ggml_graph_compute(ctx, graph) ggml_graph_compute(ctx, graph)ggml_graph_compute(ctx, graph) ggml_graph_compute(ctx, graph)
ctx |
GGML context |
graph |
Graph object created by ggml_build_forward_expand |
NULL (invisible)
No return value, called for side effects
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) result <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx) ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_set_f32(a, 1:10) ggml_set_f32(b, 11:20) c <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, c) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(c) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) result <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx) ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_set_f32(a, 1:10) ggml_set_f32(b, 11:20) c <- ggml_add(ctx, a, b) graph <- ggml_build_forward_expand(ctx, c) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(c) ggml_free(ctx)
Computes the computation graph using the context-based method. This is an alternative to ggml_graph_compute() that uses ggml_graph_plan() and ggml_graph_compute() internally.
ggml_graph_compute_with_ctx(ctx, graph, n_threads = 0L)ggml_graph_compute_with_ctx(ctx, graph, n_threads = 0L)
ctx |
GGML context |
graph |
Graph object created by ggml_build_forward_expand |
n_threads |
Number of threads to use (0 for auto-detect, default: 0) |
No return value, called for side effects
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_set_f32(a, 1:10) c <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, c) ggml_graph_compute_with_ctx(ctx, graph) result <- ggml_get_f32(c) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_set_f32(a, 1:10) c <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, c) ggml_graph_compute_with_ctx(ctx, graph) result <- ggml_get_f32(c) ggml_free(ctx)
Copies the contents of src into dst, which must already have
room for them. Unlike ggml_graph_dup this allocates nothing.
ggml_graph_cpy(src, dst)ggml_graph_cpy(src, dst)
src |
Source graph |
dst |
Destination graph |
NULL, invisibly; dst is modified in place
Exports the computation graph to a DOT file for visualization. The DOT file can be converted to an image using Graphviz tools.
ggml_graph_dump_dot(graph, leafs = NULL, filename)ggml_graph_dump_dot(graph, leafs = NULL, filename)
graph |
Graph object |
leafs |
Optional graph with leaf tensors (NULL for none) |
filename |
Output filename (should end with .dot) |
No return value, called for side effects
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, b) ggml_graph_dump_dot(graph, NULL, tempfile(fileext = ".dot")) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, b) ggml_graph_dump_dot(graph, NULL, tempfile(fileext = ".dot")) ggml_free(ctx)
Copies graph into ctx. With force_grads = TRUE the copy
carries gradient storage even when the original does not, which is one way to
turn a plain forward graph into one that
ggml_build_backward_expand accepts.
ggml_graph_dup(ctx, graph, force_grads = FALSE)ggml_graph_dup(ctx, graph, force_grads = FALSE)
ctx |
GGML context to allocate the copy in |
graph |
Graph to duplicate |
force_grads |
Logical; give the copy gradient storage (default
|
A new graph object (external pointer)
ggml_build_forward_expand_grads
ggml_build_forward_expand() always creates a fresh graph, so it can
only express a single root. A model with several independent output
branches needs every output expanded into the same graph: an output that is
unreachable from the first root never enters the graph, the scheduler never
assigns it a buffer, and reading it back fails. This appends tensor
and its ancestors to a graph that already exists.
ggml_graph_expand(graph, tensor)ggml_graph_expand(graph, tensor)
graph |
Graph object returned by |
tensor |
Additional output tensor to expand into |
NULL, invisibly. graph is modified in place.
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, 1:4) ggml_set_f32(b, 5:8) # Two independent outputs -- neither reachable from the other. o1 <- ggml_add(ctx, a, a) o2 <- ggml_add(ctx, b, b) graph <- ggml_build_forward_expand(ctx, o1) ggml_graph_expand(graph, o2) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, 1:4) ggml_set_f32(b, 5:8) # Two independent outputs -- neither reachable from the other. o1 <- ggml_add(ctx, a, a) o2 <- ggml_add(ctx, b, b) graph <- ggml_build_forward_expand(ctx, o1) ggml_graph_expand(graph, o2) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Returns the tensor holding d(output)/d(node) after
ggml_build_backward_expand has added the backward pass and the
graph has been computed.
ggml_graph_get_grad(graph, node) ggml_graph_get_grad_acc(graph, node)ggml_graph_get_grad(graph, node) ggml_graph_get_grad_acc(graph, node)
graph |
Graph carrying a backward pass |
node |
Tensor whose gradient is wanted |
Returns NULL when the node has no gradient in this graph – because it
was never marked with ggml_set_param, or because nothing
differentiable connects it to the output.
The gradient tensor, or NULL
For ggml_graph_get_grad_acc(), the gradient accumulator
tensor, or NULL
Finds a tensor in the computation graph by its name
ggml_graph_get_tensor(graph, name)ggml_graph_get_tensor(graph, name)
graph |
Graph object |
name |
Character string with tensor name |
Tensor pointer or NULL if not found
Returns the number of computation nodes in the graph
ggml_graph_n_nodes(graph)ggml_graph_n_nodes(graph)
graph |
Graph object |
Integer number of nodes
Gets a specific node (tensor) from the computation graph by index
ggml_graph_node(graph, i)ggml_graph_node(graph, i)
graph |
Graph object |
i |
Node index (0-based, negative indices count from end) |
Tensor pointer
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_add(ctx, a, a) graph <- ggml_build_forward_expand(ctx, b) # Get the last node (output) output <- ggml_graph_node(graph, -1) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_add(ctx, a, a) graph <- ggml_build_forward_expand(ctx, b) # Get the last node (output) output <- ggml_graph_node(graph, -1) ggml_free(ctx)
Returns the memory overhead required for a computation graph
ggml_graph_overhead()ggml_graph_overhead()
Size in bytes
Prints debug information about the computation graph
ggml_graph_print(graph)ggml_graph_print(graph)
graph |
Graph object |
No return value, called for side effects
Zeroes every gradient in the graph and seeds the loss node's own gradient
with 1 – the value the chain rule starts from. Call this after
ggml_build_backward_expand and before
ggml_graph_compute: without it every gradient computes as
zero, since the backward pass multiplies through a seed that was never set.
ggml_graph_reset(graph)ggml_graph_reset(graph)
graph |
Graph object with gradients allocated |
Requires a graph built by ggml_build_forward_expand_grads;
an inference-only graph has no gradients to reset and is rejected.
No return value, called for side effects
ggml_build_backward_expand,
ggml_build_forward_expand_grads
Creates a view of a portion of a computation graph, containing nodes from index i0 to i1 (exclusive). The view shares the underlying nodes but does not include leaf tensors or gradients.
ggml_graph_view(graph, i0, i1)ggml_graph_view(graph, i0, i1)
graph |
External pointer to computation graph |
i0 |
Start index (0-based, inclusive) |
i1 |
End index (exclusive) |
External pointer to graph view
Other graph:
ggml_op_can_inplace()
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, b) n_nodes <- ggml_graph_n_nodes(graph) view <- ggml_graph_view(graph, 0, n_nodes) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) b <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, b) n_nodes <- ggml_graph_n_nodes(graph) view <- ggml_graph_view(graph, 0, n_nodes) ggml_free(ctx)
Creates a graph node for group normalization. Normalizes along ne0*ne1*n_groups dimensions. Used in Stable Diffusion and other image generation models.
ggml_group_norm(ctx, a, n_groups, eps = 1e-05)ggml_group_norm(ctx, a, n_groups, eps = 1e-05)
ctx |
GGML context |
a |
Input tensor |
n_groups |
Number of groups to divide channels into |
eps |
Epsilon for numerical stability (default 1e-5) |
Tensor representing the group norm operation
ctx <- ggml_init(16 * 1024 * 1024) # 4 channels, 2 groups (2 channels per group) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 8) ggml_set_f32(a, rnorm(32)) result <- ggml_group_norm(ctx, a, n_groups = 2) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) # 4 channels, 2 groups (2 channels per group) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 8) ggml_set_f32(a, rnorm(32)) result <- ggml_group_norm(ctx, a, n_groups = 2) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Creates a graph node for in-place group normalization.
ggml_group_norm_inplace(ctx, a, n_groups, eps = 1e-05)ggml_group_norm_inplace(ctx, a, n_groups, eps = 1e-05)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
n_groups |
Number of groups |
eps |
Epsilon for numerical stability (default 1e-5) |
View of input tensor with group norm applied
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 8) ggml_set_f32(a, rnorm(32)) result <- ggml_group_norm_inplace(ctx, a, n_groups = 2) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 8) ggml_set_f32(a, rnorm(32)) result <- ggml_group_norm_inplace(ctx, a, n_groups = 2) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Create a GRU Layer Object
ggml_gru( units, return_sequences = FALSE, activation = "tanh", recurrent_activation = "sigmoid", name = NULL, trainable = TRUE )ggml_gru( units, return_sequences = FALSE, activation = "tanh", recurrent_activation = "sigmoid", name = NULL, trainable = TRUE )
units |
Integer, number of hidden units. |
return_sequences |
Logical. |
activation |
Candidate activation (default |
recurrent_activation |
Gate activation (default |
name |
Optional character name. |
trainable |
Logical. |
A ggml_layer object.
Creates a graph node for Hard Sigmoid activation. HardSigmoid(x) = ReLU6(x + 3) / 6 = min(max(0, x + 3), 6) / 6. A computationally efficient approximation of the sigmoid function.
ggml_hardsigmoid(ctx, a)ggml_hardsigmoid(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the Hard Sigmoid operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-4, -1, 0, 1, 4)) r <- ggml_hardsigmoid(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # [0, 0.333, 0.5, 0.667, 1] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-4, -1, 0, 1, 4)) r <- ggml_hardsigmoid(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # [0, 0.333, 0.5, 0.667, 1] ggml_free(ctx)
Creates a graph node for Hard Swish activation. HardSwish(x) = x * ReLU6(x + 3) / 6 = x * min(max(0, x + 3), 6) / 6. Used in MobileNetV3 and other efficient architectures.
ggml_hardswish(ctx, a)ggml_hardswish(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the Hard Swish operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-4, -1, 0, 1, 4)) r <- ggml_hardswish(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-4, -1, 0, 1, 4)) r <- ggml_hardswish(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) ggml_free(ctx)
Transforms image data into column format for efficient convolution. This is a low-level operation used internally by convolution implementations.
ggml_im2col( ctx, a, b, s0, s1, p0, p1, d0, d1, is_2D = TRUE, dst_type = GGML_TYPE_F16 )ggml_im2col( ctx, a, b, s0, s1, p0, p1, d0, d1, is_2D = TRUE, dst_type = GGML_TYPE_F16 )
ctx |
GGML context |
a |
Convolution kernel tensor |
b |
Input data tensor |
s0 |
Stride dimension 0 |
s1 |
Stride dimension 1 |
p0 |
Padding dimension 0 |
p1 |
Padding dimension 1 |
d0 |
Dilation dimension 0 |
d1 |
Dilation dimension 1 |
is_2D |
Whether this is a 2D operation (default TRUE) |
dst_type |
Output type (default GGML_TYPE_F16) |
Transformed tensor in column format
Initialize GGML context
ggml_init(mem_size = 16 * 1024 * 1024, no_alloc = FALSE)ggml_init(mem_size = 16 * 1024 * 1024, no_alloc = FALSE)
mem_size |
Memory size in bytes |
no_alloc |
If TRUE, don't allocate memory for tensors (default: FALSE) |
GGML context pointer
ctx <- ggml_init(1024 * 1024) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) ggml_free(ctx)
Creates a context with automatically calculated size based on planned tensors.
ggml_init_auto(..., extra_mb = 10, type = GGML_TYPE_F32, no_alloc = FALSE)ggml_init_auto(..., extra_mb = 10, type = GGML_TYPE_F32, no_alloc = FALSE)
... |
Named arguments with tensor dimensions (integer vectors) |
extra_mb |
Extra megabytes to add (default: 10) |
type |
Tensor type (default: GGML_TYPE_F32) |
no_alloc |
If TRUE, don't allocate memory for tensors (default: FALSE) |
GGML context
ctx <- ggml_init_auto(mat1 = c(1000L, 1000L), mat2 = c(1000L, 1000L)) ggml_free(ctx)ctx <- ggml_init_auto(mat1 = c(1000L, 1000L), mat2 = c(1000L, 1000L)) ggml_free(ctx)
Writes a ggml_result back into the appropriate slot of a Seurat
or SingleCellExperiment object, returning the updated object. The
destination depends on
result$metadata$kind: a dimensionality reduction (the default) goes into
a DimReduc; a "transform" (normalize / scale) overwrites an assay
layer; a "graph" (neighbors) writes <assay>_nn and
<assay>_snn Graph objects into @graphs, exactly where
Seurat's FindNeighbors puts them so FindClusters can consume them.
Component standard deviations and the backend used are recorded alongside so
downstream tools and the user can see how the result was produced.
ggml_inject( x, result, reduction_name = "ggml", key = "GGML_", assay = NULL, ... ) ## S3 method for class 'Seurat' ggml_inject( x, result, reduction_name = "ggml", key = "GGML_", assay = NULL, ... ) ## S3 method for class 'SingleCellExperiment' ggml_inject( x, result, reduction_name = "ggml", key = "GGML_", assay = NULL, ... )ggml_inject( x, result, reduction_name = "ggml", key = "GGML_", assay = NULL, ... ) ## S3 method for class 'Seurat' ggml_inject( x, result, reduction_name = "ggml", key = "GGML_", assay = NULL, ... ) ## S3 method for class 'SingleCellExperiment' ggml_inject( x, result, reduction_name = "ggml", key = "GGML_", assay = NULL, ... )
x |
A |
result |
A |
reduction_name |
Name of the reduction slot to create, e.g.
|
key |
Column-name prefix for the embedding, e.g. |
assay |
Assay to associate the reduction with (Seurat). Defaults to the object's default assay. |
... |
Passed to methods. |
The updated container.
Creates a symbolic input node for the Functional API. The node records
only the shape of one sample (without batch dimension); actual
memory is allocated when ggml_compile() is called.
ggml_input(shape, name = NULL, dtype = "float32")ggml_input(shape, name = NULL, dtype = "float32")
shape |
Integer vector describing the shape of a single sample.
For flat feature vectors use a scalar, e.g. |
name |
Optional character name for the input tensor. |
dtype |
Data type of the input: |
A ggml_tensor_node object.
x <- ggml_input(shape = 64L) x <- ggml_input(shape = c(28L, 28L, 1L), name = "image") x <- ggml_input(shape = 10L, dtype = "int32") # token indicesx <- ggml_input(shape = 64L) x <- ggml_input(shape = c(28L, 28L, 1L), name = "image") x <- ggml_input(shape = 10L, dtype = "int32") # token indices
Check if GGML is available
ggml_is_available()ggml_is_available()
TRUE if GGML library is loaded
ggml_is_available()ggml_is_available()
Returns TRUE if tensor data is stored contiguously in memory
ggml_is_contiguous(tensor)ggml_is_contiguous(tensor)
tensor |
Tensor pointer |
Logical
ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_is_contiguous(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_is_contiguous(t) ggml_free(ctx)
Check if tensor is contiguous. Same as ggml_is_contiguous.
ggml_is_contiguous_0(tensor)ggml_is_contiguous_0(tensor)
tensor |
Tensor pointer |
Logical indicating contiguity
Other tensor_layout:
ggml_are_same_stride(),
ggml_can_repeat(),
ggml_count_equal(),
ggml_is_contiguous_1(),
ggml_is_contiguous_2(),
ggml_is_contiguous_channels(),
ggml_is_contiguous_rows(),
ggml_is_contiguously_allocated()
Check if tensor is contiguous for dimensions >= 1. Allows non-contiguous first dimension.
ggml_is_contiguous_1(tensor)ggml_is_contiguous_1(tensor)
tensor |
Tensor pointer |
Logical indicating contiguity for dims >= 1
Other tensor_layout:
ggml_are_same_stride(),
ggml_can_repeat(),
ggml_count_equal(),
ggml_is_contiguous_0(),
ggml_is_contiguous_2(),
ggml_is_contiguous_channels(),
ggml_is_contiguous_rows(),
ggml_is_contiguously_allocated()
Check if tensor is contiguous for dimensions >= 2. Allows non-contiguous first two dimensions.
ggml_is_contiguous_2(tensor)ggml_is_contiguous_2(tensor)
tensor |
Tensor pointer |
Logical indicating contiguity for dims >= 2
Other tensor_layout:
ggml_are_same_stride(),
ggml_can_repeat(),
ggml_count_equal(),
ggml_is_contiguous_0(),
ggml_is_contiguous_1(),
ggml_is_contiguous_channels(),
ggml_is_contiguous_rows(),
ggml_is_contiguously_allocated()
Check if tensor has contiguous channels (important for CNN operations). Data for each channel should be stored contiguously.
ggml_is_contiguous_channels(tensor)ggml_is_contiguous_channels(tensor)
tensor |
Tensor pointer |
Logical indicating channel-wise contiguity
Other tensor_layout:
ggml_are_same_stride(),
ggml_can_repeat(),
ggml_count_equal(),
ggml_is_contiguous_0(),
ggml_is_contiguous_1(),
ggml_is_contiguous_2(),
ggml_is_contiguous_rows(),
ggml_is_contiguously_allocated()
Check if tensor has contiguous rows (important for matrix operations). Each row should be stored contiguously in memory.
ggml_is_contiguous_rows(tensor)ggml_is_contiguous_rows(tensor)
tensor |
Tensor pointer |
Logical indicating row-wise contiguity
Other tensor_layout:
ggml_are_same_stride(),
ggml_can_repeat(),
ggml_count_equal(),
ggml_is_contiguous_0(),
ggml_is_contiguous_1(),
ggml_is_contiguous_2(),
ggml_is_contiguous_channels(),
ggml_is_contiguously_allocated()
Check if tensor data is contiguously allocated in memory. Different from contiguous layout - this checks the actual allocation.
ggml_is_contiguously_allocated(tensor)ggml_is_contiguously_allocated(tensor)
tensor |
Tensor pointer |
Logical indicating if data is contiguously allocated
Other tensor_layout:
ggml_are_same_stride(),
ggml_can_repeat(),
ggml_count_equal(),
ggml_is_contiguous_0(),
ggml_is_contiguous_1(),
ggml_is_contiguous_2(),
ggml_is_contiguous_channels(),
ggml_is_contiguous_rows()
Returns TRUE if tensor dimensions have been permuted
ggml_is_permuted(tensor)ggml_is_permuted(tensor)
tensor |
Tensor pointer |
Logical
ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_is_permuted(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_is_permuted(t) ggml_free(ctx)
Returns TRUE if the GGML type is a quantized format.
ggml_is_quantized(type)ggml_is_quantized(type)
type |
GGML type constant |
Logical indicating if type is quantized
Other type_system:
ggml_blck_size(),
ggml_ftype_to_ggml_type(),
ggml_type_name(),
ggml_type_sizef()
ggml_is_quantized(GGML_TYPE_F32) # FALSE ggml_is_quantized(GGML_TYPE_Q4_0) # TRUEggml_is_quantized(GGML_TYPE_F32) # FALSE ggml_is_quantized(GGML_TYPE_Q4_0) # TRUE
Returns TRUE if tensor has been transposed
ggml_is_transposed(tensor)ggml_is_transposed(tensor)
tensor |
Tensor pointer |
Logical
ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_is_transposed(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_is_transposed(t) ggml_free(ctx)
Creates a graph node for L2 normalization (unit norm). Normalizes vectors to unit length: x / ||x||_2. Used in RWKV v7 and embedding normalization.
ggml_l2_norm(ctx, a, eps = 1e-05)ggml_l2_norm(ctx, a, eps = 1e-05)
ctx |
GGML context |
a |
Input tensor |
eps |
Epsilon for numerical stability (default 1e-5) |
Tensor representing the L2 norm operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(3, 0, 0, 4)) # Length = 5 result <- ggml_l2_norm(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [0.6, 0, 0, 0.8] unit vector ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(3, 0, 0, 4)) # Length = 5 result <- ggml_l2_norm(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [0.6, 0, 0, 0.8] unit vector ggml_free(ctx)
Creates a graph node for in-place L2 normalization.
ggml_l2_norm_inplace(ctx, a, eps = 1e-05)ggml_l2_norm_inplace(ctx, a, eps = 1e-05)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
eps |
Epsilon for numerical stability (default 1e-5) |
View of input tensor with L2 norm applied
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(3, 0, 0, 4)) result <- ggml_l2_norm_inplace(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(3, 0, 0, 4)) result <- ggml_l2_norm_inplace(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Adds two (or more) tensor nodes element-wise. All tensors must have the same shape. This is the functional equivalent of a residual / skip connection.
ggml_layer_add(tensors, name = NULL)ggml_layer_add(tensors, name = NULL)
tensors |
A list of |
name |
Optional character name for the layer. |
A new ggml_tensor_node representing the sum.
x <- ggml_input(shape = 64L) a <- x |> ggml_layer_dense(64, activation = "relu") b <- x |> ggml_layer_dense(64) out <- ggml_layer_add(list(a, b))x <- ggml_input(shape = 64L) a <- x |> ggml_layer_dense(64, activation = "relu") b <- x |> ggml_layer_dense(64) out <- ggml_layer_add(list(a, b))
Scaled dot-product attention with n_heads heads, in the pipe-style
form that appends a layer to a functional graph. For the reusable
layer object that shares one set of weights between several
applications – how an encoder block is reused across a stack – see
ggml_attention.
ggml_layer_attention( x, d_model, n_heads = 1L, causal = FALSE, context = NULL, mask = NULL, rope = FALSE, rope_base = 10000, dropout = 0, bias = TRUE, name = NULL, trainable = TRUE )ggml_layer_attention( x, d_model, n_heads = 1L, causal = FALSE, context = NULL, mask = NULL, rope = FALSE, rope_base = 10000, dropout = 0, bias = TRUE, name = NULL, trainable = TRUE )
x |
A |
d_model |
Integer, model width. Must be divisible by |
n_heads |
Integer, number of attention heads (default 1). |
causal |
Logical: mask out positions after the current one, making the
layer autoregressive (default |
context |
Optional |
mask |
Optional |
rope |
Logical: apply rotary position embedding to the queries and keys
(default |
rope_base |
Rotation base for |
dropout |
Dropout rate applied to the attention weights during training (default 0, no dropout). used by LLaMA-family models). Larger values stretch the rotation over longer contexts. |
bias |
Logical: add a bias to the output projection (default
|
name |
Optional layer name. |
trainable |
Logical. |
A new ggml_tensor_node of shape c(seq_len, d_model).
x must be a sequence node of shape c(seq_len, d_model), the
same layout ggml_layer_lstm takes. The output has that same
shape, so attention blocks stack directly and a residual connection
(ggml_layer_add(list(x, attn_out))) needs no reshaping.
All heads are computed in one batched pass rather than in a loop: the
projections are reshaped to [d_head, n_heads, seq, batch] and
permuted so that ggml_mul_mat() batches over the head and batch axes.
The node count is therefore independent of n_heads.
Pass a list of two nodes, ggml_layer_attention(list(query, context),
...), to take queries from the first and keys/values from the second. The
two may differ in sequence length but must share d_model. A single
node is self-attention.
ggml_attention for the shared-weights layer object.
# A transformer encoder block: attention + residual, then a feed-forward # sublayer + residual. x <- ggml_input(shape = c(10L, 32L)) attn <- x |> ggml_layer_attention(d_model = 32L, n_heads = 4L) h <- ggml_layer_add(list(x, attn)) ff <- h |> ggml_layer_dense(32L, activation = "relu") out <- ggml_layer_add(list(h, ff)) # GPT-style causal self-attention. dec <- x |> ggml_layer_attention(32L, n_heads = 4L, causal = TRUE) # Padding mask, fed as a second input alongside the sequence. pad <- ggml_input(shape = c(10L, 10L), name = "mask") enc <- x |> ggml_layer_attention(32L, n_heads = 4L, mask = pad)# A transformer encoder block: attention + residual, then a feed-forward # sublayer + residual. x <- ggml_input(shape = c(10L, 32L)) attn <- x |> ggml_layer_attention(d_model = 32L, n_heads = 4L) h <- ggml_layer_add(list(x, attn)) ff <- h |> ggml_layer_dense(32L, activation = "relu") out <- ggml_layer_add(list(h, ff)) # GPT-style causal self-attention. dec <- x |> ggml_layer_attention(32L, n_heads = 4L, causal = TRUE) # Padding mask, fed as a second input alongside the sequence. pad <- ggml_input(shape = c(10L, 10L), name = "mask") enc <- x |> ggml_layer_attention(32L, n_heads = 4L, mask = pad)
Applies normalization: RMS-normalizes the input, then scales by gamma
and shifts by beta (both learnable). Uses ggml_rms_norm which
supports backward pass for training.
ggml_layer_batch_norm(model, eps = 1e-05, name = NULL, trainable = TRUE)ggml_layer_batch_norm(model, eps = 1e-05, name = NULL, trainable = TRUE)
model |
A ggml_sequential_model object |
eps |
Small constant for numerical stability (default 1e-5) |
name |
Optional character name for the layer. |
trainable |
Logical; whether the layer weights are updated during training. |
The model object with the batch_norm layer appended (invisibly).
model <- ggml_model_sequential() |> ggml_layer_dense(128, input_shape = 784) |> ggml_layer_batch_norm() |> ggml_layer_dense(10, activation = "softmax")model <- ggml_model_sequential() |> ggml_layer_dense(128, input_shape = 784) |> ggml_layer_batch_norm() |> ggml_layer_dense(10, activation = "softmax")
Concatenates two or more tensor nodes along the specified axis.
ggml_layer_concatenate(tensors, axis = 0L, name = NULL)ggml_layer_concatenate(tensors, axis = 0L, name = NULL)
tensors |
A list of |
axis |
Integer axis along which to concatenate (0-based, ggml convention).
Default |
name |
Optional character name for the layer. |
A new ggml_tensor_node representing the concatenated tensor.
x <- ggml_input(shape = 32L) y <- ggml_input(shape = 32L) out <- ggml_layer_concatenate(list(x, y), axis = 0L)x <- ggml_input(shape = 32L) y <- ggml_input(shape = 32L) out <- ggml_layer_concatenate(list(x, y), axis = 0L)
Add 1D Convolution Layer
ggml_layer_conv_1d( model, filters, kernel_size, activation = NULL, input_shape = NULL, strides = 1L, padding = "valid", name = NULL, trainable = TRUE )ggml_layer_conv_1d( model, filters, kernel_size, activation = NULL, input_shape = NULL, strides = 1L, padding = "valid", name = NULL, trainable = TRUE )
model |
A ggml_sequential_model object |
filters |
Number of output filters |
kernel_size |
Integer kernel size |
activation |
Activation function name: "relu", "sigmoid", "tanh", "softmax", or NULL |
input_shape |
Input shape c(L, C) - required for first layer only (length, channels) |
strides |
Integer stride (default 1) |
padding |
"valid" (no padding) or "same" (preserve length) |
name |
Optional character name for the layer. |
trainable |
Logical; whether the layer weights are updated during training. |
The model object with the conv_1d layer appended (invisibly).
model <- ggml_model_sequential() |> ggml_layer_conv_1d(32, 3, activation = "relu", input_shape = c(100, 1))model <- ggml_model_sequential() |> ggml_layer_conv_1d(32, 3, activation = "relu", input_shape = c(100, 1))
Add 2D Convolution Layer
ggml_layer_conv_2d( model, filters, kernel_size, activation = NULL, input_shape = NULL, strides = c(1L, 1L), padding = "valid", name = NULL, trainable = TRUE )ggml_layer_conv_2d( model, filters, kernel_size, activation = NULL, input_shape = NULL, strides = c(1L, 1L), padding = "valid", name = NULL, trainable = TRUE )
model |
A ggml_sequential_model object |
filters |
Number of output filters |
kernel_size |
Integer or vector of 2 integers for kernel height and width |
activation |
Activation function name: "relu", "sigmoid", "tanh", "softmax", or NULL |
input_shape |
Input shape c(H, W, C) - required for first layer only |
strides |
Integer or vector of 2 integers for stride |
padding |
"valid" (no padding) or "same" (preserve spatial dims) |
name |
Optional character name for the layer. |
trainable |
Logical; whether the layer weights are updated during training. |
The model object with the conv_2d layer appended (invisibly).
model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1))model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1))
Turns an arbitrary composition of ggml operations into a reusable functional
layer. ggml_layer_custom() does not build anything itself — it
returns a layer function that behaves like any other
ggml_layer_*() and can be piped into a model.
ggml_layer_custom(forward, name = NULL, output_shape = NULL)ggml_layer_custom(forward, name = NULL, output_shape = NULL)
forward |
A function of |
name |
Optional character name for the layer. Used as the base for auto-generated node names. |
output_shape |
Output shape excluding the batch dimension. Defaults to
|
The forward function is called once while the compute graph is being
built, with the active ggml_context and the parent tensor. Because
the returned tensor becomes an ordinary graph node, gradients flow through a
custom layer automatically — as long as every operation used inside
forward implements a backward pass.
A layer function of (x, name = NULL) that appends a custom
node to the graph and returns the new ggml_tensor_node.
Custom layers are stateless: forward maps activations to activations
and owns no trainable parameters. Put learnable weights in a neighbouring
ggml_layer_dense(), or use the autograd API (ag_*) when a layer
needs its own parameters.
# Mish activation: x * tanh(softplus(x)) layer_mish <- ggml_layer_custom( name = "mish", forward = function(ctx, x) ggml_mul(ctx, x, ggml_tanh(ctx, ggml_softplus(ctx, x))) ) inp <- ggml_input(shape = 8L) out <- inp |> ggml_layer_dense(16L) |> layer_mish() |> ggml_layer_dense(2L, activation = "softmax") model <- ggml_model(inputs = inp, outputs = out)# Mish activation: x * tanh(softplus(x)) layer_mish <- ggml_layer_custom( name = "mish", forward = function(ctx, x) ggml_mul(ctx, x, ggml_tanh(ctx, ggml_softplus(ctx, x))) ) inp <- ggml_input(shape = 8L) out <- inp |> ggml_layer_dense(16L) |> layer_mish() |> ggml_layer_dense(2L, activation = "softmax") model <- ggml_model(inputs = inp, outputs = out)
Add Dense (Fully Connected) Layer
ggml_layer_dense( model, units, activation = NULL, input_shape = NULL, name = NULL, trainable = TRUE, time_distributed = FALSE )ggml_layer_dense( model, units, activation = NULL, input_shape = NULL, name = NULL, trainable = TRUE, time_distributed = FALSE )
model |
A ggml_sequential_model object |
units |
Number of output units |
activation |
Activation function name: "relu", "sigmoid", "tanh", "softmax", or NULL |
input_shape |
Integer or integer vector specifying the input shape (only needed for the first layer) |
name |
Optional character name for the layer. |
trainable |
Logical; whether the layer weights are updated during training. |
time_distributed |
Logical; apply the kernel per position of a sequence
input rather than flattening it (default |
The model object with the dense layer appended (invisibly).
By default a dense layer flattens its input, so a sequence node of shape
c(seq_len, features) becomes a single vector of units and the
sequence axis is gone. With time_distributed = TRUE the same kernel
is applied independently at every position instead, sharing one set of
weights across the sequence, and the output keeps its length:
c(seq_len, units). This is the semantics of Keras'
TimeDistributed, and it is what the position-wise feed-forward
sublayer of a transformer block needs – without it, a dense layer after
ggml_layer_attention would collapse every position into one
vector.
It is computed as a single batched ggml_mul_mat() over the combined
position/batch axes, not as a loop over positions.
Functional API only: a sequential model carries a flat running shape, so there is no sequence axis to distribute over.
model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_flatten() |> ggml_layer_dense(128, activation = "relu") # Position-wise feed-forward, as in a transformer block. x <- ggml_input(shape = c(10L, 32L)) ff <- x |> ggml_layer_dense(64L, activation = "relu", time_distributed = TRUE)model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_flatten() |> ggml_layer_dense(128, activation = "relu") # Position-wise feed-forward, as in a transformer block. x <- ggml_input(shape = c(10L, 32L)) ff <- x |> ggml_layer_dense(64L, activation = "relu", time_distributed = TRUE)
Applies dropout regularization. During training, multiplies all activations
by (1 - rate) (deterministic expected-value scaling).
During inference (training = FALSE), the layer is an identity (no change).
ggml_layer_dropout( model, rate, stochastic = FALSE, name = NULL, trainable = FALSE )ggml_layer_dropout( model, rate, stochastic = FALSE, name = NULL, trainable = FALSE )
model |
A |
rate |
Dropout rate in |
stochastic |
Logical. If |
name |
Optional layer name. |
trainable |
Ignored for dropout (no weights); kept for API consistency. |
The model with the dropout layer appended, or a new tensor node.
Keras implements inverted dropout: during training it applies a random
Bernoulli mask and scales surviving activations up by
1 / (1 - rate), so the expected value of each unit is preserved and
no scaling is needed at inference.
This implementation uses deterministic scaling (multiply by
(1 - rate) at training, identity at inference) — equivalent in
expected value but without stochastic noise. Consequences:
No random mask → the regularization signal is weaker (no co-adaptation breaking).
Activations at training are scaled down, not up — the magnitude seen by subsequent layers differs from Keras behaviour.
Results are fully deterministic and reproducible without setting a seed.
With stochastic = TRUE the Bernoulli mask is regenerated once
per epoch (not per batch), because ggml_opt_fit processes all
batches inside a single C call. This is weaker than per-batch dropout
but stronger than the deterministic variant.
model <- ggml_model_sequential() |> ggml_layer_dense(128, activation = "relu", input_shape = 784L) |> ggml_layer_dropout(0.5, stochastic = TRUE) |> ggml_layer_dense(10, activation = "softmax")model <- ggml_model_sequential() |> ggml_layer_dense(128, activation = "relu", input_shape = 784L) |> ggml_layer_dropout(0.5, stochastic = TRUE) |> ggml_layer_dense(10, activation = "softmax")
Looks up dense vectors for integer token indices. The input must be an
integer matrix of 0-based indices in [0, vocab_size - 1] (use
ggml_input(shape, dtype = "int32") in Functional mode).
ggml_layer_embedding( model, vocab_size, dim, name = NULL, trainable = TRUE, input_shape = NULL )ggml_layer_embedding( model, vocab_size, dim, name = NULL, trainable = TRUE, input_shape = NULL )
model |
A |
vocab_size |
Number of distinct tokens (vocabulary size). |
dim |
Embedding dimension (vector length per token). |
name |
Optional layer name. |
trainable |
Logical; whether embedding weights are updated during training. |
input_shape |
Sequence length, for an embedding that is the first layer
of a sequential model (its input is then a matrix of token indices,
one row per sample). Ignored in the functional API, where
|
The model with the embedding layer appended, or a new tensor node.
The layer's output shape is c(seq_len, dim) – one row per position,
as in Keras – which is the same convention a declared sequence input
(ggml_input(shape = c(seq_len, features))) uses. So the output feeds
ggml_layer_attention, ggml_layer_gru and
ggml_layer_lstm directly, with no axis juggling in between.
The underlying tensor is [dim, seq_len, N]: ggml is column-major, so
an R shape c(a, b) is the tensor [b, a, N] throughout the
package. That matters only when reading raw output tensors –
ggml_layer_flatten() gives the same vector either way.
Indices must be in [0, vocab_size - 1]. Out-of-range values cause
undefined behaviour inside the ggml kernel (no bounds check is performed at
the R level).
inp <- ggml_input(shape = 10L, dtype = "int32") out <- inp |> ggml_layer_embedding(vocab_size = 1000L, dim = 32L) |> ggml_layer_flatten() |> ggml_layer_dense(10L, activation = "softmax") model <- ggml_model(inputs = inp, outputs = out)inp <- ggml_input(shape = 10L, dtype = "int32") out <- inp |> ggml_layer_embedding(vocab_size = 1000L, dim = 32L) |> ggml_layer_flatten() |> ggml_layer_dense(10L, activation = "softmax") model <- ggml_model(inputs = inp, outputs = out)
Flattens the spatial dimensions into a single vector per sample.
ggml_layer_flatten(model, name = NULL, trainable = TRUE)ggml_layer_flatten(model, name = NULL, trainable = TRUE)
model |
A ggml_sequential_model object |
name |
Optional character name for the layer. |
trainable |
Logical; reserved for API consistency (no weights). |
The model object with the flatten layer appended (invisibly).
model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_flatten()model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_flatten()
Reduces a [H, W, C] feature map to [C] by averaging all
spatial positions per channel. Equivalent to Keras
GlobalAveragePooling2D().
ggml_layer_global_average_pooling_2d(model, name = NULL, trainable = TRUE)ggml_layer_global_average_pooling_2d(model, name = NULL, trainable = TRUE)
model |
A |
name |
Optional character name for the layer. |
trainable |
Logical; reserved for API consistency (no weights). |
Updated model or a new ggml_tensor_node.
model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_global_average_pooling_2d() |> ggml_layer_dense(10, activation = "softmax")model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_global_average_pooling_2d() |> ggml_layer_dense(10, activation = "softmax")
Reduces a [H, W, C] feature map to [C] by taking the maximum
value per channel across all spatial positions. Equivalent to Keras
GlobalMaxPooling2D().
ggml_layer_global_max_pooling_2d(model, name = NULL, trainable = TRUE)ggml_layer_global_max_pooling_2d(model, name = NULL, trainable = TRUE)
model |
A |
name |
Optional character name for the layer. |
trainable |
Logical; reserved for API consistency (no weights). |
Updated model or a new ggml_tensor_node.
model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_global_max_pooling_2d() |> ggml_layer_dense(10, activation = "softmax")model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_global_max_pooling_2d() |> ggml_layer_dense(10, activation = "softmax")
Gated Recurrent Unit recurrent layer. Implemented as an unrolled computation graph (BPTT).
ggml_layer_gru( model, units, return_sequences = FALSE, activation = "tanh", recurrent_activation = "sigmoid", input_shape = NULL, name = NULL, trainable = TRUE )ggml_layer_gru( model, units, return_sequences = FALSE, activation = "tanh", recurrent_activation = "sigmoid", input_shape = NULL, name = NULL, trainable = TRUE )
model |
A |
units |
Integer, number of hidden units. |
return_sequences |
Logical; return all hidden states or only the last. |
activation |
Activation for the candidate hidden state ( |
recurrent_activation |
Activation for z/r gates ( |
input_shape |
Input shape |
name |
Optional layer name. |
trainable |
Logical. |
Updated model or a new ggml_tensor_node.
W_zh [input_size, 2*units] — input kernel for z and r
gates.
U_zh [units, 2*units] — recurrent kernel for z and r.
b_zh [2*units] — bias for z and r.
W_n [input_size, units] — input kernel for candidate.
U_n [units, units] — recurrent kernel for candidate.
b_n [units] — bias for candidate.
model <- ggml_model_sequential() |> ggml_layer_gru(64L, input_shape = c(10L, 32L)) |> ggml_layer_dense(10L, activation = "softmax")model <- ggml_model_sequential() |> ggml_layer_gru(64L, input_shape = c(10L, 32L)) |> ggml_layer_dense(10L, activation = "softmax")
Normalizes each sample over its own features – subtract the mean, divide by
the standard deviation – then scales by gamma and shifts by beta (both
learnable). Like ggml_layer_rms_norm it keeps no running
statistics and does not couple the samples in a batch, but unlike it, it does
centre the input. This is the normalization of the original transformer.
ggml_layer_layer_norm(model, eps = 1e-05, name = NULL, trainable = TRUE)ggml_layer_layer_norm(model, eps = 1e-05, name = NULL, trainable = TRUE)
model |
A |
eps |
Small constant for numerical stability (default 1e-5) |
name |
Optional character name for the layer. |
trainable |
Logical; whether the layer weights are updated during training. |
The model with the layer appended, or a new ggml_tensor_node
in the functional API.
ggml_layer_rms_norm, ggml_layer_batch_norm
x <- ggml_input(shape = c(10L, 32L)) h <- x |> ggml_layer_layer_norm() h <- h |> ggml_layer_attention(d_model = 32L, n_heads = 4L)x <- ggml_input(shape = c(10L, 32L)) h <- x |> ggml_layer_layer_norm() h <- h |> ggml_layer_attention(d_model = 32L, n_heads = 4L)
Long Short-Term Memory recurrent layer. Implemented as an unrolled computation graph (BPTT) so that ggml's automatic differentiation works without any C extensions.
ggml_layer_lstm( model, units, return_sequences = FALSE, activation = "tanh", recurrent_activation = "sigmoid", input_shape = NULL, name = NULL, trainable = TRUE )ggml_layer_lstm( model, units, return_sequences = FALSE, activation = "tanh", recurrent_activation = "sigmoid", input_shape = NULL, name = NULL, trainable = TRUE )
model |
A |
units |
Integer, number of hidden units. |
return_sequences |
Logical; if |
activation |
Activation for the cell gate (default |
recurrent_activation |
Activation for the recurrent step (default
|
input_shape |
Input shape |
name |
Optional layer name. |
trainable |
Logical. |
Updated model or a new ggml_tensor_node.
W_gates [input_size, 4*units] — input kernel for all
four gates (i, f, g, o) concatenated.
U_gates [units, 4*units] — recurrent kernel.
b_gates [4*units] — bias.
Input: [seq_len, input_size] per sample (R row-major), or a 3-D
array [N, seq_len, input_size]. In the Functional API the input
node shape should be c(seq_len, input_size).
Output (Sequential): [units] per sample when
return_sequences = FALSE (default), or c(seq_len, units)
when return_sequences = TRUE.
model <- ggml_model_sequential() |> ggml_layer_lstm(64L, input_shape = c(10L, 32L)) |> ggml_layer_dense(10L, activation = "softmax")model <- ggml_model_sequential() |> ggml_layer_lstm(64L, input_shape = c(10L, 32L)) |> ggml_layer_dense(10L, activation = "softmax")
Add 2D Max Pooling Layer
ggml_layer_max_pooling_2d( model, pool_size = c(2L, 2L), strides = NULL, name = NULL, trainable = TRUE )ggml_layer_max_pooling_2d( model, pool_size = c(2L, 2L), strides = NULL, name = NULL, trainable = TRUE )
model |
A ggml_sequential_model object |
pool_size |
Integer or vector of 2 integers for pool height and width |
strides |
Integer or vector of 2 integers (defaults to pool_size) |
name |
Optional character name for the layer. |
trainable |
Logical; reserved for API consistency (no weights). |
The model object with the max pooling layer appended (invisibly).
model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_max_pooling_2d(c(2, 2))model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_max_pooling_2d(c(2, 2))
Reorder a sample's axes. The batch axis is not part of dims and is
never moved – dims indexes the per-sample shape, exactly as
input_shape does.
ggml_layer_permute( model, dims, input_shape = NULL, name = NULL, trainable = TRUE )ggml_layer_permute( model, dims, input_shape = NULL, name = NULL, trainable = TRUE )
model |
A ggml_sequential_model object, or a |
dims |
Integer vector, a permutation of |
input_shape |
Input shape, required for the first layer only. |
name |
Optional character name for the layer. |
trainable |
Logical; reserved for API consistency (no weights). |
The model object with the permute layer appended (invisibly), or a
ggml_tensor_node in the functional API.
dims follows base R's aperm: dims[i] names
the source axis that ends up at position i. So
dims = c(2, 1) means "the new first axis is the old second one".
This is deliberately the inverse of the underlying
ggml_permute, whose arguments are destination
positions; the layer converts between the two so that callers can think in
the R idiom. Read ggml_permute itself as "source axis i goes
to position p_i" – mixing the two conventions up silently inverts
every non-trivial permutation.
Reordering a sample's axes to match what the next layer reads: a
c(features, seq_len) tensor into the c(seq_len, features) that
attention and the recurrent layers expect, or the reverse.
Note that ggml_layer_embedding does not need this –
it already reports c(seq_len, dim) and feeds attention, GRU and LSTM
directly. Adding a permute there transposes the underlying tensor and breaks
the graph.
ggml_permute for the low-level op and its
destination-position convention.
# Swap a c(features, seq_len) sample into c(seq_len, features). model <- ggml_model_sequential() |> ggml_layer_permute(c(2, 1), input_shape = c(32, 16))# Swap a c(features, seq_len) sample into c(seq_len, features). model <- ggml_model_sequential() |> ggml_layer_permute(c(2, 1), input_shape = c(32, 16))
Adds a learned vector to each position of a sequence, so that attention –
which is order-blind on its own – can tell one position from another. The
table is [d_model, seq_len], one row per position, added to the input
and broadcast over the batch.
ggml_layer_positional_embedding(model, name = NULL, trainable = TRUE)ggml_layer_positional_embedding(model, name = NULL, trainable = TRUE)
model |
A |
name |
Optional character name for the layer. |
trainable |
Logical; whether the table is updated during training. |
Unlike ggml_layer_embedding there are no indices to look up:
the position IS the place in the sequence, so this layer takes the sequence
itself and returns it with the positional term added. The shape is
unchanged.
The model with the layer appended, or a new ggml_tensor_node
of the same shape as the input.
ggml_layer_attention, ggml_layer_embedding
# Without this the encoder cannot distinguish "ab" from "ba". x <- ggml_input(shape = c(10L, 32L)) h <- x |> ggml_layer_positional_embedding() h <- h |> ggml_layer_attention(d_model = 32L, n_heads = 4L)# Without this the encoder cannot distinguish "ab" from "ba". x <- ggml_input(shape = c(10L, 32L)) h <- x |> ggml_layer_positional_embedding() h <- h |> ggml_layer_attention(d_model = 32L, n_heads = 4L)
Reinterpret a sample's shape without moving any data. The batch axis is not
part of shape and is never reshaped – shape describes one
sample, exactly as input_shape does.
ggml_layer_reshape( model, shape, input_shape = NULL, name = NULL, trainable = TRUE )ggml_layer_reshape( model, shape, input_shape = NULL, name = NULL, trainable = TRUE )
model |
A ggml_sequential_model object, or a |
shape |
Integer vector, the new per-sample shape. At most one entry may
be |
input_shape |
Input shape, required for the first layer only. |
name |
Optional character name for the layer. |
trainable |
Logical; reserved for API consistency (no weights). |
The model object with the reshape layer appended (invisibly), or a
ggml_tensor_node in the functional API.
One entry of shape may be -1, in which case it is computed
from the element count: ggml_layer_reshape(c(-1, 32)) on a
c(4, 8, 16) input gives c(16, 32). At most one axis may be
-1, and the remaining sizes must divide the element count exactly.
Reshape only relabels the axes; the elements keep their memory order. To
actually reorder axes – swapping c(a, b) into c(b, a), say –
use ggml_layer_permute. Using reshape for that silently
interleaves the data instead.
ggml_reshape_*() requires contiguous data, and a permuted tensor is
not contiguous. The layer inserts ggml_cont when needed, so it
works after a permute; the cost is one copy.
ggml_layer_permute to reorder axes,
ggml_layer_flatten to collapse them all.
# Split a flat feature vector into a sequence of 32-wide steps. model <- ggml_model_sequential() |> ggml_layer_dense(512, input_shape = 128) |> ggml_layer_reshape(c(-1, 32))# Split a flat feature vector into a sequence of 32-wide steps. model <- ggml_model_sequential() |> ggml_layer_dense(512, input_shape = 128) |> ggml_layer_reshape(c(-1, 32))
Normalizes each sample by the root mean square of its features, then scales
by gamma and shifts by beta (both learnable). Unlike
ggml_layer_batch_norm it does not subtract the mean and keeps
no running statistics, so it behaves identically in training and inference
and does not couple the samples in a batch. This is the normalization used
by transformer blocks.
ggml_layer_rms_norm(model, eps = 1e-05, name = NULL, trainable = TRUE)ggml_layer_rms_norm(model, eps = 1e-05, name = NULL, trainable = TRUE)
model |
A |
eps |
Small constant for numerical stability (default 1e-5) |
name |
Optional character name for the layer. |
trainable |
Logical; whether the layer weights are updated during training. |
The model with the layer appended, or a new ggml_tensor_node
in the functional API.
# Functional API: the normalization of a transformer block. x <- ggml_input(shape = c(10L, 32L)) h <- x |> ggml_layer_rms_norm() h <- h |> ggml_layer_attention(d_model = 32L, n_heads = 4L)# Functional API: the normalization of a transformer block. x <- ggml_input(shape = c(10L, 32L)) h <- x |> ggml_layer_rms_norm() h <- h |> ggml_layer_attention(d_model = 32L, n_heads = 4L)
Collapses the sequence axis, turning c(seq_len, d_model) into
d_model: the step that lets a classification or regression head sit
on top of an encoder. ggml_layer_flatten() also produces a flat
vector, but keeps every position separately (seq_len * d_model
features), so its width – and the head above it – depends on the sequence
length. Pooling does not.
ggml_layer_sequence_pooling(model, mode = c("mean", "first"), name = NULL)ggml_layer_sequence_pooling(model, mode = c("mean", "first"), name = NULL)
model |
A |
mode |
|
name |
Optional character name for the layer. |
mode = "mean" averages over the positions; every position
contributes equally, which is the usual choice for an encoder without a
dedicated summary token. mode = "first" takes position 1 and ignores
the rest – the CLS-token convention, where attention is expected to have
gathered what matters into that position.
The model with the layer appended, or a new ggml_tensor_node
of shape d_model.
ggml_layer_flatten, ggml_layer_attention
# An encoder with a regression head: the head's width is d_model, whatever # the sequence length turns out to be. x <- ggml_input(shape = c(10L, 32L)) h <- x |> ggml_layer_attention(d_model = 32L, n_heads = 4L) h <- h |> ggml_layer_sequence_pooling() y <- h |> ggml_layer_dense(1L)# An encoder with a regression head: the head's width is d_model, whatever # the sequence length turns out to be. x <- ggml_input(shape = c(10L, 32L)) h <- x |> ggml_layer_attention(d_model = 32L, n_heads = 4L) h <- h |> ggml_layer_sequence_pooling() y <- h |> ggml_layer_dense(1L)
The standard pre-LN encoder block, in one call: normalize, attend, add the residual, normalize again, run the position-wise feed-forward, add the second residual. Composed from the existing layers – there is no new node type – so the pieces can still be wired by hand when a block needs to differ.
ggml_layer_transformer_block( x, d_model, n_heads = 1L, ff_dim = NULL, activation = "silu", norm = c("rms", "layer"), causal = FALSE, mask = NULL, rope = FALSE, dropout = 0, attn_dropout = dropout, name = NULL )ggml_layer_transformer_block( x, d_model, n_heads = 1L, ff_dim = NULL, activation = "silu", norm = c("rms", "layer"), causal = FALSE, mask = NULL, rope = FALSE, dropout = 0, attn_dropout = dropout, name = NULL )
x |
A |
d_model |
Integer, model width. Must match the input's feature
dimension and be divisible by |
n_heads |
Integer, number of attention heads (default 1). |
ff_dim |
Integer, width of the feed-forward hidden layer. Defaults to
|
activation |
Feed-forward activation (default |
norm |
|
causal |
Logical: causal self-attention (default |
mask |
Optional attention mask node, |
rope |
Logical: rotary position embedding on queries and keys
(default |
dropout |
Dropout rate applied to each sublayer's output before it is added back to the residual (default 0, meaning no dropout). |
attn_dropout |
Dropout rate for the attention weights specifically;
defaults to |
name |
Optional prefix for the layers inside the block. |
A new ggml_tensor_node of shape c(seq_len, d_model),
so blocks stack directly.
Normalization comes BEFORE each sublayer, not after, which is what makes deep stacks trainable without a warmup schedule: the residual path stays un-normalized from input to output, so the gradient reaches the early layers undiminished. The original 2017 transformer normalized after; every large model since does it this way.
ggml_layer_attention,
ggml_layer_sequence_pooling
# A two-block encoder with a pooled regression head. x <- ggml_input(shape = c(10L, 32L)) h <- x |> ggml_layer_positional_embedding() for (i in 1:2) h <- h |> ggml_layer_transformer_block(32L, n_heads = 4L) y <- h |> ggml_layer_sequence_pooling() |> ggml_layer_dense(1L)# A two-block encoder with a pooled regression head. x <- ggml_input(shape = c(10L, 32L)) h <- x |> ggml_layer_positional_embedding() for (i in 1:2) h <- h |> ggml_layer_transformer_block(32L, n_heads = 4L) y <- h |> ggml_layer_sequence_pooling() |> ggml_layer_dense(1L)
Creates a graph node for Leaky ReLU activation. LeakyReLU(x) = x if x > 0, else negative_slope * x. Unlike standard ReLU, Leaky ReLU allows a small gradient for negative values.
ggml_leaky_relu(ctx, a, negative_slope = 0.01, inplace = FALSE)ggml_leaky_relu(ctx, a, negative_slope = 0.01, inplace = FALSE)
ctx |
GGML context |
a |
Input tensor |
negative_slope |
Slope for negative values (default: 0.01) |
inplace |
If TRUE, operation is performed in-place (default: FALSE) |
Tensor representing the Leaky ReLU operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) r <- ggml_leaky_relu(ctx, a, negative_slope = 0.1) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # [-0.2, -0.1, 0, 1, 2] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) r <- ggml_leaky_relu(ctx, a, negative_slope = 0.1) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # [-0.2, -0.1, 0, 1, 2] ggml_free(ctx)
Restores a model previously saved with ggml_save_model(). The
returned model is compiled and ready for ggml_predict() /
ggml_evaluate(). Call ggml_fit() again to continue training.
ggml_load_model(path, backend = "auto")ggml_load_model(path, backend = "auto")
path |
File path to an RDS file written by |
backend |
Backend selection: |
A compiled model object.
model <- ggml_model_sequential() |> ggml_layer_dense(16L, activation = "relu", input_shape = 4L) |> ggml_layer_dense(2L, activation = "softmax") model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy") x <- matrix(runif(64 * 4), 64, 4) y <- matrix(c(rep(c(1,0), 32), rep(c(0,1), 32)), 64, 2) model <- ggml_fit(model, x, y, epochs = 1L, batch_size = 32L, verbose = 0L) tmp <- tempfile(fileext = ".rds") ggml_save_model(model, tmp) model2 <- ggml_load_model(tmp)model <- ggml_model_sequential() |> ggml_layer_dense(16L, activation = "relu", input_shape = 4L) |> ggml_layer_dense(2L, activation = "softmax") model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy") x <- matrix(runif(64 * 4), 64, 4) y <- matrix(c(rep(c(1,0), 32), rep(c(0,1), 32)), 64, 2) model <- ggml_fit(model, x, y, epochs = 1L, batch_size = 32L, verbose = 0L) tmp <- tempfile(fileext = ".rds") ggml_save_model(model, tmp) model2 <- ggml_load_model(tmp)
Loads previously saved weights into a compiled model. The model architecture must match the saved weights (same layer types, sizes, and shapes).
ggml_load_weights(model, path)ggml_load_weights(model, path)
model |
A compiled ggml_sequential_model (same architecture as saved) |
path |
File path to load weights from |
The model with loaded weights.
Creates a graph node for element-wise natural logarithm: log(x)
ggml_log(ctx, a)ggml_log(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the log operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 3) ggml_set_f32(a, c(1, exp(1), exp(2))) result <- ggml_log(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [0, 1, 2] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 3) ggml_set_f32(a, c(1, exp(1), exp(2))) result <- ggml_log(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [0, 1, 2] ggml_free(ctx)
Creates a graph node for in-place element-wise natural logarithm.
ggml_log_inplace(ctx, a)ggml_log_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with log values
Check if R Logging is Enabled
ggml_log_is_r_enabled()ggml_log_is_r_enabled()
Logical indicating if R-compatible logging is active
Other logging:
ggml_abort_is_r_enabled(),
ggml_log_set_default(),
ggml_log_set_r(),
ggml_set_abort_callback_default(),
ggml_set_abort_callback_r()
Restores GGML to default logging behavior (stderr output).
ggml_log_set_default()ggml_log_set_default()
NULL invisibly
Other logging:
ggml_abort_is_r_enabled(),
ggml_log_is_r_enabled(),
ggml_log_set_r(),
ggml_set_abort_callback_default(),
ggml_set_abort_callback_r()
Redirects GGML log messages to R's message system: - INFO/DEBUG messages go to stdout (via Rprintf) - WARN/ERROR messages go to stderr (via REprintf)
ggml_log_set_r()ggml_log_set_r()
NULL invisibly
Other logging:
ggml_abort_is_r_enabled(),
ggml_log_is_r_enabled(),
ggml_log_set_default(),
ggml_set_abort_callback_default(),
ggml_set_abort_callback_r()
ggml_log_set_r() # Now GGML messages will appear in R consoleggml_log_set_r() # Now GGML messages will appear in R console
Create an LSTM Layer Object
ggml_lstm( units, return_sequences = FALSE, activation = "tanh", recurrent_activation = "sigmoid", name = NULL, trainable = TRUE )ggml_lstm( units, return_sequences = FALSE, activation = "tanh", recurrent_activation = "sigmoid", name = NULL, trainable = TRUE )
units |
Integer, number of hidden units. |
return_sequences |
Logical. |
activation |
Cell gate activation (default |
recurrent_activation |
Recurrent gate activation (default |
name |
Optional character name. |
trainable |
Logical. |
A ggml_layer object.
Serializes a trained sequential or functional ggmlR model into a self-describing raw container suitable for transport between R sessions or parallel workers (e.g. for mlr3 parallel resampling and tuning).
ggml_marshal_model(model)ggml_marshal_model(model)
model |
A compiled |
The container wraps the bytes produced by ggml_save_model
together with a format tag, schema version, package/R versions, a SHA-256
integrity checksum, and a timestamp. Autograd modules are not
supported in this version and cause the function to signal an error; the
mlr3 learners catch this and fall back to marshaled = FALSE.
A named list with class "ggmlR_marshaled" containing the
serialized payload and metadata. Pass it to
ggml_unmarshal_model to reconstruct the model.
ggml_unmarshal_model, ggml_save_model
%*%)Computes A %*% B on the Vulkan GPU and returns an ordinary R matrix,
with a transparent CPU fallback. A drop-in accelerator for a large matrix
multiply: no autograd, no wrapper types required — plain matrices in and out.
ggml_matmul(A, B, device = "auto", prec = "f32")ggml_matmul(A, B, device = "auto", prec = "f32")
A, B
|
Numeric matrices with |
device |
|
prec |
|
Precision: R multiplies in double precision (f64); a GPU offers f32 at best,
so the GPU result never matches R to full double precision. prec = "f32"
(the default) requests the f32 accumulation kernel, but how close the result
actually lands depends on the Vulkan driver: some accumulate mul_mat in
f16 regardless (e.g. RADV / Mesa), giving a relative error around
1e-3 either way. Treat the GPU path as a fast, approximate multiply —
typically 1e-3, better on drivers with true f32 accumulation — not a
bit-for-bit replacement for %*%. prec = "f16" only ever lowers
precision; use it when speed matters more than the last digits.
The product A %*% B as a numeric matrix.
ggml_crossprod, ggml_tcrossprod,
as_gpu_matrix
A <- matrix(rnorm(4), 2); B <- matrix(rnorm(4), 2) ggml_matmul(A, B, device = "cpu")A <- matrix(rnorm(4), 2); B <- matrix(rnorm(4), 2) ggml_matmul(A, B, device = "cpu")
Computes A %*% B on the Vulkan GPU in full double precision
(fp64), returning an ordinary R matrix, with a transparent CPU fallback.
Unlike ggml_matmul (which is f32/f16 and approximate), this
matches R's %*% to machine precision (~1e-15), because the
kernel accumulates in double throughout.
ggml_matmul_f64(A, B, device = "auto")ggml_matmul_f64(A, B, device = "auto")
A, B
|
Numeric matrices with |
device |
|
fp64 on the GPU is only worthwhile on hardware with fast double throughput:
data-centre cards (NVIDIA Tesla P100/V100, AMD Instinct) run fp64 at 1:2 of
fp32, whereas consumer GPUs cripple it (RDNA ~1:16, GeForce ~1:64), where the
CPU (multithreaded double BLAS) is usually faster. Use this when you need
bit-accurate double results on a capable GPU, or for numerically sensitive
work where the f32 path's error is unacceptable; otherwise prefer
ggml_matmul. Requires the shaderFloat64 device feature — without
it (or without a GPU), the multiply silently falls back to the CPU.
The product A %*% B as a numeric matrix, accurate to
double precision.
ggml_matmul (faster f32/f16 approximate path).
A <- matrix(rnorm(4), 2); B <- matrix(rnorm(4), 2) ggml_matmul_f64(A, B, device = "cpu")A <- matrix(rnorm(4), 2); B <- matrix(rnorm(4), 2) ggml_matmul_f64(A, B, device = "cpu")
An S4 class holding an ordinary numeric matrix, tagged so that %*%,
crossprod() and tcrossprod() run on the GPU (see
ggml_matmul). Wrap a matrix with as_gpu_matrix.
## S4 method for signature 'ggml_matrix,ANY' x %*% y ## S4 method for signature 'ANY,ggml_matrix' x %*% y ## S4 method for signature 'ggml_matrix,ggml_matrix' x %*% y ## S4 method for signature 'ggml_matrix' crossprod(x, y) ## S4 method for signature 'ggml_matrix' tcrossprod(x, y) ## S4 method for signature 'ggml_matrix' dim(x) ## S4 method for signature 'ggml_matrix' show(object)## S4 method for signature 'ggml_matrix,ANY' x %*% y ## S4 method for signature 'ANY,ggml_matrix' x %*% y ## S4 method for signature 'ggml_matrix,ggml_matrix' x %*% y ## S4 method for signature 'ggml_matrix' crossprod(x, y) ## S4 method for signature 'ggml_matrix' tcrossprod(x, y) ## S4 method for signature 'ggml_matrix' dim(x) ## S4 method for signature 'ggml_matrix' show(object)
x, y
|
Operands; at least one is a |
object |
A |
%*%, crossprod and tcrossprod return a plain
numeric matrix (the GPU result); dim an integer vector.
dataThe underlying numeric matrix.
deviceDispatch preference: "auto", "gpu" or "cpu".
precPrecision of the GPU path: "f32" or "f16".
Creates a graph node that computes the mean of all elements.
ggml_mean(ctx, a)ggml_mean(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Scalar tensor with the mean
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(2, 4, 6, 8, 10)) result <- ggml_mean(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # 6 ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(2, 4, 6, 8, 10)) result <- ggml_mean(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # 6 ggml_free(ctx)
Assembles a ggml_functional_model from symbolic input and output
nodes produced by ggml_input() and ggml_layer_*() calls.
ggml_model(inputs, outputs)ggml_model(inputs, outputs)
inputs |
A |
outputs |
A |
A ggml_functional_model object.
x <- ggml_input(shape = 64L) out <- x |> ggml_layer_dense(10, activation = "softmax") model <- ggml_model(inputs = x, outputs = out)x <- ggml_input(shape = 64L) out <- x |> ggml_layer_dense(10, activation = "softmax") model <- ggml_model(inputs = x, outputs = out)
Reports the backend the model was actually compiled onto, making a
silent backend = "auto" fallback to CPU (when no GPU is available)
inspectable. Works on a raw sequential/functional model or a fitted parsnip
engine object.
ggml_model_backend(object, verbose = FALSE)ggml_model_backend(object, verbose = FALSE)
object |
A compiled/fitted |
verbose |
If |
If verbose = FALSE, a length-1 character: the backend in use
("vulkan" or "cpu"). If verbose = TRUE, a list with:
requested (what was asked: "auto"/"cpu"/"vulkan"),
used ("vulkan"/"cpu"), device (GPU device
description, or "cpu") and fallback (logical: TRUE when
a non-CPU backend was requested but CPU was used instead).
Creates an empty sequential model that layers can be added to using
pipe (|>) operators.
ggml_model_sequential()ggml_model_sequential()
A ggml_sequential_model object
## Not run: model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_max_pooling_2d(c(2,2)) |> ggml_layer_flatten() |> ggml_layer_dense(128, activation = "relu") |> ggml_layer_dense(10, activation = "softmax") ## End(Not run)## Not run: model <- ggml_model_sequential() |> ggml_layer_conv_2d(32, c(3,3), activation = "relu", input_shape = c(28, 28, 1)) |> ggml_layer_max_pooling_2d(c(2,2)) |> ggml_layer_flatten() |> ggml_layer_dense(128, activation = "relu") |> ggml_layer_dense(10, activation = "softmax") ## End(Not run)
Creates a graph node for element-wise multiplication.
ggml_mul(ctx, a, b) ggml_mul(ctx, a, b)ggml_mul(ctx, a, b) ggml_mul(ctx, a, b)
ctx |
GGML context |
a |
First tensor |
b |
Second tensor (same shape as a) |
Tensor representing the multiplication operation
Tensor representing the multiplication operation
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(2, 2, 2, 2, 2)) result <- ggml_mul(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx) ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(2, 2, 2, 2, 2)) result <- ggml_mul(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(2, 2, 2, 2, 2)) result <- ggml_mul(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx) ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) ggml_set_f32(b, c(2, 2, 2, 2, 2)) result <- ggml_mul(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)
Creates a graph node for in-place element-wise multiplication. Result is stored in tensor a, saving memory allocation.
ggml_mul_inplace(ctx, a, b)ggml_mul_inplace(ctx, a, b)
ctx |
GGML context |
a |
First tensor (will be modified in-place) |
b |
Second tensor (same shape as a) |
View of tensor a with the multiplication result
Creates a graph node for matrix multiplication. CRITICAL for LLM operations. For matrices A (m x n) and B (n x p), computes C = A * B (m x p).
ggml_mul_mat(ctx, a, b)ggml_mul_mat(ctx, a, b)
ctx |
GGML context |
a |
First matrix tensor |
b |
Second matrix tensor |
Tensor representing the matrix multiplication
ctx <- ggml_init(1024 * 1024) A <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 3) B <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 2) ggml_set_f32(A, 1:12) ggml_set_f32(B, 1:8) C <- ggml_mul_mat(ctx, A, B) graph <- ggml_build_forward_expand(ctx, C) ggml_graph_compute(ctx, graph) ggml_get_f32(C) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) A <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 3) B <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 4, 2) ggml_set_f32(A, 1:12) ggml_set_f32(B, 1:8) C <- ggml_mul_mat(ctx, A, B) graph <- ggml_build_forward_expand(ctx, C) ggml_graph_compute(ctx, graph) ggml_get_f32(C) ggml_free(ctx)
Indirect matrix multiplication for Mixture of Experts architectures. Selects expert weights based on indices and performs batched matmul.
ggml_mul_mat_id(ctx, as, b, ids)ggml_mul_mat_id(ctx, as, b, ids)
ctx |
GGML context |
as |
Stacked expert weight matrices [n_embd, n_ff, n_experts] |
b |
Input tensor |
ids |
Expert selection indices tensor (I32) |
Output tensor after expert-selected matrix multiplication
ctx <- ggml_init(64 * 1024 * 1024) # 4 experts, each with 8x16 weights (small for example) experts <- ggml_new_tensor_3d(ctx, GGML_TYPE_F32, 8, 16, 4) ggml_set_f32(experts, rnorm(8 * 16 * 4)) input <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 2) ggml_set_f32(input, rnorm(16)) # Select expert 0 for token 0, expert 2 for token 1 ids <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 2) ggml_set_i32(ids, c(0L, 2L)) output <- ggml_mul_mat_id(ctx, experts, input, ids) graph <- ggml_build_forward_expand(ctx, output) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(64 * 1024 * 1024) # 4 experts, each with 8x16 weights (small for example) experts <- ggml_new_tensor_3d(ctx, GGML_TYPE_F32, 8, 16, 4) ggml_set_f32(experts, rnorm(8 * 16 * 4)) input <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 2) ggml_set_f32(input, rnorm(16)) # Select expert 0 for token 0, expert 2 for token 1 ids <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 2) ggml_set_i32(ids, c(0L, 2L)) output <- ggml_mul_mat_id(ctx, experts, input, ids) graph <- ggml_build_forward_expand(ctx, output) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Returns the number of dimensions of a tensor
ggml_n_dims(tensor)ggml_n_dims(tensor)
tensor |
Tensor pointer |
Integer number of dimensions (1-4)
ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_n_dims(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_n_dims(t) ggml_free(ctx)
Sentinel for the n_tasks argument of ggml_custom and
ggml_custom_inplace, meaning "use as many threads as the
backend allows" (mirrors GGML_N_TASKS_MAX in ggml.h).
GGML_N_TASKS_MAXGGML_N_TASKS_MAX
An object of class integer of length 1.
Get number of bytes
Get Number of Bytes
ggml_nbytes(tensor) ggml_nbytes(tensor)ggml_nbytes(tensor) ggml_nbytes(tensor)
tensor |
Tensor |
Integer number of bytes
Integer number of bytes
ctx <- ggml_init(1024 * 1024) tensor <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_nbytes(tensor) ggml_free(ctx) ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_nbytes(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) tensor <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_nbytes(tensor) ggml_free(ctx) ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_nbytes(t) ggml_free(ctx)
Creates a graph node for element-wise negation: -x
ggml_neg(ctx, a)ggml_neg(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the negation operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, -2, 3, -4)) result <- ggml_neg(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [-1, 2, -3, 4] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, -2, 3, -4)) result <- ggml_neg(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [-1, 2, -3, 4] ggml_free(ctx)
Creates a graph node for in-place element-wise negation: -x
ggml_neg_inplace(ctx, a)ggml_neg_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with negated values
Get number of elements
Get Number of Elements
ggml_nelements(tensor) ggml_nelements(tensor)ggml_nelements(tensor) ggml_nelements(tensor)
tensor |
Tensor |
Integer number of elements
Integer number of elements
ctx <- ggml_init(1024 * 1024) tensor <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_nelements(tensor) ggml_free(ctx) ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_nelements(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) tensor <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_nelements(tensor) ggml_free(ctx) ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_nelements(t) ggml_free(ctx)
Creates a 1-element tensor containing a single float value. Useful for scalar operations, learning rates, and other scalar floats.
ggml_new_f32(ctx, value)ggml_new_f32(ctx, value)
ctx |
GGML context |
value |
Numeric value |
Tensor pointer (1-element F32 tensor)
ctx <- ggml_init(1024 * 1024) scalar <- ggml_new_f32(ctx, 3.14) ggml_get_f32(scalar) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) scalar <- ggml_new_f32(ctx, 3.14) ggml_get_f32(scalar) ggml_free(ctx)
Creates a 1-element tensor containing a single integer value. Useful for indices, counters, and other scalar integer operations.
ggml_new_i32(ctx, value)ggml_new_i32(ctx, value)
ctx |
GGML context |
value |
Integer value |
Tensor pointer (1-element I32 tensor)
ctx <- ggml_init(1024 * 1024) scalar <- ggml_new_i32(ctx, 42) ggml_get_i32(scalar) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) scalar <- ggml_new_i32(ctx, 42) ggml_get_i32(scalar) ggml_free(ctx)
Generic tensor constructor for creating tensors with 1-4 dimensions. This is more flexible than the ggml_new_tensor_Nd functions.
ggml_new_tensor(ctx, type = GGML_TYPE_F32, n_dims, ne)ggml_new_tensor(ctx, type = GGML_TYPE_F32, n_dims, ne)
ctx |
GGML context |
type |
Data type (GGML_TYPE_F32, etc.) |
n_dims |
Number of dimensions (1-4) |
ne |
Numeric vector of dimension sizes |
Tensor pointer
ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor(ctx, GGML_TYPE_F32, 3, c(10, 20, 30)) ggml_nelements(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor(ctx, GGML_TYPE_F32, 3, c(10, 20, 30)) ggml_nelements(t) ggml_free(ctx)
Create 1D tensor
Create 1D Tensor
ggml_new_tensor_1d(ctx, type = GGML_TYPE_F32, ne0) ggml_new_tensor_1d(ctx, type = GGML_TYPE_F32, ne0)ggml_new_tensor_1d(ctx, type = GGML_TYPE_F32, ne0) ggml_new_tensor_1d(ctx, type = GGML_TYPE_F32, ne0)
ctx |
GGML context |
type |
Data type |
ne0 |
Size |
Tensor pointer
Tensor pointer
ctx <- ggml_init(1024 * 1024) tensor <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_nelements(tensor) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) tensor <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_nelements(tensor) ggml_free(ctx)
Create 2D tensor
Create 2D Tensor
ggml_new_tensor_2d(ctx, type = GGML_TYPE_F32, ne0, ne1) ggml_new_tensor_2d(ctx, type = GGML_TYPE_F32, ne0, ne1)ggml_new_tensor_2d(ctx, type = GGML_TYPE_F32, ne0, ne1) ggml_new_tensor_2d(ctx, type = GGML_TYPE_F32, ne0, ne1)
ctx |
GGML context |
type |
Data type |
ne0 |
Rows |
ne1 |
Columns |
Tensor pointer
Tensor pointer
ctx <- ggml_init(1024 * 1024) tensor <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_nelements(tensor) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) tensor <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_nelements(tensor) ggml_free(ctx)
Create 3D Tensor
ggml_new_tensor_3d(ctx, type = GGML_TYPE_F32, ne0, ne1, ne2)ggml_new_tensor_3d(ctx, type = GGML_TYPE_F32, ne0, ne1, ne2)
ctx |
GGML context |
type |
Data type (default GGML_TYPE_F32) |
ne0 |
Size of dimension 0 |
ne1 |
Size of dimension 1 |
ne2 |
Size of dimension 2 |
Tensor pointer
ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_3d(ctx, GGML_TYPE_F32, 10, 20, 30) ggml_nelements(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_3d(ctx, GGML_TYPE_F32, 10, 20, 30) ggml_nelements(t) ggml_free(ctx)
Create 4D Tensor
ggml_new_tensor_4d(ctx, type = GGML_TYPE_F32, ne0, ne1, ne2, ne3)ggml_new_tensor_4d(ctx, type = GGML_TYPE_F32, ne0, ne1, ne2, ne3)
ctx |
GGML context |
type |
Data type (default GGML_TYPE_F32) |
ne0 |
Size of dimension 0 |
ne1 |
Size of dimension 1 |
ne2 |
Size of dimension 2 |
ne3 |
Size of dimension 3 |
Tensor pointer
ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 8, 8, 3, 2) ggml_nelements(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 8, 8, 3, 2) ggml_nelements(t) ggml_free(ctx)
Creates a graph node for layer normalization. Normalizes input to zero mean and unit variance.
ggml_norm(ctx, a, eps = 1e-05)ggml_norm(ctx, a, eps = 1e-05)
ctx |
GGML context |
a |
Input tensor |
eps |
Epsilon value for numerical stability (default: 1e-5) |
Tensor representing the layer normalization operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_norm(ctx, a, eps = 1e-5) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) # Normalized values ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_norm(ctx, a, eps = 1e-5) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) # Normalized values ggml_free(ctx)
Creates a graph node for the backward pass of ggml_norm, i.e.
of LayerNorm without gamma/beta. Upstream ggml has no backward for
GGML_OP_NORM; ggmlR adds it with CPU and Vulkan kernels, which is what
makes ggml_layer_layer_norm trainable.
ggml_norm_back(ctx, a, b, eps = 1e-05)ggml_norm_back(ctx, a, b, eps = 1e-05)
ctx |
GGML context |
a |
Gradient tensor (dy from the forward output) |
b |
Input tensor (x from the forward pass) |
eps |
Epsilon for numerical stability (default 1e-5) |
Tensor representing the gradient with respect to input
Creates a graph node for in-place layer normalization. Returns a view of the input tensor.
ggml_norm_inplace(ctx, a, eps = 1e-05)ggml_norm_inplace(ctx, a, eps = 1e-05)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
eps |
Epsilon value for numerical stability (default: 1e-5) |
View of input tensor with layer normalization applied
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_norm_inplace(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_norm_inplace(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Returns the number of rows in a tensor (product of all dimensions except ne[0]).
ggml_nrows(tensor)ggml_nrows(tensor)
tensor |
Tensor pointer |
Number of rows
Returns whether a GGML operation can reuse memory from its source tensors. This is useful for memory optimization.
ggml_op_can_inplace(op)ggml_op_can_inplace(op)
op |
Operation code (integer) |
Logical indicating if operation supports in-place execution
Other graph:
ggml_graph_view()
# Check if operation code 1 (ADD) can be in-place can_inplace <- ggml_op_can_inplace(1L)# Check if operation code 1 (ADD) can be in-place can_inplace <- ggml_op_can_inplace(1L)
Returns a description of the operation that produces a tensor.
ggml_op_desc(tensor)ggml_op_desc(tensor)
tensor |
Tensor pointer |
Character string describing the operation
Other op_info:
ggml_get_unary_op(),
ggml_op_name(),
ggml_op_symbol(),
ggml_unary_op_name()
Returns the string name of a GGML operation.
ggml_op_name(op)ggml_op_name(op)
op |
GGML operation constant |
Character string with operation name
Other op_info:
ggml_get_unary_op(),
ggml_op_desc(),
ggml_op_symbol(),
ggml_unary_op_name()
Returns the mathematical symbol for a GGML operation.
ggml_op_symbol(op)ggml_op_symbol(op)
op |
GGML operation constant |
Character string with operation symbol
Other op_info:
ggml_get_unary_op(),
ggml_op_desc(),
ggml_op_name(),
ggml_unary_op_name()
Returns the registry of operations the single-cell adapter can dispatch. Use
this to check capabilities (and required parameters) before building a
ggml_task — capability is declared, never discovered at runtime.
ggml_ops_registry(op = NULL)ggml_ops_registry(op = NULL)
op |
Optional operation name. If supplied, returns that single entry (or
|
A list describing the operation(s): op, params
(required parameter names) and desc.
ggml_ops_registry() ggml_ops_registry("embed")ggml_ops_registry() ggml_ops_registry("embed")
Must be called before ggml_opt_eval. Allocates forward or forward+backward graph.
ggml_opt_alloc(opt_ctx, backward = TRUE)ggml_opt_alloc(opt_ctx, backward = TRUE)
opt_ctx |
External pointer to optimizer context |
backward |
Whether to allocate backward graph (for training) |
NULL invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Get optimizer type from context
ggml_opt_context_optimizer_type(opt_ctx)ggml_opt_context_optimizer_type(opt_ctx)
opt_ctx |
External pointer to optimizer context |
Integer optimizer type constant
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the underlying data tensor with shape [ne_datapoint, ndata].
ggml_opt_dataset_data(dataset)ggml_opt_dataset_data(dataset)
dataset |
External pointer to dataset |
External pointer to data tensor
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Releases memory associated with a dataset.
ggml_opt_dataset_free(dataset)ggml_opt_dataset_free(dataset)
dataset |
External pointer to dataset |
NULL invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Copies a batch of data and labels to the provided tensors.
ggml_opt_dataset_get_batch(dataset, data_batch, labels_batch = NULL, ibatch)ggml_opt_dataset_get_batch(dataset, data_batch, labels_batch = NULL, ibatch)
dataset |
External pointer to dataset |
data_batch |
Tensor to receive data batch |
labels_batch |
Tensor to receive labels batch (can be NULL) |
ibatch |
Batch index |
NULL invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Multi-output counterpart of ggml_opt_dataset_get_batch(). The
dataset's labels hold every head's labels concatenated along the first
dimension; this copies just the slice belonging to one head.
ggml_opt_dataset_get_batch_head( dataset, data_batch, labels_batch, labels_off, ibatch )ggml_opt_dataset_get_batch_head( dataset, data_batch, labels_batch, labels_off, ibatch )
dataset |
Dataset pointer |
data_batch |
Input batch tensor, or NULL to copy only the labels (the caller copies the inputs once, with the first head) |
labels_batch |
This head's labels batch tensor |
labels_off |
Offset of this head within a label row, 0-based, in elements |
ibatch |
Batch index, 0-based |
NULL, invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Creates a dataset for training with specified data and label types.
ggml_opt_dataset_init( type_data, type_label, ne_datapoint, ne_label, ndata, ndata_shard = 1 )ggml_opt_dataset_init( type_data, type_label, ne_datapoint, ne_label, ndata, ndata_shard = 1 )
type_data |
GGML type for data tensor (e.g., GGML_TYPE_F32) |
type_label |
GGML type for label tensor (e.g., GGML_TYPE_F32) |
ne_datapoint |
Number of elements per datapoint |
ne_label |
Number of elements per label (0 if no labels) |
ndata |
Total number of datapoints |
ndata_shard |
Shard size for shuffling (1 is fine for most cases) |
External pointer to dataset
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the underlying labels tensor with shape [ne_label, ndata].
ggml_opt_dataset_labels(dataset)ggml_opt_dataset_labels(dataset)
dataset |
External pointer to dataset |
External pointer to labels tensor, or NULL if no labels
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Get number of datapoints in dataset
ggml_opt_dataset_ndata(dataset)ggml_opt_dataset_ndata(dataset)
dataset |
External pointer to dataset |
Number of datapoints
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Shuffles the dataset using the RNG from the optimizer context.
ggml_opt_dataset_shuffle(opt_ctx, dataset, idata = -1)ggml_opt_dataset_shuffle(opt_ctx, dataset, idata = -1)
opt_ctx |
External pointer to optimizer context |
dataset |
External pointer to dataset |
idata |
Number of datapoints to shuffle (-1 for all) |
NULL invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the (lazily allocated) loss weights tensor with shape
[ne0, ndata]. The first call allocates it; fill it via
ggml_backend_tensor_set_data(). Used together with
ggml_opt_loss_type_weighted_mse.
ggml_opt_dataset_weights(dataset, ne0 = 1L)ggml_opt_dataset_weights(dataset, ne0 = 1L)
dataset |
External pointer to dataset |
ne0 |
Width of the weight tensor: 1 for per-datapoint weights, or the label width for a per-output mask. |
ne0 = 1 gives one weight per datapoint, broadcast over the outputs –
the original behaviour. ne0 = ne_label gives a per-output-element
mask, which is what a Q-model needs: weight 1 on the action that has a
TD target, 0 elsewhere, so only that coordinate is trained.
The mask does not change the denominator: the loss stays
sum(w * (pred - y)^2) / nelements(pred), so zeroed elements drop out
of the numerator without renormalising over the active coordinates.
Calling this again with a different ne0 than the allocation is an
error rather than a silent return of the cached tensor – both shapes
broadcast, so a mismatch would otherwise weight the wrong thing unnoticed.
External pointer to weights tensor
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns a list with default optimization parameters.
ggml_opt_default_params(sched, loss_type)ggml_opt_default_params(sched, loss_type)
sched |
Backend scheduler |
loss_type |
Loss type constant |
List with loss_type, build_type, opt_period, optimizer
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Performs training on the front portion of the dataset and evaluation on the back portion. This gives more control than ggml_opt_fit.
ggml_opt_epoch( opt_ctx, dataset, result_train = NULL, result_eval = NULL, idata_split, callback_train = TRUE, callback_eval = TRUE )ggml_opt_epoch( opt_ctx, dataset, result_train = NULL, result_eval = NULL, idata_split, callback_train = TRUE, callback_eval = TRUE )
opt_ctx |
External pointer to optimizer context |
dataset |
External pointer to dataset |
result_train |
Result object to accumulate training stats (or NULL) |
result_eval |
Result object to accumulate evaluation stats (or NULL) |
idata_split |
Data index at which to split training and evaluation |
callback_train |
Callback for training: TRUE for progress bar, FALSE for none, or a function(train, ibatch, ibatch_max, t_start_us, result) |
callback_eval |
Callback for evaluation: TRUE for progress bar, FALSE for none, or a function(train, ibatch, ibatch_max, t_start_us, result) |
NULL invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
# Requires full optimizer setup - see ggml_opt_fit() for simpler API if (FALSE) { result_train <- ggml_opt_result_init() result_eval <- ggml_opt_result_init() ggml_opt_epoch(opt_ctx, dataset, result_train, result_eval, idata_split = 900, callback_train = TRUE) ggml_opt_result_free(result_train) ggml_opt_result_free(result_eval) }# Requires full optimizer setup - see ggml_opt_fit() for simpler API if (FALSE) { result_train <- ggml_opt_result_init() result_eval <- ggml_opt_result_init() ggml_opt_epoch(opt_ctx, dataset, result_train, result_eval, idata_split = 900, callback_train = TRUE) ggml_opt_result_free(result_train) ggml_opt_result_free(result_eval) }
Performs forward pass, optionally increments result, and does backward pass if allocated.
ggml_opt_eval(opt_ctx, result = NULL)ggml_opt_eval(opt_ctx, result = NULL)
opt_ctx |
External pointer to optimizer context |
result |
External pointer to result object (optional) |
NULL invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
High-level function to train a model on a dataset. This is the recommended way to train models.
ggml_opt_fit( sched, ctx_compute, inputs, outputs, dataset, loss_type = ggml_opt_loss_type_mse(), optimizer = ggml_opt_optimizer_type_adamw(), nepoch = 1, nbatch_logical = 32, val_split = 0, silent = FALSE )ggml_opt_fit( sched, ctx_compute, inputs, outputs, dataset, loss_type = ggml_opt_loss_type_mse(), optimizer = ggml_opt_optimizer_type_adamw(), nepoch = 1, nbatch_logical = 32, val_split = 0, silent = FALSE )
sched |
Backend scheduler |
ctx_compute |
Compute context (for temporary tensors) |
inputs |
Input tensor with shape [ne_datapoint, batch_size] |
outputs |
Output tensor with shape [ne_label, batch_size] |
dataset |
Dataset created with ggml_opt_dataset_init |
loss_type |
Loss type (default: MSE) |
optimizer |
Optimizer type (default: AdamW) |
nepoch |
Number of epochs |
nbatch_logical |
Logical batch size (for gradient accumulation) |
val_split |
Fraction of data for validation (0.0 to 1.0) |
silent |
Whether to suppress progress output |
NULL invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
# Full training requires building a computation graph # See package vignettes for complete examples if (FALSE) { cpu <- ggml_backend_cpu_init() sched <- ggml_backend_sched_new(list(cpu)) dataset <- ggml_opt_dataset_init(GGML_TYPE_F32, GGML_TYPE_F32, 10, 1, 1000) # ... build model graph with ctx_compute, inputs, outputs ... ggml_opt_fit(sched, ctx_compute, inputs, outputs, dataset, nepoch = 10, val_split = 0.1) ggml_opt_dataset_free(dataset) ggml_backend_sched_free(sched) ggml_backend_free(cpu) }# Full training requires building a computation graph # See package vignettes for complete examples if (FALSE) { cpu <- ggml_backend_cpu_init() sched <- ggml_backend_sched_new(list(cpu)) dataset <- ggml_opt_dataset_init(GGML_TYPE_F32, GGML_TYPE_F32, 10, 1, 1000) # ... build model graph with ctx_compute, inputs, outputs ... ggml_opt_fit(sched, ctx_compute, inputs, outputs, dataset, nepoch = 10, val_split = 0.1) ggml_opt_dataset_free(dataset) ggml_backend_sched_free(sched) ggml_backend_free(cpu) }
Trains a model with several output heads. Each head has its own labels,
loss type and weight; the optimized total is
sum(loss_weights[i] * loss_i).
ggml_opt_fit_multi( sched, ctx_compute, inputs, outputs, dataset, loss_types, loss_weights = rep(1, length(outputs)), labels_offs = NULL, optimizer = ggml_opt_optimizer_type_adamw(), nepoch = 1, nbatch_logical = 32, val_split = 0, silent = FALSE )ggml_opt_fit_multi( sched, ctx_compute, inputs, outputs, dataset, loss_types, loss_weights = rep(1, length(outputs)), labels_offs = NULL, optimizer = ggml_opt_optimizer_type_adamw(), nepoch = 1, nbatch_logical = 32, val_split = 0, silent = FALSE )
sched |
Backend scheduler |
ctx_compute |
Compute context holding the model graph |
inputs |
Input tensor |
outputs |
List of output tensors, one per head |
dataset |
Dataset with data and concatenated labels |
loss_types |
Integer vector of loss type constants, one per head |
loss_weights |
Numeric vector of head weights (default: all 1) |
labels_offs |
Numeric vector of label offsets, 0-based, one per head |
optimizer |
Optimizer type constant |
nepoch |
Number of epochs |
nbatch_logical |
Logical batch size |
val_split |
Fraction of the data used for validation |
silent |
Suppress progress output |
All heads share the dataset's datapoints. The dataset's labels hold every
head's labels concatenated along the first dimension: head i occupies
labels_offs[i] + seq_len(width_i) of each label row.
Named list with train_loss, train_accuracy, val_loss, val_accuracy, head_loss (a nepoch x nhead matrix of unweighted per-head losses) and head_accuracy (same shape; NA for heads whose loss is not cross-entropy)
ggml_fit_opt_multi, which this wraps and which adds
callbacks and shuffling control
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Releases memory associated with an optimizer context.
ggml_opt_free(opt_ctx)ggml_opt_free(opt_ctx)
opt_ctx |
External pointer to optimizer context |
NULL invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Get current learning rate from optimizer context
ggml_opt_get_lr(lr_ud)ggml_opt_get_lr(lr_ud)
lr_ud |
LR userdata pointer (from 'ggml_opt_init_for_fit()$lr_ud') |
Named numeric vector with 'adamw' and 'sgd' LR values
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the gradient accumulator tensor for a node from the forward graph.
ggml_opt_grad_acc(opt_ctx, node)ggml_opt_grad_acc(opt_ctx, node)
opt_ctx |
External pointer to optimizer context |
node |
External pointer to tensor node |
External pointer to gradient accumulator tensor, or NULL if not found
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Creates a new optimizer context for training.
ggml_opt_init( sched, loss_type, optimizer = ggml_opt_optimizer_type_adamw(), opt_period = 1L, ctx_compute = NULL, inputs = NULL, outputs = NULL )ggml_opt_init( sched, loss_type, optimizer = ggml_opt_optimizer_type_adamw(), opt_period = 1L, ctx_compute = NULL, inputs = NULL, outputs = NULL )
sched |
Backend scheduler |
loss_type |
Loss type (use ggml_opt_loss_type_* functions) |
optimizer |
Optimizer type (use ggml_opt_optimizer_type_* functions) |
opt_period |
Gradient accumulation steps before optimizer step |
ctx_compute |
Compute context for static graph mode (or NULL) |
inputs |
Input tensor for static graph mode (or NULL) |
outputs |
Output tensor for static graph mode (or NULL) |
External pointer to optimizer context
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns a list with 'opt_ctx' and 'lr_ud' (learning rate userdata pointer). Use 'ggml_opt_set_lr()' to update LR between epochs. The optimizer state (momentum) is preserved across epochs.
ggml_opt_init_for_fit( sched, loss_type, optimizer = ggml_opt_optimizer_type_adamw(), opt_period = 1L, ctx_compute = NULL, inputs = NULL, outputs = NULL, loss_mask_ne0 = NULL, weight_decay = 0 )ggml_opt_init_for_fit( sched, loss_type, optimizer = ggml_opt_optimizer_type_adamw(), opt_period = 1L, ctx_compute = NULL, inputs = NULL, outputs = NULL, loss_mask_ne0 = NULL, weight_decay = 0 )
sched |
Backend scheduler |
loss_type |
Loss type constant |
optimizer |
Optimizer type constant |
opt_period |
Gradient accumulation period |
ctx_compute |
Compute context (for static graphs) |
inputs |
Input tensor (for static graphs) |
outputs |
Output tensor (for static graphs) |
loss_mask_ne0 |
Width of the weighted-MSE weight tensor: |
weight_decay |
Decoupled weight decay, in |
List with elements 'opt_ctx' and 'lr_ud'
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Multi-output counterpart of ggml_opt_init_for_fit(). Returns the same
opt_ctx/lr_ud pair, so ggml_opt_set_lr() and hence LR
schedulers work identically; in addition it records each head's loss type,
weight and label offset, which ggml_opt_fit_multi() would otherwise
set internally.
ggml_opt_init_for_fit_multi( sched, loss_types, loss_weights, labels_offs, optimizer = ggml_opt_optimizer_type_adamw(), opt_period = 1L, ctx_compute = NULL, inputs = NULL, outputs = NULL, weight_decay = 0 )ggml_opt_init_for_fit_multi( sched, loss_types, loss_weights, labels_offs, optimizer = ggml_opt_optimizer_type_adamw(), opt_period = 1L, ctx_compute = NULL, inputs = NULL, outputs = NULL, weight_decay = 0 )
sched |
Backend scheduler |
loss_types |
Integer vector of loss type constants, one per head |
loss_weights |
Numeric vector of head weights, one per head |
labels_offs |
Numeric vector of 0-based label offsets, one per head |
optimizer |
Optimizer type constant |
opt_period |
Gradient accumulation period |
ctx_compute |
Compute context holding the model graph |
inputs |
Input tensor |
outputs |
List of output tensors, one per head |
weight_decay |
As for |
List with elements 'opt_ctx' and 'lr_ud'
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Get inputs tensor from optimizer context
ggml_opt_inputs(opt_ctx)ggml_opt_inputs(opt_ctx)
opt_ctx |
External pointer to optimizer context |
External pointer to inputs tensor
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Get labels tensor from optimizer context
ggml_opt_labels(opt_ctx)ggml_opt_labels(opt_ctx)
opt_ctx |
External pointer to optimizer context |
External pointer to labels tensor
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Get loss tensor from optimizer context
ggml_opt_loss(opt_ctx)ggml_opt_loss(opt_ctx)
opt_ctx |
External pointer to optimizer context |
External pointer to loss tensor
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the head's own loss, before it is multiplied by its
loss_weights entry and summed into the total. ggml_opt_loss()
returns the weighted total that is actually optimized.
ggml_opt_loss_i(opt_ctx, ihead)ggml_opt_loss_i(opt_ctx, ihead)
opt_ctx |
External pointer to optimizer context |
ihead |
Output head index, 1-based |
External pointer to the head's loss tensor
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the constant for element-wise binary cross-entropy,
mean(-[y*log(p) + (1-y)*log(1-p)]).
ggml_opt_loss_type_binary_cross_entropy()ggml_opt_loss_type_binary_cross_entropy()
Unlike ggml_opt_loss_type_cross_entropy, which softmaxes its
own input across a class axis and therefore expects logits, this one treats
every output as an independent Bernoulli and expects probabilities –
so the model's last layer should end in a sigmoid. It is the loss for
multi-label targets and for a single-unit binary output. Inputs are clamped
away from 0 and 1 internally, since log(0) would otherwise poison the
whole batch.
Integer constant for binary cross-entropy loss
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the constant for cross entropy loss type. Use for classification tasks.
ggml_opt_loss_type_cross_entropy()ggml_opt_loss_type_cross_entropy()
Integer constant for cross entropy loss
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the constant for the Huber loss with delta = 1:
0.5 * e^2 where |e| <= 1 and |e| - 0.5 beyond it.
Quadratic near zero, so unlike MAE the gradient vanishes at the optimum, and
linear far out, so unlike MSE a single outlier cannot dominate the batch.
ggml_opt_loss_type_huber()ggml_opt_loss_type_huber()
delta is fixed at 1, the usual default.
Integer constant for Huber loss
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the constant for MAE (L1) loss, mean(|pred - y|). Its
gradient is sgn(pred - y), so a far-off datapoint pulls no harder than
a near one – which is what makes it more robust to outliers than MSE, at the
cost of a gradient that does not vanish as the fit improves.
ggml_opt_loss_type_mae()ggml_opt_loss_type_mae()
Integer constant for MAE loss
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the constant for mean loss type. Custom loss - reduces outputs to mean value.
ggml_opt_loss_type_mean()ggml_opt_loss_type_mean()
Integer constant for mean loss
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the constant for MSE loss type. Use for regression tasks.
ggml_opt_loss_type_mse()ggml_opt_loss_type_mse()
Integer constant for MSE loss
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the constant for sum loss type. Custom loss - reduces outputs to sum value.
ggml_opt_loss_type_sum()ggml_opt_loss_type_sum()
Integer constant for sum loss
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the constant for per-datapoint weighted MSE loss type. Computes
sum(w * (pred - y)^2) / nelements, where w is a per-sample
weight supplied via ggml_opt_dataset_weights.
ggml_opt_loss_type_weighted_mse()ggml_opt_loss_type_weighted_mse()
Integer constant for weighted MSE loss
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Number of output heads in an optimizer context
ggml_opt_n_loss(opt_ctx)ggml_opt_n_loss(opt_ctx)
opt_ctx |
External pointer to optimizer context |
Integer number of output heads (1 for single-output models)
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Get number of correct predictions tensor
ggml_opt_ncorrect(opt_ctx)ggml_opt_ncorrect(opt_ctx)
opt_ctx |
External pointer to optimizer context |
External pointer to ncorrect tensor
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Get optimizer name
ggml_opt_optimizer_name(optimizer_type)ggml_opt_optimizer_name(optimizer_type)
optimizer_type |
Integer optimizer type constant |
Character string with optimizer name
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the constant for AdamW optimizer. Adam with weight decay - recommended for most tasks.
ggml_opt_optimizer_type_adamw()ggml_opt_optimizer_type_adamw()
Integer constant for AdamW optimizer
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the constant for SGD optimizer. Stochastic gradient descent - simpler but may require tuning.
ggml_opt_optimizer_type_sgd()ggml_opt_optimizer_type_sgd()
Integer constant for SGD optimizer
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Get outputs tensor from optimizer context
ggml_opt_outputs(opt_ctx)ggml_opt_outputs(opt_ctx)
opt_ctx |
External pointer to optimizer context |
External pointer to outputs tensor
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Per-head counterparts of ggml_opt_outputs() and friends, for
multi-output models. ihead is 1-based; head 1 is what the
single-head accessors return.
ggml_opt_outputs_i(opt_ctx, ihead) ggml_opt_labels_i(opt_ctx, ihead) ggml_opt_loss_weights_i(opt_ctx, ihead) ggml_opt_pred_i(opt_ctx, ihead) ggml_opt_ncorrect_i(opt_ctx, ihead)ggml_opt_outputs_i(opt_ctx, ihead) ggml_opt_labels_i(opt_ctx, ihead) ggml_opt_loss_weights_i(opt_ctx, ihead) ggml_opt_pred_i(opt_ctx, ihead) ggml_opt_ncorrect_i(opt_ctx, ihead)
opt_ctx |
External pointer to optimizer context |
ihead |
Output head index, 1-based |
External pointer to the tensor, or NULL if this head has none
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Get predictions tensor from optimizer context
ggml_opt_pred(opt_ctx)ggml_opt_pred(opt_ctx)
opt_ctx |
External pointer to optimizer context |
External pointer to predictions tensor
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Must be called before ggml_opt_alloc when not using static graphs. Sets up the optimizer context with the computation graph and input/output tensors.
ggml_opt_prepare_alloc(opt_ctx, ctx_compute, graph, inputs, outputs)ggml_opt_prepare_alloc(opt_ctx, ctx_compute, graph, inputs, outputs)
opt_ctx |
External pointer to optimizer context |
ctx_compute |
Compute context for temporary tensors |
graph |
Computation graph (from ggml_build_forward_expand) |
inputs |
Input tensor |
outputs |
Output tensor |
NULL invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Resets gradients to zero, initializes loss, and optionally resets optimizer state.
ggml_opt_reset(opt_ctx, optimizer = FALSE)ggml_opt_reset(opt_ctx, optimizer = FALSE)
opt_ctx |
External pointer to optimizer context |
optimizer |
Whether to also reset optimizer state (momentum, etc.) |
NULL invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Get accuracy from result
ggml_opt_result_accuracy(result)ggml_opt_result_accuracy(result)
result |
External pointer to result object |
Named numeric vector with 'accuracy' and 'uncertainty'
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Free optimization result
ggml_opt_result_free(result)ggml_opt_result_free(result)
result |
External pointer to result object |
NULL invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Creates a new result object to accumulate training statistics.
ggml_opt_result_init()ggml_opt_result_init()
External pointer to result object
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Get loss from result
ggml_opt_result_loss(result)ggml_opt_result_loss(result)
result |
External pointer to result object |
Named numeric vector with 'loss' and 'uncertainty'
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Per-head counterparts of ggml_opt_result_loss() and
ggml_opt_result_accuracy(), which report the weighted total the
optimizer minimizes. These report a single head on its own, before
weighting, for the training history. ihead is 1-based.
ggml_opt_result_loss_i(result, ihead) ggml_opt_result_accuracy_i(result, ihead)ggml_opt_result_loss_i(result, ihead) ggml_opt_result_accuracy_i(result, ihead)
result |
External pointer to result object |
ihead |
Output head index, 1-based |
Heads whose loss is not cross-entropy have no accuracy; those return
NA.
Named numeric vector with the value and its 'uncertainty'
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Zero before the first epoch has run, so callers must treat a freshly reset result as holding no per-head data yet.
ggml_opt_result_n_loss(result)ggml_opt_result_n_loss(result)
result |
External pointer to result object |
Integer number of heads
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Get number of datapoints from result
ggml_opt_result_ndata(result)ggml_opt_result_ndata(result)
result |
External pointer to result object |
Number of datapoints processed
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Returns the predictions as an integer vector. The length equals the number of datapoints processed.
ggml_opt_result_pred(result)ggml_opt_result_pred(result)
result |
External pointer to result object |
Integer vector of predictions
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Reset optimization result
ggml_opt_result_reset(result)ggml_opt_result_reset(result)
result |
External pointer to result object |
NULL invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Updates the LR used for subsequent backward passes. Can be called between epochs to implement LR scheduling.
ggml_opt_set_lr(lr_ud, adamw_lr = NA, sgd_lr = NA)ggml_opt_set_lr(lr_ud, adamw_lr = NA, sgd_lr = NA)
lr_ud |
LR userdata pointer (from 'ggml_opt_init_for_fit()$lr_ud') |
adamw_lr |
New AdamW learning rate (NA to keep current) |
sgd_lr |
New SGD learning rate (NA to keep current) |
NULL invisibly
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_static_graphs(),
ggml_snapshot_free(),
ggml_trainer()
Check if using static graphs
ggml_opt_static_graphs(opt_ctx)ggml_opt_static_graphs(opt_ctx)
opt_ctx |
External pointer to optimizer context |
Logical indicating if graphs are statically allocated
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_snapshot_free(),
ggml_trainer()
Builds a node that applies one AdamW step when the graph is computed:
m <- beta1 * m + (1 - beta1) * grad v <- beta2 * v + (1 - beta2) * grad^2 a <- a * (1 - alpha * wd) - alpha * (m * beta1h) / (sqrt(v * beta2h) + eps)
a, m and v are updated in place; the returned
tensor is a view of a. Because one kernel reads and writes each
element, the update is free of the read-before-write ordering that a chain of
separate ops would need. With wd = 0 the decay factor is exactly 1.
ggml_opt_step_adamw(ctx, a, grad, m, v, adamw_params)ggml_opt_step_adamw(ctx, a, grad, m, v, adamw_params)
ctx |
GGML context |
a |
Weight tensor, F32, contiguous. It need not be marked with
|
grad |
Gradient, same shape as |
m |
First moment, same shape as |
v |
Second moment, same shape as |
adamw_params |
F32 tensor of 7 elements:
|
Precision: the kernel receives beta2 as a float and forms
1 - beta2 itself, so the weight of grad^2 in v is
1 - fl(beta2), not 1 - beta2. Its relative error is
9.5e-7 for 0.99, 1.29e-5 for 0.999 and 1.66e-4 for 0.9999 – the closer
beta2 is to 1, the larger. The effect on the weight is at most about
half of that. Expect agreement with a double-precision Adam to that level,
not to the last bit.
Bias correction is passed in, not derived: for step t,
beta1h = 1 / (1 - beta1^t) and beta2h = 1 / (1 - beta2^t).
Only F32 is supported (the CPU and Vulkan kernels are F32-only); any other type or a shape mismatch is an R error.
A view of a; expand it into a graph to run the step
ggml_set_param, optimizer_adam
Computes the outer product of two vectors: C = a * b^T For vectors a[m] and b[n], produces matrix C[m, n].
ggml_out_prod(ctx, a, b)ggml_out_prod(ctx, a, b)
ctx |
GGML context |
a |
First vector tensor |
b |
Second vector tensor |
Matrix tensor representing the outer product
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 3) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3)) ggml_set_f32(b, c(1, 2, 3, 4)) c <- ggml_out_prod(ctx, a, b) # Result: 3x4 matrix graph <- ggml_build_forward_expand(ctx, c) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 3) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3)) ggml_set_f32(b, c(1, 2, 3, 4)) c <- ggml_out_prod(ctx, a, b) # Result: 3x4 matrix graph <- ggml_build_forward_expand(ctx, c) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Pads tensor dimensions with zeros on the right side. Useful for aligning tensor sizes in attention operations.
ggml_pad(ctx, a, p0 = 0L, p1 = 0L, p2 = 0L, p3 = 0L)ggml_pad(ctx, a, p0 = 0L, p1 = 0L, p2 = 0L, p3 = 0L)
ctx |
GGML context |
a |
Input tensor to pad |
p0 |
Padding for dimension 0 (default 0) |
p1 |
Padding for dimension 1 (default 0) |
p2 |
Padding for dimension 2 (default 0) |
p3 |
Padding for dimension 3 (default 0) |
Padded tensor with shape [ne0+p0, ne1+p1, ne2+p2, ne3+p3]
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 5, 3) ggml_set_f32(a, 1:15) # Pad to 8x4 b <- ggml_pad(ctx, a, 3, 1) # Add 3 zeros to dim0, 1 to dim1 graph <- ggml_build_forward_expand(ctx, b) ggml_graph_compute(ctx, graph) # Result shape: [8, 4] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 5, 3) ggml_set_f32(a, 1:15) # Pad to 8x4 b <- ggml_pad(ctx, a, 3, 1) # Add 3 zeros to dim0, 1 to dim1 graph <- ggml_build_forward_expand(ctx, b) ggml_graph_compute(ctx, graph) # Result shape: [8, 4] ggml_free(ctx)
Like ggml_pad, but each dimension can also be padded before its
first element.
ggml_pad_ext(ctx, a, lp = c(0L, 0L, 0L, 0L), rp = c(0L, 0L, 0L, 0L))ggml_pad_ext(ctx, a, lp = c(0L, 0L, 0L, 0L), rp = c(0L, 0L, 0L, 0L))
ctx |
GGML context |
a |
Input tensor to pad |
lp |
Integer vector of length 4: zeros added before dimensions 0..3 |
rp |
Integer vector of length 4: zeros added after dimensions 0..3 |
Padded tensor with shape [ne0+lp0+rp0, ne1+lp1+rp1, ...]
Pads the first dimension of a tensor using reflection of its values.
ggml_pad_reflect_1d(ctx, a, p0, p1)ggml_pad_reflect_1d(ctx, a, p0, p1)
ctx |
GGML context |
a |
Input tensor |
p0 |
Left padding |
p1 |
Right padding |
Padded tensor
Permutes the tensor dimensions according to specified axes. CRITICAL for attention mechanisms in transformers.
ggml_permute(ctx, a, axis0, axis1, axis2, axis3)ggml_permute(ctx, a, axis0, axis1, axis2, axis3)
ctx |
GGML context |
a |
Input tensor |
axis0 |
New position for axis 0 |
axis1 |
New position for axis 1 |
axis2 |
New position for axis 2 |
axis3 |
New position for axis 3 |
Permuted tensor
ctx <- ggml_init(16 * 1024 * 1024) # Create 4D tensor: (2, 3, 4, 5) t <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 2, 3, 4, 5) # Swap axes 0 and 1: result shape (3, 2, 4, 5) t_perm <- ggml_permute(ctx, t, 1, 0, 2, 3) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) # Create 4D tensor: (2, 3, 4, 5) t <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 2, 3, 4, 5) # Swap axes 0 and 1: result shape (3, 2, 4, 5) t_perm <- ggml_permute(ctx, t, 1, 0, 2, 3) ggml_free(ctx)
Applies 1D pooling operation for downsampling.
ggml_pool_1d(ctx, a, op, k0, s0 = k0, p0 = 0L) GGML_OP_POOL_MAX GGML_OP_POOL_AVGggml_pool_1d(ctx, a, op, k0, s0 = k0, p0 = 0L) GGML_OP_POOL_MAX GGML_OP_POOL_AVG
ctx |
GGML context |
a |
Input tensor |
op |
Pool operation constant (see details) |
k0 |
Kernel size (window size) |
s0 |
Stride (default = k0 for non-overlapping windows) |
p0 |
Padding (default 0) |
An object of class integer of length 1.
An object of class integer of length 1.
Pool operation constants:
GGML_OP_POOL_MAX (0): Max pooling - takes maximum value in each window
GGML_OP_POOL_AVG (1): Average pooling - takes mean of values in each window
Pooled tensor with reduced dimensions
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 8) ggml_set_f32(a, c(1, 3, 2, 4, 5, 2, 8, 1)) # Max pooling with kernel 2, stride 2 max_pool <- ggml_pool_1d(ctx, a, GGML_OP_POOL_MAX, k0 = 2) # Result: [3, 4, 5, 8] (max of each pair) # Average pooling with kernel 2, stride 2 avg_pool <- ggml_pool_1d(ctx, a, GGML_OP_POOL_AVG, k0 = 2) # Result: [2, 3, 3.5, 4.5] (mean of each pair) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 8) ggml_set_f32(a, c(1, 3, 2, 4, 5, 2, 8, 1)) # Max pooling with kernel 2, stride 2 max_pool <- ggml_pool_1d(ctx, a, GGML_OP_POOL_MAX, k0 = 2) # Result: [3, 4, 5, 8] (max of each pair) # Average pooling with kernel 2, stride 2 avg_pool <- ggml_pool_1d(ctx, a, GGML_OP_POOL_AVG, k0 = 2) # Result: [2, 3, 3.5, 4.5] (mean of each pair) ggml_free(ctx)
Applies 2D pooling operation.
ggml_pool_2d(ctx, a, op, k0, k1, s0 = k0, s1 = k1, p0 = 0, p1 = 0)ggml_pool_2d(ctx, a, op, k0, k1, s0 = k0, s1 = k1, p0 = 0, p1 = 0)
ctx |
GGML context |
a |
Input tensor |
op |
Pool operation: GGML_OP_POOL_MAX (0) or GGML_OP_POOL_AVG (1) |
k0 |
Kernel size dimension 0 |
k1 |
Kernel size dimension 1 |
s0 |
Stride dimension 0 (default = k0) |
s1 |
Stride dimension 1 (default = k1) |
p0 |
Padding dimension 0 (default 0) |
p1 |
Padding dimension 1 (default 0) |
Pooled tensor
Removes the last layer from the model. The model must not be compiled.
ggml_pop_layer(model)ggml_pop_layer(model)
model |
A ggml_sequential_model object |
The model with the last layer removed.
model <- ggml_model_sequential() |> ggml_layer_dense(64, activation = "relu") |> ggml_layer_dense(10, activation = "softmax") model <- ggml_pop_layer(model) length(model$layers) # 1model <- ggml_model_sequential() |> ggml_layer_dense(64, activation = "relu") |> ggml_layer_dense(10, activation = "softmax") model <- ggml_pop_layer(model) length(model$layers) # 1
Runs data parallelism over the batch across replicas of a pipeline-parallel
model: the batch is split into contiguous shards, one per replica, and each
replica runs the whole model as a device-by-layer pipeline on its own
set of GPUs (via ggml_pp_forward). This is the pipeline
counterpart of ggml_tp_dp_forward — use it when the model is too
large for one card (split it by layers) and you also want to raise throughput
by replicating the pipeline across independent GPU sets.
ggml_pp_dp_forward( make_stages, x, replicas, out_ncol, mem_per_stage = 16L * 1024L * 1024L )ggml_pp_dp_forward( make_stages, x, replicas, out_ncol, mem_per_stage = 16L * 1024L * 1024L )
make_stages |
A function |
x |
Input activation matrix, |
replicas |
A list of integer vectors, each the GPU-index pipeline for one
replica, e.g. |
out_ncol |
Number of columns |
mem_per_stage |
Bytes of ggml context metadata per stage (see
|
On a 4-GPU box the natural layout is PP=2 x DP=2: replica A pipelines
its layers across GPUs c(0, 1), replica B across c(2, 3), and the
two replicas each handle half the batch with no cross-replica traffic.
The M x N result matrix, the batch-concatenation of each
replica's pipeline output.
ggml_pp_forward, ggml_tp_dp_forward.
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 4) { K <- 64L W1 <- matrix(rnorm(K * K), K); W2 <- matrix(rnorm(K * K), K) make_stages <- function(devices, m) { mk <- function(dev, Wt, relu) list( device = dev, in_shape = c(K, m), build = function(ctx, input) { w <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, K, K) z <- ggml_mul_mat(ctx, w, input) list(output = if (relu) ggml_relu(ctx, z) else z, set_weights = function() ggml_backend_tensor_set_data(w, as.numeric(Wt))) }) list(mk(devices[1], W1, TRUE), mk(devices[2], W2, FALSE)) } X <- matrix(rnorm(8 * K), nrow = 8) # batch of 8 Y <- ggml_pp_dp_forward(make_stages, X, replicas = list(c(0, 1), c(2, 3)), out_ncol = K) }if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 4) { K <- 64L W1 <- matrix(rnorm(K * K), K); W2 <- matrix(rnorm(K * K), K) make_stages <- function(devices, m) { mk <- function(dev, Wt, relu) list( device = dev, in_shape = c(K, m), build = function(ctx, input) { w <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, K, K) z <- ggml_mul_mat(ctx, w, input) list(output = if (relu) ggml_relu(ctx, z) else z, set_weights = function() ggml_backend_tensor_set_data(w, as.numeric(Wt))) }) list(mk(devices[1], W1, TRUE), mk(devices[2], W2, FALSE)) } X <- matrix(rnorm(8 * K), nrow = 8) # batch of 8 Y <- ggml_pp_dp_forward(make_stages, X, replicas = list(c(0, 1), c(2, 3)), out_ncol = K) }
Runs a forward pass split BY LAYERS across devices (pipeline parallelism), the
complement of ggml_vulkan_split_mul_mat's split-by-matrix tensor
parallelism. Each stage owns a contiguous block of the model's layers on
one GPU; the activation tensor is handed from one stage to the next exactly once
per pass (a single cross-device copy), versus TP's per-layer gather. This suits
models too large for one card's VRAM: at ~1 GB/s host staging the single
handoff costs only ~10-20 ms per pass.
ggml_pp_forward(stages, x, out_shape, mem_per_stage = 16L * 1024L * 1024L)ggml_pp_forward(stages, x, out_shape, mem_per_stage = 16L * 1024L * 1024L)
stages |
A list of stage descriptors (see Details), in pipeline order. |
x |
A numeric vector/array of input activations for the first stage,
laid out in ggml (column-major) order matching |
out_shape |
Integer vector: the ggml |
mem_per_stage |
Bytes of ggml context metadata to reserve per stage (default 16 MiB) — raise it for stages with very many ops. |
Each stage is a list with:
devicePhysical GPU index (0-based) the stage runs on.
buildA function function(ctx, input) that builds this
stage's sub-graph from the input tensor using the ggml_* ops
and returns either the output tensor, or a list
list(output = <tensor>, set_weights = function() ...). The
set_weights closure (if given) is called AFTER the stage's tensors
are allocated on the device, and is where weight tensors created inside
build get their values (via ggml_backend_tensor_set_data).
in_shapeInteger vector: the ggml ne shape of this
stage's input tensor (fastest dim first). For stage 1 this must match the
supplied x; for later stages it must match the previous stage's
output shape.
A numeric vector of the final stage's output activations (ggml order).
ggml_tp_dp_forward for the tensor-parallel counterpart.
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) { K <- 32L; M <- 4L W1 <- matrix(rnorm(K * K), K); W2 <- matrix(rnorm(K * K), K) stage <- function(dev, Wt) list( device = dev, in_shape = c(K, M), build = function(ctx, input) { w <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, K, K) ggml_mul_mat(ctx, w, input) # (weights set inside build via attr, see vignette) }) # See inst/examples/pp_pipeline.R for a complete runnable stage definition. }if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) { K <- 32L; M <- 4L W1 <- matrix(rnorm(K * K), K); W2 <- matrix(rnorm(K * K), K) stage <- function(dev, Wt) list( device = dev, in_shape = c(K, M), build = function(ctx, input) { w <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, K, K) ggml_mul_mat(ctx, w, input) # (weights set inside build via attr, see vignette) }) # See inst/examples/pp_pipeline.R for a complete runnable stage definition. }
Returns predicted class indices (1-based) by applying argmax
to the output of ggml_predict().
ggml_predict_classes(model, x, batch_size = 32L)ggml_predict_classes(model, x, batch_size = 32L)
model |
A trained ggml_sequential_model |
x |
Input data (matrix or array) |
batch_size |
Batch size for inference |
Integer vector of predicted class indices (1-based)
Runs forward pass on input data and returns prediction probabilities
(or raw output values for regression). Unlike ggml_evaluate(), this
does not require labels.
## S3 method for class 'ggml_functional_model' ggml_predict(model, x, batch_size = 32L, matmul_precision = NULL, ...) ggml_predict(model, ...) ## S3 method for class 'ggml_sequential_model' ggml_predict(model, x, batch_size = 32L, matmul_precision = NULL, ...)## S3 method for class 'ggml_functional_model' ggml_predict(model, x, batch_size = 32L, matmul_precision = NULL, ...) ggml_predict(model, ...) ## S3 method for class 'ggml_sequential_model' ggml_predict(model, x, batch_size = 32L, matmul_precision = NULL, ...)
model |
A trained ggml_sequential_model |
x |
Input data (matrix or array) |
batch_size |
Batch size for inference |
matmul_precision |
|
... |
Additional arguments (ignored). |
Matrix of predictions with shape [N, output_units]
On a Vulkan device with fp16 support the matrix multiplies keep their
tiles in f16 by default. Results then carry a relative error near 1e-3
that depends on the batch size (a single row takes an exact f32 path,
larger batches the tiled one), so the same input can predict slightly
differently in a batch of 1 and a batch of 64. "f32" marks every
multiply GGML_PREC_F32 and selects f32 tiles: the result then
matches the CPU to about 1e-6, at a cost in matmul throughput that depends
on the device and driver. The CPU backend is exact either way.
Which setting wins: the matmul_precision argument of the call, then
the model's predict_precision, then "default".
Helper to print memory usage information
ggml_print_mem_status(ctx)ggml_print_mem_status(ctx)
ctx |
GGML context |
List with total, used, free memory (invisible)
ctx <- ggml_init(1024 * 1024) ggml_print_mem_status(ctx) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) ggml_print_mem_status(ctx) ggml_free(ctx)
Debug function to print all objects (tensors) in the context
ggml_print_objects(ctx)ggml_print_objects(ctx)
ctx |
GGML context |
NULL (invisible)
ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_print_objects(ctx) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_print_objects(ctx) ggml_free(ctx)
Returns information about a quantization type including name, type size, block size, and whether it's quantized.
ggml_quant_block_info(type)ggml_quant_block_info(type)
type |
GGML type constant |
List with type_name, type_size, block_size, is_quantized
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Quantizes a chunk of floating-point data to a lower precision format.
ggml_quantize_chunk(type, src, nrows, n_per_row)ggml_quantize_chunk(type, src, nrows, n_per_row)
type |
Target GGML type (e.g., GGML_TYPE_Q4_0) |
src |
Source numeric vector (F32 data) |
nrows |
Number of rows |
n_per_row |
Number of elements per row |
Raw vector containing quantized data
# Quantize 256 floats to Q8_0 (block size 32) data <- rnorm(256) quantized <- ggml_quantize_chunk(GGML_TYPE_Q8_0, data, 1, 256) ggml_quantize_free() # Clean up# Quantize 256 floats to Q8_0 (block size 32) data <- rnorm(256) quantized <- ggml_quantize_chunk(GGML_TYPE_Q8_0, data, 1, 256) ggml_quantize_free() # Clean up
Frees any memory allocated by quantization. Call at end of program to avoid memory leaks.
ggml_quantize_free()ggml_quantize_free()
NULL invisibly
Initializes quantization tables for a given type. Called automatically by ggml_quantize_chunk, but can be called manually.
ggml_quantize_init(type)ggml_quantize_init(type)
type |
GGML type (e.g., GGML_TYPE_Q4_0) |
NULL invisibly
Some quantization types require an importance matrix for optimal quality.
ggml_quantize_requires_imatrix(type)ggml_quantize_requires_imatrix(type)
type |
GGML type |
TRUE if importance matrix is required
Creates a graph node for ReGLU operation. ReGLU uses ReLU as the activation function on the first half.
ggml_reglu(ctx, a)ggml_reglu(ctx, a)
ctx |
GGML context |
a |
Input tensor (first dimension must be even) |
Formula: output = ReLU(x) * gate
Tensor with half the first dimension of input
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 3) ggml_set_f32(a, rnorm(24)) r <- ggml_reglu(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # Shape: 4x3 ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 3) ggml_set_f32(a, rnorm(24)) r <- ggml_reglu(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # Shape: 4x3 ggml_free(ctx)
Creates a graph node for ReGLU with separate input and gate tensors.
ggml_reglu_split(ctx, a, b)ggml_reglu_split(ctx, a, b)
ctx |
GGML context |
a |
Input tensor (the values to be gated) |
b |
Gate tensor (same shape as a) |
Formula: output = ReLU(a) * b
Tensor with same shape as input tensors
Creates a graph node for ReLU (Rectified Linear Unit) activation: max(0, x)
ggml_relu(ctx, a)ggml_relu(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the ReLU operation
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) result <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) result <- ggml_relu(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)
Creates a graph node for in-place ReLU activation: max(0, x)
ggml_relu_inplace(ctx, a)ggml_relu_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with ReLU applied
Creates a graph node that repeats tensor 'a' to match shape of tensor 'b'.
ggml_repeat(ctx, a, b)ggml_repeat(ctx, a, b)
ctx |
GGML context |
a |
Tensor to repeat |
b |
Target tensor (defines output shape) |
Tensor with repeated values
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 1, 2) ggml_set_f32(a, c(1, 2)) b <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 2) result <- ggml_repeat(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [1, 1, 1, 2, 2, 2] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 1, 2) ggml_set_f32(a, c(1, 2)) b <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 2) result <- ggml_repeat(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [1, 1, 1, 2, 2, 2] ggml_free(ctx)
Backward pass for repeat operation - sums repetitions back to original shape. Used for gradient computation during training.
ggml_repeat_back(ctx, a, b)ggml_repeat_back(ctx, a, b)
ctx |
GGML context |
a |
Input tensor (gradients from repeated tensor) |
b |
Target shape tensor (original tensor before repeat) |
Tensor with summed gradients matching shape of b
Clears all tensor allocations in the context memory pool. The context can be reused without recreating it. This is more efficient than free + init for temporary operations.
ggml_reset(ctx)ggml_reset(ctx)
ctx |
GGML context pointer |
NULL (invisible)
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) ggml_reset(ctx) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 200) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) ggml_reset(ctx) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 200) ggml_free(ctx)
Reshapes tensor to 1D with ne0 elements
ggml_reshape_1d(ctx, a, ne0)ggml_reshape_1d(ctx, a, ne0)
ctx |
GGML context |
a |
Input tensor |
ne0 |
Size of dimension 0 |
Reshaped tensor
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 4) ggml_set_f32(a, 1:12) result <- ggml_reshape_1d(ctx, a, 12) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 4) ggml_set_f32(a, 1:12) result <- ggml_reshape_1d(ctx, a, 12) ggml_free(ctx)
Reshapes tensor to 2D with shape (ne0, ne1)
ggml_reshape_2d(ctx, a, ne0, ne1)ggml_reshape_2d(ctx, a, ne0, ne1)
ctx |
GGML context |
a |
Input tensor |
ne0 |
Size of dimension 0 |
ne1 |
Size of dimension 1 |
Reshaped tensor
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 12) ggml_set_f32(a, 1:12) result <- ggml_reshape_2d(ctx, a, 3, 4) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 12) ggml_set_f32(a, 1:12) result <- ggml_reshape_2d(ctx, a, 3, 4) ggml_free(ctx)
Reshapes tensor to 3D with shape (ne0, ne1, ne2)
ggml_reshape_3d(ctx, a, ne0, ne1, ne2)ggml_reshape_3d(ctx, a, ne0, ne1, ne2)
ctx |
GGML context |
a |
Input tensor |
ne0 |
Size of dimension 0 |
ne1 |
Size of dimension 1 |
ne2 |
Size of dimension 2 |
Reshaped tensor
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 24) ggml_set_f32(a, 1:24) result <- ggml_reshape_3d(ctx, a, 2, 3, 4) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 24) ggml_set_f32(a, 1:24) result <- ggml_reshape_3d(ctx, a, 2, 3, 4) ggml_free(ctx)
Reshapes tensor to 4D with shape (ne0, ne1, ne2, ne3)
ggml_reshape_4d(ctx, a, ne0, ne1, ne2, ne3)ggml_reshape_4d(ctx, a, ne0, ne1, ne2, ne3)
ctx |
GGML context |
a |
Input tensor |
ne0 |
Size of dimension 0 |
ne1 |
Size of dimension 1 |
ne2 |
Size of dimension 2 |
ne3 |
Size of dimension 3 |
Reshaped tensor
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 120) ggml_set_f32(a, 1:120) result <- ggml_reshape_4d(ctx, a, 2, 3, 4, 5) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 120) ggml_set_f32(a, 1:120) result <- ggml_reshape_4d(ctx, a, 2, 3, 4, 5) ggml_free(ctx)
A ggml_result is the contract object returned by the dispatch layer
and consumed by the injection layer. The embedding is stored cell-by-component
(cells in rows), ready to drop into reducedDim() / a Seurat reduction.
ggml_result(embedding, metadata = list(), timings = numeric(0))ggml_result(embedding, metadata = list(), timings = numeric(0))
embedding |
A numeric matrix, cells in rows, components in columns. |
metadata |
Named list (e.g. |
timings |
Named numeric vector of elapsed seconds per stage. |
An object of class ggml_result.
Creates a graph node for RMS (Root Mean Square) normalization. Normalizes by x / sqrt(mean(x^2) + eps). CRITICAL for LLaMA models.
ggml_rms_norm(ctx, a, eps = 1e-05)ggml_rms_norm(ctx, a, eps = 1e-05)
ctx |
GGML context |
a |
Input tensor |
eps |
Epsilon value for numerical stability (default: 1e-5) |
Tensor representing the RMS normalization operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_rms_norm(ctx, a, eps = 1e-5) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # sqrt(mean(output^2)) should be ~1 ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_rms_norm(ctx, a, eps = 1e-5) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # sqrt(mean(output^2)) should be ~1 ggml_free(ctx)
Creates a graph node for backward pass of RMS normalization. Used in training for computing gradients.
ggml_rms_norm_back(ctx, a, b, eps = 1e-05)ggml_rms_norm_back(ctx, a, b, eps = 1e-05)
ctx |
GGML context |
a |
Gradient tensor (dy from the forward output) |
b |
Input tensor (x from the forward pass) |
eps |
Epsilon for numerical stability (default 1e-5) |
Tensor representing the gradient with respect to input
Creates a graph node for in-place RMS normalization. Returns a view of the input tensor. CRITICAL for LLaMA models when memory efficiency is important.
ggml_rms_norm_inplace(ctx, a, eps = 1e-05)ggml_rms_norm_inplace(ctx, a, eps = 1e-05)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
eps |
Epsilon value for numerical stability (default: 1e-5) |
View of input tensor with RMS normalization applied
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_rms_norm_inplace(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_rms_norm_inplace(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Circularly shifts tensor elements along dimensions 0..3.
ggml_roll(ctx, a, shift0 = 0L, shift1 = 0L, shift2 = 0L, shift3 = 0L)ggml_roll(ctx, a, shift0 = 0L, shift1 = 0L, shift2 = 0L, shift3 = 0L)
ctx |
GGML context |
a |
Input tensor |
shift0, shift1, shift2, shift3
|
Shift amount along each dimension |
Rolled tensor
Creates a graph node for RoPE (Rotary Position Embedding). RoPE is the dominant position encoding method in modern LLMs like LLaMA, Mistral, and many others.
ggml_rope(ctx, a, b, n_dims, mode = 0L)ggml_rope(ctx, a, b, n_dims, mode = 0L)
ctx |
GGML context |
a |
Input tensor of shape [head_dim, n_head, seq_len, batch] |
b |
Position tensor (int32) of shape [seq_len] containing position indices |
n_dims |
Number of dimensions to apply rotation to (usually head_dim) |
mode |
RoPE mode: GGML_ROPE_TYPE_NORM (0), GGML_ROPE_TYPE_NEOX (2), etc. |
RoPE encodes position information by rotating pairs of dimensions in the embedding space. The rotation angle depends on position and dimension index.
Key benefits of RoPE: - Relative position information emerges naturally from rotation - Better extrapolation to longer sequences than absolute embeddings - No additional parameters needed
Tensor with same shape as input, with rotary embeddings applied
ctx <- ggml_init(16 * 1024 * 1024) # Query tensor: head_dim=8, n_head=4, seq_len=16, batch=1 q <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 8, 4, 16, 1) ggml_set_f32(q, rnorm(8 * 4 * 16)) # Position indices pos <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 16) ggml_set_i32(pos, 0:15) # Apply RoPE q_rope <- ggml_rope(ctx, q, pos, 8, GGML_ROPE_TYPE_NORM) graph <- ggml_build_forward_expand(ctx, q_rope) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) # Query tensor: head_dim=8, n_head=4, seq_len=16, batch=1 q <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 8, 4, 16, 1) ggml_set_f32(q, rnorm(8 * 4 * 16)) # Position indices pos <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 16) ggml_set_i32(pos, 0:15) # Apply RoPE q_rope <- ggml_rope(ctx, q, pos, 8, GGML_ROPE_TYPE_NORM) graph <- ggml_build_forward_expand(ctx, q_rope) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Creates a graph node for extended RoPE with frequency scaling parameters. Supports context extension techniques like YaRN, Linear Scaling, etc.
ggml_rope_ext( ctx, a, b, c = NULL, n_dims, mode = 0L, n_ctx_orig = 0L, freq_base = 10000, freq_scale = 1, ext_factor = 0, attn_factor = 1, beta_fast = 32, beta_slow = 1 )ggml_rope_ext( ctx, a, b, c = NULL, n_dims, mode = 0L, n_ctx_orig = 0L, freq_base = 10000, freq_scale = 1, ext_factor = 0, attn_factor = 1, beta_fast = 32, beta_slow = 1 )
ctx |
GGML context |
a |
Input tensor |
b |
Position tensor (int32) |
c |
Optional frequency factors tensor (NULL for default) |
n_dims |
Number of dimensions to apply rotation to |
mode |
RoPE mode |
n_ctx_orig |
Original context length the model was trained on |
freq_base |
Base frequency for RoPE (default 10000 for most models) |
freq_scale |
Frequency scale factor (1.0 = no scaling) |
ext_factor |
YaRN extension factor (0.0 to disable) |
attn_factor |
Attention scale factor (typically 1.0) |
beta_fast |
YaRN parameter for fast dimensions |
beta_slow |
YaRN parameter for slow dimensions |
This extended version supports various context extension techniques:
- **Linear Scaling**: Set freq_scale = original_ctx / new_ctx - **YaRN**: Set ext_factor > 0 with appropriate beta_fast/beta_slow - **NTK-aware**: Adjust freq_base for NTK-style scaling
Common freq_base values: - LLaMA 1/2: 10000 - LLaMA 3: 500000 - Mistral: 10000 - Phi-3: 10000
Tensor with extended RoPE applied
ctx <- ggml_init(16 * 1024 * 1024) q <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 64, 8, 32, 1) ggml_set_f32(q, rnorm(64 * 8 * 32)) pos <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 32) ggml_set_i32(pos, 0:31) # Standard RoPE with default freq_base q_rope <- ggml_rope_ext(ctx, q, pos, NULL, n_dims = 64, mode = 0L, n_ctx_orig = 4096, freq_base = 10000, freq_scale = 1.0, ext_factor = 0.0, attn_factor = 1.0, beta_fast = 32, beta_slow = 1) graph <- ggml_build_forward_expand(ctx, q_rope) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) q <- ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 64, 8, 32, 1) ggml_set_f32(q, rnorm(64 * 8 * 32)) pos <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 32) ggml_set_i32(pos, 0:31) # Standard RoPE with default freq_base q_rope <- ggml_rope_ext(ctx, q, pos, NULL, n_dims = 64, mode = 0L, n_ctx_orig = 4096, freq_base = 10000, freq_scale = 1.0, ext_factor = 0.0, attn_factor = 1.0, beta_fast = 32, beta_slow = 1) graph <- ggml_build_forward_expand(ctx, q_rope) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Backward pass for extended RoPE (Rotary Position Embedding). Used during training to compute gradients through RoPE.
ggml_rope_ext_back( ctx, a, b, c = NULL, n_dims, mode = 0L, n_ctx_orig = 0L, freq_base = 10000, freq_scale = 1, ext_factor = 0, attn_factor = 1, beta_fast = 32, beta_slow = 1 )ggml_rope_ext_back( ctx, a, b, c = NULL, n_dims, mode = 0L, n_ctx_orig = 0L, freq_base = 10000, freq_scale = 1, ext_factor = 0, attn_factor = 1, beta_fast = 32, beta_slow = 1 )
ctx |
GGML context |
a |
Gradient tensor from upstream (gradients of ggml_rope_ext result) |
b |
Position tensor (same as forward pass) |
c |
Optional frequency factors tensor (NULL for default) |
n_dims |
Number of dimensions for rotation |
mode |
RoPE mode |
n_ctx_orig |
Original context length |
freq_base |
Base frequency |
freq_scale |
Frequency scale factor |
ext_factor |
Extension factor (YaRN) |
attn_factor |
Attention factor |
beta_fast |
YaRN fast beta |
beta_slow |
YaRN slow beta |
Gradient tensor for the input
Creates a graph node for extended RoPE, modifying input tensor in place. Returns a view of the input tensor.
ggml_rope_ext_inplace( ctx, a, b, c = NULL, n_dims, mode = 0L, n_ctx_orig = 0L, freq_base = 10000, freq_scale = 1, ext_factor = 0, attn_factor = 1, beta_fast = 32, beta_slow = 1 )ggml_rope_ext_inplace( ctx, a, b, c = NULL, n_dims, mode = 0L, n_ctx_orig = 0L, freq_base = 10000, freq_scale = 1, ext_factor = 0, attn_factor = 1, beta_fast = 32, beta_slow = 1 )
ctx |
GGML context |
a |
Input tensor |
b |
Position tensor (int32) |
c |
Optional frequency factors tensor (NULL for default) |
n_dims |
Number of dimensions to apply rotation to |
mode |
RoPE mode |
n_ctx_orig |
Original context length the model was trained on |
freq_base |
Base frequency for RoPE (default 10000 for most models) |
freq_scale |
Frequency scale factor (1.0 = no scaling) |
ext_factor |
YaRN extension factor (0.0 to disable) |
attn_factor |
Attention scale factor (typically 1.0) |
beta_fast |
YaRN parameter for fast dimensions |
beta_slow |
YaRN parameter for slow dimensions |
View of input tensor with RoPE applied in place
Other rope:
ggml_rope_multi(),
ggml_rope_multi_inplace()
In-place version of ggml_rope. Returns a view of the input tensor.
ggml_rope_inplace(ctx, a, b, n_dims, mode = 0L)ggml_rope_inplace(ctx, a, b, n_dims, mode = 0L)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
b |
Position tensor (int32) |
n_dims |
Number of dimensions to apply rotation to |
mode |
RoPE mode |
View of input tensor with RoPE applied
Creates a graph node for multi-dimensional RoPE (MRoPE) used in vision transformers. Supports separate rotation for different positional dimensions (e.g., height, width, time).
ggml_rope_multi( ctx, a, b, c = NULL, n_dims, sections = c(0L, 0L, 0L, 0L), mode = 0L, n_ctx_orig = 0L, freq_base = 10000, freq_scale = 1, ext_factor = 0, attn_factor = 1, beta_fast = 32, beta_slow = 1 )ggml_rope_multi( ctx, a, b, c = NULL, n_dims, sections = c(0L, 0L, 0L, 0L), mode = 0L, n_ctx_orig = 0L, freq_base = 10000, freq_scale = 1, ext_factor = 0, attn_factor = 1, beta_fast = 32, beta_slow = 1 )
ctx |
GGML context |
a |
Input tensor |
b |
Position tensor (int32) |
c |
Optional frequency factors tensor (NULL for default) |
n_dims |
Number of dimensions to apply rotation to |
sections |
Integer vector of length 4 specifying dimension sections for MRoPE |
mode |
RoPE mode |
n_ctx_orig |
Original context length the model was trained on |
freq_base |
Base frequency for RoPE (default 10000 for most models) |
freq_scale |
Frequency scale factor (1.0 = no scaling) |
ext_factor |
YaRN extension factor (0.0 to disable) |
attn_factor |
Attention scale factor (typically 1.0) |
beta_fast |
YaRN parameter for fast dimensions |
beta_slow |
YaRN parameter for slow dimensions |
Tensor with multi-dimensional RoPE applied
Other rope:
ggml_rope_ext_inplace(),
ggml_rope_multi_inplace()
Gradient node for ggml_rope_multi. Takes the same geometry
arguments as the forward op, which is what makes the rotation invertible.
ggml_rope_multi_back( ctx, a, b, c = NULL, n_dims, sections = c(0L, 0L, 0L, 0L), mode = 0L, n_ctx_orig = 0L, freq_base = 10000, freq_scale = 1, ext_factor = 0, attn_factor = 1, beta_fast = 32, beta_slow = 1 )ggml_rope_multi_back( ctx, a, b, c = NULL, n_dims, sections = c(0L, 0L, 0L, 0L), mode = 0L, n_ctx_orig = 0L, freq_base = 10000, freq_scale = 1, ext_factor = 0, attn_factor = 1, beta_fast = 32, beta_slow = 1 )
ctx |
GGML context |
a |
Input tensor |
b |
Position tensor (int32) |
c |
Optional frequency factors tensor (NULL for default) |
n_dims |
Number of dimensions to apply rotation to |
sections |
Integer vector of length 4 specifying dimension sections for MRoPE |
mode |
RoPE mode |
n_ctx_orig |
Original context length the model was trained on |
freq_base |
Base frequency for RoPE (default 10000 for most models) |
freq_scale |
Frequency scale factor (1.0 = no scaling) |
ext_factor |
YaRN extension factor (0.0 to disable) |
attn_factor |
Attention scale factor (typically 1.0) |
beta_fast |
YaRN parameter for fast dimensions |
beta_slow |
YaRN parameter for slow dimensions |
Gradient w.r.t. the rotated input
Creates a graph node for multi-dimensional RoPE, modifying input in place.
ggml_rope_multi_inplace( ctx, a, b, c = NULL, n_dims, sections = c(0L, 0L, 0L, 0L), mode = 0L, n_ctx_orig = 0L, freq_base = 10000, freq_scale = 1, ext_factor = 0, attn_factor = 1, beta_fast = 32, beta_slow = 1 )ggml_rope_multi_inplace( ctx, a, b, c = NULL, n_dims, sections = c(0L, 0L, 0L, 0L), mode = 0L, n_ctx_orig = 0L, freq_base = 10000, freq_scale = 1, ext_factor = 0, attn_factor = 1, beta_fast = 32, beta_slow = 1 )
ctx |
GGML context |
a |
Input tensor |
b |
Position tensor (int32) |
c |
Optional frequency factors tensor (NULL for default) |
n_dims |
Number of dimensions to apply rotation to |
sections |
Integer vector of length 4 specifying dimension sections for MRoPE |
mode |
RoPE mode |
n_ctx_orig |
Original context length the model was trained on |
freq_base |
Base frequency for RoPE (default 10000 for most models) |
freq_scale |
Frequency scale factor (1.0 = no scaling) |
ext_factor |
YaRN extension factor (0.0 to disable) |
attn_factor |
Attention scale factor (typically 1.0) |
beta_fast |
YaRN parameter for fast dimensions |
beta_slow |
YaRN parameter for slow dimensions |
View of input tensor with MRoPE applied in place
Other rope:
ggml_rope_ext_inplace(),
ggml_rope_multi()
Creates a graph node for element-wise rounding: round(x)
ggml_round(ctx, a)ggml_round(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the round operation
Creates a graph node for in-place element-wise rounding.
ggml_round_inplace(ctx, a)ggml_round_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with rounded values
Validates a ggml_task against ggml_ops_registry,
resolves the compute device, invokes the registered engine and returns a
ggml_result. This is the dispatch layer: the only place that
decides GPU vs CPU. device = "auto" (the task default) uses Vulkan when
a GPU is available and CPU otherwise; device = "vulkan" on a machine
with no GPU degrades to CPU with a message rather than failing.
ggml_run(task, backend = NULL, ...)ggml_run(task, backend = NULL, ...)
task |
A |
backend |
Optional override of |
... |
Additional parameters merged over |
A ggml_result.
ggml_rwkv_wkv6, ggml_rwkv_wkv7 and
ggml_gated_linear_attn all return one
[S * H, n_tokens + S * n_seqs] tensor holding the per-token outputs
followed by the final state. These helpers view either half, so the column
arithmetic lives in one place rather than at every call site.
ggml_rwkv_output(ctx, result, k) ggml_rwkv_state(ctx, result, k, state)ggml_rwkv_output(ctx, result, k) ggml_rwkv_state(ctx, result, k, state)
ctx |
GGML context |
result |
The tensor returned by one of the RWKV / GLA ops |
k |
The |
state |
The |
Both are graph views sharing memory with result and add no
computation. k must be the same tensor the op was built with, since
S, H and n_tokens are read from it.
For ggml_rwkv_output(), a tensor [S * H, n_tokens]
For ggml_rwkv_state(), a tensor [S * H, S * n_seqs]
Other state-space:
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
The weighted key-value recurrence of RWKV-6: a linear-attention-style scan
whose decay is per-token (td) on top of a per-channel bonus
(tf). Runs in O(n_tokens) rather than the O(n_tokens^2) of ordinary
attention, which is what makes RWKV a recurrent model at inference time.
ggml_rwkv_wkv6(ctx, k, v, r, tf, td, state)ggml_rwkv_wkv6(ctx, k, v, r, tf, td, state)
ctx |
GGML context |
k |
Keys, |
v |
Values, |
r |
Receptance, |
tf |
Per-channel decay bonus, |
td |
Per-token decay, |
state |
Initial state, |
A tensor [S * H, n_tokens + S * n_seqs]
With head size S, H heads, n_tokens tokens and
n_seqs sequences: k, v, r and td are
[S, H, n_tokens], tf is [S, H], and state holds
S * S * H * n_seqs elements (typically [S * S * H, n_seqs]).
All must be contiguous.
n_tokens counts the tokens of all sequences together, so it
must be a multiple of n_seqs: the kernel derives the per-sequence
length as n_tokens / n_seqs and restarts the state at each boundary.
The result is [S * H, n_tokens + S * n_seqs]: the per-token outputs
occupy the first n_tokens columns and the final state the remaining
S * n_seqs. ggml_rwkv_output and
ggml_rwkv_state view either half.
GGML_OP_RWKV_WKV6 has no backward pass in ggml.
ggml_rwkv_wkv7, ggml_rwkv_output
Other state-space:
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Gradients of ggml_rwkv_wkv6 with respect to all six of its
inputs, given grad, the gradient of the loss with respect to the
recurrence's packed forward result.
ggml_rwkv_wkv6_back(ctx, k, v, r, tf, td, state, grad)ggml_rwkv_wkv6_back(ctx, k, v, r, tf, td, state, grad)
ctx |
GGML context |
k, v, r, tf, td, state
|
The tensors passed to the forward recurrence |
grad |
Gradient w.r.t. the packed forward result |
Training reaches this automatically through the backward graph; it is exported so the gradient can be checked against a numeric one.
A flat tensor holding the six gradients, concatenated
All six gradients come back in one flat tensor, in the order the forward op
takes its inputs: [d_k | d_v | d_r | d_tf | d_td | d_state].
Other state-space:
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
The RWKV-7 ("Goose") recurrence. Where RWKV-6 decays the state by a vector,
RWKV-7 applies a rank-1 update built from a and b alongside the
decay w, giving the state a richer transition.
ggml_rwkv_wkv7(ctx, r, w, k, v, a, b, state)ggml_rwkv_wkv7(ctx, r, w, k, v, a, b, state)
ctx |
GGML context |
r |
Receptance, |
w |
Decay, |
k |
Keys, |
v |
Values, |
a |
Removal (state update term), |
b |
Addition (state update term), |
state |
Initial state, |
A tensor [S * H, n_tokens + S * n_seqs]
r, w, k, v, a and b are
[S, H, n_tokens]; state holds S * S * H * n_seqs
elements. All must be contiguous. The result is
[S * H, n_tokens + S * n_seqs], outputs then final state – see
ggml_rwkv_output.
n_tokens counts the tokens of all sequences together, so it
must be a multiple of n_seqs.
GGML_OP_RWKV_WKV7 has no backward pass in ggml.
ggml_rwkv_wkv6, ggml_rwkv_output
Other state-space:
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Gradients of ggml_rwkv_wkv7 with respect to all seven of its
inputs, packed in forward source order:
[d_r | d_w | d_k | d_v | d_a | d_b | d_state].
ggml_rwkv_wkv7_back(ctx, r, w, k, v, a, b, state, grad)ggml_rwkv_wkv7_back(ctx, r, w, k, v, a, b, state, grad)
ctx |
GGML context |
r, w, k, v, a, b, state
|
The tensors passed to the forward recurrence |
grad |
Gradient w.r.t. the packed forward result |
A flat tensor holding the seven gradients, concatenated
Other state-space:
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Saves both the architecture and trained weights of a model to an RDS file.
Unlike ggml_save_weights(), which requires the model to be manually
reconstructed before loading, ggml_save_model() saves everything
needed to restore the model with a single call to ggml_load_model().
ggml_save_model(model, path)ggml_save_model(model, path)
model |
A trained |
path |
File path (typically |
The model (invisibly).
ggml_sequential_model — input shape, layer configs, trained
weights, and compilation settings are all saved.
ggml_functional_model — input/output node graphs (pure R
lists, no ggml pointers) and trained node_weights are saved.
model <- ggml_model_sequential() |> ggml_layer_dense(16L, activation = "relu", input_shape = 4L) |> ggml_layer_dense(2L, activation = "softmax") model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy") x <- matrix(runif(64 * 4), 64, 4) y <- matrix(c(rep(c(1,0), 32), rep(c(0,1), 32)), 64, 2) model <- ggml_fit(model, x, y, epochs = 1L, batch_size = 32L, verbose = 0L) tmp <- tempfile(fileext = ".rds") ggml_save_model(model, tmp) model2 <- ggml_load_model(tmp)model <- ggml_model_sequential() |> ggml_layer_dense(16L, activation = "relu", input_shape = 4L) |> ggml_layer_dense(2L, activation = "softmax") model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy") x <- matrix(runif(64 * 4), 64, 4) y <- matrix(c(rep(c(1,0), 32), rep(c(0,1), 32)), 64, 2) model <- ggml_fit(model, x, y, epochs = 1L, batch_size = 32L, verbose = 0L) tmp <- tempfile(fileext = ".rds") ggml_save_model(model, tmp) model2 <- ggml_load_model(tmp)
Saves the trained weights of a sequential model to an RDS file. The file includes both weights and architecture metadata for validation when loading.
ggml_save_weights(model, path)ggml_save_weights(model, path)
model |
A trained ggml_sequential_model |
path |
File path to save weights (typically with .rds extension) |
The model (invisibly).
Creates a graph node for scaling tensor by a scalar: x * s
ggml_scale(ctx, a, s)ggml_scale(ctx, a, s)
ctx |
GGML context |
a |
Input tensor |
s |
Scalar value to multiply by |
Tensor representing the scaled values
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_scale(ctx, a, 2.0) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [2, 4, 6, 8] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_scale(ctx, a, 2.0) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [2, 4, 6, 8] ggml_free(ctx)
Creates a graph node computing s * a + b for scalar s and
b. Unlike ggml_add1() with ggml_new_f32(), this adds a
constant without materialising a data-carrying tensor, so it is usable while
building a graph in a no_alloc context.
ggml_scale_bias(ctx, a, s, b)ggml_scale_bias(ctx, a, s, b)
ctx |
GGML context |
a |
Input tensor |
s |
Scale factor (numeric scalar) |
b |
Bias added after scaling (numeric scalar) |
Tensor representing s * a + b
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_scale_bias(ctx, a, 2.0, 1.0) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [3, 5, 7, 9] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_scale_bias(ctx, a, 2.0, 1.0) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [3, 5, 7, 9] ggml_free(ctx)
Creates a graph node for in-place scaling: a * s
ggml_scale_inplace(ctx, a, s)ggml_scale_inplace(ctx, a, s)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
s |
Scalar value to multiply by |
View of tensor a with scaled values
Copies 'data', then writes each element of 'updates' into the copy at the position given by the matching element of 'indices' along 'axis'. Every other coordinate is the element's own, so 'updates' and 'indices' share a shape and only their position along 'axis' is redirected.
ggml_scatter_elements(ctx, data, updates, indices, reduction = 0L, axis = 0L)ggml_scatter_elements(ctx, data, updates, indices, reduction = 0L, axis = 0L)
ctx |
A ggml context |
data |
Base tensor (F32); the result starts as a copy of it |
updates |
Values to write (F32), same shape as 'indices' |
indices |
Destination positions along 'axis' (I32) |
reduction |
0 to overwrite, 1 to add |
axis |
Axis to scatter along, in ggml dim order (0 = ne[0]) |
A new tensor with the shape of 'data'
ctx <- ggml_init(16 * 1024 * 1024) base <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 4) # 4 rows of 3 ggml_set_f32(base, rep(0, 12)) upd <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 2) ggml_set_f32(upd, c(1, 2, 3, 4, 5, 6)) idx <- ggml_new_tensor_2d(ctx, GGML_TYPE_I32, 3, 2) ggml_set_i32(idx, c(3L, 3L, 3L, 0L, 0L, 0L)) # rows 3 and 0 out <- ggml_scatter_elements(ctx, base, upd, idx, 0L, 1L) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) base <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 4) # 4 rows of 3 ggml_set_f32(base, rep(0, 12)) upd <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 2) ggml_set_f32(upd, c(1, 2, 3, 4, 5, 6)) idx <- ggml_new_tensor_2d(ctx, GGML_TYPE_I32, 3, 2) ggml_set_i32(idx, c(3L, 3L, 3L, 0L, 0L, 0L)) # rows 3 and 0 out <- ggml_scatter_elements(ctx, base, upd, idx, 0L, 1L) ggml_free(ctx)
Anneals LR from initial value to 'eta_min' following a cosine curve.
ggml_schedule_cosine_decay(eta_min = 0, T_max = NULL)ggml_schedule_cosine_decay(eta_min = 0, T_max = NULL)
eta_min |
Minimum LR at end of schedule |
T_max |
Total number of epochs (defaults to nepoch from fit state) |
List with on_epoch_begin function
Other callbacks:
ggml_callback_early_stopping(),
ggml_schedule_reduce_on_plateau(),
ggml_schedule_step_decay()
Reduces LR when a metric stops improving.
ggml_schedule_reduce_on_plateau( monitor = "val_loss", factor = 0.5, patience = 5, min_lr = 1e-07, min_delta = 1e-04, mode = "auto" )ggml_schedule_reduce_on_plateau( monitor = "val_loss", factor = 0.5, patience = 5, min_lr = 1e-07, min_delta = 1e-04, mode = "auto" )
monitor |
Metric to monitor: "val_loss", "train_loss", etc. |
factor |
Factor to reduce LR by |
patience |
Epochs with no improvement before reducing |
min_lr |
Minimum LR |
min_delta |
Minimum change to qualify as improvement |
mode |
"min" or "max". "auto" infers from monitor name. |
List with on_epoch_end function
Other callbacks:
ggml_callback_early_stopping(),
ggml_schedule_cosine_decay(),
ggml_schedule_step_decay()
Reduces LR by a factor every 'step_size' epochs.
ggml_schedule_step_decay(step_size = 10, gamma = 0.1)ggml_schedule_step_decay(step_size = 10, gamma = 0.1)
step_size |
Reduce LR every this many epochs |
gamma |
Multiplicative factor of LR reduction |
List with on_epoch_begin function
Other callbacks:
ggml_callback_early_stopping(),
ggml_schedule_cosine_decay(),
ggml_schedule_reduce_on_plateau()
Copies tensor b into tensor a at a specified offset. This allows writing to a portion of a tensor.
ggml_set(ctx, a, b, nb1, nb2, nb3, offset)ggml_set(ctx, a, b, nb1, nb2, nb3, offset)
ctx |
GGML context |
a |
Destination tensor |
b |
Source tensor (data to copy) |
nb1 |
Stride for dimension 1 (in bytes) |
nb2 |
Stride for dimension 2 (in bytes) |
nb3 |
Stride for dimension 3 (in bytes) |
offset |
Byte offset in destination tensor |
Tensor representing the set operation
Simplified 1D version of ggml_set. Copies tensor b into tensor a starting at offset.
ggml_set_1d(ctx, a, b, offset)ggml_set_1d(ctx, a, b, offset)
ctx |
GGML context |
a |
Destination tensor |
b |
Source tensor |
offset |
Byte offset in destination tensor |
Tensor representing the set operation
Simplified 2D version of ggml_set.
ggml_set_2d(ctx, a, b, nb1, offset)ggml_set_2d(ctx, a, b, nb1, offset)
ctx |
GGML context |
a |
Destination tensor |
b |
Source tensor |
nb1 |
Stride for dimension 1 (in bytes) |
offset |
Byte offset in destination tensor |
Tensor representing the set operation
Restores GGML to default abort behavior (prints to stderr and aborts).
ggml_set_abort_callback_default()ggml_set_abort_callback_default()
NULL invisibly
Other logging:
ggml_abort_is_r_enabled(),
ggml_log_is_r_enabled(),
ggml_log_set_default(),
ggml_log_set_r(),
ggml_set_abort_callback_r()
Converts GGML abort calls into R errors (via Rf_error). This allows R to catch GGML failures with tryCatch.
ggml_set_abort_callback_r()ggml_set_abort_callback_r()
NULL invisibly
Other logging:
ggml_abort_is_r_enabled(),
ggml_log_is_r_enabled(),
ggml_log_set_default(),
ggml_log_set_r(),
ggml_set_abort_callback_default()
ggml_set_abort_callback_r() # Now GGML aborts will become R errors result <- tryCatch({ # ... ggml operations that might fail ... }, error = function(e) { message("GGML error caught: ", e$message) })ggml_set_abort_callback_r() # Now GGML aborts will become R errors result <- tryCatch({ # ... ggml operations that might fail ... }, error = function(e) { message("GGML error caught: ", e$message) })
Set F32 data
Set F32 Data
ggml_set_f32(tensor, data) ggml_set_f32(tensor, data)ggml_set_f32(tensor, data) ggml_set_f32(tensor, data)
tensor |
Tensor |
data |
Numeric vector |
NULL (invisible)
NULL (invisible)
ctx <- ggml_init(1024 * 1024) tensor <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(tensor, c(1, 2, 3, 4, 5)) ggml_get_f32(tensor) ggml_free(ctx) ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(t, c(1, 2, 3, 4, 5)) ggml_get_f32(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) tensor <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(tensor, c(1, 2, 3, 4, 5)) ggml_get_f32(tensor) ggml_free(ctx) ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(t, c(1, 2, 3, 4, 5)) ggml_get_f32(t) ggml_free(ctx)
Sets a single f32 value in the tensor at position [i0, i1, i2, i3]. This is a direct data write, not a graph operation.
ggml_set_f32_nd(tensor, i0, i1 = 0, i2 = 0, i3 = 0, value)ggml_set_f32_nd(tensor, i0, i1 = 0, i2 = 0, i3 = 0, value)
tensor |
Tensor pointer |
i0, i1, i2, i3
|
Indices (0-based) |
value |
Float value to set |
NULL (invisible)
Sets integer data in an I32 tensor. Used for indices (ggml_get_rows) and position tensors (ggml_rope).
ggml_set_i32(tensor, data)ggml_set_i32(tensor, data)
tensor |
Tensor of type GGML_TYPE_I32 |
data |
Integer vector |
NULL (invisible)
ctx <- ggml_init(1024 * 1024) pos <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 10) ggml_set_i32(pos, 0:9) ggml_get_i32(pos) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) pos <- ggml_new_tensor_1d(ctx, GGML_TYPE_I32, 10) ggml_set_i32(pos, 0:9) ggml_get_i32(pos) ggml_free(ctx)
Sets a single i32 value in the tensor at position [i0, i1, i2, i3].
ggml_set_i32_nd(tensor, i0, i1 = 0, i2 = 0, i3 = 0, value)ggml_set_i32_nd(tensor, i0, i1 = 0, i2 = 0, i3 = 0, value)
tensor |
Tensor pointer |
i0, i1, i2, i3
|
Indices (0-based) |
value |
Integer value to set |
NULL (invisible)
Mark Tensor as Input
ggml_set_input(tensor)ggml_set_input(tensor)
tensor |
Tensor pointer |
The tensor (for chaining)
Marks a tensor as the quantity to differentiate.
ggml_build_backward_expand requires the graph to contain one –
together with at least one ggml_set_param tensor, these are the
two ends of the backward pass.
ggml_set_loss(tensor)ggml_set_loss(tensor)
tensor |
Tensor pointer, usually a scalar produced by
|
A graph missing either is rejected with an error rather than aborting inside ggml.
The tensor (for chaining)
ggml_set_param,
ggml_build_forward_expand_grads
Set the number of threads for GGML operations
ggml_set_n_threads(n_threads)ggml_set_n_threads(n_threads)
n_threads |
Number of threads to use |
Number of threads set
# Use 4 threads ggml_set_n_threads(4) # Use all available cores ggml_set_n_threads(parallel::detectCores())# Use 4 threads ggml_set_n_threads(4) # Use all available cores ggml_set_n_threads(parallel::detectCores())
Assigns a name to a tensor. Useful for debugging and graph visualization.
ggml_set_name(tensor, name)ggml_set_name(tensor, name)
tensor |
Tensor pointer |
name |
Character string name |
The tensor (for chaining)
When enabled, tensor creation will not allocate memory for data. Useful for creating computation graphs without allocating storage.
ggml_set_no_alloc(ctx, no_alloc)ggml_set_no_alloc(ctx, no_alloc)
ctx |
GGML context |
no_alloc |
Logical, TRUE to disable allocation |
NULL (invisible)
ctx <- ggml_init(1024 * 1024) ggml_set_no_alloc(ctx, TRUE) ggml_get_no_alloc(ctx) ggml_set_no_alloc(ctx, FALSE) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) ggml_set_no_alloc(ctx, TRUE) ggml_get_no_alloc(ctx) ggml_set_no_alloc(ctx, FALSE) ggml_free(ctx)
Directly calls omp_set_num_threads() to limit OpenMP parallelism. Useful in tests to comply with CRAN policy on core usage.
ggml_set_omp_threads(n)ggml_set_omp_threads(n)
n |
Number of threads |
NULL invisibly
Sets the raw op_params bytes for a tensor.
ggml_set_op_params(tensor, params)ggml_set_op_params(tensor, params)
tensor |
External pointer to tensor |
params |
Raw vector of parameters (max 64 bytes) |
NULL invisibly
Other tensor:
ggml_are_same_layout(),
ggml_get_op_params(),
ggml_get_op_params_f32(),
ggml_get_op_params_i32(),
ggml_set_op_params_f32(),
ggml_set_op_params_i32()
Sets a single float value in tensor op_params at given index.
ggml_set_op_params_f32(tensor, index, value)ggml_set_op_params_f32(tensor, index, value)
tensor |
External pointer to tensor |
index |
0-based index (0-15 for 64-byte op_params) |
value |
Numeric value to set |
NULL invisibly
Other tensor:
ggml_are_same_layout(),
ggml_get_op_params(),
ggml_get_op_params_f32(),
ggml_get_op_params_i32(),
ggml_set_op_params(),
ggml_set_op_params_i32()
Sets a single int32 value in tensor op_params at given index.
ggml_set_op_params_i32(tensor, index, value)ggml_set_op_params_i32(tensor, index, value)
tensor |
External pointer to tensor |
index |
0-based index (0-15 for 64-byte op_params) |
value |
Integer value to set |
NULL invisibly
Other tensor:
ggml_are_same_layout(),
ggml_get_op_params(),
ggml_get_op_params_f32(),
ggml_get_op_params_i32(),
ggml_set_op_params(),
ggml_set_op_params_f32()
Mark Tensor as Output
ggml_set_output(tensor)ggml_set_output(tensor)
tensor |
Tensor pointer |
The tensor (for chaining)
Marks a tensor as a trainable parameter for backpropagation. The optimizer will compute gradients for this tensor during training.
ggml_set_param(tensor)ggml_set_param(tensor)
tensor |
Tensor pointer |
The tensor (for chaining)
ggml_set_loss, which marks the other end of the
backward pass, and ggml_build_backward_expand.
Fixes the random seed used by ggmlR for everything that is stochastic:
weight initialisation (sequential, functional and autograd layers),
dropout masks (training-time),
data shuffling in the autograd dataloader / training loops.
ggml_set_seed(seed)ggml_set_seed(seed)
seed |
A single integer (or value coercible to integer) used as the RNG
seed. |
This is a thin wrapper around set.seed: all randomness in
ggmlR is produced by the base R RNG, so a fixed seed gives identical starting
weights, dropout masks and batch ordering across runs. It is the single point
of control used by the mlr3 learners (seed hyperparameter) and
the parsnip "ggml" engine (seed engine argument).
GPU note: this controls the random *inputs* to the computation, not the floating-point arithmetic itself. GPU (Vulkan) kernels are run-to-run stable on a given device/driver for the standard forward/backward paths, but ggmlR does not guarantee bit-for-bit identical results across different devices, drivers or backends (CPU vs Vulkan). Reproducibility is at the level of training dynamics, not exact bits.
Invisibly returns seed.
ggml_set_seed(42) a <- runif(3) ggml_set_seed(42) b <- runif(3) identical(a, b) # TRUEggml_set_seed(42) a <- runif(3) ggml_set_seed(42) b <- runif(3) identical(a, b) # TRUE
Sets all elements of a tensor to zero. This is more efficient than manually setting all elements.
ggml_set_zero(tensor)ggml_set_zero(tensor)
tensor |
Tensor to zero out |
NULL (invisible)
ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_set_f32(t, 1:10) ggml_set_zero(t) ggml_get_f32(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_set_f32(t, 1:10) ggml_set_zero(t) ggml_get_f32(t) ggml_free(ctx)
Creates a graph node for element-wise sign function. sgn(x) = -1 if x < 0, 0 if x == 0, 1 if x > 0
ggml_sgn(ctx, a)ggml_sgn(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the sign operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -0.5, 0, 0.5, 2)) r <- ggml_sgn(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # c(-1, -1, 0, 1, 1) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -0.5, 0, 0.5, 2)) r <- ggml_sgn(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # c(-1, -1, 0, 1, 1) ggml_free(ctx)
Creates a graph node for sigmoid activation: 1 / (1 + exp(-x))
ggml_sigmoid(ctx, a)ggml_sigmoid(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the sigmoid operation
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) result <- ggml_sigmoid(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) result <- ggml_sigmoid(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)
Creates a graph node for in-place sigmoid activation: 1 / (1 + e^(-x))
ggml_sigmoid_inplace(ctx, a)ggml_sigmoid_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with sigmoid applied
Creates a graph node for SiLU (Sigmoid Linear Unit) activation, also known as Swish. CRITICAL for LLaMA models.
ggml_silu(ctx, a)ggml_silu(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the SiLU operation
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) result <- ggml_silu(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) result <- ggml_silu(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)
Computes the backward pass for SiLU (Swish) activation. Used during training for gradient computation.
ggml_silu_back(ctx, a, b)ggml_silu_back(ctx, a, b)
ctx |
GGML context |
a |
Forward input tensor |
b |
Gradient tensor from upstream |
Gradient tensor for the input
Creates a graph node for in-place SiLU (Sigmoid Linear Unit) activation. CRITICAL for LLaMA models with memory efficiency.
ggml_silu_inplace(ctx, a)ggml_silu_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with SiLU applied
Creates a graph node for element-wise sine: sin(x)
ggml_sin(ctx, a)ggml_sin(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the sin operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(0, pi/6, pi/2, pi)) result <- ggml_sin(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [0, 0.5, 1, 0] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(0, pi/6, pi/2, pi)) result <- ggml_sin(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [0, 0.5, 1, 0] ggml_free(ctx)
Frees the weights buffer and scheduler of a model returned by a
ggml_trainer's $snapshot(). Calling it twice is
harmless. The snapshot must not be used afterwards.
ggml_snapshot_free(model)ggml_snapshot_free(model)
model |
A snapshot from |
NULL, invisibly.
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_trainer()
Creates a graph node for softmax operation. CRITICAL for attention mechanisms.
ggml_soft_max(ctx, a)ggml_soft_max(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the softmax operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_soft_max(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # Output sums to 1.0 ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_soft_max(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # Output sums to 1.0 ggml_free(ctx)
The ggml_soft_max_ext counterpart of
ggml_flash_attn_ext_add_sinks.
ggml_soft_max_add_sinks(tensor, sinks)ggml_soft_max_add_sinks(tensor, sinks)
tensor |
A softmax node |
sinks |
Sink logits, one per head |
The tensor (for chaining)
Creates a graph node for fused softmax operation with optional masking and ALiBi (Attention with Linear Biases) support. Computes: softmax(a * scale + mask * (ALiBi slope)) CRITICAL for efficient attention computation in transformers.
ggml_soft_max_ext(ctx, a, mask = NULL, scale = 1, max_bias = 0)ggml_soft_max_ext(ctx, a, mask = NULL, scale = 1, max_bias = 0)
ctx |
GGML context |
a |
Input tensor (typically attention scores) |
mask |
Optional attention mask tensor (F16 or F32). NULL for no mask. Shape must be broadcastable to input tensor. |
scale |
Scaling factor, typically 1/sqrt(head_dim) |
max_bias |
Maximum ALiBi bias (0.0 to disable ALiBi) |
This extended softmax is commonly used in transformer attention: 1. Scale attention scores by 1/sqrt(d_k) for numerical stability 2. Apply attention mask (e.g., causal mask, padding mask) 3. Optionally apply ALiBi position bias 4. Compute softmax
All these operations are fused for efficiency.
Tensor representing the scaled and masked softmax
ctx <- ggml_init(16 * 1024 * 1024) scores <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 10) ggml_set_f32(scores, rnorm(100)) attn <- ggml_soft_max_ext(ctx, scores, NULL, 1.0, max_bias = 0.0) graph <- ggml_build_forward_expand(ctx, attn) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) scores <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 10) ggml_set_f32(scores, rnorm(100)) attn <- ggml_soft_max_ext(ctx, scores, NULL, 1.0, max_bias = 0.0) graph <- ggml_build_forward_expand(ctx, attn) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Backward pass for extended softmax operation.
ggml_soft_max_ext_back(ctx, a, b, scale = 1, max_bias = 0)ggml_soft_max_ext_back(ctx, a, b, scale = 1, max_bias = 0)
ctx |
GGML context |
a |
Softmax output tensor (from forward pass) |
b |
Gradient tensor from upstream |
scale |
Scale factor (same as forward pass) |
max_bias |
Maximum ALiBi bias (same as forward pass) |
Gradient tensor for the input
Creates a graph node for the backward pass of extended softmax, modifying in place.
ggml_soft_max_ext_back_inplace(ctx, a, b, scale = 1, max_bias = 0)ggml_soft_max_ext_back_inplace(ctx, a, b, scale = 1, max_bias = 0)
ctx |
GGML context |
a |
Gradient tensor from upstream |
b |
Softmax output from forward pass |
scale |
Scaling factor used in forward pass |
max_bias |
Maximum ALiBi bias used in forward pass |
View of input tensor with gradient computed in place
Other softmax:
ggml_soft_max_ext_inplace()
Creates a graph node for extended softmax, modifying input tensor in place. Returns a view of the input tensor.
ggml_soft_max_ext_inplace(ctx, a, mask = NULL, scale = 1, max_bias = 0)ggml_soft_max_ext_inplace(ctx, a, mask = NULL, scale = 1, max_bias = 0)
ctx |
GGML context |
a |
Input tensor (typically attention scores) |
mask |
Optional attention mask tensor (F16 or F32). NULL for no mask. Shape must be broadcastable to input tensor. |
scale |
Scaling factor, typically 1/sqrt(head_dim) |
max_bias |
Maximum ALiBi bias (0.0 to disable ALiBi) |
View of input tensor with softmax applied in place
Other softmax:
ggml_soft_max_ext_back_inplace()
Creates a graph node for in-place softmax operation. Returns a view of the input tensor.
ggml_soft_max_inplace(ctx, a)ggml_soft_max_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of input tensor with softmax applied
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_soft_max_inplace(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_soft_max_inplace(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Creates a graph node for Softplus activation. Softplus(x) = log(1 + exp(x)). A smooth approximation of ReLU.
ggml_softplus(ctx, a)ggml_softplus(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the Softplus operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) r <- ggml_softplus(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) r <- ggml_softplus(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) ggml_free(ctx)
Creates a graph node for in-place softplus activation: log(1 + e^x)
ggml_softplus_inplace(ctx, a)ggml_softplus_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with softplus applied
Sort Order Constants
GGML_SORT_ORDER_ASC GGML_SORT_ORDER_DESCGGML_SORT_ORDER_ASC GGML_SORT_ORDER_DESC
Integer constants
An object of class integer of length 1.
Constants for specifying sort order in argsort operations.
GGML_SORT_ORDER_ASC (0): Ascending order (smallest first)
GGML_SORT_ORDER_DESC (1): Descending order (largest first)
An integer constant representing a sort order
GGML_SORT_ORDER_ASC # 0 - Ascending order GGML_SORT_ORDER_DESC # 1 - Descending order # Usage with ggml_argsort ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(3, 1, 4, 1, 5)) # Get ascending sort indices idx_asc <- ggml_argsort(ctx, a, GGML_SORT_ORDER_ASC) # Get descending sort indices idx_desc <- ggml_argsort(ctx, a, GGML_SORT_ORDER_DESC) ggml_free(ctx)GGML_SORT_ORDER_ASC # 0 - Ascending order GGML_SORT_ORDER_DESC # 1 - Descending order # Usage with ggml_argsort ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(3, 1, 4, 1, 5)) # Get ascending sort indices idx_asc <- ggml_argsort(ctx, a, GGML_SORT_ORDER_ASC) # Get descending sort indices idx_desc <- ggml_argsort(ctx, a, GGML_SORT_ORDER_DESC) ggml_free(ctx)
Creates a graph node for element-wise squaring: x^2
ggml_sqr(ctx, a)ggml_sqr(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the square operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_sqr(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [1, 4, 9, 16] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 2, 3, 4)) result <- ggml_sqr(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [1, 4, 9, 16] ggml_free(ctx)
Creates a graph node for in-place element-wise square: x^2
ggml_sqr_inplace(ctx, a)ggml_sqr_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with squared values
Creates a graph node for element-wise square root: sqrt(x)
ggml_sqrt(ctx, a)ggml_sqrt(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the sqrt operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 4, 9, 16)) result <- ggml_sqrt(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [1, 2, 3, 4] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4) ggml_set_f32(a, c(1, 4, 9, 16)) result <- ggml_sqrt(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [1, 2, 3, 4] ggml_free(ctx)
Creates a graph node for in-place element-wise square root.
ggml_sqrt_inplace(ctx, a)ggml_sqrt_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with square root values
Depthwise causal convolution over a sequence, the convolution branch of a
Mamba block. Each of the d_inner channels is convolved with its own
d_conv-tap kernel; the leading d_conv - 1 positions of
sx are the carried-over context from the previous chunk, which is
what makes the op resumable across chunks.
ggml_ssm_conv(ctx, sx, c)ggml_ssm_conv(ctx, sx, c)
ctx |
GGML context |
sx |
Input with carried context, |
c |
Convolution kernels, |
A tensor [d_inner, n_t, n_s]
sx is [d_conv - 1 + n_t, d_inner, n_s] and c is
[d_conv, d_inner]; the result is [d_inner, n_t, n_s], where
n_t is the number of tokens per sequence and n_s the number of
sequences. All tensors are F32.
GGML_OP_SSM_CONV has no backward pass in ggml, so a graph containing
it cannot be trained – see ggml_ssm_scan.
Other state-space:
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Gradients of ggml_ssm_conv with respect to both of its inputs,
given grad, the gradient of the loss with respect to the convolution's
output.
ggml_ssm_conv_back(ctx, sx, c, grad)ggml_ssm_conv_back(ctx, sx, c, grad)
ctx |
GGML context |
sx |
The |
c |
The |
grad |
Gradient w.r.t. the output, shaped like it: |
Training reaches this automatically through the backward graph – an
ggml_ssm_conv node is differentiable, unlike the other
state-space and RWKV ops. It is exported so the gradient can be checked
directly against a numeric one.
A flat tensor holding d_sx followed by d_c
Both gradients come back in one flat tensor, [d_sx | d_c]: the first
ggml_nelements(sx) elements are the gradient with respect to
sx, the remaining ggml_nelements(c) the gradient with respect
to c. A single op returns a single tensor, and packing them keeps the
convolution from being run once per input.
Other state-space:
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_scan(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
The recurrent core of a Mamba block: a linear state-space recurrence whose
transition is modulated per token by dt, B and C.
ids selects which state row each sequence continues from, so several
sequences can share one state buffer.
ggml_ssm_scan(ctx, s, x, dt, A, B, C, ids)ggml_ssm_scan(ctx, s, x, dt, A, B, C, ids)
ctx |
GGML context |
s |
Initial states, |
x |
Inputs, |
dt |
Per-token timestep, |
A |
State decay, |
B |
Input projection, |
C |
Output projection, same shape as |
ids |
Sequence-to-state mapping, |
A flat tensor: outputs followed by final states
With d_state the state width, head_dim the per-head channel
count, n_head heads, n_seq_tokens tokens and n_seqs
sequences:
s — [d_state, head_dim, n_head, n_seqs_total], contiguous
x — [head_dim, n_head, n_seq_tokens, n_seqs]
dt — [n_head, n_seq_tokens, n_seqs], contiguous
A — [1, n_head] (Mamba-2) or [d_state, n_head] (Mamba-1)
B, C — [d_state, n_group, n_seq_tokens, n_seqs], same shape
ids — [n_seqs], type GGML_TYPE_I32, 0-based
The result is a flat tensor holding the outputs followed by the final
states: ggml_nelements(x) output elements, then
d_state * head_dim * n_head * n_seqs state elements. Use
ggml_ssm_scan_output and ggml_ssm_scan_state to
view either half rather than computing the offsets by hand – getting them
wrong silently reads the state as output.
GGML_OP_SSM_SCAN has no backward pass in ggml. A graph containing it
runs forward on CPU and Vulkan, but cannot be trained: ggml_fit() and
the autograd engine will not produce gradients for it.
ggml_ssm_scan_output, ggml_ssm_scan_state,
ggml_ssm_conv
Other state-space:
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan_back(),
ggml_ssm_scan_output()
Gradients of ggml_ssm_scan with respect to all six of its
differentiable inputs, given grad, the gradient of the loss with
respect to the scan's packed forward result.
ggml_ssm_scan_back(ctx, s, x, dt, A, B, C, ids, grad)ggml_ssm_scan_back(ctx, s, x, dt, A, B, C, ids, grad)
ctx |
GGML context |
s |
Initial states, as passed to the forward scan |
x |
Inputs, as passed to the forward scan |
dt |
Per-token timestep, as passed to the forward scan |
A |
State decay, as passed to the forward scan |
B |
Input projection, as passed to the forward scan |
C |
Output projection, as passed to the forward scan |
ids |
Sequence-to-state mapping, as passed to the forward scan |
grad |
Gradient w.r.t. the packed forward result |
Training reaches this automatically through the backward graph. It is exported so the gradient can be checked directly against a numeric one.
A flat tensor holding the six gradients, concatenated
All six gradients come back in one flat tensor, in the same order the scan
takes its inputs: [d_s | d_x | d_dt | d_A | d_B | d_C]. ids is
integer and has no gradient. A single op returns a single tensor, and packing
them keeps the recurrence from being replayed once per input.
The forward pass does not keep its intermediate states, so this replays the recurrence before walking it back – roughly twice the forward work, and no extra memory beyond one state buffer.
Other state-space:
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_output()
ggml_ssm_scan returns the outputs and the final states in one
flat tensor. These helpers view either half with the right shape, so the
offset arithmetic lives in one place: ggml_ssm_scan_output() yields
the outputs shaped like x, and ggml_ssm_scan_state() the final
states shaped like the leading n_seqs rows of s.
ggml_ssm_scan_output(ctx, result, x) ggml_ssm_scan_state(ctx, result, s, ids)ggml_ssm_scan_output(ctx, result, x) ggml_ssm_scan_state(ctx, result, s, ids)
ctx |
GGML context |
result |
The tensor returned by |
x |
The |
s |
The |
ids |
The |
Both are graph views sharing memory with result; they add no
computation. x, s and ids must be the same tensors the
scan was built with, since their shapes are what the offsets are derived
from.
For ggml_ssm_scan_output(), a tensor shaped like x
For ggml_ssm_scan_state(), a tensor
[d_state, head_dim, n_head, n_seqs]
Other state-space:
ggml_gated_linear_attn(),
ggml_gated_linear_attn_back(),
ggml_rwkv_output(),
ggml_rwkv_wkv6(),
ggml_rwkv_wkv6_back(),
ggml_rwkv_wkv7(),
ggml_rwkv_wkv7_back(),
ggml_ssm_conv(),
ggml_ssm_conv_back(),
ggml_ssm_scan(),
ggml_ssm_scan_back()
Creates a graph node for element-wise step function. step(x) = 0 if x <= 0, 1 if x > 0 Also known as the Heaviside step function.
ggml_step(ctx, a)ggml_step(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the step operation
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -0.5, 0, 0.5, 2)) r <- ggml_step(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # c(0, 0, 0, 1, 1) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -0.5, 0, 0.5, 2)) r <- ggml_step(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # c(0, 0, 0, 1, 1) ggml_free(ctx)
Creates a graph node for element-wise subtraction.
ggml_sub(ctx, a, b)ggml_sub(ctx, a, b)
ctx |
GGML context |
a |
First tensor |
b |
Second tensor (same shape as a) |
Tensor representing the subtraction operation (a - b)
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(5, 4, 3, 2, 1)) ggml_set_f32(b, c(1, 1, 1, 1, 1)) result <- ggml_sub(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) b <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(5, 4, 3, 2, 1)) ggml_set_f32(b, c(1, 1, 1, 1, 1)) result <- ggml_sub(ctx, a, b) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)
Creates a graph node for in-place element-wise subtraction. Result is stored in tensor a, saving memory allocation.
ggml_sub_inplace(ctx, a, b)ggml_sub_inplace(ctx, a, b)
ctx |
GGML context |
a |
First tensor (will be modified in-place) |
b |
Second tensor (same shape as a) |
View of tensor a with the subtraction result
Creates a graph node that computes the sum of all elements.
ggml_sum(ctx, a)ggml_sum(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Scalar tensor with the sum
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) result <- ggml_sum(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # 15 ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(1, 2, 3, 4, 5)) result <- ggml_sum(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # 15 ggml_free(ctx)
Creates a graph node that computes the sum along rows.
ggml_sum_rows(ctx, a)ggml_sum_rows(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor with row sums
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 2) ggml_set_f32(a, c(1, 2, 3, 4, 5, 6)) result <- ggml_sum_rows(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [6, 15] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 2) ggml_set_f32(a, c(1, 2, 3, 4, 5, 6)) result <- ggml_sum_rows(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) output <- ggml_get_f32(result) # [6, 15] ggml_free(ctx)
Creates a graph node for SwiGLU operation. SwiGLU uses SiLU (Swish) as the activation function on the first half. CRITICAL for LLaMA, Mistral, and many modern LLMs.
ggml_swiglu(ctx, a)ggml_swiglu(ctx, a)
ctx |
GGML context |
a |
Input tensor (first dimension must be even) |
Formula: output = SiLU(x) * gate
Tensor with half the first dimension of input
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 3) ggml_set_f32(a, rnorm(24)) r <- ggml_swiglu(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # Shape: 4x3 ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 3) ggml_set_f32(a, rnorm(24)) r <- ggml_swiglu(ctx, a) graph <- ggml_build_forward_expand(ctx, r) ggml_graph_compute(ctx, graph) result <- ggml_get_f32(r) # Shape: 4x3 ggml_free(ctx)
Creates a graph node for SwiGLU with separate input and gate tensors.
ggml_swiglu_split(ctx, a, b)ggml_swiglu_split(ctx, a, b)
ctx |
GGML context |
a |
Input tensor (the values to be gated) |
b |
Gate tensor (same shape as a) |
Formula: output = SiLU(a) * b
Tensor with same shape as input tensors
Creates a graph node for hyperbolic tangent activation.
ggml_tanh(ctx, a)ggml_tanh(ctx, a)
ctx |
GGML context |
a |
Input tensor |
Tensor representing the tanh operation
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) result <- ggml_tanh(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 5) ggml_set_f32(a, c(-2, -1, 0, 1, 2)) result <- ggml_tanh(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) ggml_get_f32(result) ggml_free(ctx)
Creates a graph node for in-place hyperbolic tangent activation.
ggml_tanh_inplace(ctx, a)ggml_tanh_inplace(ctx, a)
ctx |
GGML context |
a |
Input tensor (will be modified in-place) |
View of tensor a with tanh applied
A ggml_task is the contract object passed from the extraction layer to
the dispatch layer. It bundles the operation name, the dense feature matrix
(features in rows, cells in columns — the single-cell convention), the
operation parameters and the requested device. It performs no computation.
ggml_task(op, matrix, params = list(), device = c("auto", "vulkan", "cpu"))ggml_task(op, matrix, params = list(), device = c("auto", "vulkan", "cpu"))
op |
Operation name; must be registered in
|
matrix |
A numeric |
params |
Named list of operation parameters (e.g. |
device |
|
An object of class ggml_task.
tcrossprod)Computes A %*% t(B) (or A %*% t(A) when B is
NULL) on the Vulkan GPU, returning an R matrix, with a CPU fallback.
See ggml_matmul for the precision and dispatch notes.
ggml_tcrossprod(A, B = NULL, device = "auto", prec = "f32")ggml_tcrossprod(A, B = NULL, device = "auto", prec = "f32")
A |
A numeric matrix. |
B |
A numeric matrix with |
device, prec
|
See |
A %*% t(B) as a numeric matrix.
Copies raw data from src tensor to dst tensor (must be same size).
ggml_tensor_copy(dst, src)ggml_tensor_copy(dst, src)
dst |
Destination tensor |
src |
Source tensor |
NULL (invisible)
Returns the address of the tensor's data buffer as a numeric value. Useful for checking whether two tensors alias the same memory – a view and its source share a pointer.
ggml_tensor_data_ptr(tensor)ggml_tensor_data_ptr(tensor)
tensor |
Tensor pointer |
The data address, as a double
Returns the byte strides for each dimension of the tensor.
ggml_tensor_nb(tensor)ggml_tensor_nb(tensor)
tensor |
Tensor pointer |
Numeric vector of 4 stride values (nb0, nb1, nb2, nb3)
Counts the number of tensors allocated in a context.
ggml_tensor_num(ctx)ggml_tensor_num(ctx)
ctx |
GGML context |
Number of tensors
Returns the memory overhead (metadata) for each tensor in bytes
ggml_tensor_overhead()ggml_tensor_overhead()
Size in bytes
ggml_tensor_overhead()ggml_tensor_overhead()
Sets all elements of a f32 tensor to a single value.
ggml_tensor_set_f32_scalar(tensor, value)ggml_tensor_set_f32_scalar(tensor, value)
tensor |
Tensor pointer |
value |
Float value to fill with |
NULL (invisible)
Returns the shape of a tensor as a numeric vector of 4 elements (ne0, ne1, ne2, ne3)
ggml_tensor_shape(tensor)ggml_tensor_shape(tensor)
tensor |
Tensor pointer |
Numeric vector of length 4 with dimensions
ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_tensor_shape(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 10, 20) ggml_tensor_shape(t) ggml_free(ctx)
Returns the data type of a tensor as an integer code
ggml_tensor_type(tensor)ggml_tensor_type(tensor)
tensor |
Tensor pointer |
Integer type code (0 = F32, 1 = F16, etc.)
ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_tensor_type(t) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) t <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_tensor_type(t) ggml_free(ctx)
Runs GGML library self-test and prints version info.
ggml_test()ggml_test()
TRUE if test passed
ggml_test()ggml_test()
Runs ggml_opt_step_adamw() for several steps on backend and on
the CPU from the same starting weights and the same gradients, and reports the
weight checksum after each step.
ggml_test_adamw_steps(backend)ggml_test_adamw_steps(backend)
backend |
Backend pointer to compare against the CPU. |
This needs its own function rather than a case in
ggml_test_backend_ops: the op updates the weights and both
moment buffers in place, so step N depends on every step before it, and the
bias-correction terms it receives change every step. Comparing a single
execution proves nothing here – the first Adam step agrees bit-for-bit on
both backends and the disagreement only starts at the second.
A data frame with step, w_cpu, w_backend and
abs_diff, or a one-column data frame with note when the op
could not be run.
if (ggml_vulkan_available()) { be <- ggml_vulkan_init(0) print(ggml_test_adamw_steps(be)) ggml_backend_free(be) }if (ggml_vulkan_available()) { be <- ggml_vulkan_init(0) print(ggml_test_adamw_steps(be)) ggml_backend_free(be) }
Runs each supported operation on backend and on the CPU with identical
inputs, and reports the normalized mean squared error between the two. This is
the differential test used to find operations whose accelerated implementation
disagrees with the reference one.
ggml_test_backend_ops(backend, filter = NULL)ggml_test_backend_ops(backend, filter = NULL)
backend |
Backend pointer, e.g. from |
filter |
Optional substring; only operations whose name contains it are
run. |
Each case is executed three times against the same allocated graph and the final run is the one compared, so faults that only appear when a graph is re-executed are visible. That matters: both backend defects found in this package so far were of that kind – the first run was exact and later ones were not.
A data frame with columns op, nmse and note.
nmse is NA when the case could not be run, and note
then says why.
if (ggml_vulkan_available()) { be <- ggml_vulkan_init(0) res <- ggml_test_backend_ops(be) res[res$nmse > 1e-6, ] # operations that disagree ggml_backend_free(be) }if (ggml_vulkan_available()) { be <- ggml_vulkan_init(0) res <- ggml_test_backend_ops(be) res[res$nmse > 1e-6, ] # operations that disagree ggml_backend_free(be) }
Initializes the GGML timing system. Call this once at the beginning of the program before using ggml_time_ms() or ggml_time_us().
ggml_time_init()ggml_time_init()
NULL (invisible)
ggml_time_init() start <- ggml_time_ms() Sys.sleep(0.01) elapsed <- ggml_time_ms() - startggml_time_init() start <- ggml_time_ms() Sys.sleep(0.01) elapsed <- ggml_time_ms() - start
Returns the current time in milliseconds since the timer was initialized.
ggml_time_ms()ggml_time_ms()
Numeric value representing milliseconds
ggml_time_init() start <- ggml_time_ms() Sys.sleep(0.01) elapsed <- ggml_time_ms() - startggml_time_init() start <- ggml_time_ms() Sys.sleep(0.01) elapsed <- ggml_time_ms() - start
Returns the current time in microseconds since the timer was initialized. More precise than ggml_time_ms() for micro-benchmarking.
ggml_time_us()ggml_time_us()
Numeric value representing microseconds
ggml_time_init() start <- ggml_time_us() Sys.sleep(0.001) elapsed <- ggml_time_us() - startggml_time_init() start <- ggml_time_us() Sys.sleep(0.001) elapsed <- ggml_time_us() - start
Creates sinusoidal timestep embeddings as used in diffusion models. Reference: CompVis/stable-diffusion util.py timestep_embedding
ggml_timestep_embedding(ctx, timesteps, dim, max_period = 10000L)ggml_timestep_embedding(ctx, timesteps, dim, max_period = 10000L)
ctx |
GGML context |
timesteps |
Input tensor of timestep values [N] |
dim |
Embedding dimension |
max_period |
Maximum period for sinusoidal embedding (default 10000) |
Tensor of shape [N, dim] with timestep embeddings
Returns the indices of top K elements per row. Useful for sampling strategies in language models (top-k sampling). Note: the resulting indices are in no particular order within top-k.
ggml_top_k(ctx, a, k)ggml_top_k(ctx, a, k)
ctx |
GGML context |
a |
Input tensor (F32) |
k |
Number of top elements to return per row |
Tensor containing I32 indices of top-k elements (not values)
ctx <- ggml_init(16 * 1024 * 1024) # Logits from model output logits <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) ggml_set_f32(logits, rnorm(100)) # Get top 5 logits for sampling top5 <- ggml_top_k(ctx, logits, 5) graph <- ggml_build_forward_expand(ctx, top5) ggml_graph_compute(ctx, graph) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) # Logits from model output logits <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) ggml_set_f32(logits, rnorm(100)) # Get top 5 logits for sampling top5 <- ggml_top_k(ctx, logits, 5) graph <- ggml_build_forward_expand(ctx, top5) ggml_graph_compute(ctx, graph) ggml_free(ctx)
Computes Y = X %*% t(W) as a hybrid of tensor parallelism and data
parallelism: the weight matrix W is replicated across replicas
device groups (data parallelism over the batch X), and within each group
the weight rows are split across that group's GPUs (tensor parallelism, via
ggml_vulkan_split_mul_mat). Each replica computes a contiguous
shard of the batch rows; the shards are concatenated back into the full result.
ggml_tp_dp_forward( W, X, replicas, weights = NULL, transport = c("host-staging", "opaque-fd", "device-group") )ggml_tp_dp_forward( W, X, replicas, weights = NULL, transport = c("host-staging", "opaque-fd", "device-group") )
W |
Weight matrix, |
X |
Activation matrix, |
replicas |
A list of integer vectors, each a group of physical GPU indices
(0-based) forming one replica, e.g. |
weights |
Optional numeric vector (length = group size) giving the
per-device TP row weighting inside each group (applied to every group).
|
transport |
Cross-device TP gather transport (see
|
This mirrors the inference layout for the 4x P100 target (2 replicas x TP=2): there is no cross-replica communication, so throughput scales with the replica count while each replica's cross-device TP gather stays local to its group.
The M x N result matrix, equal to X %*% t(W) up to the
GPU's floating-point accumulation.
ggml_vulkan_split_mul_mat for the single-group TP path.
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 4) { W <- matrix(rnorm(2048 * 64), nrow = 2048) X <- matrix(rnorm(8 * 64), nrow = 8) # batch of 8 # 2 replicas x TP=2: batch split 4/4, weights split within {0,1} and {2,3} Y <- ggml_tp_dp_forward(W, X, replicas = list(c(0, 1), c(2, 3))) max(abs(Y - X %*% t(W))) }if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 4) { W <- matrix(rnorm(2048 * 64), nrow = 2048) X <- matrix(rnorm(8 * 64), nrow = 8) # batch of 8 # 2 replicas x TP=2: batch split 4/4, weights split within {0,1} and {2,3} Y <- ggml_tp_dp_forward(W, X, replicas = list(c(0, 1), c(2, 3))) max(abs(Y - X %*% t(W))) }
Builds the graph and optimizer context for a compiled model and hands them
back, so a training loop can be written in R rather than handed to
ggml_fit. The context lives until $free(), keeping
weights, Adam moments and the compute graph across as many steps as the loop
takes.
ggml_trainer( model, batch_size = 32L, nbatch_logical = NULL, loss_mask = NULL, weight_decay = NULL )ggml_trainer( model, batch_size = 32L, nbatch_logical = NULL, loss_mask = NULL, weight_decay = NULL )
model |
A compiled |
batch_size |
Samples per step. The graph is built for exactly this many;
every batch handed to |
nbatch_logical |
Logical batch size for gradient accumulation. Defaults
to |
loss_mask |
|
weight_decay |
Decoupled weight decay in |
This is the level below ggml_fit_opt_gen. Reach for it when the
loop is not a pass over data: reinforcement learning, where experience comes
from a replay buffer the agent is still filling and a target network is
refreshed every so many steps; curriculum schemes that change the data
distribution as they go; data generated fresh for every step; or anything
where "epoch" is not the unit of progress.
Do not emulate this by calling ggml_fit once per batch: each
such call builds a new optimizer context, so the Adam moments are reset and
every step is the optimizer's first.
An object of class ggml_trainer: a list of functions
$step(x, y, sample_weight = NULL),
$eval(x, y, sample_weight = NULL),
$predict(x, batch_size = NULL, matmul_precision = NULL), $snapshot(),
$update_target(target, tau = 1), $model(),
$calibrate_bn(x), $set_lr(lr), $n_steps(), $free().
tr <- ggml_trainer(model, batch_size = 32L)
on.exit(tr$free())
for (i in seq_len(10000L)) {
b <- sample_batch(replay, 32L) # your own bookkeeping
loss <- tr$step(b$x, b$y)
if (i %% 500L == 0L) target <- tr$model() # refresh a target network
}
model <- tr$model() # weights back into an R model object
tr$free()
$step(x, y, sample_weight) runs one forward and backward pass and
returns that step's loss. Gradient accumulation, if nbatch_logical
asked for it, is tracked inside the context: the optimizer updates the
weights only on the period boundary, and the intervening steps only
accumulate. $eval(x, y, sample_weight) is the same without the
backward pass, for measuring on held-out data.
With loss_mask set, the loss is weighted mean squared error,
sum(w * (pred - y)^2) / (batch * outputs), and $step() and
$eval() take a sample_weight matrix:
loss_mask = 1: sample_weight is [batch, 1] (or
a vector of length batch) – one weight per sample, as in Keras.
loss_mask = <outputs>: sample_weight is
[batch, outputs] – one weight per output element. This is
not Keras' sample_weight: a 0/1 matrix is a mask that
trains only the outputs whose label is known (e.g. one action of a
Q-network), and a 0 removes that element's gradient exactly. The loss is
not renormalised by the number of active elements.
Omitting sample_weight on a weighted trainer means all ones. Only
loss = "mse" supports weights.
Sequential models and functional models (ggml_model) with
one input and one output are supported. Functional models with
several inputs or outputs are refused. For a functional model with dropout
the dropout masks are redrawn before every $step().
$predict(x, batch_size) returns predictions from the current weights,
as ggml_predict would, in eval mode (no dropout). It runs on a
scheduler of its own, created on first use: the weights are copied into a
separate buffer for the call and released after it, so neither the training
graph's buffers nor the optimizer state are touched. batch_size
defaults to the trainer's; x may have any number of rows.
$snapshot() returns an independent copy of the model: its own weight
tensors in its own buffer and its own scheduler, sharing nothing mutable with
the trainer. Later steps do not move it, ggml_predict on it
never touches the training path, and it outlives $free(). It is built
in eval mode, so dropout is off in it. Release it with
ggml_snapshot_free (a finaliser is only a backstop); after that
ggml_predict and $update_target on it are refused with an error.
$update_target(target, tau = 1) moves a snapshot towards the current
weights, :
tau = 1 is the hard copy of DQN, a small tau the Polyak
average of DDPG/SAC. Non-trainable buffers (a sequential batch-norm's
running mean and variance) are copied whole, never averaged. The target
must have the same weights, by name and shape, or it is refused. The update
goes through host memory, one read and one write per tensor.
The context holds GPU buffers and must be released with $free();
calling it twice is harmless. A trainer that goes out of scope without it is
cleaned up when garbage collected, but that moment is not predictable, so
free it yourself. After $free() every other method errors rather than
touching released memory.
The model object passed in is unchanged – R semantics being what they are,
the trainer works on its own copy of the weights. $model() returns a
model carrying the current ones.
The returned model shares the trainer's backend scheduler, and predicting
with it re-lays the scheduler's buffers for another graph. The trainer
therefore re-allocates its own graph on the first step after any
$model() call; weights and Adam moments are kept. $predict()
needs none of this. Predicting with the
original model object while the trainer is alive is not covered by
this and can crash the session on the next step – take the model from
$model() instead.
A step normalizes each batch by its own statistics and does not update the
running ones a sequential batch_norm predicts with; ggml_fit()
recalibrates them after training, a trainer does not. Call
$calibrate_bn(x) with representative inputs (up to 1024 rows are
used) before predicting, snapshotting or saving – or fix the statistics
with ggml_freeze_bn_stats. Creating a trainer for a model
with unfixed batch-norm layers warns once.
ggml_fit_opt_gen for a generator-driven epoch loop,
ggml_fit for the ordinary in-memory path.
Other optimization:
ggml_fit_opt(),
ggml_fit_opt_gen(),
ggml_fit_opt_multi(),
ggml_opt_alloc(),
ggml_opt_context_optimizer_type(),
ggml_opt_dataset_data(),
ggml_opt_dataset_free(),
ggml_opt_dataset_get_batch(),
ggml_opt_dataset_get_batch_head(),
ggml_opt_dataset_init(),
ggml_opt_dataset_labels(),
ggml_opt_dataset_ndata(),
ggml_opt_dataset_shuffle(),
ggml_opt_dataset_weights(),
ggml_opt_default_params(),
ggml_opt_epoch(),
ggml_opt_eval(),
ggml_opt_fit(),
ggml_opt_fit_multi(),
ggml_opt_free(),
ggml_opt_get_lr(),
ggml_opt_grad_acc(),
ggml_opt_init(),
ggml_opt_init_for_fit(),
ggml_opt_init_for_fit_multi(),
ggml_opt_inputs(),
ggml_opt_labels(),
ggml_opt_loss(),
ggml_opt_loss_i(),
ggml_opt_loss_type_binary_cross_entropy(),
ggml_opt_loss_type_cross_entropy(),
ggml_opt_loss_type_huber(),
ggml_opt_loss_type_mae(),
ggml_opt_loss_type_mean(),
ggml_opt_loss_type_mse(),
ggml_opt_loss_type_sum(),
ggml_opt_loss_type_weighted_mse(),
ggml_opt_n_loss(),
ggml_opt_ncorrect(),
ggml_opt_optimizer_name(),
ggml_opt_optimizer_type_adamw(),
ggml_opt_optimizer_type_sgd(),
ggml_opt_outputs(),
ggml_opt_outputs_i(),
ggml_opt_pred(),
ggml_opt_prepare_alloc(),
ggml_opt_reset(),
ggml_opt_result_accuracy(),
ggml_opt_result_free(),
ggml_opt_result_init(),
ggml_opt_result_loss(),
ggml_opt_result_loss_i(),
ggml_opt_result_n_loss(),
ggml_opt_result_ndata(),
ggml_opt_result_pred(),
ggml_opt_result_reset(),
ggml_opt_set_lr(),
ggml_opt_static_graphs(),
ggml_snapshot_free()
ggml_set_n_threads(1L) # deterministic, single OpenMP pool n <- 64 x <- matrix(runif(n * 4), nrow = n, ncol = 4) y <- matrix(0, nrow = n, ncol = 2) for (i in seq_len(n)) { y[i, if (sum(x[i, ]) > 2) 1L else 2L] <- 1 } model <- ggml_model_sequential() |> ggml_layer_dense(8, activation = "relu") |> ggml_layer_dense(2, activation = "softmax") model$input_shape <- 4L model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy") tr <- ggml_trainer(model, batch_size = 16L) for (i in 1:4) { idx <- ((i - 1L) * 16L + 1L):(i * 16L) tr$step(x[idx, , drop = FALSE], y[idx, , drop = FALSE]) } model <- tr$model() tr$free()ggml_set_n_threads(1L) # deterministic, single OpenMP pool n <- 64 x <- matrix(runif(n * 4), nrow = n, ncol = 4) y <- matrix(0, nrow = n, ncol = 2) for (i in seq_len(n)) { y[i, if (sum(x[i, ]) > 2) 1L else 2L] <- 1 } model <- ggml_model_sequential() |> ggml_layer_dense(8, activation = "relu") |> ggml_layer_dense(2, activation = "softmax") model$input_shape <- 4L model <- ggml_compile(model, optimizer = "adam", loss = "categorical_crossentropy") tr <- ggml_trainer(model, batch_size = 16L) for (i in 1:4) { idx <- ((i - 1L) * 16L + 1L):(i * 16L) tr$step(x[idx, , drop = FALSE], y[idx, , drop = FALSE]) } model <- tr$model() tr$free()
Returns the per-epoch loss / accuracy curve recorded during
ggml_fit, in a tidy data frame. This is the standard accessor
for the loss curve; it works on a raw sequential/functional model or on a
fitted parsnip engine object (e.g. from extract_fit_engine()).
ggml_training_history(object, format = c("wide", "long"), ...)ggml_training_history(object, format = c("wide", "long"), ...)
object |
A fitted |
format |
|
... |
Unused; for extensibility. |
A data frame (tibble if tibble is installed). Wide columns:
epoch, train_loss, train_accuracy, and
val_loss / val_accuracy when a validation split was used.
Returns NULL with a warning if the model has no recorded history
(e.g. not yet fitted).
Creates a graph node for matrix transpose operation.
ggml_transpose(ctx, a)ggml_transpose(ctx, a)
ctx |
GGML context |
a |
Input tensor (2D matrix) |
Tensor representing the transposed matrix
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 2) ggml_set_f32(a, 1:6) result <- ggml_transpose(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) shape <- ggml_tensor_shape(result) # [2, 3] ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 3, 2) ggml_set_f32(a, 1:6) result <- ggml_transpose(ctx, a) graph <- ggml_build_forward_expand(ctx, result) ggml_graph_compute(ctx, graph) shape <- ggml_tensor_shape(result) # [2, 3] ggml_free(ctx)
Constants representing different data types supported by GGML.
GGML_TYPE_F32 GGML_TYPE_F16 GGML_TYPE_Q4_0 GGML_TYPE_Q4_1 GGML_TYPE_Q8_0 GGML_TYPE_Q2_K GGML_TYPE_Q3_K GGML_TYPE_Q4_K GGML_TYPE_Q5_K GGML_TYPE_Q6_K GGML_TYPE_I32 GGML_TYPE_BF16GGML_TYPE_F32 GGML_TYPE_F16 GGML_TYPE_Q4_0 GGML_TYPE_Q4_1 GGML_TYPE_Q8_0 GGML_TYPE_Q2_K GGML_TYPE_Q3_K GGML_TYPE_Q4_K GGML_TYPE_Q5_K GGML_TYPE_Q6_K GGML_TYPE_I32 GGML_TYPE_BF16
Integer constants
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
GGML_TYPE_F32: 32-bit floating point (default)
GGML_TYPE_F16: 16-bit floating point (half precision)
GGML_TYPE_Q4_0: 4-bit quantization type 0
GGML_TYPE_Q4_1: 4-bit quantization type 1
GGML_TYPE_Q8_0: 8-bit quantization type 0
GGML_TYPE_I32: 32-bit integer
GGML_TYPE_BF16: 16-bit brain float (bfloat16)
An integer constant representing a GGML data type
GGML_TYPE_F32 GGML_TYPE_F16 GGML_TYPE_I32GGML_TYPE_F32 GGML_TYPE_F16 GGML_TYPE_I32
Returns the string name of a GGML type.
ggml_type_name(type)ggml_type_name(type)
type |
GGML type constant (e.g., GGML_TYPE_F32) |
Character string with type name
Other type_system:
ggml_blck_size(),
ggml_ftype_to_ggml_type(),
ggml_is_quantized(),
ggml_type_sizef()
ggml_type_name(GGML_TYPE_F32) # "f32" ggml_type_name(GGML_TYPE_Q4_0) # "q4_0"ggml_type_name(GGML_TYPE_F32) # "f32" ggml_type_name(GGML_TYPE_Q4_0) # "q4_0"
Returns the size in bytes for all elements in a block for a given type.
ggml_type_size(type)ggml_type_size(type)
type |
GGML type constant (e.g., GGML_TYPE_F32) |
Size in bytes
Returns the size in bytes of a GGML type as a floating-point number. For quantized types, this is the average bytes per element.
ggml_type_sizef(type)ggml_type_sizef(type)
type |
GGML type constant |
Numeric size in bytes (can be fractional for quantized types)
Other type_system:
ggml_blck_size(),
ggml_ftype_to_ggml_type(),
ggml_is_quantized(),
ggml_type_name()
ggml_type_sizef(GGML_TYPE_F32) # 4.0 ggml_type_sizef(GGML_TYPE_F16) # 2.0ggml_type_sizef(GGML_TYPE_F32) # 4.0 ggml_type_sizef(GGML_TYPE_F16) # 2.0
Returns the string name of a GGML unary operation.
ggml_unary_op_name(op)ggml_unary_op_name(op)
op |
GGML unary operation constant |
Character string with operation name
Other op_info:
ggml_get_unary_op(),
ggml_op_desc(),
ggml_op_name(),
ggml_op_symbol()
Sets trainable = TRUE on layers, chosen as in
ggml_freeze_weights: by layer_names and/or
prefix, else by from / to, else all of them.
ggml_unfreeze_weights( model, from = 1L, to = length(model$layers), layer_names = NULL, prefix = NULL, ... )ggml_unfreeze_weights( model, from = 1L, to = length(model$layers), layer_names = NULL, prefix = NULL, ... )
model |
A model object (ggml_sequential_model or ggml_functional_model) |
from |
Integer index of the first layer to unfreeze (default: 1) |
to |
Integer index of the last layer to unfreeze (default: last layer) |
layer_names |
Character vector of layer names to unfreeze (overrides from/to) |
prefix |
Character vector: unfreeze every layer whose name starts with
one of these. With |
... |
Additional arguments passed to methods |
The model with selected layers unfrozen.
model <- ggml_model_sequential() |> ggml_layer_dense(64, activation = "relu") |> ggml_layer_dense(10, activation = "softmax") model <- ggml_freeze_weights(model) model <- ggml_unfreeze_weights(model, from = 2) # unfreeze last layer onlymodel <- ggml_model_sequential() |> ggml_layer_dense(64, activation = "relu") |> ggml_layer_dense(10, activation = "softmax") model <- ggml_freeze_weights(model) model <- ggml_unfreeze_weights(model, from = 2) # unfreeze last layer only
Reconstructs a ggmlR model previously produced by
ggml_marshal_model. Validates the container's format tag,
schema version, and (if digest is installed) the SHA-256 checksum of
the payload before deserializing.
ggml_unmarshal_model(x, backend = NULL)ggml_unmarshal_model(x, backend = NULL)
x |
A |
backend |
Backend selection passed through to
|
A compiled ggmlR model object (sequential or functional).
ggml_marshal_model, ggml_load_model
Upscales tensor by multiplying ne0 and ne1 by scale factor. Supports different interpolation modes for image upscaling.
ggml_upscale(ctx, a, scale_factor, mode = 0L) GGML_SCALE_MODE_NEAREST GGML_SCALE_MODE_BILINEAR GGML_SCALE_MODE_BICUBICggml_upscale(ctx, a, scale_factor, mode = 0L) GGML_SCALE_MODE_NEAREST GGML_SCALE_MODE_BILINEAR GGML_SCALE_MODE_BICUBIC
ctx |
GGML context |
a |
Input tensor (typically 2D or 4D for images) |
scale_factor |
Integer scale factor (e.g., 2 = double size) |
mode |
Scale mode constant (see details) |
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
Scale mode constants:
GGML_SCALE_MODE_NEAREST (0): Nearest neighbor interpolation - fastest, pixelated
GGML_SCALE_MODE_BILINEAR (1): Bilinear interpolation - smooth, good balance
GGML_SCALE_MODE_BICUBIC (2): Bicubic interpolation - smoothest, most compute
Upscaled tensor with dimensions multiplied by scale_factor
ctx <- ggml_init(16 * 1024 * 1024) img <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 8) ggml_set_f32(img, rnorm(64)) # Nearest neighbor (fastest, pixelated) up_nearest <- ggml_upscale(ctx, img, 2, GGML_SCALE_MODE_NEAREST) # Bilinear (smooth) up_bilinear <- ggml_upscale(ctx, img, 2, GGML_SCALE_MODE_BILINEAR) # Bicubic (smoothest) up_bicubic <- ggml_upscale(ctx, img, 2, GGML_SCALE_MODE_BICUBIC) graph <- ggml_build_forward_expand(ctx, up_nearest) ggml_graph_compute(ctx, graph) # Result is 16x16 ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) img <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 8) ggml_set_f32(img, rnorm(64)) # Nearest neighbor (fastest, pixelated) up_nearest <- ggml_upscale(ctx, img, 2, GGML_SCALE_MODE_NEAREST) # Bilinear (smooth) up_bilinear <- ggml_upscale(ctx, img, 2, GGML_SCALE_MODE_BILINEAR) # Bicubic (smoothest) up_bicubic <- ggml_upscale(ctx, img, 2, GGML_SCALE_MODE_BICUBIC) graph <- ggml_build_forward_expand(ctx, up_nearest) ggml_graph_compute(ctx, graph) # Result is 16x16 ggml_free(ctx)
Returns the amount of memory currently used in the context
ggml_used_mem(ctx)ggml_used_mem(ctx)
ctx |
GGML context |
Used memory in bytes
ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) ggml_used_mem(ctx) ggml_free(ctx)ctx <- ggml_init(1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) ggml_used_mem(ctx) ggml_free(ctx)
Get GGML version
ggml_version()ggml_version()
Character string with GGML version
ggml_version()ggml_version()
Creates a 1D view of a tensor starting at a byte offset. The view shares memory with the source tensor.
ggml_view_1d(ctx, a, ne0, offset = 0)ggml_view_1d(ctx, a, ne0, offset = 0)
ctx |
GGML context |
a |
Source tensor |
ne0 |
Number of elements in the view |
offset |
Byte offset from the start of tensor data |
View tensor
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) # View elements 10-19 (offset = 10 * 4 bytes = 40) v <- ggml_view_1d(ctx, a, 10, 40) ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 100) # View elements 10-19 (offset = 10 * 4 bytes = 40) v <- ggml_view_1d(ctx, a, 10, 40) ggml_free(ctx)
Creates a 2D view of a tensor starting at a byte offset. The view shares memory with the source tensor.
ggml_view_2d(ctx, a, ne0, ne1, nb1, offset = 0)ggml_view_2d(ctx, a, ne0, ne1, nb1, offset = 0)
ctx |
GGML context |
a |
Source tensor |
ne0 |
Size of dimension 0 |
ne1 |
Size of dimension 1 |
nb1 |
Stride for dimension 1 (in bytes) |
offset |
Byte offset from the start of tensor data |
View tensor
Creates a 3D view of a tensor starting at a byte offset. The view shares memory with the source tensor.
ggml_view_3d(ctx, a, ne0, ne1, ne2, nb1, nb2, offset = 0)ggml_view_3d(ctx, a, ne0, ne1, ne2, nb1, nb2, offset = 0)
ctx |
GGML context |
a |
Source tensor |
ne0 |
Size of dimension 0 |
ne1 |
Size of dimension 1 |
ne2 |
Size of dimension 2 |
nb1 |
Stride for dimension 1 (in bytes) |
nb2 |
Stride for dimension 2 (in bytes) |
offset |
Byte offset from the start of tensor data |
View tensor
Creates a 4D view of a tensor starting at a byte offset. The view shares memory with the source tensor. CRITICAL for KV-cache operations in transformers.
ggml_view_4d(ctx, a, ne0, ne1, ne2, ne3, nb1, nb2, nb3, offset = 0)ggml_view_4d(ctx, a, ne0, ne1, ne2, ne3, nb1, nb2, nb3, offset = 0)
ctx |
GGML context |
a |
Source tensor |
ne0 |
Size of dimension 0 |
ne1 |
Size of dimension 1 |
ne2 |
Size of dimension 2 |
ne3 |
Size of dimension 3 |
nb1 |
Stride for dimension 1 (in bytes) |
nb2 |
Stride for dimension 2 (in bytes) |
nb3 |
Stride for dimension 3 (in bytes) |
offset |
Byte offset from the start of tensor data |
View tensor
Creates a view of the tensor (shares data, no copy)
ggml_view_tensor(ctx, src)ggml_view_tensor(ctx, src)
ctx |
GGML context |
src |
Source tensor |
View tensor (shares data with src)
ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) view <- ggml_view_tensor(ctx, a) # view shares data with a ggml_free(ctx)ctx <- ggml_init(16 * 1024 * 1024) a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) view <- ggml_view_tensor(ctx, a) # view shares data with a ggml_free(ctx)
Returns TRUE if the package was compiled with Vulkan support. To enable Vulkan, install libvulkan-dev and glslc, then reinstall ggmlR.
ggml_vulkan_available()ggml_vulkan_available()
Logical indicating if Vulkan is available
ggml_vulkan_available()ggml_vulkan_available()
Returns the name of the Vulkan backend (includes device info).
ggml_vulkan_backend_name(backend)ggml_vulkan_backend_name(backend)
backend |
Vulkan backend pointer |
Character string with backend name
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { backend <- ggml_vulkan_init(0) print(ggml_vulkan_backend_name(backend)) ggml_vulkan_free(backend) }if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { backend <- ggml_vulkan_init(0) print(ggml_vulkan_backend_name(backend)) ggml_vulkan_free(backend) }
Returns hardware capabilities for the specified Vulkan device.
ggml_vulkan_device_caps(device = 0L)ggml_vulkan_device_caps(device = 0L)
device |
Device index (0-based, default 0) |
Named list: coopmat_support, coopmat1_fa_support, fp16, subgroup_size, subgroup_no_shmem
Returns the number of available Vulkan-capable GPU devices.
ggml_vulkan_device_count()ggml_vulkan_device_count()
Integer count of Vulkan devices (0 if Vulkan not available)
if (ggml_vulkan_available()) { ggml_vulkan_device_count() }if (ggml_vulkan_available()) { ggml_vulkan_device_count() }
Returns a human-readable description of the specified Vulkan device.
ggml_vulkan_device_description(device = 0L)ggml_vulkan_device_description(device = 0L)
device |
Device index (0-based) |
Character string with device description
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { ggml_vulkan_device_description(0) }if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { ggml_vulkan_device_description(0) }
Enumerates Vulkan device groups (VK_KHR_device_group, also known as
Linked Device Adapter / LDA) and probes whether the driver reports direct
peer memory access between the physical GPUs in each group.
ggml_vulkan_device_groups()ggml_vulkan_device_groups()
A device group with more than one device and peer copy/generic memory features is the prerequisite for true GPU-to-GPU transfers routed over NVLink (or PCIe peer-to-peer) through a single device-group logical device — as opposed to sharing memory as an opaque fd between independent devices, which the driver may route through host memory.
This is a diagnostic only: it does not create any long-lived device group. On a machine with a single GPU it reports zero multi-device groups. Device groups for compute are effectively an NVIDIA feature; AMD/RADV typically reports only single-device groups.
A named list with n_groups (integer, number of device groups
reported by the driver) and report (character, a human-readable
per-group diagnostic including peer memory features). Use cat() on
report to read it.
if (ggml_vulkan_available()) { g <- ggml_vulkan_device_groups() cat(g$report) }if (ggml_vulkan_available()) { g <- ggml_vulkan_device_groups() cat(g$report) }
Returns free and total memory for the specified Vulkan device.
ggml_vulkan_device_memory(device = 0L)ggml_vulkan_device_memory(device = 0L)
device |
Device index (0-based) |
Named list with 'free' and 'total' memory in bytes
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { mem <- ggml_vulkan_device_memory(0) cat("Free:", mem$free / 1e9, "GB\n") cat("Total:", mem$total / 1e9, "GB\n") }if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { mem <- ggml_vulkan_device_memory(0) cat("Free:", mem$free / 1e9, "GB\n") cat("Total:", mem$total / 1e9, "GB\n") }
Releases resources associated with the Vulkan backend.
ggml_vulkan_free(backend)ggml_vulkan_free(backend)
backend |
Vulkan backend pointer from ggml_vulkan_init() |
NULL (invisible)
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { backend <- ggml_vulkan_init(0) ggml_vulkan_free(backend) }if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { backend <- ggml_vulkan_init(0) ggml_vulkan_free(backend) }
Reports whether this build of ggmlR was compiled with
-DGGML_VK_HARD_EXIT, which is what makes
ggml_vulkan_shutdown(hard = TRUE) actually call _exit().
ggml_vulkan_hard_exit_available()ggml_vulkan_hard_exit_available()
The hard-exit path is disabled by default: CRAN Repository Policy
forbids a package from terminating the user's R session, so the released
package must not link _exit(). Builds from source can opt in with
R CMD INSTALL . --configure-args="--enable-hard-exit" (on Windows, set
Sys.setenv(GGML_VK_HARD_EXIT = "1") before installing, because R there
ignores configure.args).
When it is not compiled in, ggml_vulkan_shutdown(hard = TRUE) performs
the normal teardown and emits a warning rather than silently
ignoring the request.
TRUE if the hard-exit path is available, otherwise FALSE.
ggml_vulkan_hard_exit_available()ggml_vulkan_hard_exit_available()
Creates a Vulkan backend for the specified device. The backend must be freed with ggml_vulkan_free() when done.
ggml_vulkan_init(device = 0L)ggml_vulkan_init(device = 0L)
device |
Device index (0-based, default 0) |
Vulkan backend pointer
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { backend <- ggml_vulkan_init(0) print(ggml_vulkan_backend_name(backend)) ggml_vulkan_free(backend) }if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { backend <- ggml_vulkan_init(0) print(ggml_vulkan_backend_name(backend)) ggml_vulkan_free(backend) }
Returns TRUE if the given backend is a Vulkan backend.
ggml_vulkan_is_backend(backend)ggml_vulkan_is_backend(backend)
backend |
Backend pointer |
Logical indicating if backend is Vulkan
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { vk_backend <- ggml_vulkan_init(0) cpu_backend <- ggml_backend_cpu_init() ggml_vulkan_is_backend(vk_backend) # TRUE ggml_vulkan_is_backend(cpu_backend) # FALSE ggml_vulkan_free(vk_backend) ggml_backend_free(cpu_backend) }if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { vk_backend <- ggml_vulkan_init(0) cpu_backend <- ggml_backend_cpu_init() ggml_vulkan_is_backend(vk_backend) # TRUE ggml_vulkan_is_backend(cpu_backend) # FALSE ggml_vulkan_free(vk_backend) ggml_backend_free(cpu_backend) }
Returns detailed information about all available Vulkan devices.
ggml_vulkan_list_devices()ggml_vulkan_list_devices()
List of device information (index, name, memory)
if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { devices <- ggml_vulkan_list_devices() print(devices) }if (ggml_vulkan_available() && ggml_vulkan_device_count() > 0) { devices <- ggml_vulkan_list_devices() print(devices) }
Exercises the VK_KHR_external_memory_fd transport used by Vulkan tensor
parallelism to move data between GPUs. A byte pattern is written on the source
device, its memory is exported as an opaque fd and imported on the destination
device, copied into a local buffer there, then read back and compared.
ggml_vulkan_p2p_selftest( src_device, dst_device, bytes = 64L * 1024L * 1024L, iters = 50L, transport = c("host-staging", "opaque-fd", "device-group") )ggml_vulkan_p2p_selftest( src_device, dst_device, bytes = 64L * 1024L * 1024L, iters = 50L, transport = c("host-staging", "opaque-fd", "device-group") )
src_device |
Source GPU index (0-based). |
dst_device |
Destination GPU index (0-based). Equal to |
bytes |
Transfer size in bytes (default 64 MiB). |
iters |
Number of copies to time for the bandwidth measurement (default 50). |
transport |
Cross-device transport to exercise: |
When src_device == dst_device the test runs in loopback mode
(export and import on the same GPU) — a sanity check of the fd mechanism that
touches no inter-device link. When the devices differ it runs
cross-device: after verifying correctness it times iters
device-to-device copies and reports the achieved bandwidth.
Interpreting bandwidth: a measured rate above the PCIe 3.0 x16 ceiling (~16 GB/s) is empirical evidence that a faster physical link (e.g. NVLink) carried the bytes. It is not a claim that Vulkan used an NVLink API — Vulkan exposes no call to query the route, so the conclusion is inferred from the rate alone, never asserted from the API.
A named list: status (integer, 0 = data verified, <0 = failure),
gbps (numeric, measured cross-device bandwidth; 0 for loopback or on
failure) and report (character diagnostic, incl. the NVLink-vs-PCIe
inference).
if (ggml_vulkan_available()) { # loopback sanity check on device 0 r <- ggml_vulkan_p2p_selftest(0L, 0L) cat(r$report) # cross-device P2P (requires >= 2 GPUs) if (ggml_vulkan_device_count() >= 2) { r <- ggml_vulkan_p2p_selftest(0L, 1L) cat(r$report) } }if (ggml_vulkan_available()) { # loopback sanity check on device 0 r <- ggml_vulkan_p2p_selftest(0L, 0L) cat(r$report) # cross-device P2P (requires >= 2 GPUs) if (ggml_vulkan_device_count() >= 2) { r <- ggml_vulkan_p2p_selftest(0L, 1L) cat(r$report) } }
Waits for all Vulkan devices to go idle, then destroys the devices and the Vulkan instance. Call this at the end of a script that used multiple GPUs (tensor / pipeline parallelism), before the process exits.
ggml_vulkan_shutdown(hard = FALSE, status = 0L)ggml_vulkan_shutdown(hard = FALSE, status = 0L)
hard |
Logical. If |
status |
Integer process exit code used only when |
Why this exists: with several Vulkan devices active, R's own teardown at
process exit can run after the Vulkan loader / Mesa ICD libraries have
been unmapped, so the device destructors call into unmapped memory and the
process crashes with a segfault after your results were already printed.
Calling ggml_vulkan_shutdown() while the process is still alive runs the
teardown while the loader is still mapped, avoiding that crash. It is only
needed for exit cleanliness — your computed results are unaffected either way.
Idempotent and a no-op if Vulkan was never initialized. After this call the next Vulkan operation transparently re-initializes the instance from scratch, so it is safe to call mid-session too.
The plain teardown alone does not always win the exit-time race: with backends
created inside pipeline / tensor-parallel forwards, the loader-static-destruction
segfault (address ending f1ba0) can still fire flakily after your
results are printed, because no R exit hook runs before R unmaps the loader.
For a guaranteed-clean exit, pass hard = TRUE as the last
statement of a standalone script: after teardown it calls _exit(0),
terminating the process immediately without running R's / the C runtime's exit
handlers or unmapping any shared library — so there is no loader teardown phase
left to crash. Because it bypasses R's normal shutdown, only use
hard = TRUE at the very end of a script, never mid-session or in a
package/interactive context.
Invisibly NULL. Does not return when hard = TRUE and
the hard-exit path is compiled in (see the section above).
The _exit() path is compiled out by default, because CRAN
Repository Policy forbids a package from terminating the user's R session. In
such a build hard = TRUE performs the normal teardown and emits a
warning — it is never silently ignored — so the exit-time race
described above can still fire. To compile it in, build from source with
R CMD INSTALL . --configure-args="--enable-hard-exit" (on Windows set
Sys.setenv(GGML_VK_HARD_EXIT = "1") first, since R ignores
configure.args there). Check the current build with
ggml_vulkan_hard_exit_available().
ggml_vulkan_hard_exit_available
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) { W <- matrix(rnorm(2048 * 64), nrow = 2048) X <- matrix(rnorm(4 * 64), nrow = 4) Y <- ggml_vulkan_split_mul_mat(W, X, n_devices = 2) ggml_vulkan_shutdown() # clean up before the script exits }if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) { W <- matrix(rnorm(2048 * 64), nrow = 2048) X <- matrix(rnorm(4 * 64), nrow = 4) Y <- ggml_vulkan_split_mul_mat(W, X, n_devices = 2) ggml_vulkan_shutdown() # clean up before the script exits }
Builds (or fetches from a cache) a ggml buffer type that row-splits a
weight tensor across n_devices GPUs for tensor parallelism. This is the
factory the graph allocator talks to (Stage E4): a weight placed in this buffer
type has its rows distributed across the devices, each holding a contiguous
band (see ggml_vulkan_split_row_ranges). Non-split tensors fall
back to main_device.
ggml_vulkan_split_buffer_type( n_devices = ggml_vulkan_device_count(), main_device = 0L, weights = NULL, device_ids = NULL, transport = c("host-staging", "opaque-fd", "device-group"), probe = NULL )ggml_vulkan_split_buffer_type( n_devices = ggml_vulkan_device_count(), main_device = 0L, weights = NULL, device_ids = NULL, transport = c("host-staging", "opaque-fd", "device-group"), probe = NULL )
n_devices |
Number of GPUs to split across (default: all available).
Ignored if |
main_device |
Device index (0-based) holding non-split fallbacks (default 0). |
weights |
Optional numeric vector of length |
device_ids |
Optional integer vector of physical GPU indices (0-based) the
split slots map onto, e.g. |
transport |
Cross-device gather transport for the eventual compute:
|
probe |
A numeric vector |
The buffer type is cached in C and lives for the rest of the session; the returned external pointer has no finalizer and must not be freed.
A named list: ptr (external pointer to the cached buffer type),
name (its config-derived name) and alloc_size (sum of the
padded per-device slice sizes for probe, in bytes; 0 if no probe).
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) { bt <- ggml_vulkan_split_buffer_type(n_devices = 2, probe = c(2048, 64)) bt$name bt$alloc_size # total bytes for a 2048x64 f32 weight across 2 devices }if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) { bt <- ggml_vulkan_split_buffer_type(n_devices = 2, probe = c(2048, 64)) bt$name bt$alloc_size # total bytes for a 2048x64 f32 weight across 2 devices }
Computes Y = X %*% t(W) with the rows of the weight matrix W
split across n_devices GPUs (true tensor parallelism), the activations
X broadcast to every device, and the resulting column slices gathered
back into a single matrix. This is Stage E3 of the Vulkan tensor-parallelism
work: it exercises the row-split + per-device compute + cross-device gather
path end to end on real hardware.
ggml_vulkan_split_mul_mat( W, X, n_devices = ggml_vulkan_device_count(), weights = NULL, device_ids = NULL, transport = c("host-staging", "opaque-fd", "device-group") )ggml_vulkan_split_mul_mat( W, X, n_devices = ggml_vulkan_device_count(), weights = NULL, device_ids = NULL, transport = c("host-staging", "opaque-fd", "device-group") )
W |
Weight matrix, |
X |
Activation matrix, |
n_devices |
Number of GPUs to split across (default: all available).
Ignored if |
weights |
Optional numeric vector of length |
device_ids |
Optional integer vector of physical GPU indices (0-based) to
split across, e.g. |
transport |
Cross-device gather transport: |
Each device owns a contiguous band of W's rows (via the same row-split
math as ggml_vulkan_split_row_ranges) and produces the matching
band of Y's columns; the bands are gathered through the transport (host
staging by default). This is a validation / correctness entry point, not a hot
inference path — it stands up one backend per device per call and round-trips
through host memory.
The M x N result matrix, equal to X %*% t(W) up to the
GPU's floating-point accumulation. Errors if the split compute fails; the
C-level diagnostic is attached as attribute "report".
if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) { W <- matrix(rnorm(8 * 4), nrow = 8) # 8 output features x 4 inputs X <- matrix(rnorm(3 * 4), nrow = 3) # 3 samples x 4 inputs Y <- ggml_vulkan_split_mul_mat(W, X, n_devices = 2) max(abs(Y - X %*% t(W))) # ~1e-3 (GPU f16 accumulation) }if (ggml_vulkan_available() && ggml_vulkan_device_count() >= 2) { W <- matrix(rnorm(8 * 4), nrow = 8) # 8 output features x 4 inputs X <- matrix(rnorm(3 * 4), nrow = 3) # 3 samples x 4 inputs Y <- ggml_vulkan_split_mul_mat(W, X, n_devices = 2) max(abs(Y - X %*% t(W))) # ~1e-3 (GPU f16 accumulation) }
Pure arithmetic helper for the Vulkan tensor-parallel split buffer type: given
a number of tensor rows and a per-device weight vector, returns the half-open
row range [row_low, row_high) owned by each device. Row boundaries are
rounded down to a fixed granularity, and the last device always covers up to
nrows, so the ranges are contiguous, non-overlapping and cover every
row exactly once.
ggml_vulkan_split_row_ranges(nrows, n_devices, weights = NULL)ggml_vulkan_split_row_ranges(nrows, n_devices, weights = NULL)
nrows |
Number of rows in the tensor (integer). |
n_devices |
Number of devices to split across (integer, >= 1). |
weights |
Optional numeric vector of length |
This touches no GPU and is exposed mainly to verify the split logic. It is the math behind row-split tensor parallelism (a weight matrix distributed across several GPUs), independent of any actual multi-GPU allocation.
A named list with row_low and row_high: numeric vectors
of length n_devices holding 0-based, half-open row ranges.
if (ggml_vulkan_available()) { # 4096 rows across 2 devices, evenly ggml_vulkan_split_row_ranges(4096L, 2L) # weighted 3:1 ggml_vulkan_split_row_ranges(4096L, 2L, weights = c(3, 1)) }if (ggml_vulkan_available()) { # 4096 rows across 2 devices, evenly ggml_vulkan_split_row_ranges(4096L, 2L) # weighted 3:1 ggml_vulkan_split_row_ranges(4096L, 2L, weights = c(3, 1)) }
Prints information about Vulkan availability and devices, and returns the same information as a list so it can be used programmatically.
ggml_vulkan_status()ggml_vulkan_status()
Invisibly, a list with components:
Logical, whether Vulkan support is compiled in and usable
Integer, number of Vulkan devices (0 if unavailable)
List of per-device property lists as returned by
ggml_vulkan_list_devices; an empty list if there are none
Status is also printed to the console.
ggml_vulkan_status() st <- ggml_vulkan_status() if (st$available && st$n_devices > 0) { print(st$devices[[1]]$name) }ggml_vulkan_status() st <- ggml_vulkan_status() if (st$available && st$n_devices > 0) { print(st$devices[[1]]$name) }
Partitions a tensor into non-overlapping windows of size w.
ggml_win_part(ctx, a, w)ggml_win_part(ctx, a, w)
ctx |
GGML context |
a |
Input tensor |
w |
Window size |
Partitioned tensor
Reassembles windowed partitions produced by ggml_win_part.
ggml_win_unpart(ctx, a, w0, h0, w)ggml_win_unpart(ctx, a, w0, h0, w)
ctx |
GGML context |
a |
Input tensor |
w0 |
Original width |
h0 |
Original height |
w |
Window size |
Un-partitioned tensor
Creates a temporary context, executes code, and frees it automatically. Useful when you need to create large temporary tensors.
ggml_with_temp_ctx(mem_size, expr)ggml_with_temp_ctx(mem_size, expr)
mem_size |
Context memory size in bytes |
expr |
Expression to evaluate with the temporary context |
Result of the expression
# Create tensors in temporary context result <- ggml_with_temp_ctx(1024 * 1024, { a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_set_f32(a, 1:10) ggml_get_f32(a) })# Create tensors in temporary context result <- ggml_with_temp_ctx(1024 * 1024, { a <- ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 10) ggml_set_f32(a, 1:10) ggml_get_f32(a) })
Explicitly frees the internal GGUF context. Called automatically by the garbage collector, but can be called manually to release memory sooner.
gguf_free(x)gguf_free(x)
x |
A |
Called for its side effect (releases the GGUF context); invisibly returns NULL.
Opens a GGUF file and reads its metadata. By default also reads tensor data
into memory; with meta_only = TRUE only the header and key-value
metadata are read (no tensor data is allocated), which is cheap and enough
for inspecting architecture / type fields. Returns an S3 object of class
"gguf" wrapping the internal pointer.
gguf_load(path, meta_only = FALSE)gguf_load(path, meta_only = FALSE)
path |
Path to a .gguf file. |
meta_only |
If |
An object of class "gguf".
Returns all key-value metadata pairs from a GGUF file as a named list.
gguf_metadata(x)gguf_metadata(x)
x |
A |
A named list of metadata values.
Dequantizes (if needed) and returns tensor weights as an R numeric array with dimensions matching the tensor shape.
gguf_tensor_data(x, name)gguf_tensor_data(x, name)
x |
A |
name |
Tensor name (character). |
A numeric array.
Returns name, shape, type, and size in bytes for a single tensor.
gguf_tensor_info(x, name)gguf_tensor_info(x, name)
x |
A |
name |
Tensor name (character). |
When the file was opened with meta_only = TRUE, the per-dimension
shape is NA (the public GGUF API does not expose tensor
dimensions without allocating tensors); name, type and
size_bytes are still returned.
A list with elements name, shape, type,
size_bytes.
List Tensor Names in a GGUF File
gguf_tensor_names(x)gguf_tensor_names(x)
x |
A |
Character vector of tensor names.
Returns a single-row [tibble][tibble::tibble] summarising the fitted model, in broom 'glance()' style.
## S3 method for class 'ggmlr_parsnip_model' glance(x, ...)## S3 method for class 'ggmlr_parsnip_model' glance(x, ...)
x |
A fitted 'ggmlr_parsnip_model' (the engine object inside a parsnip fit; e.g. from 'extract_fit_engine()'). |
... |
Unused; for generic compatibility. |
A one-row tibble with columns: 'mode', 'n_features', 'n_layers', 'total_params', 'optimizer', 'loss', 'backend', 'epochs', 'fit_time' (wall seconds) and 'final_loss' (last training loss, 'NA' if no history).
ggml_set_n_threads(1L) # deterministic, single OpenMP pool spec <- parsnip::mlp(hidden_units = 8L, epochs = 3L) |> parsnip::set_engine("ggml", backend = "cpu") |> parsnip::set_mode("regression") fit_obj <- parsnip::fit(spec, mpg ~ ., data = mtcars) generics::glance(parsnip::extract_fit_engine(fit_obj))ggml_set_n_threads(1L) # deterministic, single OpenMP pool spec <- parsnip::mlp(hidden_units = 8L, epochs = 3L) |> parsnip::set_engine("ggml", backend = "cpu") |> parsnip::set_mode("regression") fit_obj <- parsnip::fit(spec, mpg ~ ., data = mtcars) generics::glance(parsnip::extract_fit_engine(fit_obj))
Frees lookup tables for IQ2 quantization types.
iq2xs_free_impl(type)iq2xs_free_impl(type)
type |
GGML type constant |
NULL invisibly
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Initializes lookup tables for IQ2 quantization types. Must be called before using iq2_xxs, iq2_xs, or iq2_s quantization.
iq2xs_init_impl(type)iq2xs_init_impl(type)
type |
GGML type constant (e.g., GGML_TYPE_IQ2_XXS()) |
NULL invisibly
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Frees lookup tables for IQ3 quantization types.
iq3xs_free_impl(grid_size)iq3xs_free_impl(grid_size)
grid_size |
Grid size for IQ3 |
NULL invisibly
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Initializes lookup tables for IQ3 quantization types. Must be called before using iq3_xxs or iq3_s quantization.
iq3xs_init_impl(grid_size)iq3xs_init_impl(grid_size)
grid_size |
Grid size for IQ3 (typically 256) |
NULL invisibly
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Varies the learning rate following a cosine curve from lr_max down to
lr_min over T_max steps. Restarts (SGDR-style) if
restart = TRUE.
lr_scheduler_cosine(optimizer, T_max, lr_min = 0, restart = FALSE, T_mult = 1)lr_scheduler_cosine(optimizer, T_max, lr_min = 0, restart = FALSE, T_mult = 1)
optimizer |
Optimizer environment. |
T_max |
Number of steps for the first cosine cycle. |
lr_min |
Minimum learning rate (default 0). |
restart |
Logical; if |
T_mult |
Cycle-length multiplier applied after every restart (default
1, i.e. constant period). Only meaningful when |
With restart = TRUE and T_mult > 1 this is the full
CosineAnnealingWarmRestarts (SGDR) schedule: the first cycle lasts
T_max steps and every subsequent cycle is T_mult times longer
than the previous one. T_mult = 1 (the default) keeps every cycle the
same length.
An lr_scheduler_cosine environment
Loshchilov, I. & Hutter, F. (2017). SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR.
w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_adam(list(w = w), lr = 0.1) sch <- lr_scheduler_cosine(opt, T_max = 50L) for (epoch in 1:50) sch$step() # SGDR with doubling cycle length: 10, 20, 40, ... opt2 <- optimizer_adam(list(w = w), lr = 0.1) sch2 <- lr_scheduler_cosine(opt2, T_max = 10L, restart = TRUE, T_mult = 2) for (epoch in 1:70) sch2$step()w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_adam(list(w = w), lr = 0.1) sch <- lr_scheduler_cosine(opt, T_max = 50L) for (epoch in 1:50) sch$step() # SGDR with doubling cycle length: 10, 20, 40, ... opt2 <- optimizer_adam(list(w = w), lr = 0.1) sch2 <- lr_scheduler_cosine(opt2, T_max = 10L, restart = TRUE, T_mult = 2) for (epoch in 1:70) sch2$step()
Cycles the learning rate linearly between base_lr and max_lr
over a triangular cycle of 2 * step_size_up steps (Smith, 2017).
Each call to $step() advances one step — call it per batch,
not per epoch.
lr_scheduler_cyclic( optimizer, base_lr, max_lr, step_size_up = 2000L, step_size_down = NULL, mode = c("triangular", "triangular2", "exp_range"), gamma = 1 )lr_scheduler_cyclic( optimizer, base_lr, max_lr, step_size_up = 2000L, step_size_down = NULL, mode = c("triangular", "triangular2", "exp_range"), gamma = 1 )
optimizer |
Optimizer environment. |
base_lr |
Lower learning rate bound. |
max_lr |
Upper learning rate bound. |
step_size_up |
Steps in the increasing half of a cycle (default 2000). |
step_size_down |
Steps in the decreasing half. |
mode |
One of |
gamma |
Decay factor for |
An lr_scheduler_cyclic environment with $step() and
$get_lr().
Smith, L. N. (2017). Cyclical Learning Rates for Training Neural Networks. WACV.
w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_sgd(list(w = w), lr = 0.01) sch <- lr_scheduler_cyclic(opt, base_lr = 0.001, max_lr = 0.01, step_size_up = 10L) for (i in 1:40) sch$step()w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_sgd(list(w = w), lr = 0.01) sch <- lr_scheduler_cyclic(opt, base_lr = 0.001, max_lr = 0.01, step_size_up = 10L) for (i in 1:40) sch$step()
Implements the 1cycle policy (Smith, 2018): the learning rate rises from
max_lr / div_factor to max_lr during the first
pct_start fraction of training, then anneals down to
max_lr / final_div_factor. Call $step() per batch;
total_steps is the total number of batches over the whole run
(epochs * batches_per_epoch).
lr_scheduler_onecycle( optimizer, max_lr, total_steps, pct_start = 0.3, div_factor = 25, final_div_factor = 10000, anneal_strategy = c("cos", "linear"), cycle_momentum = TRUE, base_momentum = 0.85, max_momentum = 0.95 )lr_scheduler_onecycle( optimizer, max_lr, total_steps, pct_start = 0.3, div_factor = 25, final_div_factor = 10000, anneal_strategy = c("cos", "linear"), cycle_momentum = TRUE, base_momentum = 0.85, max_momentum = 0.95 )
optimizer |
Optimizer environment. |
max_lr |
Peak learning rate. |
total_steps |
Total number of |
pct_start |
Fraction of |
div_factor |
Initial LR is |
final_div_factor |
Final LR is |
anneal_strategy |
|
cycle_momentum |
Logical; cycle momentum inversely to LR (default
|
base_momentum, max_momentum
|
Momentum bounds used when
|
When cycle_momentum = TRUE the momentum-like hyper-parameter moves
inversely to the learning rate: $momentum for SGD, $beta1 for
Adam. For other optimizers momentum cycling is skipped.
An lr_scheduler_onecycle environment with $step() and
$get_lr().
Smith, L. N. (2018). A disciplined approach to neural network hyper-parameters. arXiv:1803.09820.
w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_sgd(list(w = w), lr = 0.01, momentum = 0.9) sch <- lr_scheduler_onecycle(opt, max_lr = 0.1, total_steps = 100L) for (i in 1:100) sch$step()w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_sgd(list(w = w), lr = 0.01, momentum = 0.9) sch <- lr_scheduler_onecycle(opt, max_lr = 0.1, total_steps = 100L) for (i in 1:100) sch$step()
Multiplies the optimizer learning rate by gamma every
step_size calls to $step().
lr_scheduler_step(optimizer, step_size, gamma = 0.1)lr_scheduler_step(optimizer, step_size, gamma = 0.1)
optimizer |
An |
step_size |
Decay every this many steps (epochs). |
gamma |
Multiplicative decay factor (default 0.1). |
An lr_scheduler_step environment
w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_adam(list(w = w), lr = 0.1) sch <- lr_scheduler_step(opt, step_size = 10L, gamma = 0.5) for (epoch in 1:30) sch$step() opt$lr # 0.1 * 0.5^3 = 0.0125w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_adam(list(w = w), lr = 0.1) sch <- lr_scheduler_step(opt, step_size = 10L, gamma = 0.5) for (epoch in 1:30) sch$step() opt$lr # 0.1 * 0.5^3 = 0.0125
Raises the learning rate linearly from warmup_start_lr to the
optimizer's initial lr over warmup_steps steps, then anneals it
to lr_min following a cosine curve over the remaining
total_steps - warmup_steps steps. This is the standard transformer
training schedule. Call $step() per batch.
lr_scheduler_warmup_cosine( optimizer, warmup_steps, total_steps, lr_min = 0, warmup_start_lr = 0 )lr_scheduler_warmup_cosine( optimizer, warmup_steps, total_steps, lr_min = 0, warmup_start_lr = 0 )
optimizer |
Optimizer environment. Its current |
warmup_steps |
Number of linear warmup steps. |
total_steps |
Total number of |
lr_min |
Final learning rate after annealing (default 0). |
warmup_start_lr |
Learning rate at step 0 (default 0). |
An lr_scheduler_warmup_cosine environment with $step()
and $get_lr().
w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_adam(list(w = w), lr = 1e-3) sch <- lr_scheduler_warmup_cosine(opt, warmup_steps = 10L, total_steps = 100L) for (i in 1:100) sch$step()w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_adam(list(w = w), lr = 1e-3) sch <- lr_scheduler_warmup_cosine(opt, warmup_steps = 10L, total_steps = 100L) for (i in 1:100) sch$step()
Topologically sort nodes reachable from output nodes
nn_topo_sort(outputs)nn_topo_sort(outputs)
outputs |
List of output ggml_tensor_node objects |
Named list: nodes in topological order (inputs first, outputs last)
Returns information about backend placement: which backends are available, how the scheduler splits the graph, and how many ops are supported by GPU vs CPU-only.
onnx_device_info(model)onnx_device_info(model)
model |
An |
A list with:
Character vector of backend names (e.g. "Vulkan0", "CPU")
Number of backends
Number of scheduler splits (1 = all on one backend)
Total graph nodes
Ops supported by GPU backend
Ops that can only run on CPU
Named integer vector: op type => count (empty if all on GPU)
The parsed graph as a table: one row per node, with its operator type and the names of its inputs and outputs. Use it to find which node produces a given edge, or what an edge feeds into, without re-parsing the file.
onnx_graph_info(model, producer_of = NULL, consumer_of = NULL)onnx_graph_info(model, producer_of = NULL, consumer_of = NULL)
model |
An |
producer_of |
Optional edge name. When given, returns the rows whose node lists that name among its outputs. |
consumer_of |
Optional edge name. When given, returns the rows whose node lists that name among its inputs. |
Reading the structure beats searching the .onnx binary for a name:
a name occurs both as a producer's output and as its consumers' input, and
telling those apart by hand means getting the protobuf field numbers right
every time.
A data frame with columns index, op_type,
output (the first output) and list-columns outputs and
inputs.
if (file.exists("model.onnx")) { m <- onnx_load("model.onnx") g <- onnx_graph_info(m) head(g) onnx_graph_info(m, producer_of = "881") }if (file.exists("model.onnx")) { m <- onnx_load("model.onnx") g <- onnx_graph_info(m) head(g) onnx_graph_info(m, producer_of = "881") }
Returns the names and shapes of model inputs (excluding weight
initializers). Use this to know what to pass to onnx_run().
onnx_inputs(model)onnx_inputs(model)
model |
An |
A named list where names are input tensor names and values are integer vectors of dimension sizes (-1 for dynamic dimensions).
Parses an .onnx file, builds a ggml computation graph, and allocates tensors on the specified device. Weights are loaded via memory-mapped file (zero-copy where possible).
onnx_load( path, device = NULL, input_shapes = NULL, n_threads = NULL, dtype = "f32" )onnx_load( path, device = NULL, input_shapes = NULL, n_threads = NULL, dtype = "f32" )
path |
Path to .onnx file. |
device |
Backend device: |
input_shapes |
Optional named list of integer vectors specifying
fixed shapes for inputs with dynamic dimensions. Names must match
input tensor names. Each shape must include all dimensions including
batch, e.g. |
n_threads |
Number of CPU threads. |
dtype |
Weight precision: |
An opaque model object (external pointer) for use with
onnx_run(), onnx_summary(), and onnx_inputs().
Run ONNX model inference
onnx_run(model, inputs)onnx_run(model, inputs)
model |
An |
inputs |
A named list of numeric vectors/matrices.
Names must match the model's input tensor names.
Use |
A named list of output tensors: numeric vectors carrying a
dim attribute when the output has more than one dimension.
The buffers are row-major, as ONNX stores them, so dim is given in
storage order – fastest-varying axis first – which is the reverse of the
shape the ONNX file declares. An output declared [nbox, 4] arrives
as dim = c(4, nbox), one detection per column. This keeps dim
an honest description of the memory, so ordinary R indexing is correct;
use predict.onnx_model to get arrays in the declared ONNX
order instead.
Returns metadata about a loaded ONNX model.
onnx_summary(model)onnx_summary(model)
model |
An |
A list with ir_version, opset_version,
producer, graph_name, n_nodes,
n_initializers, and ops.
Create an Adam optimizer
optimizer_adam( params, lr = 0.001, beta1 = 0.9, beta2 = 0.999, eps = 1e-08, accumulate_steps = 1L, average = TRUE )optimizer_adam( params, lr = 0.001, beta1 = 0.9, beta2 = 0.999, eps = 1e-08, accumulate_steps = 1L, average = TRUE )
params |
List of ag_param tensors, named or not (names are used only for display). The same tensor listed twice is kept once, with a warning. |
lr |
Learning rate (default 1e-3) |
beta1 |
First moment decay (default 0.9) |
beta2 |
Second moment decay (default 0.999) |
eps |
Stability constant (default 1e-8) |
accumulate_steps |
Number of |
average |
When accumulating, divide the accumulated gradient by
|
An optimizer environment with step(grads), zero_grad(),
state() and load_state(state). state() returns a
versioned list (moments as host double matrices, t,
accum_count); load_state() checks the type, parameter
count, names (when both sides have them) and shapes before writing
anything, restores t and accum_count, and does NOT restore
hyperparameters (it warns when they differ). Exact continuation holds on
the same device and dtype; CPU <-> GPU in f32 continues within float
tolerance; across f16 and f32 it cannot be exact. Only read saved states
you trust (readRDS).
w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_adam(list(w = w), lr = 1e-3)w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_adam(list(w = w), lr = 1e-3)
Create an SGD optimizer
optimizer_sgd( params, lr = 0.01, momentum = 0, accumulate_steps = 1L, average = TRUE )optimizer_sgd( params, lr = 0.01, momentum = 0, accumulate_steps = 1L, average = TRUE )
params |
List of ag_param tensors, named or not (names are used only for display). The same tensor listed twice is kept once, with a warning. |
lr |
Learning rate (default 0.01) |
momentum |
Momentum factor (default 0) |
accumulate_steps |
Number of |
average |
When accumulating, divide the accumulated gradient by
|
An optimizer environment with step(grads), zero_grad(),
state() and load_state(state). state() returns a
versioned list (moments as host double matrices, t,
accum_count); load_state() checks the type, parameter
count, names (when both sides have them) and shapes before writing
anything, restores t and accum_count, and does NOT restore
hyperparameters (it warns when they differ). Exact continuation holds on
the same device and dtype; CPU <-> GPU in f32 continues within float
tolerance; across f16 and f32 it cannot be exact. Only read saved states
you trust (readRDS).
w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_sgd(list(w = w), lr = 0.01) # accumulate 4 micro-batches per update opt4 <- optimizer_sgd(list(w = w), lr = 0.01, accumulate_steps = 4L)w <- ag_param(matrix(runif(4), 2, 2)) opt <- optimizer_sgd(list(w = w), lr = 0.01) # accumulate 4 micro-batches per update opt4 <- optimizer_sgd(list(w = w), lr = 0.01, accumulate_steps = 4L)
Plots loss and accuracy curves over epochs.
## S3 method for class 'ggml_history' plot(x, ...)## S3 method for class 'ggml_history' plot(x, ...)
x |
A ggml_history object |
... |
Additional arguments (ignored) |
The history object (invisibly).
Generates predictions from a trained model. Uses the standard R
predict generic for compatibility with keras3 and
the broader R ecosystem.
## S3 method for class 'ggml_sequential_model' predict(object, x, batch_size = NULL, ...) ## S3 method for class 'ggml_functional_model' predict(object, x, batch_size = NULL, ...)## S3 method for class 'ggml_sequential_model' predict(object, x, batch_size = NULL, ...) ## S3 method for class 'ggml_functional_model' predict(object, x, batch_size = NULL, ...)
object |
A trained model object. |
x |
Input data (matrix, array, or list for multi-input models). |
batch_size |
Batch size for inference. |
... |
Additional arguments (ignored). |
Matrix of predictions.
Runs inference over x, batching it to the fixed input shape the
model was loaded with. This is the keras-compatible entry point: together
with the ggml_sequential_model and ggml_functional_model
methods it gives ggmlR and ONNX models a single predict() interface.
## S3 method for class 'onnx_model' predict(object, x, batch_size = NULL, ...)## S3 method for class 'onnx_model' predict(object, x, batch_size = NULL, ...)
object |
An |
x |
Input data: an array or matrix whose first dimension indexes samples, or a named list of such arrays for multi-input models. |
batch_size |
Ignored, with a warning if not |
... |
Ignored. |
Unlike onnx_run, which takes a named list of inputs and always
returns a named list of outputs, this method takes an array (or a named list
for multi-input models) and unwraps a single output. The output keeps the
dimensions the model produced: a 2D output comes back as a matrix, a 4D
output stays 4D.
An ONNX graph is built for one fixed batch size at onnx_load
time and cannot be resized afterwards, so x is split into chunks of
that size. A trailing partial chunk is padded to a full batch and the
padding is dropped from the result.
For a single-output model, an array with the model's output
dimensions and nrow(x) samples in the first one (a matrix when the
output is 2D). For a multi-output model, a named list of such arrays.
onnx_run for the lower-level named-list interface.
Print method for ag_tensor
## S3 method for class 'ag_tensor' print(x, ...)## S3 method for class 'ag_tensor' print(x, ...)
x |
An |
... |
Ignored |
The input x, returned invisibly (called for its side effect of printing).
Print method for ggml_functional_model
## S3 method for class 'ggml_functional_model' print(x, ...)## S3 method for class 'ggml_functional_model' print(x, ...)
x |
A ggml_functional_model object |
... |
Additional arguments (ignored) |
The model object (invisibly).
Print method for ggml_history
## S3 method for class 'ggml_history' print(x, ...)## S3 method for class 'ggml_history' print(x, ...)
x |
A ggml_history object |
... |
Additional arguments (ignored) |
The history object (invisibly).
Prints a summary of the model architecture including layer types, output shapes, and parameter counts.
## S3 method for class 'ggml_sequential_model' print(x, ...)## S3 method for class 'ggml_sequential_model' print(x, ...)
x |
A ggml_sequential_model object |
... |
Additional arguments (ignored) |
The model object (invisibly).
Print ONNX model summary
## S3 method for class 'onnx_model' print(x, ...)## S3 method for class 'onnx_model' print(x, ...)
x |
An |
... |
Ignored. |
Invisibly returns x.
Which stickers make up each piece, for a cube written as 96 stickers in face order. Pieces come in three kinds and the model is told which is which: 8 corners of three stickers, 24 wings of two, 24 centres of one. The short pieces are padded to three slots and the padding masked off.
pt_cube4_layout()pt_cube4_layout()
Sticker numbers are 0-based, as the checkpoint's own layout table has them.
A list with positions (a 56 x 3 integer matrix of
sticker indices), mask (56 x 3 logical, which slots are
real), types (length-56 integer, 0 corner / 1 wing / 2 centre),
state_size and num_classes.
lay <- pt_cube4_layout() dim(lay$positions) table(lay$types)lay <- pt_cube4_layout() dim(lay$positions) table(lay$types)
Reads a state, or a batch of them, and returns a value per action.
pt_forward(model, state)pt_forward(model, state)
model |
A |
state |
Colours in |
For one state, a numeric vector of n_actions values, one per
move. For a batch, a matrix with one row per state and one column per
action.
A batch is not a convenience here, it is the way to get the work done quickly. Every layer but the attention treats a row on its own, so a batch of states is the same arithmetic on taller matrices — and a tall matmul is what BLAS is good at, where a short one spends its time in the call rather than the multiply. Scoring the ten states of a beam in one call is several times faster than scoring them one after another.
## Not run: model <- pt_transformer_load("model.pth") solved <- rep(0:5, each = 16) pt_forward(model, solved) # 24 values pt_forward(model, rbind(solved, solved)) # 2 x 24 ## End(Not run)## Not run: model <- pt_transformer_load("model.pth") solved <- rep(0:5, each = 16) pt_forward(model, solved) # 24 values pt_forward(model, rbind(solved, solved)) # 2 x 24 ## End(Not run)
Creates the model with its weights at their initial values. To run a trained
one, use pt_transformer_load, which builds this and then fills
it from a checkpoint.
pt_transformer( layout, d_model = 256L, n_heads = 8L, n_layers = 4L, ff_dim = 1024L, n_actions = 24L, activation = "relu" )pt_transformer( layout, d_model = 256L, n_heads = 8L, n_layers = 4L, ff_dim = 1024L, n_actions = 24L, activation = "relu" )
layout |
A layout list, as |
d_model |
Width of the model (default 256). |
n_heads |
Attention heads (default 8). |
n_layers |
Encoder blocks (default 4). |
ff_dim |
Width of the feed-forward hidden layer (default 1024). |
n_actions |
Size of the output — one value per action (default 24). |
activation |
Feed-forward activation, |
A pt_transformer object.
pt_transformer_load, pt_forward
Reads a .pth saved by torch.save() and returns the model ready
to run. Everything that differs between torch's conventions and this
package's is resolved here rather than at inference time.
pt_transformer_load(path, layout = pt_cube4_layout(), ...)pt_transformer_load(path, layout = pt_cube4_layout(), ...)
path |
Path to the |
layout |
A layout list; defaults to |
... |
Passed to |
A pt_transformer with the checkpoint's weights.
pth_catalogue to see what a checkpoint holds.
## Not run: model <- pt_transformer_load("model.pth") q <- pt_forward(model, rep(0:5, each = 16)) which.max(q) ## End(Not run)## Not run: model <- pt_transformer_load("model.pth") q <- pt_forward(model, rep(0:5, each = 16)) which.max(q) ## End(Not run)
Reads the metadata of a .pth file — the tensor names, shapes and
dtypes — without reading the tensors. A checkpoint is a zip holding one
pickle of metadata and one entry per tensor, so the catalogue is cheap
whatever the file weighs.
pth_catalogue(path)pth_catalogue(path)
path |
Path to a |
What comes back is the checkpoint's own state_dict order, which is
the order the model's modules were built in.
A data.frame with one row per tensor and the columns name,
shape (as "a x b"), dtype, n (element count),
bytes, and entry (the zip entry holding the data).
The attribute "file" carries the path.
## Not run: cat <- pth_catalogue("model.pth") head(cat) sum(cat$n) # total parameters ## End(Not run)## Not run: cat <- pth_catalogue("model.pth") head(cat) sum(cat$n) # total parameters ## End(Not run)
Reads a .pth file saved by torch.save() into plain R arrays.
No Python and no torch: a checkpoint is a zip of one metadata pickle and one
entry per tensor, and both are read here directly.
pth_load(path, names = NULL)pth_load(path, names = NULL)
path |
Path to a |
names |
Optional character vector of tensor names to read. The default
reads all of them. Use |
A named list of numeric arrays, in the checkpoint's own order.
Scalars come back as length-1 vectors; everything else carries a
dim. The attribute "file" carries the path.
PyTorch stores a tensor row-major and R stores an array column-major, so the
same bytes describe transposed arrays in the two languages. The tensors come
back with their dimensions reversed — a PyTorch [768, 256]
weight arrives as a 256 x 768 R matrix — which keeps the elements
where they belong without moving a single one. For a linear layer that is
the happy case: x %*% w is then the layer's own arithmetic, no
transpose needed.
pth_catalogue to list a checkpoint without reading it.
## Not run: w <- pth_load("model.pth") dim(w$output_layer.weight) # just the one tensor pth_load("model.pth", names = "output_layer.bias") ## End(Not run)## Not run: w <- pth_load("model.pth") dim(w$output_layer.weight) # just the one tensor pth_load("model.pth", names = "output_layer.bias") ## End(Not run)
Quantizes float data to IQ format. IQ formats require importance matrix initialization before use (see iq2xs_init_impl, iq3xs_init_impl).
quantize_iq2_xxs(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq2_xs(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq2_s(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq3_xxs(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq3_s(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq1_s(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq1_m(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq4_nl(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq4_xs(src_data, n_rows, n_per_row, imatrix = NULL)quantize_iq2_xxs(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq2_xs(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq2_s(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq3_xxs(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq3_s(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq1_s(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq1_m(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq4_nl(src_data, n_rows, n_per_row, imatrix = NULL) quantize_iq4_xs(src_data, n_rows, n_per_row, imatrix = NULL)
src_data |
Numeric vector of float values to quantize |
n_rows |
Number of rows |
n_per_row |
Number of elements per row |
imatrix |
Optional importance matrix (numeric vector or NULL) |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Quantizes float data to MXFP4 (microscaling FP4) format.
quantize_mxfp4(src_data, n_rows, n_per_row, imatrix = NULL)quantize_mxfp4(src_data, n_rows, n_per_row, imatrix = NULL)
src_data |
Numeric vector of float values to quantize |
n_rows |
Number of rows |
n_per_row |
Number of elements per row |
imatrix |
Optional importance matrix (numeric vector or NULL) |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Quantizes float data to NVFP4 format (NVIDIA FP4 with UE4M3 per-sub-block scale).
quantize_nvfp4(src_data, n_rows, n_per_row, imatrix = NULL)quantize_nvfp4(src_data, n_rows, n_per_row, imatrix = NULL)
src_data |
Numeric vector of float values to quantize |
n_rows |
Number of rows |
n_per_row |
Number of elements per row (must be multiple of 64) |
imatrix |
Optional importance matrix (currently ignored) |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Quantizes float data to Q1_0 format (1-bit-per-weight sign quantization).
quantize_q1_0(src_data, n_rows, n_per_row, imatrix = NULL)quantize_q1_0(src_data, n_rows, n_per_row, imatrix = NULL)
src_data |
Numeric vector of float values to quantize |
n_rows |
Number of rows |
n_per_row |
Number of elements per row (must be multiple of 128) |
imatrix |
Optional importance matrix (currently ignored) |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Quantizes float data to K-quant format with optional importance matrix. K-quants provide better quality/size tradeoffs than basic quants.
quantize_q2_K(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q3_K(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q4_K(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q5_K(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q6_K(src_data, n_rows, n_per_row, imatrix = NULL)quantize_q2_K(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q3_K(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q4_K(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q5_K(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q6_K(src_data, n_rows, n_per_row, imatrix = NULL)
src_data |
Numeric vector of float values to quantize |
n_rows |
Number of rows |
n_per_row |
Number of elements per row |
imatrix |
Optional importance matrix (numeric vector or NULL) |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Quantizes float data to Q4_0 format with optional importance matrix.
quantize_q4_0(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q4_1(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q5_0(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q5_1(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q8_0(src_data, n_rows, n_per_row, imatrix = NULL)quantize_q4_0(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q4_1(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q5_0(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q5_1(src_data, n_rows, n_per_row, imatrix = NULL) quantize_q8_0(src_data, n_rows, n_per_row, imatrix = NULL)
src_data |
Numeric vector of float values to quantize |
n_rows |
Number of rows |
n_per_row |
Number of elements per row |
imatrix |
Optional importance matrix (numeric vector or NULL) |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Basic row-level IQ quantization.
quantize_row_iq3_xxs_ref(src_data, n_elements) quantize_row_iq4_nl_ref(src_data, n_elements) quantize_row_iq4_xs_ref(src_data, n_elements) quantize_row_iq3_s_ref(src_data, n_elements) quantize_row_iq2_s_ref(src_data, n_elements)quantize_row_iq3_xxs_ref(src_data, n_elements) quantize_row_iq4_nl_ref(src_data, n_elements) quantize_row_iq4_xs_ref(src_data, n_elements) quantize_row_iq3_s_ref(src_data, n_elements) quantize_row_iq2_s_ref(src_data, n_elements)
src_data |
Numeric vector of float values to quantize |
n_elements |
Number of elements to quantize |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Basic row-level MXFP4 quantization.
quantize_row_mxfp4_ref(src_data, n_elements)quantize_row_mxfp4_ref(src_data, n_elements)
src_data |
Numeric vector of float values to quantize |
n_elements |
Number of elements to quantize |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Basic row-level NVFP4 quantization.
quantize_row_nvfp4_ref(src_data, n_elements)quantize_row_nvfp4_ref(src_data, n_elements)
src_data |
Numeric vector of float values to quantize |
n_elements |
Number of elements to quantize |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Basic row-level Q1_0 quantization.
quantize_row_q1_0_ref(src_data, n_elements)quantize_row_q1_0_ref(src_data, n_elements)
src_data |
Numeric vector of float values to quantize |
n_elements |
Number of elements to quantize |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Basic row-level K-quant quantization without importance matrix.
quantize_row_q2_K_ref(src_data, n_elements) quantize_row_q3_K_ref(src_data, n_elements) quantize_row_q4_K_ref(src_data, n_elements) quantize_row_q5_K_ref(src_data, n_elements) quantize_row_q6_K_ref(src_data, n_elements) quantize_row_q8_K_ref(src_data, n_elements)quantize_row_q2_K_ref(src_data, n_elements) quantize_row_q3_K_ref(src_data, n_elements) quantize_row_q4_K_ref(src_data, n_elements) quantize_row_q5_K_ref(src_data, n_elements) quantize_row_q6_K_ref(src_data, n_elements) quantize_row_q8_K_ref(src_data, n_elements)
src_data |
Numeric vector of float values to quantize |
n_elements |
Number of elements to quantize |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Basic row-level quantization without importance matrix. These are reference implementations.
quantize_row_q4_0_ref(src_data, n_elements) quantize_row_q4_1_ref(src_data, n_elements) quantize_row_q5_0_ref(src_data, n_elements) quantize_row_q5_1_ref(src_data, n_elements) quantize_row_q8_0_ref(src_data, n_elements) quantize_row_q8_1_ref(src_data, n_elements)quantize_row_q4_0_ref(src_data, n_elements) quantize_row_q4_1_ref(src_data, n_elements) quantize_row_q5_0_ref(src_data, n_elements) quantize_row_q5_1_ref(src_data, n_elements) quantize_row_q8_0_ref(src_data, n_elements) quantize_row_q8_1_ref(src_data, n_elements)
src_data |
Numeric vector of float values to quantize |
n_elements |
Number of elements to quantize |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_tq1_0_ref(),
quantize_tq1_0()
Basic row-level ternary quantization.
quantize_row_tq1_0_ref(src_data, n_elements) quantize_row_tq2_0_ref(src_data, n_elements)quantize_row_tq1_0_ref(src_data, n_elements) quantize_row_tq2_0_ref(src_data, n_elements)
src_data |
Numeric vector of float values to quantize |
n_elements |
Number of elements to quantize |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_tq1_0()
Quantizes float data to ternary format with optional importance matrix.
quantize_tq1_0(src_data, n_rows, n_per_row, imatrix = NULL) quantize_tq2_0(src_data, n_rows, n_per_row, imatrix = NULL)quantize_tq1_0(src_data, n_rows, n_per_row, imatrix = NULL) quantize_tq2_0(src_data, n_rows, n_per_row, imatrix = NULL)
src_data |
Numeric vector of float values to quantize |
n_rows |
Number of rows |
n_per_row |
Number of elements per row |
imatrix |
Optional importance matrix (numeric vector or NULL) |
Raw vector of quantized data
Other quantization:
dequantize_row_iq2_xxs(),
dequantize_row_mxfp4(),
dequantize_row_nvfp4(),
dequantize_row_q1_0(),
dequantize_row_q2_K(),
dequantize_row_q4_0(),
dequantize_row_tq1_0(),
ggml_quant_block_info(),
iq2xs_free_impl(),
iq2xs_init_impl(),
iq3xs_free_impl(),
iq3xs_init_impl(),
quantize_iq2_xxs(),
quantize_mxfp4(),
quantize_nvfp4(),
quantize_q1_0(),
quantize_q2_K(),
quantize_q4_0(),
quantize_row_iq3_xxs_ref(),
quantize_row_mxfp4_ref(),
quantize_row_nvfp4_ref(),
quantize_row_q1_0_ref(),
quantize_row_q2_K_ref(),
quantize_row_q4_0_ref(),
quantize_row_tq1_0_ref()
RoPE (Rotary Position Embedding) Type Constants
GGML_ROPE_TYPE_NORM GGML_ROPE_TYPE_NEOX GGML_ROPE_TYPE_MROPE GGML_ROPE_TYPE_VISIONGGML_ROPE_TYPE_NORM GGML_ROPE_TYPE_NEOX GGML_ROPE_TYPE_MROPE GGML_ROPE_TYPE_VISION
Integer constants
An object of class integer of length 1.
An object of class integer of length 1.
An object of class integer of length 1.
Constants for RoPE (Rotary Position Embedding) modes used in transformer models. Different models use different RoPE implementations.
GGML_ROPE_TYPE_NORM (0): Standard RoPE as in original paper (LLaMA, Mistral)
GGML_ROPE_TYPE_NEOX (2): GPT-NeoX style RoPE with different interleaving
GGML_ROPE_TYPE_MROPE (8): Multi-RoPE for multimodal models (Qwen2-VL)
GGML_ROPE_TYPE_VISION (24): Vision model RoPE variant
An integer constant representing a RoPE type
GGML_ROPE_TYPE_NORM # 0 - Standard RoPE (LLaMA, Mistral) GGML_ROPE_TYPE_NEOX # 2 - GPT-NeoX style GGML_ROPE_TYPE_MROPE # 8 - Multi-RoPE (Qwen2-VL) GGML_ROPE_TYPE_VISION # 24 - Vision modelsGGML_ROPE_TYPE_NORM # 0 - Standard RoPE (LLaMA, Mistral) GGML_ROPE_TYPE_NEOX # 2 - GPT-NeoX style GGML_ROPE_TYPE_MROPE # 8 - Multi-RoPE (Qwen2-VL) GGML_ROPE_TYPE_VISION # 24 - Vision models
High-level, Seurat-style entry point: extracts the expression matrix from the
object, runs the requested operation on the GGML backend (Vulkan GPU with CPU
fallback) and writes the result back into the object. Returns the updated
object, so it slots into a %>% / |> pipeline next to
Seurat::RunPCA(). Methods exist for Seurat and
SingleCellExperiment objects (and a .default for bare matrices).
## S3 method for class 'SingleCellExperiment' RunGGML( object, op = "embed", assay = NULL, layer = NULL, n_components = 50L, reduction_name = "ggml", device = "auto", genes = NULL, cells = NULL, reduction = NULL, dims = NULL, ... ) RunGGML( object, op = "embed", assay = NULL, layer = NULL, n_components = 50L, reduction_name = "ggml", device = "auto", genes = NULL, cells = NULL, reduction = NULL, dims = NULL, ... ) ## Default S3 method: RunGGML( object, op = "embed", assay = NULL, layer = NULL, n_components = 50L, reduction_name = "ggml", device = "auto", genes = NULL, cells = NULL, ... ) ## S3 method for class 'Seurat' RunGGML( object, op = "embed", assay = NULL, layer = NULL, n_components = 50L, reduction_name = "ggml", device = "auto", genes = NULL, cells = NULL, reduction = NULL, dims = NULL, ... )## S3 method for class 'SingleCellExperiment' RunGGML( object, op = "embed", assay = NULL, layer = NULL, n_components = 50L, reduction_name = "ggml", device = "auto", genes = NULL, cells = NULL, reduction = NULL, dims = NULL, ... ) RunGGML( object, op = "embed", assay = NULL, layer = NULL, n_components = 50L, reduction_name = "ggml", device = "auto", genes = NULL, cells = NULL, reduction = NULL, dims = NULL, ... ) ## Default S3 method: RunGGML( object, op = "embed", assay = NULL, layer = NULL, n_components = 50L, reduction_name = "ggml", device = "auto", genes = NULL, cells = NULL, ... ) ## S3 method for class 'Seurat' RunGGML( object, op = "embed", assay = NULL, layer = NULL, n_components = 50L, reduction_name = "ggml", device = "auto", genes = NULL, cells = NULL, reduction = NULL, dims = NULL, ... )
object |
A |
op |
Operation name; see |
assay |
Assay to read (Seurat); defaults to the object's default assay. |
layer |
Layer/slot to read; default |
n_components |
Number of components for |
reduction_name |
Name of the reduction slot to create. Default
|
device |
|
genes, cells
|
Optional feature/cell subsets passed to extraction. |
reduction |
Optional name of an existing reduction to use as the input
feature space (e.g. |
dims |
Optional integer vector selecting columns of |
... |
Additional parameters forwarded to the engine. A notable one is
|
Supported operations include "embed" (PCA), "normalize",
"scale", "umap", "neighbors" and "largest_gene"
(per-cell highest-expressed gene QC metric); see
ggml_ops_registry.
For a Seurat object, the updated object with a new reduction. For a
bare matrix, a ggml_result.
## Not run: library(Seurat) pbmc <- RunGGML(pbmc, op = "embed", n_components = 30) DimPlot(pbmc, reduction = "ggml") ## End(Not run)## Not run: library(Seurat) pbmc <- RunGGML(pbmc, op = "embed", n_components = 30) DimPlot(pbmc, reduction = "ggml") ## End(Not run)
Prints a detailed summary including input shape, layer details, trainable/non-trainable parameter counts, and memory estimate.
## S3 method for class 'ggml_sequential_model' summary(object, ...)## S3 method for class 'ggml_sequential_model' summary(object, ...)
object |
A ggml_sequential_model object |
... |
Additional arguments (ignored) |
The model object (invisibly).
Returns one row per layer of the underlying sequential network, in broom style. Useful for comparing architectures across experiments in a R Markdown / Quarto report.
## S3 method for class 'ggmlr_parsnip_model' tidy(x, ...)## S3 method for class 'ggmlr_parsnip_model' tidy(x, ...)
x |
A fitted 'ggmlr_parsnip_model' (the engine object inside a parsnip fit; e.g. from 'extract_fit_engine()'). |
... |
Unused; for generic compatibility. |
A [tibble][tibble::tibble] with columns: 'layer' (name), 'type', 'units' (output units, 'NA' if not applicable), 'activation', 'output_shape' (character), 'params' (trainable parameter count) and 'trainable' (logical).
ggml_set_n_threads(1L) # deterministic, single OpenMP pool spec <- parsnip::mlp(hidden_units = 8L, epochs = 3L) |> parsnip::set_engine("ggml", backend = "cpu") |> parsnip::set_mode("regression") fit_obj <- parsnip::fit(spec, mpg ~ ., data = mtcars) generics::tidy(parsnip::extract_fit_engine(fit_obj))ggml_set_n_threads(1L) # deterministic, single OpenMP pool spec <- parsnip::mlp(hidden_units = 8L, epochs = 3L) |> parsnip::set_engine("ggml", backend = "cpu") |> parsnip::set_mode("regression") fit_obj <- parsnip::fit(spec, mpg ~ ., data = mtcars) generics::tidy(parsnip::extract_fit_engine(fit_obj))
Records all ag_* operations inside expr for later backward().
When the default device is "gpu", the ggml context is reset at the
start of each tape.
with_grad_tape(expr)with_grad_tape(expr)
expr |
Expression to evaluate under gradient tape |
Value of last expression in expr (invisibly)
The reset frees every intermediate result of earlier ag_* calls; only
ag_param weights (and tensors made from host matrices, which
keep a host copy) survive it. An input built from ag_* operations on
the GPU – an observation normalised with ag_sub/ag_mul, say –
must therefore be built inside expr: built before the tape, it
is destroyed on entry and the forward pass fails with "buffer was freed by a
tape reset".
w <- ag_param(matrix(c(1, 0, 0, 1), 2, 2)) x <- ag_tensor(matrix(c(1, 2), 2, 1)) y <- ag_tensor(matrix(c(1, 2), 2, 1)) with_grad_tape({ out <- ag_matmul(w, x) loss <- ag_mse_loss(out, y) }) backward(loss)w <- ag_param(matrix(c(1, 0, 0, 1), 2, 2)) x <- ag_tensor(matrix(c(1, 2), 2, 1)) y <- ag_tensor(matrix(c(1, 2), 2, 1)) with_grad_tape({ out <- ag_matmul(w, x) loss <- ag_mse_loss(out, y) }) backward(loss)