sd2R

sd2R is an R package that provides a native, GPU-accelerated Stable Diffusion pipeline by wrapping the C++ implementation from stable-diffusion.cpp and using ggmlR as the tensor backend.

Overview

sd2R exposes a high-level R interface for text-to-image and image-to-image generation, while all heavy computation (tokenization, encoders, denoiser, sampler, VAE, model loading) is implemented in C++. Supports SD 1.x, SD 2.x, SDXL, Flux, FLUX.2 (Klein), and FLUX.3 (Image Klein) model families. Targets local inference on Linux with Vulkan-enabled AMD GPUs (with automatic CPU fallback via ggml), without relying on external Python or web APIs.

Architecture

Flux without Python:

R  →  sd2R  →  ggmlR  →  Vulkan  →  GPU

Key Features

Shiny GUI

Launch an interactive web interface for image generation:

# From an R session
sd_app()                                # random port, opens browser
sd_app(model_dir = "/path/to/models")   # pre-scan a model folder
sd_app(port = 3838, host = "127.0.0.1") # fixed port/host

From the terminal (one-liners):

# Simplest
Rscript -e 'sd2R::sd_app()'

# Fixed port + local host, open browser
Rscript -e 'sd2R::sd_app(port = 3838, host = "127.0.0.1", launch.browser = TRUE)'

# Equivalent low-level call (no sd2R helpers)
Rscript -e "shiny::runApp(system.file('shiny/sd2R_app', package = 'sd2R'), port = 3838, host = '127.0.0.1', launch.browser = TRUE)"

Features:

Pipeline Example

pipe <- sd_pipeline(
  sd_node("txt2img", prompt = "a cat in space", width = 512, height = 512),
  sd_node("upscale", factor = 2),
  sd_node("img2img", strength = 0.3),
  sd_node("save", path = "output.png")
)

# Save / load as JSON
sd_save_pipeline(pipe, "my_pipeline.json")
pipe <- sd_load_pipeline("my_pipeline.json")

# Run
ctx <- sd_ctx("model.safetensors")
sd_run_pipeline(pipe, ctx, upscaler_ctx = upscaler)

Quick Start: Download a Ready-to-Use FLUX 2 Model

New to sd2R? Grab a ready-made FLUX 2 model in one line — no Kaggle account, no Python, no manual file juggling. sd_download_model() downloads the bundle from a public Kaggle dataset and unpacks it for you:

# Download FLUX 2 (GGUF) into ./models/flux2
sd_download_model(dest = "models/flux2", verbose = TRUE)

# Then launch the GUI pointed at that folder
sd_app(model_dir = "models/flux2")

That's it — the app auto-detects the model and you can start generating. Re-running sd_download_model() is safe: it skips the download if the folder is already populated.

FLUX.3 Image Klein

FLUX.3 has its own DiT, uses the FLUX.2 VAE and a Qwen3-VL 4B LLM text encoder (not the plain Qwen3-4B of FLUX.2 Klein). Defaults: 16 steps, cfg_scale = 1.0, Euler.

ctx <- sd_ctx(
  diffusion_model_path = "models/flux3_image_klein_sd_Q8_0.gguf",
  vae_path             = "models/flux2-vae.safetensors",
  llm_path             = "models/Qwen3VL-4B-Instruct-Q4_K_M.gguf",
  model_type           = "flux3"
)
img <- sd_generate(ctx, "a cat sitting on a chair, oil painting",
                   width = 768L, height = 768L)

# Edit with up to 10 reference images
edit <- sd_generate_multiref(ctx, "turn the cat into a dog",
                             refs = list(img[[1]]), width = 768L, height = 768L)

VAE tile size (vae_tile_size) is given in image pixels since 0.2.2 (default 512) and converted internally with the model's VAE scale factor (8 for SD/FLUX.1, 16 for FLUX.2/FLUX.3). Older code that passed latent cells (e.g. 64) gets a warning.

Implementation Details

CRAN Readiness

Installation

Linux

# Install ggmlR first (sd2R >= 0.2.2 needs ggmlR >= 0.8.8)
install.packages("ggmlR", configure.args = "--with-simd")

# Install sd2R
install.packages("sd2R")

Linux (detailed)

Full step-by-step setup on Ubuntu/Debian, from a clean system to a working GPU build.

1. R and the Vulkan loader/tools:

sudo apt install -y r-base

sudo apt install vulkan-tools libvulkan-dev

2. The glslc shader compiler (needed to build ggmlR's Vulkan backend):

# Ubuntu 24.04 (Noble)
sudo add-apt-repository universe
sudo apt update
sudo apt install glslc

# Ubuntu 22.04 (Jammy) — install the LunarG Vulkan SDK instead
wget -qO- https://packages.lunarg.com/lunarg-signing-key-pub.asc | \
  sudo tee /etc/apt/trusted.gpg.d/lunarg.asc

sudo wget -qO /etc/apt/sources.list.d/lunarg-vulkan-jammy.list \
  https://packages.lunarg.com/vulkan/lunarg-vulkan-jammy.list

sudo apt update
sudo apt install -y vulkan-sdk

3. Verify the GPU is visible to Vulkan:

vulkaninfo --summary

4. Install ggmlR (the tensor/Vulkan backend) with CPU SIMD acceleration:

sudo Rscript -e 'install.packages("ggmlR", configure.args = "--with-simd")'

5. Confirm GPU support from R:

Rscript -e 'library(ggmlR)
ggml_vulkan_status()'

6. Install sd2R:

sudo Rscript -e 'install.packages("sd2R")'

or, to build the development version from GitHub:

sudo apt install -y libcurl4-openssl-dev libssl-dev libgit2-dev
sudo Rscript -e 'install.packages("remotes")'
sudo Rscript -e '# remotes::install_github("Zabis13/sd2R")
install.packages("sd2R", repos = c('https://zabis13.r-universe.dev', 'https://cloud.r-project.org'))'

Launch the GUI from a terminal:

Rscript -e "sd2R::sd_app()"

During installation, the configure script automatically downloads tokenizer vocabulary files (~128 MB total) from GitHub Releases. This requires curl or wget.

Windows (step-by-step)

Tested configuration:

Install R, Rtools45, and the Vulkan SDK (use the default install paths).

From CRAN — from source with SIMD (recommended; required for GPU):

Requires Rtools45. Build from source if you want Vulkan GPU acceleration: the build enables Vulkan only when the Vulkan SDK is present at compile time (configure.win auto-detects VULKAN_SDK), so install the Vulkan SDK before running the commands below.

SIMD is a ggmlR build option, enabled via the GGML_USE_SIMD environment variable. There is no --with-simd / --configure-args="..." flag — configure.win does not parse those, so set the environment variable instead.

# --- ggmlR (tensor/Vulkan backend) with SIMD ---
unlink("C:/Program Files/R/R-4.6.0/library/00LOCK-ggmlR", recursive = TRUE)
Sys.setenv(GGML_USE_SIMD = "1")
install.packages("ggmlR", type = "source")

# --- sd2R ---
unlink("C:/Program Files/R/R-4.6.0/library/00LOCK-sd2R", recursive = TRUE)
Sys.setenv(MAKEFLAGS = "-j8")   # parallel compile; lower on fewer cores
install.packages("sd2R", type = "source")

Launch the GUI from a terminal:

"C:\Program Files\R\R-4.6.0\bin\Rscript.exe" -e "library(sd2R); sd_app()"

System Requirements

Benchmarks

FLUX.1-dev Q4_K_S — 10 steps

CLIP-L + T5-XXL text encoders, VAE. sample_steps = 10.

Test AMD RX 9070 (16 GB) Tesla P100 (16 GB) 2x Tesla T4 (16 GB)
1. 768x768 direct 13.72 s 94.0 s 62.0 s
2. 1024x1024 tiled VAE 24.84 s 151.4 s 105.6 s
3. 2048x1024 highres fix 42.70 s 312.5 s 222.0 s
4. img2img 768x768 direct 8.16 s 51.0 s 32.8 s
5. 1024x1024 direct 24.90 s 152.2 s 112.1 s
6. Multi-GPU 4 prompts -- -- 141.7 s (4 img)

FLUX.2 Klein 4B — 4 steps

Qwen3 LLM text encoder + FLUX.2 VAE. sample_steps = 4.

RTX 3090 system: CPU Xeon E5-2666 v3, 32 GB RAM (Windows).

Test AMD RX 9070 (16 GB) RTX 3090 (24 GB) (old)
1. 768x768 direct 3.80 s 5.10 s
2. 1024x1024 tiled VAE 6.70 s 8.59 s
3. 2048x1024 highres fix 13.99 s 23.54 s
4. img2img 768x768 direct 2.92 s 4.34 s
5. 1024x1024 direct 6.76 s 8.74 s

FLUX.3 Image Klein Q8_0 — 16 steps

Qwen3-VL 4B (Q4_K_M) LLM text encoder + FLUX.2 VAE. sample_steps = 16, Euler. Tiled VAE uses 512 px tiles.

Test AMD RX 9070 (16 GB)
1. 768x768 direct 11.35 s
2. 1024x1024 tiled VAE 23.70 s
3. 2048x1024 highres fix 38.69 s
4. img2img 768x768 direct 5.82 s
5. 1024x1024 direct 22.46 s
6. Edit 768x768 (1 reference image) 25.39 s

Model size comparison

SD 1.5 Flux Q4_K_S
Diffusion params ~860 MB ~6.5 GB
Text encoders CLIP ~240 MB CLIP-L + T5-XXL ~3.9 GB
Sampling per step (768x768) ~0.1–0.3 s ~3.9 s
Architecture UNet MMDiT (57 blocks)

Examples

For a live, runnable demo see the Kaggle notebook: Stable Diffusion in R (ggmlR + Vulkan GPU).

See Also

License

MIT