A heterogeneous, capability-aware work queue. Producers submit payloads; the queue matches each item to workers that have the hardware and software capabilities to handle it.
The system is built around three programs:
garage-queue-server— accepts producer requests via HTTP, matches items to workers, holds connections open while waiting for results.garage-queue-worker— connects to the server via SSE, receives matching work items, delegates them to a configured HTTP backend, and returns results. Workers can process multiple items concurrently via per-queue concurrency limits.garage-queue-cli— inspection and management tool.
The first delegator type is HTTP forwarding, intended for Ollama inference requests. The program has no knowledge of Ollama, VRAM, or language models; those concerns live entirely in configuration.
Workers connect via SSE and advertise their capabilities and per-queue concurrency limits. The server pushes work items as SSE events only when the worker has available concurrency slots, giving pull semantics over a push transport.
garage-queue-server --config /etc/garage-queue/server.tomlConfig file and log level can also be set via environment variables:
CONFIG_FILE=/etc/garage-queue/server.toml \
LOG_LEVEL=debug \
garage-queue-servergarage-queue-worker --config ~/.config/garage-queue/worker.tomlCheck whether the server is reachable:
garage-queue-cli healthSubmit a generate request and wait for the result (reads JSON from stdin):
echo '{"model":"llama3.2:8b","prompt":"Why is the sky blue?"}' \
| garage-queue-cli generatePass the payload inline with --payload:
garage-queue-cli generate --payload '{"model":"llama3.2:8b","prompt":"Hello"}'Override the server URL for a single invocation:
garage-queue-cli --server-url http://192.168.1.10:9090 healthEvery queue item carries a list of requirements extracted from its payload at intake time. Every worker advertises its capabilities when connecting. The server only offers an item to a worker whose capabilities satisfy all of the item’s requirements.
A tag requirement is an exact string match. The worker must list the tag among its capabilities.
A scalar requirement is a named numeric threshold. The worker must advertise a
value for that name that is greater than or equal to the item’s required value.
This models resource capacity: a worker with vram_mb = 16384 satisfies any
item requiring vram_mb <= 16384.
log_level = "info"
log_format = "text"
[server]
# Use a Unix socket, or "0.0.0.0:9090" for TCP.
listen = "unix:/run/garage-queue/server.sock"
[queues.ollama]
# HTTP path at which this queue accepts intake requests.
route = "/api/generate"
method = "post"
[queues.ollama.extractors.model_tag]
kind = "tag"
capability = "model"
jq_exp = ".model"
[queues.ollama.extractors.vram]
kind = "scalar"
capability = "vram_mb"
# Complex expressions belong in a separate file.
jq_file = "extractors/vram.jq"[worker]
server_url = "http://192.168.1.10:9090"
id = "titanium"
reconnect_interval_ms = 1000
[control]
host = "127.0.0.1"
port = 9091
[capabilities]
tags = ["llama3.2:8b", "gemma2:9b"]
[capabilities.scalars]
vram_mb = 16384
[concurrency]
# Default concurrency for all queues (sequential processing).
default = 1
# Override for specific queues. The tags queue is cheap, allow 4 in-flight.
ollama-tags = 4
[delegator]
kind = "http"
url = "http://127.0.0.1:11434/api/generate"Capability extractors use jq expressions (evaluated by the jaq Rust library). Simple fields can be expressed inline:
jq_exp = ".model"More complex expressions belong in a .jq file pointed to by jq_file:
# extractors/vram.jq
#
# Estimate VRAM requirement from the model name.
# Parses parameter count (e.g. "8b" → 8) and quantisation level
# (e.g. "q4" → 600 MB/B, default q4 if absent), then adds overhead.
.model as $m |
({ "q4": 600, "q5": 700, "q8": 1000, "f16": 2000 }) as $mult |
({"mixtral:8x7b": 26000}) as $overrides |
$overrides[$m] //
(
($m | capture("(?<n>[0-9]+\\.?[0-9]*)b").n | tonumber) as $params |
($m | (match("(q[0-9]+|f[0-9]+)").captures[0].string) // "q4") as $quant |
($mult[$quant] // 600) as $mbPerB |
($params * $mbPerB + 1500)
)Workers expose a local HTTP control server (default 127.0.0.1:9091) for
integration with process supervisors such as sytter:
| Endpoint | Effect |
|---|---|
POST /control/pause | Stop accepting new items; finish in-flight items |
POST /control/resume | Resume accepting items |
POST /control/stop | Finish in-flight items, then exit |
POST /control/stop/immediate | Exit immediately; in-flight items are abandoned |
Import the server module on the machine running the queue server, and the worker module on each worker machine:
# flake.nix (consumer)
inputs.garage-queue.url = "github:you/garage-queue";
# NixOS configuration
imports = [ garage-queue.nixosModules.server ];
services.garage-queue-server = {
enable = true;
# Defaults to a Unix domain socket at
# /run/garage-queue-server/garage-queue-server.sock via systemd
# socket activation. Set socket = null; host = "0.0.0.0"; port = 9090;
# to use TCP instead.
settings = {
queues.ollama = {
route = "/api/generate";
extractors.model_tag = {
kind = "tag";
capability = "model";
jq_exp = ".model";
};
};
};
};# Worker NixOS configuration
imports = [ garage-queue.nixosModules.worker ];
services.garage-queue-worker.workers.gpu = {
enable = true;
settings = {
worker = {
server_url = "http://192.168.1.10:9090";
id = "titanium";
reconnect_interval_ms = 1000;
};
control = { host = "127.0.0.1"; port = 9091; };
capabilities = {
tags = [ "llama3.2:8b" ];
scalars.vram_mb = 8192;
};
concurrency = {
default = 1;
ollama-tags = 4;
};
delegator = { kind = "http"; url = "http://localhost:11434/api/generate"; };
};
};The worker runs as a user-level launchd agent so it shares the login session with Ollama. The server runs as a system daemon.
# darwin configuration
imports = [ garage-queue.darwinModules.worker ];
services.garage-queue-worker = {
enable = true;
settings = {
worker.server_url = "http://192.168.1.10:9090";
capabilities = {
tags = [ "llama3.2:8b" "llama3.2:3b" ];
scalars.vram_mb = 8192;
};
concurrency = {
default = 1;
ollama-tags = 4;
};
delegator = { kind = "http"; url = "http://localhost:11434"; };
};
};Currently each extractor produces a single tag value. A plausible future
extension is a kind = "tag_list" extractor whose jq expression returns a JSON
array of strings, each becoming a separate tag requirement.
The current decision was to defer this because there is no known use case, and because allowing an expression to return either a string or an array (polymorphic output) was considered unfriendly to operators writing expressions.
The system forces "stream": false on all delegated requests. Supporting
streaming would require the server to proxy chunked output from the worker back
to the waiting producer connection. This is architecturally straightforward but
adds complexity and was deferred pending a concrete need.
Items whose delegator call fails are silently dropped; the producer times out. A dead letter queue with configurable retry policy is a natural next step.
Scalar matching currently always uses >= semantics (worker capacity >=
item requirement). If a use case arises where a different comparator is
needed (e.g. <= for a budget that must not be exceeded), the config should
gain an explicit compare field on scalar extractors and a corresponding
field on worker scalar declarations, rather than hard-coding the direction.