infra-ai — PCT Local Inference Server

Local LLM inference on pct-inf-one (192.168.50.250), a Minisforum MS-S1 MAX with AMD Ryzen AI MAX+ 395 (Strix Halo) and 128 GB unified RAM.

Hardware

Component	Spec
APU	AMD Ryzen AI MAX+ 395 (Strix Halo)
CPU	16C/32T Zen 5, up to 5.1 GHz
GPU	Radeon 8060S, 40 RDNA 3.5 CUs, gfx1151
RAM	128 GB LPDDR5X-8000 unified (shared CPU/GPU)
Memory BW	~212 GB/s GPU
OS	CachyOS (Arch-based), kernel 6.19.x
GPU stack	ROCm 7.0 + Vulkan RADV
Shell	fish

Inference Stacks

Three independent inference engines run on this host. They are not sub-components of each other.

Lemonade (port 8000)

AMD's inference server. Manages a catalog of 55+ models and routes requests to llama.cpp backends.

• Service: lemonade-server.service (user: lemonade)
• Version: 10.2.0 (embeddable Lemonade runtime, Vulkan backend)
• Config: /etc/lemonade/lemonade.conf
• Context: 131072 tokens
• Backend: LEMONADE_LLAMACPP=vulkan
• Architecture: lemonade-router (:8000) dispatches to llama-server workers (:8001 text, :8002 embeddings)
• Production model: Qwen3-Coder-30B-A3B-Instruct Q4_K_M — 84.9 tok/s TG
• Model cache: /var/lib/lemonade/.cache/
• API: OpenAI-compatible (/v1/chat/completions, /v1/models, /v1/embeddings)

Gemma 4 (port 8081)

Standalone llama.cpp server for Google Gemma 4 with multimodal (vision) support.

• Service: gemma4-server.service (user: pctadmin)
• Binary: /opt/llama.cpp/llama-server (master, commit a1cfb64, Vulkan)
• Model: gemma4-26b-a4b-q4km.gguf + mmproj-26b-a4b-bf16.gguf
• Context: 32768 tokens
• KV cache: q4_0 keys / q8_0 values (asymmetric, saves RAM)
• Flags: --flash-attn on --jinja --reasoning-budget 0
• Performance: ~57 tok/s TG, ~1230 tok/s PP | 4.7 GB memory
• Pre-warm: /opt/llama.cpp/prewarm.sh runs on service start
• API: OpenAI-compatible (/v1/chat/completions)
• Vision: Working — accepts base64 images in message content

Ollama (port 11434)

General-purpose model runner. ROCm backend. Currently idle (used for ad-hoc model testing).

• Service: ollama.service
• Version: 0.18.3
• Backend: ROCm (Vulkan available but not active — ROCm takes priority)
• Host: 0.0.0.0:11434

Benchmarks (2026-04-02, Vulkan RADV)

Model	Params (active)	Quant	TG tok/s	Notes
Qwen3-Coder-30B-A3B	30B (3B)	Q4_K_M	84.9	Production pick for text/code
gpt-oss-20b	20B	mxfp4	78.0	Microsoft, fast reasoning
Qwen3.5-35B-A3B	35B (3B)	Q4_K_M	51.8	Best tool-calling (BFCL 67.3)
Gemma 4 26B-A4B	26B (4B)	Q4_K_M	57.2	Vision, multimodal
Gemma 4 E4B	— (4B)	Q4_K_M	56.7	Smaller variant

Reference benchmarks from kyuz0/amd-strix-halo-toolboxes (same hardware, Fedora 43):

• Vulkan RADV consistently beats ROCm 7.2 for token generation (~20% faster)
• ROCm wins prompt prefill by ~10%
• See also: strixhalo.wiki

Consumers

Service	Uses	Port	Purpose
stack-dashboard chat	Lemonade (text) + Gemma 4 (vision)	:3007	23 chat tools, dual-model routing
RC-AT server	Lemonade	:3011	Ticket summarization
meeting-notes	Lemonade (Whisper)	—	STT transcription pipeline

Key Gotchas

• fish shell — Can't inline bash syntax over SSH. Use bash -c '...' or SCP+execute.
• Qwen3.x thinking suppression — Use chat_template_kwargs: {"enable_thinking": false} in API calls. Not /no_think prefix.
• Gemma 4 reasoning drain — Without --reasoning-budget 0, model generates invisible reasoning tokens that consume the entire output budget, producing empty or <unused24> responses. See ggml-org/llama.cpp#21321.
• Ollama Vulkan — OLLAMA_VULKAN=1 does not force Vulkan when ROCm is detected. Ollama always prefers ROCm.
• pip ROCm wheels — Fail on gfx1151. Use pacman native builds instead.
• Speculative decoding — Incompatible with --mmproj (multimodal). Can't use both.
• Context vs speed — 131K context is ~20% slower than 32K for the same model. Keep 131K for Lemonade (capability), 32K for Gemma 4 (speed).

Fine-Tuning

Unsloth QLoRA is operational on gfx1151 (first confirmed instance on this GPU).

• Venv: /home/pctadmin/unsloth-venv
• Source: /home/pctadmin/unsloth-src
• System torch: python-pytorch-opt-rocm 2.11.0 (pacman)
• gfx1151 patch: From PR #4720
• Test result: 36.5s / 10 steps, 8.8 GB VRAM

Repo Contents

Path	Description
README.md	This file — current state of localinf
references.md	Curated external links: Strix Halo, Lemonade, Unsloth, HuggingFace, llama.cpp issues
benchmarks.md	Consolidated benchmark history (all phases, March-April 2026)
conda-envs.md	Python environments on localinf
services/	Systemd unit files, lemonade.conf, prewarm.sh — server config backup
benchmarking/tested_models.json	Structured data: 40+ models across inference, vision, coding, embedding, whisper, imagegen, TTS
benchmarking/	Raw benchmark results and reports
baseline-results/	March 3 baseline data (single-request, context scaling, concurrency)
../infra-benchmarks/	Canonical benchmark runner repo: suites, schemas, tests, and structured results/YYYY-MM-DD/<suite>/... outputs with results/latest-summary.json
architecture.md	Speculative 4-node cluster design (planning doc)
journal.md	Session notes: model benchmarks, thinking suppression, RC-AT tuning
server_spec.md	Original Lemonade server specification (67 KB)
optimize.md	ROCm/Vulkan optimization notes
summarization-benchmark.md	RC-AT ticket summarization benchmark (March 4)
further-exploration.md	Research notes on model families and capabilities
mcp-config/	MCP server registry and sync script
archive/vllm/	Legacy vLLM scripts and results (superseded by Lemonade)

Tracking

• OneDev: int-infra-ai (project id: 20)
• Strategy issue: int-infra-ai#1 — 6 workstreams (model selection, inference optimization, quality, fine-tuning, multi-model routing, monitoring)