Inference Lab / Benchmarks

Local inference benchmark and evaluation harness for OpenAI-compatible text, tool-calling, vision, audio/STT, embeddings, TTS, and image generation endpoints.

This repo was split out from the older infra-ai benchmark work and rehydrated around Rowan's local Gemma 4 12B multimodal llama.cpp endpoint.

See also:

• docs/rowan-gemma4-12b-rehydration.md
• endpoints.json
• scripts/run-rowan-gemma4-smoke.sh
• scripts/run-rowan-gemma4-openai-tools.py

---

PCT Agent Benchmarks

Benchmark framework for testing PCT AI agents (VMIDs 200-206) against standardized test suites.

Overview

This framework evaluates agent performance across three key categories:

• Basic Reasoning: Mathematical problems, logical deduction, reading comprehension
• Tool Calling: Function calling format validation, parameter extraction, API interaction patterns
• Code Generation: Python, PowerShell, and bash snippet generation and correctness

Prerequisites

• Python 3.8+
• requests library (pip install requests)
• Access to LiteLLM endpoint at 192.168.50.250:8000
• OpenAI-compatible API format

Running Benchmarks

Basic Usage

python runner.py --suite suites/basic-reasoning.json --endpoint http://192.168.50.250:8000

Options

• --suite <path>: Path to suite JSON file (required)
• --endpoint <url>: LiteLLM endpoint URL (default: http://192.168.50.250:8000)
• --model <name>: Model name to use (default: gpt-4-turbo)
• --max-retries <n>: Maximum retries per test (default: 1)

Quality benchmarks (Phase 2)

# Dry-run validation (no endpoint needed — validates suite + grader):
python quality_runner.py --suite suites/mmlu-pro.json --benchmark mmlu-pro --dry-run
python quality_runner.py --suite suites/gpqa-d.json --benchmark gpqa-d --dry-run
python quality_runner.py --suite suites/humaneval-plus.json --benchmark humaneval-plus --dry-run
python quality_runner.py --suite suites/math-500.json --benchmark math-500 --dry-run

# Live run against main endpoint:
python quality_runner.py --suite suites/mmlu-pro.json --benchmark mmlu-pro \
    --endpoint http://192.168.50.250:8000 --model Qwen3-14B-GGUF

# Bonsai-8B target:
python quality_runner.py --suite suites/mmlu-pro.json --benchmark mmlu-pro \
    --endpoint http://192.168.50.250:8050 --model prism-ml/Bonsai-8B-gguf

Current modalities

• Text/inference benchmarks are runnable today via runner.py
• Embeddings are runnable via embeddings_runner.py and save schema-validated outputs through save_embeddings_results(...)
• Vision benchmarks are runnable via vision_runner.py using image inputs and schema-validated saved outputs
• Speech-to-text benchmarks are runnable via stt_runner.py using local audio assets, WER/word-accuracy scoring, and schema-validated saved outputs
• Text-to-speech benchmarks are runnable via tts_runner.py using prompt/voice matrices, timing metrics, and schema-validated saved outputs
• Image-generation benchmarks are runnable via imagegen_runner.py using prompt suites, saved PNG artifacts, and schema-validated saved outputs

Example

python runner.py --suite suites/tool-calling.json --endpoint http://192.168.50.250:8000 --model gpt-4

Embeddings examples

python embeddings_runner.py --suite suites/embeddings-retrieval.json --endpoint http://192.168.50.250:8000 --model nomic-embed-text-v2-moe-GGUF --embedding-dim 768 --index-time-s 5.57
python embeddings_runner.py --suite suites/embeddings-retrieval-canonical.json --endpoint http://192.168.50.250:8000 --model nomic-embed-text-v2-moe-GGUF --embedding-dim 768 --index-time-s 5.57 --historical-metrics-file ../infra-ai/benchmarking/embed-2026-03-04.json

Vision examples

# Current local benchmark waves run VLM suites against the main endpoint on :8000.
# A separate :8081 vision endpoint may not be present on every host.
python vision_runner.py --suite suites/vision-basic.json --endpoint http://192.168.50.250:8000 --model Qwen3-VL-4B-Instruct-GGUF
python vision_runner.py --suite suites/vision-ocr.json --endpoint http://192.168.50.250:8000 --model Qwen3-VL-4B-Instruct-GGUF
python vision_runner.py --suite suites/vision-ui.json --endpoint http://192.168.50.250:8000 --model Qwen3-VL-4B-Instruct-GGUF
python vision_runner.py --suite suites/vision-diagram.json --endpoint http://192.168.50.250:8000 --model Qwen3-VL-4B-Instruct-GGUF
python vision_runner.py --suite suites/vision-ticket-attachments.json --endpoint http://192.168.50.250:8000 --model Qwen3-VL-4B-Instruct-GGUF
python vision_runner.py --suite suites/vision-document-classification.json --endpoint http://192.168.50.250:8000 --model Qwen3-VL-4B-Instruct-GGUF

STT example

python stt_runner.py --suite suites/stt-basic.json --endpoint http://192.168.50.250:8000 --model Whisper-Base --chunk-seconds 25

TTS example

python tts_runner.py --suite suites/tts-basic.json --endpoint http://192.168.50.250:8000 --model kokoro-v1

Image generation examples

# Turbo preview models only
python imagegen_runner.py --suite suites/imagegen-basic.json --endpoint http://192.168.50.250:8000 --model SD-Turbo

# SD-1.5 quality rerun
python imagegen_runner.py --suite suites/imagegen-quality-rerun.json --endpoint http://192.168.50.250:8000 --model SD-1.5

# SDXL base quality rerun
python imagegen_runner.py --suite suites/imagegen-quality-rerun-sdxl.json --endpoint http://192.168.50.250:8000 --model SDXL-Base-1.0

Suite Format

Each suite is a JSON array of test case objects:

[
  {
    "id": "test_001",
    "prompt": "What is 2 + 2?",
    "expected_contains": ["4"],
    "max_tokens": 100,
    "category": "basic-reasoning"
  }
]

Test Case Fields

Field	Type	Description
id	string	Unique test identifier
prompt	string	Input prompt to send to agent
expected_contains	array	At least one string must appear in response (OR condition)
expected_all_contains	array	All listed strings must appear in response
exact_match	string	Response must exactly match this value after trim
regex_match	string	Python regex that must match the response
forbidden_contains	array	Strings that must not appear in response
tool_call	object	Structured tool-call expectation with name, arguments, optional forbidden_arguments, and optional forbid_extra_arguments
code	object	Executable code validation config with language and exec
max_tokens	integer	Maximum tokens for response
category	string	Test category (basic-reasoning, tool-calling, code-generation)

You can combine assertion styles in one test case. All supplied checks must pass.

Results

Saved results now use a structured layout:

results/
  YYYY-MM-DD/
    <suite-name>/
      <suite-name>-YYYYMMDD_HHMMSS.json

Before writing a result file, the runner validates the payload against:

• schemas/benchmark-run.schema.json
• schemas/inference.schema.json

It also writes a convenience file at:

• results/latest-summary.json

Schema files:

• schemas/benchmark-run.schema.json — base canonical runner output
• schemas/inference.schema.json — current text/inference benchmark schema
• schemas/embeddings.schema.json — retrieval benchmark schema for the embeddings modality
• schemas/vision.schema.json — dedicated schema for image-input vision benchmark runs
• schemas/stt.schema.json — dedicated schema for audio-input speech-to-text benchmark runs
• schemas/tts.schema.json — dedicated schema for text-to-speech benchmark runs
• schemas/imagegen.schema.json — dedicated schema for image-generation benchmark runs

Output format:

results/
  2026-04-09/
    basic-reasoning/
      basic-reasoning-20260409_143022.json

Output format:

{
  "suite": "basic-reasoning.json",
  "model": "gpt-4-turbo",
  "endpoint": "http://192.168.50.250:8000",
  "timestamp": "2024-03-05T14:30:22Z",
  "total_tests": 5,
  "passed": 4,
  "failed": 1,
  "pass_rate": 80.0,
  "total_prompt_tokens": 40,
  "total_completion_tokens": 17,
  "total_tokens": 57,
  "total_wall_s": 1.75,
  "avg_wall_s": 0.35,
  "avg_tps": 18.7,
  "results": [
    {
      "id": "test_001",
      "category": "basic-reasoning",
      "prompt": "What is 2 + 2?",
      "passed": true,
      "response": "2 + 2 equals 4.",
      "expected_contains": ["4"],
      "expected_all_contains": [],
      "exact_match": null,
      "regex_match": null,
      "forbidden_contains": [],
      "matched_strings": ["4"],
      "matched_all_strings": [],
      "forbidden_found": [],
      "exact_match_passed": null,
      "regex_match_passed": null,
      "finish_reason": "stop",
      "prompt_tokens": 9,
      "completion_tokens": 5,
      "total_tokens": 14,
      "wall_s": 0.25,
      "tps": 20.0,
      "error": null
    }
  ]
}

Additional telemetry captured when the endpoint returns it:

• Per test:
  • finish_reason
  • prompt_tokens
  • completion_tokens
  • total_tokens
  • wall_s — end-to-end request wall time
  • tps — derived completion tokens per second
  • error — timeout/HTTP/connection classification when the request fails
  • tool_call_passed / tool_call_error for structured JSON tool-call checks
  • code_validation_passed / code_validation_error for executable code checks
• Per suite:
  • total_prompt_tokens
  • total_completion_tokens
  • total_tokens
  • total_wall_s
  • avg_wall_s
  • avg_tps
• Convenience summary output:
  • results/latest-summary.json
  • contains latest run path, suite, model, pass rate, and aggregate timing fields

Infrastructure

Containers & VMIDs

• VMID 205: Ephemeral sandbox environment
• VMID 206: Secondary ephemeral sandbox
• VMIDs 200-204: Agent instances

Endpoints

Known endpoints are catalogued in endpoints.json. Use --endpoint <url> --model <name> with any runner.

Name	URL	Auth	Notes
lemonade-main	http://192.168.50.250:8000	none	Main gateway; production workload
bonsai-8b	http://192.168.50.250:8050	none	Bonsai-8B Q1_0 ternary; shares GPU

LiteLLM Endpoint (main)

Runs on 192.168.50.250:8000 with OpenAI-compatible API format.

Supports standard OpenAI parameters:

• model: Model identifier
• messages: Chat message format
• max_tokens: Response length limit
• temperature: Sampling temperature (0.0-2.0)

Bonsai-8B Endpoint

Runs on 192.168.50.250:8050. No API key required. Model: prism-ml/Bonsai-8B-gguf. Shares the GPU with the main endpoint — do not run heavy benchmarks while production traffic is active.

Troubleshooting

Connection Refused

Verify LiteLLM service is running and endpoint is accessible:

curl -I http://192.168.50.250:8000/health

Invalid JSON Response

Check that the endpoint is returning proper JSON format. LiteLLM should return:

{
  "choices": [
    {
      "message": {
        "content": "response text"
      }
    }
  ]
}

Files

• runner.py - Main text/inference benchmark runner script
• embeddings_runner.py - Embeddings retrieval benchmark runner
• vision_runner.py - Vision benchmark runner for image-input evaluation
• quality_runner.py - Phase-2 quality benchmark runner (MMLU-Pro, GPQA-D, HumanEval+, MATH-500)
• suites/basic-reasoning.json - Basic reasoning test cases
• suites/tool-calling.json - Tool calling test cases
• suites/code-generation.json - Code generation test cases
• suites/mmlu-pro.json - MMLU-Pro 5-choice MCQ sample (15 questions, 10 subjects)
• suites/gpqa-d.json - GPQA-Diamond 4-choice MCQ sample (10 questions, graduate science)
• suites/humaneval-plus.json - HumanEval+ Python code generation sample (10 problems)
• suites/math-500.json - MATH-500 competition math sample (10 problems, Levels 1-4)
• suites/embeddings-retrieval.json - Suite skeleton for embeddings retrieval benchmarks
• suites/vision-basic.json - Starter suite for image-input vision benchmarks
• schemas/ - JSON schemas for benchmark result formats
• vendored/ - Vendored grader code for public benchmarks (mmlu-pro, gpqa-d, humaneval-plus, math-500)
• endpoints.json - Known endpoint catalogue (lemonade-main, bonsai-8b)
• docs/ - Per-benchmark documentation (scoring, runtime, usage)
• .gitignore - Ignores Python cache files and generated results/
• results/ - Output directory for benchmark results