InferBench

curl -fsSL https://cyben.sixbit.fun/init.sh | sh

📖 Detailed User Manual

1. Introduction & Overview

InferBench is a lightweight, cross-platform AI local benchmark CLI suite designed to help developers and server operators validate hardware capabilities, estimate model memory footprints, and measure reasoning engine throughput (tokens/sec) and latency. Key benefits include finding the most cost-effective and profitable LLM configuration for your hardware and verifying execution stack health in seconds.

Linux and macOS users can run curl -fsSL https://cyben.sixbit.fun/init.sh | sh. After the verified installation, the terminal wizard can initialize dependencies and complete either a benchmark report or a managed Linux CyAI worker setup.

2. Supported Inference Engines

• vLLM: High-throughput batching engine, optimal for enterprise serving on NVIDIA GPUs.
• SGLang: Next-gen execution backend optimized for rapid structured generation.
• llama.cpp: Minimalist runtime optimal for CPUs, Intel GPUs, and macOS Apple Silicon (Metal).
• Ollama: Tight integration with native Ollama background daemon services.

3. Command Reference & Parameters

InferBench includes commands for diagnostics, fitting, benchmarking, local services, authentication, and CyAI workers:

• inferbench doctor: Runs system diagnostics (checks OS, Python environment, engine installations, and hub connectivity). Pass --auto-fix-deps to automatically resolve errors.
• inferbench scan: Displays local hardware details (CPUs, GPUs, VRAM capacity, and driver versions). Pass --json for machine-readable output.
• inferbench fit: Evaluates which registered models fit into your local GPU VRAM. Pass --all-quantizations to see full tables, or --bench-cmd to print copy-pasteable execution commands.
• inferbench bench: Starts local inference servers and runs concurrency sweeps. --model is optional in an interactive terminal, where a model menu is shown. --quantization defaults to fp16; pass auto for automatic fit selection. Other options include --engine-type, --model-source, --quick-test, and --inferbench-submit=false to keep reports local.
• inferbench gui: Launches the interactive dashboard. Pass --addr to bind to a specific address, and --db to specify the SQLite storage file.

4. Platform-Specific Setup Guides

macOS (Apple Silicon): We recommend using llama.cpp or ollama. Install compilation prerequisites via Homebrew, then run ./inferbench doctor --auto-fix-deps --engine-type=llama.cpp to build a clean local environment.

Windows (x64): Supports native CUDA execution. Run .\inferbench.exe doctor --auto-fix-deps --engine-type=ollama to automatically install Python via Winget, configure virtualenvs, and locate Ollama installation paths.

Linux (x64 / amd64): Ideal for high-throughput CUDA serving. Run ./inferbench doctor --auto-fix-deps --engine-type=vllm to automatically run package-manager upgrades, configure local virtual environments, and install vllm or sglang libraries using fast uv pip utilities.

5. CyAI Worker on Debian or Ubuntu

Run sudo inferbench cyai install for interactive setup. InferBench asks for the API key, uses the hostname as the default worker name, lists supported models, and can manage a local vLLM service with systemd. Use --model-service=existing --model-port=8000 to connect to an OpenAI-compatible server you already operate.

Check services with sudo inferbench cyai status, sudo journalctl -u cyai-model.service -f, and sudo journalctl -u cyai.service -f.

ice -f.

6. China Mainland Optimization

If network diagnostics detect that HuggingFace Hub is blocked but ModelScope Hub is open, InferBench will automatically recommend downloading models via ModelScope. You can force this configuration at any time by specifying --model-source=modelscope in the command line or UI.