Technical Documentation

What is OpenBalancer? #

OpenBalancer is an ultra-fast, asynchronous load balancer and API reverse proxy built specifically for high-throughput AI model inference clusters, microservice meshes, and mission-critical API routing. Written from the ground up on non-blocking Python asyncio raw sockets, OpenBalancer eliminates unnecessary middleware layers to deliver sub-millisecond p99 routing overhead.

Core Architectural Highlights

• Non-Blocking Asynchronous Socket Loop: Pipes incoming client requests to upstream backends asynchronously using raw TCP stream readers and writers without threading locks.
• AI LLM Streaming & SSE Passthrough: Zero response buffering for text/event-stream and application/grpc, ensuring instant token-by-token streaming with constant O(1) memory consumption.
• Active Background Health Probing: Continuously interrogates upstream backend health paths at configurable intervals with sub-millisecond latency tracking.
• Automatic Circuit Breaking: Immediately isolates flapping or failing upstream nodes after consecutive probe failures, preventing cascading microservice collapse.
• Prometheus Telemetry & JSON Status: Native built-in endpoints at /openbalancer/status and /metrics for turnkey Grafana observability.

Quickstart & Installation #

Get OpenBalancer running in under 60 seconds using Docker, Docker Compose, or standalone Python.

1. Run via Docker (Recommended)

docker run -d \
  --name openbalancer \
  -p 8080:8080 \
  -v $(pwd)/config.json:/app/config.json \
  openbalancer/core:latest

2. Run with Docker Compose (Full Multi-Backend Test)

git clone https://github.com/incontrolplus/openbalancer.git
cd openbalancer/core
docker compose up -d

3. Standalone Python (Zero Dependencies)

# Requires Python 3.11+ (Standard Library only - 0 pip packages required)
python3 openbalancer.py config.json

4. Verify Ingress & Status

# Test Load Balancer Ingress
curl -i http://localhost:8080/

# Query JSON Cluster Telemetry
curl http://localhost:8080/openbalancer/status

# Scrape Prometheus Metrics
curl http://localhost:8080/metrics

Kernel & Event Loop Architecture #

OpenBalancer utilizes single-threaded non-blocking cooperative multitasking via Python's asyncio socket event loop. By bypassing traditional thread pools and GIL contention, it maintains a lightweight footprint (~18MB RSS) capable of handling tens of thousands of concurrent open TCP streams.

Circuit Breaker & Health Probing State Machine

Every configured upstream node is evaluated across a robust state machine:

Routing Algorithms #

OpenBalancer provides 6 production-ready load balancing strategies switchable at runtime or via configuration:

Setting the Algorithm in config.json

{
  "algorithm": "least_latency"
}

Configuration Schema (`config.json`) #

OpenBalancer reads its initial topology from config.json. All parameters can be hot-reloaded without downtime via SIGHUP.

Environment Variable Overrides

For Twelve-Factor App compliance and container environments, all primary options can be overridden via environment variables:

Interactive JSON Configuration Builder #

Configure your cluster parameters visually below. The schema updates in real time with 1-click clipboard copy and direct JSON file download.

Custom Error Page Studio #

Design, live-preview, and export branded, lightweight standalone HTML error pages for 429 Rate Limiting, 502 Bad Gateway, and 503 Outage scenarios. Zero external dependencies, automatic countdown retry logic, and pixel-perfect design.

Telemetry & Prometheus API #

OpenBalancer features built-in, zero-dependency observability endpoints natively exposing cluster telemetry in both structured JSON and standard Prometheus scrape formats.

1. Status & Topology API (`GET /openbalancer/status`)

Returns real-time cluster uptime, total proxied requests, selected algorithm, and per-node health status, latency, and circuit breaker trip counts.

{
  "system": "OpenBalancer Core",
  "operator": "INCONTROL PLUS EOOD",
  "license": "MIT",
  "uptime_seconds": 14280,
  "total_proxied_requests": 849200,
  "algorithm": "least_latency",
  "backends": [
    {
      "url": "http://10.0.1.10:8000",
      "healthy": true,
      "weight": 3,
      "total_requests": 424600,
      "last_latency_ms": 1.18,
      "circuit_trips": 0
    },
    {
      "url": "http://10.0.1.11:8000",
      "healthy": true,
      "weight": 2,
      "total_requests": 283100,
      "last_latency_ms": 1.42,
      "circuit_trips": 0
    },
    {
      "url": "http://10.0.1.12:8000",
      "healthy": true,
      "weight": 1,
      "total_requests": 141500,
      "last_latency_ms": 1.85,
      "circuit_trips": 0
    }
  ]
}

2. Prometheus Metrics (`GET /metrics`)

Compatible out of the box with Prometheus, VictoriaMetrics, and Grafana Agent without requiring an external exporter sidecar.

# HELP openbalancer_requests_total Total number of proxied HTTP requests
# TYPE openbalancer_requests_total counter
openbalancer_requests_total 849200

# HELP openbalancer_uptime_seconds OpenBalancer uptime in seconds
# TYPE openbalancer_uptime_seconds gauge
openbalancer_uptime_seconds 14280

# HELP openbalancer_backend_health_status Health status of backend node (1=healthy, 0=down)
# TYPE openbalancer_backend_health_status gauge
openbalancer_backend_health_status{backend="http://10.0.1.10:8000",host="10.0.1.10",port="8000"} 1
openbalancer_backend_health_status{backend="http://10.0.1.11:8000",host="10.0.1.11",port="8000"} 1

# HELP openbalancer_backend_latency_ms Last probed health check latency in milliseconds
# TYPE openbalancer_backend_latency_ms gauge
openbalancer_backend_latency_ms{backend="http://10.0.1.10:8000",host="10.0.1.10",port="8000"} 1.18
openbalancer_backend_latency_ms{backend="http://10.0.1.11:8000",host="10.0.1.11",port="8000"} 1.42

# HELP openbalancer_circuit_breaker_trips_total Total circuit breaker trip events
# TYPE openbalancer_circuit_breaker_trips_total counter
openbalancer_circuit_breaker_trips_total{backend="http://10.0.1.10:8000",host="10.0.1.10",port="8000"} 0

Real-Time Latency Heatmap & Flamegraph #

Observe real-time latency distribution across 4 discrete threshold buckets (<10ms green, 10-50ms cyan, 50-200ms yellow, >200ms red) with live percentile calculations and execution waterfall stages.

Interactive API Tester ("Try it Out") #

Test OpenBalancer management and telemetry endpoints in real time. Choose an endpoint below and click Execute Request. If running locally, you can query your live instance, or use our high-fidelity simulated sandbox engine.

Production Deployment & Docker #

Deploying OpenBalancer in mission-critical environments with automatic restart policies, healthchecks, and resource constraints.

1. Complete Production Dockerfile

FROM python:3.11-alpine

WORKDIR /app

COPY openbalancer.py config.json /app/

EXPOSE 8080

HEALTHCHECK --interval=5s --timeout=2s --start-period=3s --retries=3 \
  CMD python3 -c "import urllib.request; urllib.request.urlopen(\"http://127.0.0.1:8080/openbalancer/status\", timeout=2)" || exit 1

ENTRYPOINT ["python3", "openbalancer.py", "config.json"]

2. Production Docker Compose Stack

services:
  openbalancer:
    image: openbalancer/core:latest
    container_name: openbalancer
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - ./config.json:/app/config.json:ro
    environment:
      - PORT=8080
      - ALGORITHM=least_latency
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 128M

3. Systemd Unit File (`/etc/systemd/system/openbalancer.service`)

[Unit]
Description=OpenBalancer AI & API Reverse Proxy Core
After=network.target

[Service]
Type=simple
User=openbalancer
Group=openbalancer
WorkingDirectory=/opt/openbalancer
ExecStart=/usr/bin/python3 /opt/openbalancer/openbalancer.py /etc/openbalancer/config.json
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=3
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Zero-Downtime Hot Reload (`SIGHUP`) #

OpenBalancer listens for POSIX SIGHUP signals. When received, the core re-reads config.json from disk, reconstructs upstream pools, updates routing weights, and adjusts probing intervals without dropping active client TCP connections or interrupting streaming responses.

# Send SIGHUP to standalone process
kill -HUP $(pgrep -f openbalancer.py)

# Send SIGHUP to Docker container
docker kill -s HUP openbalancer

# OpenBalancer output confirms reload:
# 2026-08-18 10:45:00 [INFO] [OpenBalancer] SIGHUP received: Reloading config.json
# 2026-08-18 10:45:00 [INFO] [OpenBalancer] Loaded 3 backends. Strategy: least_latency. Port: 8080

Performance & Benchmarks #

Measured on bare-metal Linux (AMD EPYC 7763, 64 Cores, 10GbE network link) running wrk -t16 -c1000 -d30s against 3 upstream nodes.

Enterprise SLA & Support #

OpenBalancer is maintained and commercially backed by INCONTROL PLUS ЕООД. For production enterprise deployments requiring guaranteed uptime, sub-15 minute emergency incident escalation, and custom AI dispatcher integrations, we provide contractual B2B Master Services Agreements (MSAs).