Introducing OpenBalancer v1.5.0: The Enterprise AI Load Balancer

Why another load balancer?

While battle-tested tools like NGINX, HAProxy, and Envoy are industry standards, setting them up for local LLM clusters (vLLM, Ollama, TGI) often involves complex buffering configurations, external health check sidecars, or high base memory overhead.

OpenBalancer was engineered around a few core priorities:

• Zero-Buffer SSE / Token Streaming: Powered by native Python asyncio stream pipelines, OpenBalancer streams tokens chunk-by-chunk with constant O(1) memory overhead (~18MB base RSS).
• AI Inference Failover & Circuit Breaking: Automatically trips unhealthy backends when inference nodes run out of VRAM (OOM) or time out, instantly rerouting traffic in <50ms without dropping active streaming clients.
• Dual Observability Out-of-the-Box: Clean JSON telemetry endpoint (/openbalancer/status) and Prometheus metrics exporter (/metrics).
• Adaptive Load Balancing Strategies: Round-Robin, Weighted Round-Robin, Least Connections, Consistent IP Hash, and Power-of-Two Random Choices (P2C).
• Zero-Dependency Quickstart: Run as a standalone Python CLI, PyPI package, or ultralight Alpine container (<45MB).

🚀 10-Second Quickstart

# Install via pip
pip install openbalancer

# Launch interactive demo sandbox with 3 mock backends:
openbalancer demo

# Or run via Docker:
docker run -d -p 8088:8088 ghcr.io/incontrolplus/openbalancer:v1.5.0

You can also try the interactive traffic simulator and visualizer directly in your browser on our landing page or read the documentation.