flowchart TB
Client[Open WebUI or chat client] --> API[OpenAI-compatible API]
API --> Engine[Best-of-N engine]
Config[YAML pseudo-model definitions] -.-> Engine
Engine --> A[Candidate A]
Engine --> B[Candidate B]
Engine --> C[Candidate C]
A --> Judge[LLM judge]
B --> Judge
C --> Judge
Judge --> Winner[Selected response]
Judge -. Failure .-> Fallback[Deterministic fallback]
Fallback --> Winner
Winner --> API[Chat response]
Multi-LLM: Best-of-N Response Selection
Python
Applied AI
LLMs
FastAPI
Docker
An OpenAI-compatible chat proxy that generates candidate responses from one model or a model pool and uses a judge model to select a winner.
Overview
I built a best-of-N chat proxy to explore an alternative to returning the first response from a language model: generate several candidates, then use a judge model to choose one.
The service supports repeated sampling from one model and pools of different models. It returns the selected answer through an OpenAI-compatible endpoint, so the selection process can fit into an existing chat interface such as Open WebUI.
How it works
YAML configuration defines pseudo-model IDs such as bon/llama3.1 and bon-pool/general. Each ID maps to candidate models, sampling settings, a total candidate count, and a judge. These are strategies published by the proxy; they do not create new model weights or require an Ollama Modelfile.
Open WebUI discovers the pseudo-models through /v1/models and submits chats through /v1/chat/completions. The proxy translates each strategy into requests to Ollama’s OpenAI-compatible /v1 API and returns the chosen response under the pseudo-model ID.
The repository documents the OpenAI connection in Open WebUI, the separate URLs used by host and container processes, provider credentials, model downloads, configuration precedence, restart requirements, and persisted WebUI settings. The included Compose stack starts the proxy and WebUI; Ollama runs separately. Direct Ollama integration is disabled in that stack so model selection goes through the proxy.
Engineering decisions
- Fit existing chat clients. Configured strategies appear as model IDs behind a standard chat API, keeping model orchestration separate from the user interface.
- Make sampling explicit. Fixed candidate counts and proportional weights define a model pool; strategy and candidate settings control sampling. Candidates run concurrently with distinct seeds.
- Handle partial failures. Failed candidates do not discard successful responses. A single usable answer bypasses judging, invalid judge output triggers a deterministic fallback, and requests with no usable answers return an upstream-failure response.
- Keep providers configurable. Endpoint URLs and API credentials come from YAML and environment variables, supporting local inference and other compatible providers without machine-specific source changes.
This implementation grew out of earlier Ollama-only proxy experiments. The public version focuses on provider abstraction, concurrent generation, and response selection.
Validation and tradeoffs
The public implementation has 22 automated tests, including API requests and SSE responses, judge selection, fallback, partial and total candidate failures, configuration validation, sampling precedence, and provider URL handling. Regression cases check that missing-information and refusal phrases receive no blanket fallback penalty. Integration tests use fake providers rather than live inference.
The project demonstrates the selection mechanism. It makes no measured claim that best-of-N improves answer quality.
The default judge rubric favors supported claims and appropriate uncertainty, permitting clarification or refusal when warranted. Each request requires N candidate generations and usually another call for judging. Streaming returns the completed winner after selection. Judge decisions and the fallback heuristic can be wrong, and returned token usage covers only the winner rather than the full strategy. The proxy is intended for trusted local experiments and has no inbound authentication.
Stack: Python, FastAPI, asyncio, HTTPX, Pydantic, Ollama, Docker Compose, pytest.