Managed private deployment
Provision llmxy in a dedicated Learnmark-managed or customer-controlled environment with repeatable releases, encrypted configuration, backups, and rollback procedures.
Managed AI Infrastructure
llmxy
Operate one production gateway for multiple model providers with controlled routing, OpenAI-compatible APIs, usage visibility, and a managed path from deployment to day-two operations.
llmxy sits between your applications and model providers as a unified control plane. Teams keep a stable client protocol while operators configure upstream channels, models, route policies, quotas, and billing from one place.
- Operator-controlled routing. Configure upstream channels, model mappings, route policies, and weights without changing application clients.
- Gateway enforcement. Apply authentication, balance and quota checks, rate limiting, usage recording, and billing consistently at the gateway.
- Multi-provider access. Use an OpenAI-compatible protocol in front of OpenAI, Anthropic, Gemini, and translated upstream responses.
- Usage Analytics. Track token consumption, cost, and latency per model, key, and tenant with real-time dashboards.
- Two relay paths. Start with the direct FastAPI relay or add the optional Envoy path for higher-throughput traffic and asynchronous usage reporting.
The Learnmark product turns that open-source foundation into an operated gateway environment, with deployment engineering, provider onboarding, production controls, and an accountable support path.
Learnmark Product
What the managed llmxy product adds
The repository provides the gateway software. The Learnmark product adds the engineering and operational layer required to introduce it into a production platform and keep it healthy over time.
Provider and routing onboarding
Connect approved model providers, map model names, define route weights and policies, and validate failover behavior against your applications before rollout.
Production observability
Add environment-level metrics, alerting, capacity reviews, usage reconciliation, and operational runbooks around the gateway and its data services.
Upgrade and incident support
Receive coordinated upgrades, change reviews, troubleshooting, and a clear escalation path instead of operating the open-source stack alone.
Product Screenshots
The llmxy interface combines a self-service user console with an admin workspace for routing, monitoring, and billing operations.
User overview
Account balance, subscriptions, and quick-start API examples give users a clear path from signup to first request.
Models and API snippets
Expose available models with protocol tags and copy-ready curl, JavaScript, and Python request examples.
Smart routing rules
Configure weighted, fallback, and prompt-aware routes across upstream providers from the admin console.
Usage and billing logs
Filter usage by user, key, model, status, and label while tracking cost, latency, and token consumption.
An open foundation with an operated product path
Inspect and self-host the llmxy source code, or use the Learnmark product when your team needs implementation, environment ownership, controlled upgrades, and production support.
Open Source LLM Infrastructure
Self-hostablellmxy Open Source
A public, self-hosted LLM gateway with user and admin consoles, an OpenAI-compatible API, provider routing, quota enforcement, and usage billing.
The repository contains the complete web, administration, and API services required to operate the gateway yourself. A direct FastAPI relay is available for simple deployments, while an optional Envoy path handles higher-throughput traffic without moving billing work onto the request path.
- Repository
- wilsonwu/llmxy
- Deployment
- Docker Compose or independently operated services
- API surface
- OpenAI-compatible chat, embeddings, and model endpoints
- Languages
- Python and TypeScript
Included in the repository
Open-source capabilities
01
User console
Provide account access, plans, top-ups, API key management, model discovery, and per-user usage and billing views.
02
Administration console
Manage users, orders, upstream channels, models, rates, plans, routing policies, route weights, and operational statistics.
03
Unified API
Expose OpenAI-compatible chat completions, embeddings, and model discovery while connecting to OpenAI, Anthropic, and Gemini upstreams.
04
Gateway controls
Apply authentication, balance and quota checks, rate limiting, usage recording, and billing through one control plane.
05
Smart routing
Configure channels, models, routes, policies, and weights from the admin application, with changes propagated to running gateway instances.
06
Optional Envoy relay
Run a C++ hot path with external authorization and asynchronous gRPC access logs while retaining the direct FastAPI relay for smaller deployments.
Architecture
How the project is organized
- Website
- Next.js user application for accounts, subscriptions, keys, model access, and usage.
- Admin
- Next.js control plane for providers, routing, plans, users, billing, and Envoy instances.
- API
- FastAPI service backed by PostgreSQL and Redis for protocol translation, authorization, quotas, and billing.
- Envoy
- Optional front proxy for the high-throughput relay path, external authorization, and asynchronous usage reporting.
Technology stack
Built for self-hosting and extension
Python 3.11+FastAPINext.js 14TypeScriptTailwind CSSshadcn/uiPostgreSQL 16+Redis 7+EnvoygRPCDocker Compose
Prerequisites
- Docker with Docker Compose for the shortest setup path
- Or Python 3.11+, Node.js 18+, pnpm, PostgreSQL 16+, and Redis 7+
- Local JWT, encryption, database, and provider credentials
Quick start
git clone https://github.com/wilsonwu/llmxy.git
cd llmxy
cp .env.example .env
docker compose up -d --buildProject repositories