LiteLLM · Lumen · LLMFlux · llm-serving · IllinoisChat · MindRouter — as of July 2026
Purpose: Open-source Python library and proxy server ("LLM Gateway") providing a unified, OpenAI-compatible interface to 100+ LLM providers. Designed for developers and platform teams building multi-provider LLM applications. Ships the widest OpenAI-compatible endpoint surface of any tool here: chat, completions, embeddings, audio, fine-tuning, batches, assistants/threads, real-time WebSocket, RAG, MCP, and A2A.
Access & Auth: Virtual API keys (Bearer tokens) for all access. SSO via OAuth2/OIDC (Microsoft/Azure AD, Google, generic/Okta) — free for up to 5 users, enterprise beyond; SAML is not directly supported. Fine-grained RBAC with an Org → Team → Project → Key hierarchy (enterprise only).
Models & Routing: Native support for 100+ providers including OpenAI, Anthropic, Bedrock, Vertex AI, Mistral, Groq, Ollama, vLLM, and llama.cpp. Advanced load balancing with latency-based, cost-based, and usage-based routing strategies. Multiple backend instances of the same model are load-balanced automatically.
Observability: Integrates with 20+ platforms including Langfuse, OpenTelemetry, Datadog, and Langsmith. Prometheus metrics available in OSS. Audit-log recording is OSS (store_audit_logs); the audit-log query API and advanced per-team logging are enterprise-only. The /compliance/gdpr check endpoint is OSS.
Cost: Uses a hybrid pricing model — prices for known commercial providers are fetched on-demand from the GitHub raw model_prices_and_context_window.json (configurable via LITELLM_MODEL_COST_MAP_URL) and applied automatically per request. A bundled fallback (model_prices_and_context_window.json) is used in air-gapped environments. Self-hosted models default to zero cost; admins must set input_cost_per_token / output_cost_per_token manually in config.yaml.
Purpose: Self-hosted AI gateway for research institutions and universities. Combines a full web chat interface with an OpenAI-compatible API proxy, giving both end-users and programmatic clients governed access through a single layer.
Access & Auth: CILogon OAuth2 federated SSO for human users (institutional IdPs, included free). API keys (Bearer tokens) for programmatic/client access. Users are auto-assigned to groups based on IdP attributes such as affiliation, OU, or IdP. CSRF protection and HTTP security headers on all responses; hardened session cookie flags.
Models & Routing: Supports any OpenAI-compatible endpoint (Ollama, vLLM, llama.cpp, etc.) via config. Round-robin load balancing with health-check failover. No native provider SDKs — commercial APIs are reached via their OpenAI-compatible URLs.
Observability: Prometheus metrics endpoint covering tokens, requests, cost, endpoint health, and user counts. Full analytics dashboard (TimescaleDB) with growth charts, usage heatmap, and per-model breakdowns visible to admins. End-users see their own coin balance and API key management on their profile page, plus a dedicated /usage page with their own requests, tokens, cost, and activity heatmap.
Cost: Calculated using per-model rates in config.yaml (coins per 1M input/output tokens; per-audio-hour for ASR). As of v1.22.0, pricing and metadata (context window from vLLM/SGLang endpoints; modalities, reasoning, knowledge cutoff from models.dev) can be auto-synced via the admin config editor's Update/Update All buttons or the sync_models.py CLI. Sync is a manual trigger; startup and YAML hot-reload only re-read operator-entered YAML values (no endpoint probing).
Purpose: CLI batch processing pipeline for running LLMs on HPC clusters. Designed for researchers who need to process thousands of prompts offline as SLURM-scheduled GPU jobs using the OpenAI batch JSONL format. Also supports an llmflux serve mode that stands up a long-running OpenAI-compatible vLLM/Ollama endpoint on a SLURM compute node for ad-hoc interactive use (llmflux connect retrieves the endpoint URL and API key).
Access & Auth: No built-in authentication for batch jobs — access is controlled entirely by the HPC cluster (SLURM account credentials and file system permissions). llmflux serve jobs generate a per-session bearer API key passed to vLLM/Ollama, written to a permission-restricted (chmod 600) connection file.
Models & Routing: Supports vLLM and Ollama as inference engines on HPC nodes. Models are configured via YAML; HuggingFace models are downloaded and cached automatically. Execution is containerized with Apptainer/Singularity for reproducibility. No multi-model routing — one SLURM job per model run, with configurable multi-node/multi-GPU layouts.
Observability: Rich per-run metrics on every job: request latency p50/p95/p99, throughput (req/s, tok/s), retry/error stats, and output-token counts. For vLLM, scrapes vLLM's Prometheus /metrics endpoint (TTFT, inter-token latency, KV-cache, effective batch size). Also samples GPU utilization (avg/p50/p95/peak) via nvidia-smi. No self-hosted dashboard or Prometheus endpoint; metrics are per-run, written to the job's output JSON and shown via llmflux status.
Cost: No cost tracking or budgeting. Resource costs are managed externally by SLURM account allocations on the HPC cluster. LLMFlux has no awareness of token prices or usage quotas.
Purpose: Full-stack HPC model deployment platform for research institutions, composed of a FastAPI backend and a Next.js frontend. The backend (built on the vec-inf package) manages on-demand SLURM model deployments with full lifecycle tracking and GPU resource accounting; the frontend provides a streaming chat UI and a deployment management dashboard.
Access & Auth: CILogon OAuth2 (or local email/password fallback when CILogon is unconfigured) for web users; SHA256+pepper-hashed API keys (Bearer tokens, fixed 30-day expiry) for programmatic access. Per-deployment authorization with email-based sharing and a pending-invite workflow for users not yet registered.
Models & Routing: Models are deployed as on-demand SLURM jobs via vec-inf, each receiving a dedicated vLLM endpoint. Endpoints can be optionally exposed externally via Cloudflare Tunnel without a VPN. No cross-deployment load balancing — one endpoint per model deployment.
Observability: Real-time deployment status and log streaming directly from SLURM job output directories. Chat message voting for qualitative feedback collection. Vercel AI SDK OpenTelemetry telemetry is enabled and a per-deployment metrics endpoint exists (via vec-inf get_metrics); no Prometheus metrics or configured external exporter.
Cost: No cost tracking, token pricing, or budget enforcement. Resource costs are managed indirectly through SLURM allocations and deployment TTLs with automatic expiry-based shutdown.
Purpose: Full-stack educational AI platform for universities. Provides RAG-based Q&A over course materials, with document ingestion (PDF, DOCX, PPTX, web crawl, Canvas LMS with auto-accept enrollment, PubMed), multi-provider model routing, agentic tool use (capped at 20 steps per turn), and course-level multi-tenancy. Also powers several non-course, domain-specific corpora (patents, PubMed, agriculture "CropWizard", "Vyriad"). Built by NCSA/Illinois for student and faculty use across UIUC courses. Backend (ai-ta-backend) is tagged v0.0.13; the crawlee ingestion service declares v0.0.2.
Access & Auth: Keycloak OAuth2/OIDC for user SSO; optional email-based per-course access lists; API keys for programmatic access. Three-tier RBAC per course: owner → admin → approved users.
Models & Routing: Routes to OpenAI, Anthropic (with extended thinking), Azure OpenAI, Google Gemini, AWS Bedrock, SambaNova, Ollama, NCSA-hosted models, in-browser WebLLM (local, no-network inference via @mlc-ai/web-llm), and an OpenAI-compatible provider (OpenRouter, Groq, Cerebras, Mistral, DeepSeek, and custom endpoints). No cross-provider load balancing — per-request provider selection.
Observability: Sentry error tracking (backend only), PostHog product analytics, LangSmith conversation logging. Per-course cost totals (prompt + completion + embedding prices) tracked in PostgreSQL. Nomic Atlas embedding-map visualizations of course documents/conversations, regenerated by a daily cron. No Prometheus metrics.
Cost: Token costs tracked per course in the database (total_prompt_price, total_completions_price, total_embeddings_price). No budget enforcement or spending caps — tracking only.
Purpose: Production-ready LLM inference load balancer and translation layer (NSF-funded, University of Idaho) fronting heterogeneous Ollama and vLLM backend clusters behind a unified OpenAI/Ollama/Anthropic-compatible API. Provides a Weighted Deficit Round Robin fair-share scheduler, per-user/group token quotas, full prompt/response audit logging with tiered archive retention, real-time per-GPU telemetry via sidecar agents, and a complete admin/user/chat web dashboard.
Access & Auth: Argon2-hashed API keys (Bearer or X-API-Key, mr2_-prefixed, with a service-key variant) and signed session cookies for the dashboard; optional Azure AD/Entra ID SSO with JIT group provisioning. Group-based RBAC including a read-only auditor role; legacy student/staff/faculty/admin hierarchy mapped to scheduler weights.
Models & Routing: Ollama and vLLM backends (plus diffusion for image generation), auto-discovered via /api/tags and /v1/models. Multi-factor backend scoring (model-loaded, GPU util, latency, queue, throughput, priority) combined with a Weighted Deficit Round Robin fair-share scheduler; per-backend circuit breakers, retry-with-different-backend, and num_ctx/max_tokens context management.
Observability: Full Request/Response audit tables (user, model, tokens, prompt/response, timing), AdminAuditLog, Prometheus /metrics, OpenTelemetry (OTLP/gRPC), structlog JSON logs, per-node GPU sidecar telemetry + IPMI power, and a tiered archive database for long-term retention. Scheduler decisions persisted for fairness debugging.
Cost: Token budgets (not dollar) per user with group defaults on a rolling period; hard-stop enforcement (HTTP 429) with auto-reset. No dollar pricing or spend reporting.
| Feature | LiteLLM | Lumen | LLMFlux | llm-serving | IllinoisChat | MindRouter |
|---|---|---|---|---|---|---|
| Core | ||||||
| Primary purpose | Developer SDK + LLM gateway | Institutional AI gateway + end-user chat | CLI batch processor for HPC LLM inference | HPC model deployment manager + chat UI | Educational AI tutoring platform — RAG over course materials with multi-provider model routing | LLM inference load balancer & translation layer for Ollama/vLLM backend clusters (NSF-funded, University of Idaho) |
| OpenAI-compatible API | Yes — full /v1/* proxy |
Yes — OpenAI-compatible /v1/chat/completions, /v1/completions, /v1/models, /v1/audio/* proxy (no embeddings/images/files/assistants) |
Partial — batch mode is a client-side processor with no exposed API; llmflux serve mode starts a real network-exposed OpenAI-compatible endpoint (vLLM/Ollama) on a SLURM node |
Yes — exposes deployed vLLM endpoints as standard /v1/* |
Partial — custom /api/allNewRoutingChat endpoint; not a standard /v1/* proxy |
Yes — full /v1/* (chat, completions, embeddings, rerank, images, OCR, models); also Ollama /api/* and Anthropic /anthropic/v1/* |
| Web chat UI | Partial — full admin dashboard and playground/chat UI (Next.js app in litellm/ui/litellm-dashboard/); compiled build is mounted at /ui |
Yes — chat with history (cursor-based pagination, "Load more…"), uploads, markdown/LaTeX; model name shown on each assistant message; site-wide announcement banner | No — CLI only | Yes — Next.js streaming chat with history, artifact generation, model selection, and message voting | Yes — full Next.js chat with conversation history, model selector, file uploads, shareable links, and agent mode | Yes — Bootstrap 5 chat with conversation history, file upload + drag-and-drop, markdown/LaTeX/code highlighting, model switching, web-search toggle, thinking controls |
| Python SDK / library mode | Yes — embed directly in Python apps; also exposes OpenAI-compatible proxy | Yes — via OpenAI-compatible API; use the standard openai Python SDK pointed at Lumen |
Yes — Python CLI and importable library for submitting HPC batch jobs | Yes — via OpenAI-compatible vLLM endpoints; standard openai SDK compatible |
No — web application only; no SDK | Partial — OpenAI/Ollama/Anthropic SDK-compatible via base URL; no native Python SDK library |
| Streaming responses | Yes | Yes | No — batch-only; no real-time streaming | Yes — Vercel AI SDK SSE streaming | Yes — SSE via Vercel AI SDK across all providers | Yes — SSE (OpenAI/Anthropic), NDJSON (Ollama) |
| Reasoning / thinking support | Yes | Yes — collapsible thinking blocks in UI; thinking content saved and replayed on conversation reload; token popup shows thinking tokens separately (hidden when zero) | Partial — passes model parameters through to vLLM; no UI-level handling | Yes — dedicated collapsible, streaming-aware reasoning UI (message-reasoning.tsx); depends on deployed model emitting reasoning tokens |
Yes — Claude extended thinking, OpenAI o3/o3-mini, plus DeepSeek-R1, Qwen3, GLM, Grok, GPT-OSS, Magistral, and OLMO reasoning models — each with provider-specific reasoning-token handling | Yes — per-request thinking toggle + effort dropdown; <think> tag extraction for Qwen3; reasoning_content field normalization |
| File / document uploads | Partial — OpenAI-compatible /v1/files API backing fine-tuning/batch/assistants; playground supports image upload for vision chat |
Yes — PDF, images, text, CSV, JSON | Partial — vision images and CSV/text ingested via pre-processing converters to JSONL; no runtime upload | Partial — AI-generated document artifacts (text/code/image/sheet via ProseMirror/CodeMirror/react-data-grid); user image uploads (JPEG/PNG, ≤5 MB) to chat via S3-compatible storage | Yes — PDF, DOCX, XLSX, PPTX, images (vision); stored to S3 and ingested into RAG pipeline | Yes — chat accepts images, PDF, DOCX, XLSX, CSV, JSON, MD via button + drag-and-drop; vision model auto-handling |
| Response caching / semantic cache | Yes — Redis, Redis Cluster, S3, GCS, Azure Blob, in-memory, disk, Qdrant/Valkey semantic caches | No | No | No | No | No |
| Model Support | ||||||
| Provider breadth | 100+ providers (OpenAI, Anthropic, Bedrock, Vertex, Mistral, Groq, HuggingFace, Ollama, vLLM, llama.cpp, …) | Any OpenAI-compatible endpoint (Ollama, vLLM, llama.cpp, etc.) — no native provider SDKs | vLLM and Ollama on HPC nodes; HuggingFace model download + caching; no commercial provider support | vLLM on SLURM/HPC via vec-inf; no native commercial provider SDKs | OpenAI, Anthropic, Azure OpenAI, Google Gemini, AWS Bedrock, SambaNova, Ollama, NCSA-hosted, in-browser WebLLM, OpenAI-compatible (OpenRouter/Groq/Cerebras/Mistral/DeepSeek/custom) | Ollama and vLLM backends only (plus diffusion for image generation); no commercial provider SDKs |
| Dynamic model addition (no restart) | Yes — via /model/new API |
Yes — YAML hot-reload (~5 s) | Yes — add to YAML config and resubmit job | Yes — background model catalog sync from vec-inf (hourly) | Partial — admin configures models in course settings; requires code change for new provider types | Yes — auto-discovery polls backends; admin can add/remove backends and pull Ollama models from UI without restart |
| Model health monitoring | Partial — endpoint tracking | Yes — per-endpoint up/down with dashboard | No | Yes — full deployment status lifecycle with real-time background sync | No | Yes — per-backend health via Ollama /api/tags and vLLM /health; circuit breakers; /status cluster summary |
| Load balancing strategies | Latency-based, cost-based, usage-based, least-busy, random, priority tiers | Round-robin with health-check failover | No — one SLURM job per model run | No — one endpoint per model deployment | No — per-request provider selection; no load balancing | Multi-factor scoring (model-loaded, GPU util, latency, queue, throughput, priority) + Weighted Deficit Round Robin fair-share |
| Same-model multi-instance scaling | Yes — list the same model pointing at N backend endpoints in config.yaml; the load balancer distributes traffic across all instances automatically; new instances added via /model/new API without restart |
Yes — add multiple entries for the same model in YAML; round-robin spreads requests across all entries; hot-reloaded within ~5 s | N/A — each job processes an independent batch; submit multiple parallel jobs for the same model to scale throughput; no runtime request distribution | No — one active deployment per (model, user) pair enforced by a partial unique index; different users can each deploy the same model, and the same user can redeploy after the previous instance terminates; no automatic scaling/replication | No — delegates directly to provider APIs; no instance management | Yes — multiple backends serving the same model; scorer + WDRR distribute requests; add backends from UI without restart |
| Multi-model concurrency | Unlimited — any number of models from any provider active simultaneously; constrained only by backend capacity | Unlimited — any number of OpenAI-compatible endpoints; constrained only by backend capacity | Unlimited — each model is a separate SLURM job; constrained by cluster GPU availability and job queue | Limited by GPU pool — each deployment consumes GPUs from the SLURM cluster; the resource summary API tracks available vs. allocated capacity across all active deployments | Unlimited — any number of configured providers active simultaneously; constrained only by API rate limits of upstream providers | Unlimited — any number of models across registered backends; constrained by GPU capacity and per-backend max_concurrent |
| Authentication & Authorization | ||||||
| Authentication method | Virtual API keys (Bearer tokens, self-service); SSO/SAML (Okta, Azure AD, Google Workspace, OIDC) — enterprise only | CILogon OAuth2 (federated institutional SSO) for users; API keys (Bearer tokens) for programmatic / client access | Partial — batch jobs rely on HPC cluster auth (SLURM account, file system permissions); serve jobs add a generated per-session bearer API key |
CILogon OAuth2 + optional local email/password for web users; SHA256-hashed API keys (with expiry) for programmatic access | Keycloak OAuth2/OIDC for users; per-course email-based access lists; API keys for programmatic access | Argon2-hashed API keys (Bearer or X-API-Key, mr2_-prefixed) + signed session cookies; optional Azure AD/Entra ID SSO with JIT provisioning |
| Model access control | Allow-lists per key / team | Per-model access (allow / block) + sticky needs_ack acknowledgement + disabled flag; allow/block overridable per group / user / client (config v2 as of v1.18; legacy whitelist/graylist/blacklist still parses with a deprecation warning) |
No | Per-deployment authorized user list; email-based sharing | Per-course access control; course admins choose which models are available per course | Partial — per-model capability overrides and admin enable/disable; no per-key model allow-lists |
| Model acknowledgement / consent | No | Yes — sticky per-model needs_ack: user must acknowledge a configurable ack_message before first use; API clients exempt via api.consent |
N/A | No | No | No |
| RBAC hierarchy | Org → Team → Project → Key — enterprise only | Admin / User / Project (flat) | No | No — no admin/user role distinction; per-deployment "owner"/"user" permission only | Course owner → Course admin → Approved user (three-tier per course) | Group-based (admin / auditor / user groups) + legacy student/staff/faculty/admin hierarchy mapped to scheduler weights |
| Audit logs | Yes — recording is OSS (store_audit_logs); the audit-log query API is enterprise only |
No | No | No | No — conversation logging via LangSmith; no admin audit trail | Yes — full Request/Response audit tables (user, model, tokens, prompt/response, timing) + AdminAuditLog; configurable prompt/response logging |
| Guardrails / content moderation | Partial — large OSS guardrail catalog (~50 integrations incl. Presidio PII, Lakera, Guardrails AI, Bedrock Guardrails, OpenAI moderation); LlamaGuard, LLM Guard, banned keywords/user lists, secret detection — enterprise only | No | No | No | No | Partial — DLP scanner (regex + GLiNER NER + LLM) with severity-graded alerts and admin acknowledgement workflow |
| HTTP security headers / CSRF | Yes — built-in SecurityHeadersMiddleware (X-Frame-Options, CSP frame-ancestors, X-Content-Type-Options, opt-in HSTS); CORS configurable |
Yes — X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Strict-Transport-Security on all responses; CSRF protection on web routes; hardened session cookies (Secure, HttpOnly, SameSite=Lax, 24 h lifetime) |
N/A — CLI tool; no HTTP server | Partial — Next.js security defaults; no explicit hardened header middleware | Partial — Next.js defaults; no explicit hardened header middleware | Partial — signed + hardened session cookies; CSRF on dashboard routes; TLS via nginx/Apache reverse proxy (app does not terminate TLS) |
| Secret / vault integration | Yes — AWS KMS, AWS Secrets Manager, Azure Key Vault, GCP KMS, HashiCorp Vault, CyberArk — enterprise only | No — env vars only | No — env vars only | No — env vars only | Partial — Infisical used for secret injection in dev/build scripts; production runtime still relies on env vars | No — env vars + DB-driven config only |
| Compliance certifications | SOC 2 Type 2, ISO 27001 (hosted cloud) — enterprise only | No — self-hosted; compliance is operator's responsibility | No | No | No — self-hosted; compliance is operator's responsibility | No — self-hosted; compliance falls to the operator |
| Service-account / machine clients | Yes — virtual keys | Yes — named Project entities (formerly "Clients", renamed v1.21.0) with API keys | N/A | Yes — API keys with expiry for programmatic access | Yes — API keys stored in database per user | Yes — service keys (non-expiring, named, with data_risk_level + compliance tags) via admin-approval workflow |
| Admin role | Yes | Yes — verified per request against live config | No | No — no admin role exists in current codebase | Yes — course owner and admin roles verified per request | Yes — group is_admin flag; full admin dashboard at /admin |
| User & Team Management | ||||||
| User / team / org hierarchy | Users → Teams → Orgs | Users → Groups (with Project service accounts) | No | Users (flat; per-deployment sharing via authorized user lists) | Users → Course (owner / admin / approved-user tiers per course) | Users → Groups (one group per user; groups drive quotas, weights, admin/auditor flags) |
| Auto-assign users to groups | No | Yes — based on IdP attributes (affiliation, OU, IdP) | N/A | No | No — manual email-based access lists per course | Partial — Azure AD SSO maps job-title → group via JIT provisioning; no IdP-attribute mapping for non-SSO users |
| User management UI | Yes — admin dashboard | Yes — sortable / paginated table, per-user usage | No | Partial — basic user list via auth provider; no dedicated admin user management UI | Yes — course settings panel to manage admin list, approved users, and access control | Yes — /admin/users (search/sort/paginate, group assignment, quota mgmt, masquerade, API keys) |
| Model request workflow | No | No | No | Partial — users submit deployment requests with resource specs, date range, and purpose, recorded with a "pending" status; the schema supports a full status workflow (pending → approved/rejected → active → completed) and a generic update endpoint exists, but no approve/reject UI is implemented (the dialog shows a "Coming soon" toast) | No | Partial — quota-increase and service-key requests with admin approval workflow; no model-deployment requests |
| Cost & Budget | ||||||
| Budget model | USD spend limits per key / user / team | Coin budgets (1 coin ≈ $1) per user, group, and project | No | No | No — cost tracking only; no enforcement or caps | Token budgets (not USD) per user with group defaults on a rolling period (default 30 days) |
| Budget granularity | Per key, per user, per team, per org | Per user, per group, per project | N/A | N/A | Per course (aggregated prompt + completion + embedding costs) | Per user (group-defaulted); per-key RPM override |
| Auto-refill / reset | Yes — scheduled periodic resets via budget_duration (ResetBudgetJob/APScheduler); soft budget triggers cooldown alerts (does not block) |
Yes — configurable hourly coin refill | N/A | N/A | N/A | Yes — rolling budget period auto-resets per user |
| Cost tracking granularity | Per request, per key, per user, per team | Per request, per key, per user, per model, per project | No | No | Per course — cumulative totals stored in database; no per-user or per-request breakdown | Per request (prompt/completion/total tokens) and per-user lifetime + period totals; no dollar pricing |
| Rate Limiting | ||||||
| Rate limit types | RPM and TPM per key / team — enforces both request count and token volume | Requests-per-minute per user / client only — no token-volume (TPM) enforcement | No | No | No — no API-level rate limiting; relies on upstream provider limits | Per-user RPM (Redis sliding 60s window); per-key RPM override; no TPM |
| Distributed (multi-worker) | Yes — Redis | Yes — Redis | N/A | N/A | N/A | Yes — Redis |
| Analytics & Observability | ||||||
| End-user spend / usage visibility | No — spend data is admin-only | Yes — profile page shows each user their own coin balance, API keys, and model access list; dedicated /usage page (all logged-in users) shows their own requests, tokens, cost, model popularity, and activity heatmap |
No | No | No | Yes — user dashboard shows token usage vs. quota, lifetime totals, API keys, usage history |
| Admin analytics dashboard | Partial — spend and token summaries | Yes — growth charts, heatmap, per-model breakdowns | No | No — the active-models page shows deployments the user owns OR has been shared with them; no admin role or global cross-user view exists | Partial — conversation stats and export per course; no spend or token dashboard | Yes — /admin with audit log, conversations, queue, metrics, energy, retention pages; cluster trends/throughput APIs |
| Prometheus metrics | Yes — full callback integration; request counts, latency histograms, errors, cost, and token metrics | Yes — tokens, requests, cost, health, users | No | No | No | Yes — /metrics (requests, latency, queue, backends, tokens) |
| External observability integrations | Langfuse, OpenTelemetry, Datadog, Langsmith, MLflow, GCS, S3, and more | No — Prometheus only | No | No | LangSmith (conversation tracing), PostHog (product analytics), Sentry (error tracking) | OpenTelemetry (OTLP/gRPC auto-instrumentation); structlog JSON logs; no Langfuse/Datadog/Langsmith |
| Time-series database support | No | Yes — TimescaleDB with continuous aggregates | No | No | No | No — MariaDB app + archive DB; no TimescaleDB |
| Embedding visualization / dataset exploration | No | No | No | No | Yes — Nomic Atlas interactive 2D embedding maps of course documents/conversations, regenerated by a daily cron | No |
| Deployment | ||||||
| Docker support | Yes | Yes — Docker + Docker Compose | Partial — Apptainer/Singularity for HPC containers; standard Docker is not the primary target | No — no Dockerfile or docker-compose found in either repo | Yes — multi-stage Docker for frontend (Node 18 Alpine) and backend (Python 3.10) | Yes — Docker + Docker Compose (dev + prod); Rocky Linux 8 deployment guide |
| Kubernetes / cloud-native | Yes — first-class; scales to 1500+ req/s | Partial — Helm chart runs it on k8s (bundled TimescaleDB + optional Redis, Gateway API HTTPRoute, entrypoint migrations with a wait-for-db init container + /healthz probes, optional in-cluster vLLM/SGLang), but fixed replicaCount with no HPA/autoscaling template and single-pod bundled DBs — single-institution scale |
No — HPC/SLURM only | No — HPC/SLURM only | Partial — Dockerized and stateless; current CI/CD deploys both frontend and backend to AWS ECS Fargate (plus a separate ECS ingest-worker service); a legacy Railway config remains in the backend repo; no k8s manifests in repo | No — Docker Compose only; no Helm chart or k8s manifests |
| Database | PostgreSQL (production); SQLite supported for dev/testing (Prisma) | PostgreSQL + TimescaleDB (SQLite for dev) | None — job registry in ~/.llmflux/jobs.json |
PostgreSQL (Drizzle ORM) | PostgreSQL + Qdrant (vectors) + Redis + Neo4j (optional knowledge graph); S3-compatible object storage | MariaDB 11.2 (required); Redis 7 (optional, recommended); separate MariaDB archive DB (optional) |
| Config hot-reload | Yes — via API for models | Yes — YAML watcher (~5 s); restart for breaking changes | No — restart required | Partial — model catalog syncs hourly in background; app restart required for config changes | Partial — course metadata and model/provider config are hot-reloadable via Redis (updated live through admin APIs); environment-variable changes require a restart | Yes — DB-driven AppConfig editable from admin UI without restart; env vars for bootstrap only |
| HPC & Batch Processing | ||||||
| HPC / SLURM integration | No | No | Yes — native SLURM job submission; multi-node/multi-GPU support; per-job status, logs, cancel commands; job registry at ~/.llmflux/jobs.json |
Yes — model deployments managed as SLURM jobs via vec-inf; GPU resource allocation tracking; automatic cleanup of expired jobs | No | No |
| Batch / offline processing | No | No | Yes — JSONL batch API; processes thousands of prompts as scheduled SLURM jobs; configurable batch sizes; intermediate result saving | No — interactive inference only | No — interactive inference only | No |
| Container-based HPC execution | No | No | Yes — Apptainer/Singularity; environment isolation; CUDA device visibility; HuggingFace model cache management | Yes — vLLM runs inside Apptainer/Singularity containers (.sif images) within SLURM jobs |
No | No |
| Built-in benchmarking | No | No | Yes — LiveBench dataset auto-download; synthetic dataset generation; per-run metric collection (latency p50/p95/p99, throughput, retry/error stats, vLLM Prometheus metrics: TTFT/ITL/KV-cache, and GPU utilization via nvidia-smi); configurable SLURM overrides per benchmark | No | No | No |
| Deployment Management | ||||||
| Deployment lifecycle management | No | No | Partial — tracks SLURM job states (PENDING / RUNNING / COMPLETED / FAILED) | Yes — full lifecycle: pending → launching → ready → running → shutdown / failed / completed; background status sync every 60 s; endpoint URL provisioning when ready | N/A — hosted application; no deployment lifecycle management | Partial — manages backend nodes (register/enable/disable/drain/remove) and system-wide force-offline toggle; not a SLURM-style deploy lifecycle |
| Deployment TTL / auto-expiry | No | No | No — SLURM time limits apply at the job level externally | Yes — configurable expiry (TTL) set from SLURM job duration when ready; automatic shutdown on expiry; extend-by-N-hours API exists but no frontend UI for extension; rate-limited expiry checks | N/A | No |
| Deployment sharing | No | No | No | Yes — email-based per-deployment sharing; pending invite workflow for unregistered users; automatic invite claiming at signup | N/A | No |
| GPU resource pool management | No | No | No — SLURM manages GPU resources externally | Yes — GPU pool with allocate/release lifecycle; resource summary API (available / allocated / total); per-deployment resource recording | N/A | Yes — Node/Backend separation with gpu_indices assignment; per-GPU telemetry aggregation via sidecar agent |
| Cloudflare Tunnel integration | No | No | No | Yes — optional tunnel URL provisioning for private HPC model endpoints; external access without VPN | No | No |
| Deployment lifecycle notifications | No | No | Partial — serve jobs email the user when the model is ready (with endpoint + API key) and on SLURM failure; batch jobs have no notifications |
Yes — SMTP email to all authorized users on state transitions (ready / failed / completed); idempotent per user/deployment/type | N/A | No |
| Protocols & Integrations | ||||||
| MCP (Model Context Protocol) | Yes — MCP Gateway; list/call tools, prompts, resources; HTTP/SSE/stdio; per-key/team access control; OAuth 2.0 + AWS SigV4 auth | No | No | No | No | Yes — hosted SSE server at /mcp/sse + local stdio bridge + .mcpb Claude Desktop extension; exposes web-search tool |
| A2A (Agent-to-Agent protocol) | Yes — dedicated providers: Bedrock AgentCore, Pydantic AI, LangFlow, Watsonx Orchestrate, LangGraph (completion bridge); load balancing, cost tracking, streaming. Generic bridge can invoke any LiteLLM provider | No | No | No | No | No |
| RAG (Retrieval-Augmented Generation) | Yes — built-in /rag/ingest and /rag/query proxy endpoints with vector store integration and reranking support |
No | No | No | Yes — full RAG pipeline: Qdrant vector search, multiple embedding strategies (OpenAI, Ollama nomic, Qwen), context ranking, query rewriting, web crawling ingestion (Crawlee), Canvas LMS ingestion, PubMed ingestion | No |
| Assistants / Threads API | Yes — full OpenAI-compatible /v1/assistants, /v1/threads, /v1/threads/{id}/messages, /v1/threads/{id}/runs |
No | No | No | No — custom conversation threading (not OpenAI-compatible) | No |
| Fine-tuning API | Yes — OpenAI-compatible /v1/fine_tuning/jobs endpoint |
No | No | No | No | No |
| Audio (TTS / transcription) | Yes — /v1/audio/speech (text-to-speech) and /v1/audio/transcriptions |
Partial — /v1/audio/transcriptions + /v1/audio/translations (speech-to-text only, billed per audio-hour); no TTS |
No | No | No | Yes — /v1/audio/speech (TTS) and /v1/audio/transcriptions (STT); proxies Kokoro/OpenedAI/faster-whisper |
| Real-time / WebSocket | Yes — /v1/realtime WebSocket endpoint for streaming audio and chat |
No | No | No | No — HTTP SSE streaming only | No — SSE/NDJSON streaming only |
Batch API (/v1/batches) |
Yes — OpenAI-compatible async batch API; submit a batch of requests and retrieve results later | No | No | No | No | No |
Responses API (/v1/responses) |
Yes — OpenAI Responses API with polling, cancel, input_items, and WebSocket support | No | No | No | No | No |
| Key external dependencies | PostgreSQL (required); Redis (optional, distributed rate limiting & caching); observability tools (Langfuse, Datadog, etc.) optional | PostgreSQL + TimescaleDB (required); Redis (optional, multi-worker rate limiting); CILogon (required for SSO) | SLURM scheduler; Apptainer/Singularity; vLLM or Ollama; HuggingFace model cache | PostgreSQL (required); SLURM + vec-inf (required); vLLM (required, runs in Apptainer/Singularity container); CILogon or local email/password for auth; SMTP relay for email notifications; Cloudflare Tunnel (optional); S3-compatible storage (required for chat file/image attachments) | PostgreSQL (required); Qdrant (required, vector search); Redis (required, caching); S3-compatible storage (required); Keycloak (required, SSO); RabbitMQ (required, async ingestion queue with a dedicated ECS worker service); Beam (serverless ingestion execution, e.g. Canvas/PubMed cron jobs); Neo4j (optional, knowledge graph); Nomic Atlas (optional, embedding visualization); LangSmith + PostHog + Sentry (optional observability) | MariaDB (required); Redis (optional); Ollama and/or vLLM backends (required); nginx/Apache reverse proxy; sidecar GPU agent per node |
| Support & Cost | ||||||
| License | MIT (open-source) | Not stated in repo — no LICENSE file, no license field in pyproject.toml, and no license mention in the README |
MIT (open-source) | Inconsistent — backend pyproject.toml declares "Proprietary" but ships no LICENSE file and its README claims MIT; frontend carries an inherited Apache-2.0 LICENSE (from the Vercel ai-chatbot template) |
MIT (open-source) | Apache 2.0 (open-source) |
| Software cost | Free (open-source); Enterprise from $250/mo (Basic) to ~$2,500/mo (Premium) | Free | Free | Free | Free | Free |
| Enterprise / commercial tier | Yes — SSO, audit logs, governance, compliance (SOC2, HIPAA) assistance | No | No | No | No | No |
| Community support | GitHub issues, Discord | GitHub issues | GitHub issues | GitHub issues | GitHub issues | GitHub issues |
| Paid support / SLA | Yes — dedicated Slack channel + SLA on Premium tier | No | No | No | No | No |
| Hosting | Self-hosted (you manage infra, DB, patching) | Self-hosted (you manage infra, DB, patching) | HPC cluster (SLURM jobs; no persistent server process) | Self-hosted backend + frontend; HPC cluster for SLURM model jobs | Self-hosted or cloud — AWS ECS Fargate (frontend, backend, and ingest-worker containers), plus managed DBs (Postgres, Qdrant, Redis) | Self-hosted (Docker Compose; you manage infra, DB, patching) |
| End-User Experience | ||||||
| Accessibility (WCAG) | No — not a stated goal | Yes — WCAG 2.1 AA required; ARIA labels and live regions throughout; full keyboard navigation; auto-dismiss alerts comply with WCAG 2.2.1 | N/A — CLI tool | No — not a stated goal | No — not a stated goal | Yes — WCAG 2.1 AA claimed; ARIA labels throughout; make test-a11y tests |
| Public / guest chat access | No | No | N/A — CLI tool | Yes — unauthenticated guest chat with an always-on model (5-message cookie limit) plus a public OpenAI-compatible API-key endpoint | No — Keycloak auth required | No — login required |
| Institutional theming | Partial — UI theme/settings configurable; custom branding/swagger enterprise only | Yes — Illinois, UIC, UIS themes; custom branding | No | Partial — CILogon skin configuration (default 'illinois-chat') plus Illinois-branded UI colors throughout; no custom color theme selector | Yes — built specifically for UIUC; per-course branding and system prompt configuration | Partial — dark mode toggle (persists, respects prefers-color-scheme); no multi-institution skin selector |
| Built-in help / docs system | No — external docs site | Yes — /help with screenshots, hot-reloaded markdown; developer/internal pages hidden from sidebar by default |
Partial — CLI --help flags plus a hosted docs site (llmflux.ncsa.ai) with configuration/models/testing guides; no in-app web docs server |
No | No | Yes — /documentation in-app docs page + FastAPI /docs and /redoc; in-repo docs/ |
| Target audience | Developer / enterprise engineering teams | Research institutions / universities | HPC researchers running large-scale batch inference jobs | Research HPC teams deploying and sharing on-demand model endpoints | University students and faculty doing RAG-based Q&A over course materials | Research institutions / universities running their own GPU clusters |