LLMFlux
A command-line pipeline for running LLMs over large datasets offline — thousands of prompts as SLURM-scheduled GPU jobs on Illinois HPC, in the OpenAI batch format.
What it is
LLMFlux is a CLI batch-processing pipeline for researchers who need to run LLMs on thousands of prompts as offline, SLURM-scheduled GPU jobs. You provide prompts in the OpenAI batch JSONL format; LLMFlux schedules the work on HPC nodes, runs it against a vLLM or Ollama engine, and returns results with detailed performance metrics.
It also offers a llmflux serve mode that stands up a long-running, OpenAI-compatible endpoint on a SLURM compute node for ad-hoc interactive use — llmflux connect retrieves the endpoint URL and API key.
When to use it
Good fit
- You have a large dataset to process offline (classify, extract, transform).
- You already work on Illinois HPC with a SLURM allocation.
- You care about throughput and reproducibility, not a UI.
- You want models running entirely on University hardware.
Look elsewhere if
- You want a chat interface — see IllinoisChat or Lumen.
- You need commercial models with routing — see Lumen.
- You want a managed deployment dashboard — see LLM Hub.
- You don’t have HPC / SLURM access.
Key capabilities
| Models | Run the open-weight model you choose (any HuggingFace model) on HPC GPUs — models download and cache automatically. Runs are containerized for reproducibility and can span multiple GPUs or nodes for larger jobs. |
|---|---|
| Access | Uses your existing HPC access — your cluster account and file permissions control everything, with no separate login. In serve mode, a per-session API key is generated and stored in a private file for you. |
| Metrics | Every run reports detailed performance data — latency (p50/p95/p99), throughput (requests and tokens per second), error and retry counts, and GPU utilization — in the job’s output and via llmflux status. |
| Cost | Free to use — no charges. Compute comes from your existing HPC allocation. |
| Where it runs | Illinois / NCSA HPC. Data stays on University infrastructure. |
Data & hosting note
LLMFlux runs open-weight models on HPC nodes you already have access to, so your prompts and data stay within the cluster’s filesystem and never go to an external vendor. Access control is inherited from your SLURM account and file permissions — there is no separate login.
What’s recorded: nothing centrally — your prompts and results live only in your own files on the cluster, which you control and can delete. LLMFlux sends no telemetry.