Welcome to Aegaeon¶
Aegaeon is a blazingly fast inference system for deploying multiple AI models on shared GPUs, enabling Model-as-a-Service (MaaS) both in local and cloud environments.
Key Features¶
- Token-level autoscaling — switch models between token steps instead of waiting for whole requests to finish.
- SLO-aware scheduling — coordinates Prefill and Decode work against TTFT and TPOT objectives.
- Fast model switching — combines host-side weight caching, prefetching, a Rust quick loader, and explicit KV-cache movement.
- Multi-model runtime — supports dynamic model deployment, multiple replicas, and load-aware request routing.
- vLLM backend — inherits model and kernel support across dense, hybrid, and MoE architectures from vLLM.
- Serving and observability — provides OpenAI-style Chat and Completion APIs, SSE streaming, and runtime state telemetry.
See our SOSP '25 paper for more technical details.
Documentation¶
| Guide | Contents |
|---|---|
| Quickstart | Install, configure, and send a request |
| Deployment | Single-node and multi-node serving |
| Architecture | Request flow and runtime components |
| Benchmarking | Metrics, profiling, and performance tools |
| HTTP API | Endpoints, schemas, and responses |
| CLI | Start, deploy, and query |
See Features and Compatibility for the supported configuration.