dstack

dstack is an open-source application that provides a unified control plane for GPU provisioning and orchestration. It manages compute workloads across various GPU clouds, clusters, and on-premise servers, specifically targeting the needs of machine learning teams. The software simplifies the process of moving between different compute environments by providing a consistent interface for resource allocation.

The software is deployed as a server that can run on Linux, macOS, or Windows via WSL 2. Users interact with the system through a command line interface or a programmatic API to define and apply configurations for their infrastructure. It supports the full machine learning lifecycle, including the deployment of interactive development environments, large-scale training tasks, and production model inference services.

Key features

• Support for NVIDIA, AMD, Google TPU, and Tenstorrent accelerators
• Management of cloud and on-prem clusters through fleets
• Provisioning of interactive development environments for desktop IDEs
• Scheduling of single and distributed jobs via tasks
• Deployment of models and web apps with auto-scaling and authorization
• Management of persisted storage volumes
• Integration with AI agents via specific skill sets for CLI and config management
• Automated handling of job queuing, port-forwarding, and out-of-capacity errors

The system uses YAML files to define infrastructure as code, allowing teams to version their fleet and task configurations directly within their Git repositories. This approach enables a declarative workflow where the state of the cluster is managed via the dstack apply command. It integrates with major cloud providers including AWS, Azure, and GCP, as well as specialized GPU clouds and Slurm clusters. The architecture allows for the disaggregation of persistent disks and the use of SSH proxies to maintain secure access to remote compute nodes without complex networking overhead.

dstack is a specialized orchestrator for AI workloads, focusing on the specific requirements of model development, fine-tuning, and inference across hybrid cloud environments.