Kestra

Kestra is an open-source, self-hosted orchestration platform for data, AI, and infrastructure workflows. It uses a declarative YAML interface to unify scheduled and event-driven automation, allowing users to manage pipelines as code. The software is deployed as a server and is accessible via a web application, supporting various deployment methods including Docker, Kubernetes, and cloud templates for AWS and Google Cloud.

Users can build workflows through a built-in code editor or a visual interface that renders workflows as Directed Acyclic Graphs. The platform supports a wide range of execution environments, from local Docker containers to Kubernetes jobs and remote servers via SSH. It is designed for data engineers, DevOps teams, and AI developers who require a scalable control plane to manage complex task dependencies and automation across different teams and environments.

Key features

• Declarative YAML interface for defining workflows in a built-in code editor
• Git version control integration for managing workflows as code
• Support for both scheduled and real-time event-driven triggers
• Plugin ecosystem for extracting data from databases, cloud storage, and APIs
• Ability to run scripts in multiple languages including Python, Node.js, R, and Go
• Visual topology editor with real-time syntax validation and auto-completion
• Workflow resilience tools including retries, timeouts, and error handling
• Support for sequential, parallel, and dynamic tasks

Kestra operates as a language-agnostic orchestrator, meaning it does not lock users into a specific programming language for their task logic. It integrates with major cloud providers such as AWS, Google Cloud, and Azure, and supports big data tools like Apache Spark and Google BigQuery. The architecture utilizes namespaces for organization and isolation, which allows teams to separate development, staging, and production environments. It also includes a system for inputs and outputs that generate artifacts directly within the user interface for easier monitoring and debugging.

It is an infrastructure-as-code solution for data orchestration and pipeline management.