CloudQuery

CloudQuery is an open-source ELT platform and cloud asset inventory. It synchronizes infrastructure metadata from cloud and SaaS sources into a data warehouse to enable insights and automation for platform teams. The software allows users to unify data across various environments to create a centralized source of truth for cloud configuration.

The software is deployed as a server or via a CLI, allowing users to automate the collection of metadata from a wide array of providers. It is designed to run entirely on the user's own infrastructure, which ensures that sensitive cloud data does not leave the local environment. This architecture is specifically intended for regulated, secure, and performance critical environments where data privacy is a primary requirement.

Key features

• Support for AWS, Azure, and GCP infrastructure metadata
• Integration with over 70 cloud and SaaS sources including GitHub and Wiz
• Data normalization and rate limit handling for specialized plugins
• SQL based querying for cloud asset data
• High performance data movement powered by Apache Arrow
• Code first extensible plugin system for developers
• Compatibility with various BI tools, Slack alerts, and Jira ticketing

CloudQuery is built for developers and platform engineers who need to manage cloud asset inventories, Cloud Security Posture Management (CSPM), or Cloud FinOps. By converting API data into SQL queryable formats, it removes the need to write custom scripts to interact with poorly documented APIs. The framework is composable, allowing it to fit into existing stacks regardless of the chosen orchestrator or destination. It enables the collection and unification of billing data to assist with cloud expense reduction and the enforcement of security policies across multiple cloud providers.

It is a flexible, developer centric data integration framework for teams managing complex cloud infrastructure at scale.