Bencher

Bencher is an open-source continuous benchmarking platform that tracks performance metrics on bare metal hardware. It allows developers to run existing benchmarks locally or within CI pipelines using the same hardware environment to ensure consistent results. The software monitors performance over time and can trigger failures in pull requests when regressions are detected, preventing performance bugs from reaching production.

The system is deployed as a suite of tools including a CLI for orchestrating benchmark runs, an API server for storing and querying results, and a web console for visualization. It supports various deployment methods, including self-hosting via Docker or Kubernetes, as well as integration with GitHub Actions. This setup ensures that benchmark results are reproducible and not skewed by the noise typically found in shared CI runners.

Key features

• Execution of benchmarks on bare metal hardware to reduce variance
• Tracking and graphing of performance metrics over time via a web console
• Automated detection of performance regressions in pull requests
• CLI tool for running benchmarks and publishing results to the API
• Support for multiple languages including Rust, Go, C++, Java, and Python
• Integration with GitHub Actions for automated CI workflows
• Customizable analytics for setting performance thresholds and alerts

Bencher is designed for teams developing performance-critical software such as databases, compilers, operating systems, and cryptographic libraries. It integrates with a wide range of benchmark harnesses and supports Linux, macOS, and Windows across x86_64 and ARM64 architectures. The architecture separates the execution runner from the data storage and visualization layers, allowing for flexible on-premises or cloud configurations. By utilizing the bencher run command, engineers can specify whether the system should error on alerts or post detailed reports directly to GitHub PR comments.

This platform serves as a specialized tool for performance monitoring and automated testing within the software development lifecycle, focusing on high-precision metrics and historical data tracking.