OpenLLMetry

OpenLLMetry is an open-source web app that provides observability for LLM applications. It functions as a set of extensions built on top of OpenTelemetry to monitor performance, track costs, and debug AI applications. The software is deployed as a server-side library and SDK for Python applications, allowing developers to gain visibility into the internal workings of their generative AI pipelines.

The system integrates with existing observability stacks by outputting standard OpenTelemetry data. It can be implemented via a dedicated SDK for quick setup or by adding specific instrumentations directly to an existing OpenTelemetry configuration. This architecture allows developers to track traces across various AI providers and databases without replacing their current monitoring infrastructure. By utilizing standard protocols, it ensures that telemetry data is portable across different backend providers.

Key features

• Instrumentation for LLM providers including OpenAI, Anthropic, and Mistral AI
• Support for Vector DBs such as Chroma, Pinecone, Qdrant, and Weaviate
• Integration with frameworks like LangChain, LlamaIndex, and CrewAI
• Compatibility with the Model Context Protocol (MCP)
• Export capabilities to destinations like Grafana, Honeycomb, and SigNoz
• Support for Azure Application Insights and Google Cloud observability
• Standard OpenTelemetry semantic conventions for LLM data

OpenLLMetry is designed for AI developers and data scientists who need to monitor the execution flow of complex generative AI applications. It captures detailed data from API calls and database queries, providing visibility into how prompts and responses move through a system. The tool supports a vast array of destinations, including Axiom, Dynatrace, and Sentry, making it compatible with most enterprise monitoring environments. Because it relies on the OpenTelemetry standard, it avoids vendor lock-in and allows users to switch their observability backend as their infrastructure grows.

This tool serves as a specialized observability layer for the LLMops lifecycle, bridging the gap between standard application monitoring and the specific needs of large language model orchestration.