Envoy AI Gateway

Envoy AI Gateway is an open-source proxy platform for routing application traffic to various Generative AI services, allowing developers to manage requests between clients and Large Language Model providers. The software is deployed as a server and utilizes a two-tier gateway pattern to organize how traffic flows from the application to the inference engine.

The first tier of this architecture functions as a centralized entry point for all incoming requests. This layer is responsible for high-level management tasks such as authentication, top-level routing, and the application of global rate limits to prevent service abuse. The second tier handles ingress traffic specifically for self-hosted model serving clusters, providing more granular control over how models are accessed and utilized within a private infrastructure.

Key features

• Support for over 16 LLM providers including OpenAI, Anthropic, and Google Gemini
• Integration with AWS Bedrock and Azure OpenAI
• Centralized authentication for AI service requests
• Top-level routing and global rate limiting at the first tier
• Fine-grained control over self-hosted model access
• Endpoint picker support for LLM inference optimization
• Compatibility with Mistral, Cohere, Groq, and Together AI
• Integration with DeepInfra, DeepSeek, Hunyuan, and SambaNova

The project is built on Envoy Gateway and adheres to CNCF standards. It is designed for organizations that require a cloud-native approach to LLM orchestration, particularly those utilizing Kubernetes for their model serving environments. By implementing the two-tier pattern, the system separates the concerns of general API management from the technical requirements of LLM inference optimization. This allows operators to optimize endpoint selection and manage traffic distribution more effectively across different model providers.

Envoy AI Gateway is a specialized infrastructure layer for managing GenAI traffic within a cloud-native ecosystem, focusing on scalability and provider flexibility.