LLM Evals settings

LLM Evals settings

This is where you manage API keys for the LLM providers your organization uses. Keys are shared across all projects. Without at least one key configured, you can't run experiments or arena comparisons.

Supported providers

You can add API keys for these providers:

• OpenRouter: Routes to multiple providers through one key
• OpenAI: GPT-4o, GPT-4 and other OpenAI models
• Anthropic: Claude model family
• Custom: Any OpenAI-compatible endpoint (bring your own base URL)

Adding an API key

1. Click Settings at the bottom of the LLM Evals sidebar.
2. Click Add API key.
3. Select a provider from the list.
4. Paste your API key. The format is validated automatically (e.g., OpenAI keys must start with sk-).
5. Click Add API key to save.

Keys are encrypted before storage. The settings page only shows a masked version (e.g., sk-...abc123).

Managing keys

Each provider can have one active key at a time. To update a key, click the edit icon next to the provider card and enter the new key. To remove a key entirely, click the delete icon and confirm.

Who can manage keys

Only users with the Admin role can add, edit or delete API keys. Other roles can see which providers are configured but can't view or modify the keys themselves.