Meridian/Documentation
GitHubDashboard

A/B Experiments

Meridian lets you run A/B experiments between two LLM models without changing any application code. Route a percentage of traffic to each model and compare cost, latency, and quality side by side.

How experiments work

When a proxy request arrives, Meridian checks if you have an active experiment. If you do:

  1. The request is randomly assigned to Arm A or Arm B based on the configured split percentage.
  2. The model in the request is replaced with the model configured for that arm.
  3. The request is forwarded to the correct provider.
  4. The response is logged with experimentId and experimentArm fields.

Your application sends the same request every time. Meridian handles the model swap transparently.

Creating an experiment

  1. Navigate to the Experiments page in the dashboard.
  2. Click Create Experiment.
  3. Give it a name (e.g., “GPT-4o vs Claude Sonnet”).
  4. Select Model A and Model B from the dropdowns.
  5. Set the traffic split (default: 50/50). For example, 70/30 sends 70% of requests to Model A.
  6. Click Create.

Viewing results

The results view shows a side-by-side comparison for each arm:

GET /api/experiments/:id/results — Response
{
  "experimentId": "clx1abc123",
  "name": "GPT-4o vs Claude Sonnet",
  "modelA": "gpt-4o",
  "modelB": "claude-sonnet-4-20250514",
  "armA": {
    "requests": 523,
    "avgCostUsd": 0.0042,
    "avgLatencyMs": 1230
  },
  "armB": {
    "requests": 477,
    "avgCostUsd": 0.0038,
    "avgLatencyMs": 980
  }
}

Use cases

Cost optimization

Compare GPT-4o against GPT-4o-mini to see if the cheaper model delivers acceptable quality for your use case. A 50/50 split over 1,000 requests gives you clear cost-per-request data.

A: gpt-4oB: gpt-4o-mini

Cross-provider comparison

Route traffic between OpenAI and Anthropic to compare latency, cost, and output quality. Useful when evaluating whether to switch providers.

A: gpt-4oB: claude-sonnet-4-20250514

New model evaluation

When a provider releases a new model, send a small percentage of traffic to it while keeping most traffic on your proven model. Gradually increase the split as confidence grows.

A: gemini-2.5-proB: gemini-2.5-flash

Stopping an experiment

Toggle the experiment to Inactive from the experiments page. Once inactive, all requests use the model specified in the original request body. Historical experiment data is preserved for analysis.

Limitations

  • Only one experiment can be active at a time per user.
  • Experiments apply to all proxy requests from the user, regardless of which proxy key is used.
  • Both models must be of the same type (e.g., both chat models). Cross-type experiments (chat vs image) are not supported.