Quick Start

This guide will help you run your first Litmus test in just a few minutes.

Prerequisites

• Litmus installed on your system (see Installation)
• An OpenRouter API key, or a Cloudflare AI Gateway (see Providers)

Step 1: Set Your API Key

Set your OpenRouter API key as an environment variable:

export OPENROUTER_API_KEY="your-api-key"

To use Cloudflare AI Gateway instead, see Providers.

Step 2: Create a Test File

Create a test file called tests.json:

[
  {
    "name": "Extract person info",
    "input": "John Smith is 30 years old and works at Acme Corp",
    "expected": {
      "name": "John Smith",
      "age": 30,
      "company": "Acme Corp"
    }
  },
  {
    "name": "Extract another person",
    "input": "Jane Doe, age 25, is employed by TechStart Inc",
    "expected": {
      "name": "Jane Doe",
      "age": 25,
      "company": "TechStart Inc"
    }
  }
]

Step 3: Create a JSON Schema

Create a schema file called schema.json:

{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "age": { "type": "integer" },
    "company": { "type": "string" }
  },
  "required": ["name", "age", "company"],
  "additionalProperties": false
}

Step 4: Create a Prompt File

Create a prompt file called prompt.txt:

Extract the person's name, age, and company from the given text.

Step 5: Run Tests

Run your tests against a model:

litmus run --tests tests.json --schema schema.json --prompt-file prompt.txt --model openai/gpt-4.1-nano

Step 6: Compare Models

Run tests against multiple models to compare them:

litmus run --tests tests.json --schema schema.json --prompt-file prompt.txt --model openai/gpt-4.1-nano --model mistralai/mistral-nemo

Next Steps

• CLI Reference - Learn about all available options
• Test File Format - Understand the test file structure
• JSON Schema - Learn about JSON schema requirements