curl -X POST 'https://api.openai.com/v1/evals' \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
  "data_source_config": {
    "type": "custom",
    "item_schema": "{\n  \"type\": \"object\",\n  \"properties\": {\n    \"name\": {\"type\": \"string\"},\n    \"age\": {\"type\": \"integer\"}\n  },\n  \"required\": [\"name\", \"age\"]\n}\n",
    "include_sample_schema": false
  },
  "testing_criteria": [
    {
      "type": "label_model",
      "name": "example-name",
      "model": "example-model",
      "input": [
        {
          "role": "example-role",
          "content": "example-content"
        }
      ],
      "labels": [
        "example-labels"
      ],
      "passing_labels": [
        "example-passing-labels"
      ]
    }
  ]
}'

# POST /evals (OpenAI API)
Base URL: https://api.openai.com/v1
Auth: Authorization; OpenAI API key; required.
Request formats: application/json.
Last verified: 2026-10-20

> Create eval

## Critical Gotchas
* The Responses API is the primary interface for new text and tool workflows; Chat Completions remains a separate compatibility API.
* Audio transcription uploads use multipart form data rather than a JSON request body.

## cURL
curl -X POST 'https://api.openai.com/v1/evals' \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
  "data_source_config": {
    "type": "custom",
    "item_schema": "{\n  \"type\": \"object\",\n  \"properties\": {\n    \"name\": {\"type\": \"string\"},\n    \"age\": {\"type\": \"integer\"}\n  },\n  \"required\": [\"name\", \"age\"]\n}\n",
    "include_sample_schema": false
  },
  "testing_criteria": [
    {
      "type": "label_model",
      "name": "example-name",
      "model": "example-model",
      "input": [
        {
          "role": "example-role",
          "content": "example-content"
        }
      ],
      "labels": [
        "example-labels"
      ],
      "passing_labels": [
        "example-passing-labels"
      ]
    }
  ]
}'

## Parameters
| Name | In | Type | Required | Description |
| :--- | :--- | :--- | :--- | :--- |
| data_source_config | body | object | Yes | The configuration for the data source used for the evaluation runs. Dictates... |
| testing_criteria | body | object[] | Yes | A list of graders for all eval runs in this group. Graders can reference vari... |
| name | body | string | No | The name of the evaluation. |
| metadata | body | object | No | Set of 16 key-value pairs that can be attached to an object. This can be usef... |

## Response example
{"object":"eval","id":"string"}

---
Need an API key? Get one at https://platform.openai.com/api-keys

POST

POST /evals — OpenAI API

OpenAI API key (required); https://api.openai.com/v1

OpenAI API

Create eval

Base URL: https://api.openai.com/v1
Auth: Authorization: Bearer <OPENAI_API_KEY>
Last verified: 2026-10-20 · upstream hash matched

Actions Try in browser→

Agents: curl -H "Accept: text/markdown" this URL
→ 333 tokens · Vary: Accept

Critical gotchas

The Responses API is the primary interface for new text and tool workflows; Chat Completions remains a separate compatibility API.

Audio transcription uploads use multipart form data rather than a JSON request body.

cURL

curl -X POST 'https://api.openai.com/v1/evals' \  -H "Authorization: Bearer $OPENAI_API_KEY" \  -H 'Content-Type: application/json' \  -d '{  "data_source_config": {    "type": "custom",    "item_schema": "{\n  \"type\": \"object\",\n  \"properties\": {\n    \"name\": {\"type\": \"string\"},\n    \"age\": {\"type\": \"integer\"}\n  },\n  \"required\": [\"name\", \"age\"]\n}\n",    "include_sample_schema": false  },  "testing_criteria": [    {      "type": "label_model",      "name": "example-name",      "model": "example-model",      "input": [        {          "role": "example-role",          "content": "example-content"        }      ],      "labels": [        "example-labels"      ],      "passing_labels": [        "example-passing-labels"      ]    }  ]}'

Parameters

Name	In	Type	Required	Description
name	body	string	No	The name of the evaluation.
metadata	body	object	No	Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
data_source_config	body	object	Yes	The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
testing_criteria	body	object[]	Yes	A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`).

Response 200 OK

{
  "object": "eval",
  "id": "string",
  "name": "Chatbot effectiveness Evaluation",
  "data_source_config": {
    "type": "custom",
    "schema": {
      "type": "object",
      "properties": {
        "item": {
          "type": "object",
          "properties": {
            "label": {
              "type": "string"
            }
          },
          "required": [
            "label"
          ]
        }
      },
      "required": [
        "item"
      ]
    }
  },
  "testing_criteria": [
    {
      "type": "label_model",
      "name": "string",
      "model": "string",
      "input": [
        {
          "role": "user",
          "content": "string",
          "type": "message"
        }
      ],
      "labels": [
        "string"
      ],
      "passing_labels": [
        "string"
      ]
    }
  ],
  "created_at": 0,
  "metadata": {}
}

Get a free OpenAI API key → sponsored

Affiliate link - we may earn a commission, at no extra cost to you.