SunsettingAI Dev Tools

OpenAI Evals platform (Evals dashboard, Evals API and Datasets)

by OpenAI

Announced

2026-06-03

Complete

2026-11-30

Refund window closes

—

Refund status

No refund applicable

On June 3, 2026, OpenAI notified developers that the Evals platform is being deprecated. Per OpenAI's deprecations page, existing evals become read-only on October 31, 2026, and the Evals dashboard and API are scheduled to shut down on November 30, 2026. The notice also appears at the top of the Datasets guide, and the Graders guide says graders are being deprecated as part of the evals and fine-tuning workflows they support. OpenAI's recommended path is Promptfoo, an open-source CLI and library. OpenAI's own cookbook says the migration is a manual rebuild of each eval, not an export and import. If you use Evals to check prompt or model changes before you ship them, you have until October 31 to keep editing and running there, and until November 30 to pull anything out.

Refund flow

1
Not applicable. Eval runs are billed as model usage on your API account, with no Evals subscription or prepaid balance to refund.
2
Before October 31, 2026: list your evals with GET /v1/evals, then for each one save the eval definition (GET /v1/evals/{eval_id}), its runs (GET /v1/evals/{eval_id}/runs) and each run's output items (GET /v1/evals/{eval_id}/runs/{run_id}/output_items). These are the endpoints in OpenAI's Evals API reference, read October 19, 2026. Keep the JSON somewhere you control.
3
Also keep a copy of any test data you built in the dashboard's Datasets tab. The Datasets guide carries the same deprecation notice, so do not treat it as a place to move to.
4
Between October 31 and November 30, 2026, plan on reading only. After November 30, assume the dashboard and API are gone and the data with them.

Migration path

Recommended successor

Promptfoo (open source CLI and library)

Why this is the closest fit

It is the tool OpenAI names. The deprecations page links a cookbook titled Moving from OpenAI Evals to Promptfoo, which says OpenAI is winding down the Evals product and recommends Promptfoo for continuing and extending your evaluation workflows. Per that cookbook, both use the same building blocks: test data, prompts, providers or models, and scoring criteria.

What differs from the original

Per the cookbook, Evals lives in the OpenAI dashboard while Promptfoo is a config file (promptfooconfig.yaml) run from the CLI or in CI. Graders become Promptfoo assertions and metrics. The cookbook says there is no export feature involved: you recreate each eval's prompts, provider, test cases and assertions by hand, run it with promptfoo eval -c promptfooconfig.yaml --no-cache, and view it with promptfoo view. That new run is separate from your old Evals runs, so old scores do not carry over. OpenAI also warns that similarity scores may not match across systems and that any recreated grader, especially an LLM-as-a-judge grader, should be checked before you trust it for regression decisions. Tool, agent and custom provider workflows need their own setup.

Alternatives by use case

You want the path OpenAI documents

Rebuild each eval in Promptfoo using the cookbook, then compare the first Promptfoo run against your saved Evals output items before you rely on it

Your evals mostly use exact-match or string-check graders

These map most directly to Promptfoo assertions. Re-run a known set of inputs and confirm the pass and fail counts match your last Evals run

Your evals use a model as the grader

Recreate the grader prompt word for word and spot-check its verdicts by hand. OpenAI says LLM-as-a-judge graders are the ones to validate before trusting

You only used the Datasets tab to try prompts

Download your rows now. A spreadsheet plus a small script that calls the model can stand in until you pick a tool

What this tool meant

Evals was OpenAI's in-dashboard way to answer one question before shipping: did this prompt or model change make outputs better or worse? That question gets more urgent this quarter, because several model shutdowns land close together and each one forces a model swap. A fixed test set run before and after the switch is how you catch a regression. Losing the hosted tool does not remove the need. It means the test set and grading logic should live in your own repo, where a vendor shutdown cannot delete them. For anyone using a model to write video prompts, shot lists or captions, a short eval set of real past inputs is the cheapest way to see what a forced model change did to output.