What is LLM cost per task and how do you measure it?

LLM cost per task means what you pay for one piece of work that passes your check. You add up the token cost of every attempt, including retries and failed runs, and divide by the tasks that passed. It is not the same as price per token, which ignores how many tokens and retries a task takes.

What does DeepSeek's cost per task chart show?

On DeepSeek's chart, cost per task rises to the right and pass rate rises towards the top. The best corner is the top left. In DeepSeek's own run, its V4.1 Flash-1017 update sits there, ahead of the earlier V4.1 Flash release on both counts. DeepSeek's release note (17 October 2026) plots DeepSWE v1.1 Pass@1 against average cost per task, with cost on a log scale. Pass@1 is the share of single attempts that pass. The chart prints a pass rate of 77.0% for 1017 at its max setting. It prints no cost value.

By our reading of the chart, 1017 at max sits at about $0.20 per task and V4.1 Flash at max at about $0.50. Those two cost figures are our reading of a log-scale plot, not numbers DeepSeek published.

DeepSWE and DeepSeek's chart report average cost per task, which spreads spend over every task, passed or failed. To turn it into cost per successful task as defined above, divide by the pass rate. For 1017 at max, about $0.20 divided by 0.77 is roughly $0.26.

The sources are DeepSeek's release note (17 October 2026) and the public DeepSWE v1.1 leaderboard (updated 22 September 2026, checked 19 October 2026). The benchmark scores also appear on the Hugging Face model card. Both V4.1 Flash columns are DeepSeek's own runs. They were not on the public DeepSWE leaderboard when we checked on 19 October 2026.

DeepSeek's chart labels its own runs mini_swe_v2 and the public leaderboard points mini-swe-agent, so the two may not be like for like. DeepSeek ran Terminal-Bench with the Claude Code framework.

DeepSeek publishes no error bars for its own runs. Rows on the public board carry plus or minus 1 to 6 points. So the 2.8-point gap between 77.0% and 74.2% is not proof on its own. DeepSWE is also a coding benchmark, so your workload may differ.

On the public board, the top five closed models score 70% to 74%. They average $2.36 to $13.41 per task (leaderboard as of 19 October 2026). Divided by each model's pass rate, that is about $3.19 to $19.16 per successful task. DeepSeek's chart also includes OpenAI points taken from OpenAI's own chart. One of them, GPT-6.1 Sol at its high setting, sits near 75% at under $1 per task (read from the chart). A closed model can still be the right pick if it is the only one that passes your bar.

Why is price per token a poor guide?

Price per token is a poor guide because the bill is tokens multiplied by price, and tokens per task vary widely. DeepSeek's pricing page says "the expense = number of tokens × price".

For its Flash model, the same page lists peak prices per 1M tokens. Input costs $0.30 (cache miss) or $0.006 (cache hit), and output costs $1.20. Output costs four times as much as uncached input, and cached input costs one fiftieth of uncached input.

The public leaderboard reports output tokens and steps per task, and they differ a lot. Claude Sonnet 5 at max used 214k output tokens and 268 steps, for $26.40 per task at a 54% pass rate. Gemini 3.8 Flash at high used 143k output tokens, for $2.36 per task at 74%. Both rows come from the leaderboard as of 19 October 2026.

Effort settings add another variable. On DeepSeek's chart, the low, high and max settings for 1017 sit at different points. Low and high cost less per task and pass fewer tasks than max. One model therefore has several cost per task figures, and you need to pick the setting you will actually run.

How do you calculate cost per successful task?

Divide the total cost of every attempt by the number of tasks that passed. Retries and failed runs count, because you paid for them.

Here is the arithmetic with made-up token counts. None of it is measured data. Model A charges $1 per 1M input tokens and $4 per 1M output tokens. Model B charges $3 and $12. Each attempt sends 20,000 input tokens. Model A writes 150,000 output tokens per attempt, and Model B writes 30,000.

For one attempt, Model A costs $0.02 for input plus $0.60 for output, which is $0.62. Model B costs $0.06 for input plus $0.36 for output, which is $0.42. Now run 100 tasks through each model, with retries counted.

Model A costs a third as much per token and costs more per successful task. It writes five times the output, retries more often and passes fewer tasks. Model B costs less than half as much per successful task.

How do you build a small eval set from real work?

Build it from real inputs, write the pass check first and keep everything else fixed.

1. Collect 30 to 50 tasks from your real work. That size is our suggested starting point, not a rule.
2. Write the pass check before you run anything. A test suite, a schema check or a reviewer's checklist all work.
3. Fix the harness, prompts, tools and effort setting. Change one thing at a time.
4. Run each task more than once where you can, because results vary between runs.
5. Log input tokens, cached input tokens, output tokens, steps and retries for every attempt.
6. Work out cost as tokens multiplied by price, summed over every attempt, including failures.
7. Divide that total by the number of tasks that passed.

If a person had to fix an output before it was usable, record that too. Review time is a real cost. Then plot pass rate against cost per task for each model and effort setting.

If you are weighing whether to run models yourself, swap tokens for GPU hours in step 6. Our guide to LLM inference cost, API versus self-hosted covers that comparison.

Do shorter outputs cut cost?

Shorter outputs cut cost only if the pass rate holds. DeepSeek says 1017 gives shorter outputs and faster task completion than the previous version. That is DeepSeek's claim, and the release note gives no output-token figures.

On DeepSeek's peak pricing, output tokens cost four times as much as uncached input tokens, so output length can move the bill. In agent runs, the context sent again at each step adds input cost too. A model that stops early or skips a step can look cheap and still fail your check. Record output tokens per attempt and the pass rate across the same tasks, then compare cost per successful task.

Where does Xinference fit?

The Xinference Model API serves DeepSeek V4.1 Flash and Kimi K3 through one OpenAI-compatible API, hosted in Australia on dedicated GPUs. DeepSeek V4.1 Flash is $0.30 per 1M input tokens and $1.20 per 1M output tokens (USD), listed on the pricing page. The benchmark and cost figures above are DeepSeek's and the leaderboard's, not measurements of the Xinference Model API.

An eval harness that already uses an OpenAI-compatible API can point at it with a base URL and API key change. Our explainer on what an OpenAI-compatible API is shows what that involves. The Model API page has the details.

Don't see the model you need? Talk to us about adding it.

Get API access