By Artem Korsakov ·
When an assistant outputs text or code in a minute, it is easy to feel the acceleration. Then you have to clarify requirements, fix errors and assemble parts together. In another task, it really removes most of the routine. Therefore, it is more useful to measure a specific process than to argue about neural networks in general.
Why feelings should be checked
The METR research group compared in 2025 how experienced developers solve tasks in familiar open-source projects with and without AI. Then developers with AI worked slower than they themselves expected. Now METR calls these results outdated. METR Research, 10 July 2025.
On February 24, 2026, METR published the results of a repeat experiment and called it an unreliable assessment itself. Many developers refused to work without AI, some tasks did not make it into the sample for the same reason, and for those who run several AI agents in parallel, time is already difficult to measure. According to the authors' estimates, in early 2026, AI probably accelerates developers more than a year ago, but there is little data on the magnitude of acceleration. METR Update, 24 February 2026.
Choose one recurring task
For internal verification, prepare a standard proposal, review an incoming email or search for an answer in agreed instructions. Define the result and quality criteria in advance. Otherwise, without AI a person will make a short version, and with AI an extended one, and the comparison will lose its meaning.
Do not choose only convenient examples for the assistant. Include typical complex cases: incomplete data, contradiction in documents, out-of-scope requests. The distribution of such cases should be roughly the same in the compared groups.
What to record for each attempt
Make a simple log:
- task type and complexity;
- tool, version and date;
- employee active time;
- total time until ready;
- verification and correction time;
- accepted result and errors found;
- direct usage costs.
Active time and calendar deadline differ. While the assistant works, a person can perform another task. These minutes cannot be simultaneously fully recorded as manual work in two projects. But waiting can still delay handing over the result to a client.
How to organize an honest comparison
It is better to compare similar but different tasks. Repeating the same task without AI after a ready answer gives an advantage due to already received knowledge. Distribute tasks in advance and alternate conditions if possible, so as not to confuse the effect of a tool with employee experience.
The verifier uses identical criteria and, if possible, does not know how the result was prepared. Record incomplete attempts separately. Excluding all failures makes the assessment unreasonably optimistic.
A small internal measurement does not become a scientific study. It helps make a local decision. If there are few tasks or the spread is large, it is better to show the range of observations and continue collecting. An exact percentage for the entire business in this case is misleading.
How to calculate time and money
Training example: without AI the task took 60 minutes, with AI preparation took 20, verification 15, corrections 10. Total 45 minutes, active time decreased by 25%. The same result can be recorded as acceleration of approximately 1.33 times (60 / 45). The calculation is conditional, there is no trike_dev measurement behind it.
Подпись: Illustrative example. If you only consider preparation, the acceleration looks three times greater than it actually is.
For the budget, multiply the saved active time by the agreed hourly rate and subtract tool and maintenance costs. One-time launch costs are accounted for separately. Freed-up time gives the team a reserve for other work. Monetary expenses will decrease only if costs or workload actually change.
When to consider the pilot useful
Before launch, agree on which errors are unacceptable and what task you want to improve. Faster is not always better: reducing verification at the cost of missed obligations can worsen the result. There may be another benefit: quality has increased at the same time, but this needs to be measured separately.
There is no own completed trike_dev experiment in this article. A methodology for future verification is described. You can start with a proposal prototype: define an identical document composition, acceptance criteria and measure preparation with verification separately.
Related reading: Development with AI: where independent capabilities end; Justification for price increase: you want to understand what you are paying for.
Частые вопросы ### How many tasks are enough for a conclusion?
There is no universal number. It depends on the spread, differences in tasks and required confidence. For the first management decision, collect repeated observations, keep errors and explicitly limit the conclusion to the selected process and period.
Can you compare only subscription cost with salary?
No. A subscription itself does not perform the entire process and does not eliminate verification. Compare the total cost of an accepted result, including human work, corrections, maintenance and possible direct usage costs.