How we tested

How we tested the AI models

A plain-language explanation of the simulated negotiations, outcome scores, judge ratings, and limits.

The new outcome test

We wrote fictional instructions for a buyer and a seller. In the first published round, each of 20 models played the buyer twice against the same AI counterpart. Arken did not coach the tested model. Each side saw its own instructions and the shared conversation.

A separate check replayed each offer against the case rules, confirmed whether an agreement was valid, and calculated an outcome from 0 to 1. This number comes from the fictional case terms, not an AI judge. We show a model’s average only when both trials passed that check. Provider connection failures and offers outside the case limits remain unscored.

This first round covers one case and one role. More cases and roles are still being tested. These are simulated outcomes, not evidence of customer savings or an overall model ranking.

The earlier judge-rating test

We wrote 2 fictional negotiation situations. In each one, two AI models played opposite sides. We tested each pair twice so each model had a turn on both sides.

That gave us 10 model pairs × 2 sides × 2 situations = 40 completed conversations. Each model received 16 ratings across those conversations.

An AI judge reviewed each conversation and gave each model five ratings from 0 to 10. We averaged each model’s ratings in each area. We did not combine them into one overall score.

What each rating means

Reaching goals: Did the model work toward the goal it was given?
Decisions: Did its questions and choices make sense for the situation?
Useful ideas: Did it suggest relevant options or trade-offs?
Following limits: Did it respect the rules and boundaries it was given?
Cooperation: How did it handle agreement and disagreement with the other side?

For example, ratings of 6, 8, 9 and 7 average to 7.50. A score of 7.50 out of 10 is a judge rating, not a 75% chance of a successful deal.

What this test cannot tell us

Only 2 situations were tested, and one AI judge gave the ratings. Another judge or a larger set of situations could change the results. The published averages also do not show how much individual ratings varied.

No real customers negotiated in this test. We did not measure money saved, signed agreements, or satisfaction. You can download the published score table, but those averages alone cannot be used to re-score the conversations.