The Most Promising AI Setup In My Test Isn’t The One I Run

🔍 Read the full analysis: The Most Promising AI Setup In My Test Isn’t The One I Run on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

GPT-6 Astra scored highest in Thorsten Meyer’s test of 71 AI model configurations writing news articles, but Meyer says he kept GPT-6 Luna in production because it cost about one-hundredth as much. The test, published October 13 on ThorstenMeyerAI.com, also found that two backup models produced fabricated quotations or articles his pipeline could not use.

Meyer tested models from September 8 through October 13, 2026, using his production prompt and real news sources. The articles covered English and German editions and drew on two sources: a 6,340-character product filing and a 619-character political news item. He said the shorter source better resembled much of the material his operation handles.

In the standard test, GPT-6 Astra scored 4.00 out of 5 in both languages and received a source-fidelity score of 5.00. In a further 10-article robustness run, Astra averaged 3.93 across 12 articles, with a reported 95% bootstrap interval of 3.83–4.02. GPT-6 Luna scored 3.67 in English and 3.68 in German in the standard test; its robustness-run average was 3.67, with source fidelity of 4.71. Meyer said every Luna article in that run produced valid output.

The cost and speed gap shaped the production decision. Meyer put Astra at about $0.12 per article and 50 seconds, compared with roughly $0.0012 and 17 seconds for Luna. He also reported a production cost of about $0.0013 per Luna article over the previous week. Two AI systems, GPT-6.1 Sol and Claude Opus 5.5, judged the articles on source fidelity, fabrication, clarity, news value, structure and polish; German output was also scored for language quality.

Quality Versus Cost at Scale

The test highlights a practical choice for publishers using AI at high volume: the highest-scoring model may not be the most workable production option. If the reported per-article prices hold across a large output, Astra’s higher score comes with a substantial cost increase, while Luna’s lower score may still clear Meyer’s bar for routine use.

The results also caution against choosing a writing model solely from a general leaderboard or its reputation in another field. Meyer reported that Claude Opus 5.5, which he described as a leading coding model, scored 3.46 in English and 3.39 in German on his writing test, and 3.52 in the robustness run. Those scores were below Luna’s robustness result. This is evidence from one publisher’s task and setup, not a universal ranking of models.

For newsrooms, the fallback findings carry a separate operational lesson. An article can fail even when the primary model performs well if a backup fabricates quotations or returns unusable formatting. Meyer’s test suggests that quality checks need to cover the whole model route, including failover options, rather than only the system used most often.

How Meyer Tested News Models

Meyer said each model configuration received the same production task: turn a real source into a finished article using the prompt his pipeline uses. The test covered both English and German editions and included API, subscription, free-tier and locally run open-weight models. The two judges saw the original source and rated the resulting articles on a five-point scale.

He described the initial test as small, with only two source articles, and said score gaps below about 0.3 points should be treated as noise at that stage. To strengthen the comparison, he ran the strongest configurations on 10 additional real sources, mostly short ones, and reported 95% bootstrap intervals. He said an article counted only if it produced valid output after the same repair step used in production.

Meyer also tested GPT-6 Luna at six reasoning-effort settings. He reported that the scores did not show a reliable gain as effort rose. At the maximum setting, an article cost about 1.43 cents and took 211 seconds, versus 0.12 cents and 17 seconds at the “none” setting. The reported differences were within the test’s uncertainty, including a small apparent advantage for “high” effort in the robustness run.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, ThorstenMeyerAI.com

Limits of the Model Comparison

The findings are one publisher’s evaluation, using one production prompt, two languages, two initial sources and a limited follow-up set. The robustness run adds evidence but does not establish how the models would perform across other newsrooms, subject areas, prompts or editorial workflows. Meyer’s own rule treats small gaps as noise, and the reported scores should be read within that limitation.

The source material does not provide the complete configuration-by-configuration results, judge-by-judge scores or enough detail to independently assess every scoring decision. It also does not establish whether model versions, prices or performance will remain the same. Meyer reported that Astra’s interval did not overlap Luna’s in the robustness run, but the test does not prove that Astra is the best choice for all news production.

Meyer said GPT-4.1 nano received source-fidelity scores of 1.00 and judges flagged invented quotations, while locally run gpt-oss-120b produced zero valid articles out of four and included fabricated named quotes. Those are results from the tested configurations and samples; the article does not establish how often the same failures would occur in other settings.

Production Stays on GPT-6 Luna

Meyer’s reported decision is to keep GPT-6 Luna at the front of the news route and to leave the two weak backup models out after removing them on October 7. He did not report a plan to switch routine production to Astra, citing its higher per-article cost and slower generation time.

The test points to continued monitoring as the next practical step: model quality, cost and output validity can change, and a route that includes fallbacks needs to be checked whenever its components change. Meyer’s article does not give a date for another test or specify a new backup model, so the future failover setup remains unclear.

Key Questions

Which model scored highest in Meyer’s test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It averaged 3.93 across 12 articles in the robustness run.

Why did Meyer keep GPT-6 Luna in production?

Meyer reported that Luna cost about $0.0012 per article, compared with roughly $0.12 for Astra, while still scoring 3.67 in the robustness run and producing valid articles in that test.

Did higher reasoning effort improve the articles?

Meyer found no reliable improvement across Luna’s six tested effort settings. Maximum effort was reported to cost about 12 times as much and take about 12 times as long as the “none” setting, with score differences within noise.

What went wrong with the fallback models?

Meyer said GPT-4.1 nano produced fabricated quotations flagged by the judges, while locally run gpt-oss-120b produced zero valid articles out of four and included invented named quotes. He removed both from the route on October 7.

How broadly do these results apply?

The results apply directly to Meyer’s prompt, sources, languages and production setup. The limited test does not establish that the same ranking will hold for other publishers or tasks.

Source: ThorstenMeyerAI.com