🔍 Read the full analysis: Same Model, Different Harness: CUGA, Hermes, And Deep Agents — When To Pick CUGA? on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
IBM reports that its CUGA agent harness completed 73 of 90 AppWorld tasks using GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents. The test suggests that harness setup can affect results on this benchmark, but does not establish a general ranking or test policy enforcement.
IBM reports that its CUGA agent harness completed 73 of 90 AppWorld tasks using GLM-5.3-flash, while Hermes completed 42 and Deep Agents completed 27 with the same model. The comparison, discussed in the original analysis of these agent harnesses, suggests that the software coordinating an AI agent can affect task results, but it covers one benchmark and does not show which harness performs best across other work.
The experiment used GLM-5.3-flash across all three systems, with each running the same 90 tasks across 30 scenarios. IBM says the runs used the Harbor evaluation framework and one attempt per task. The reported task completion rates were 81.1% for CUGA, 46.7% for Hermes and 30.0% for Deep Agents.
IBM also reported scenario totals, counting a scenario as passed only when all three of its variants passed: CUGA passed 20 of 30, Hermes eight and Deep Agents two. The harness configurations differed. CUGA used task decomposition, app-focused tools, verification before execution and automatic reflection. Hermes used Harbor’s default setup with AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP alongside LangChain middleware for functions including tool selection, context summarization, task lists, retries and call limits. IBM says it made two fixes to Deep Agents so malformed model replies would not end runs.
The source says the experiment setup is available in harbor-eval, IBM’s Harbor fork. The results are IBM-reported; the supplied material does not include an independent replication or additional runs.
Harness Setup Shifts Task Results
The spread in this test makes harness design a practical part of evaluating AI agents, rather than treating a model’s benchmark score as a complete predictor of performance. A harness shapes how an agent breaks down work, which tools it can use and whether it checks actions or results. Those choices can influence whether a multi-step task is completed.
For organizations considering agents, the findings support testing the model, harness and workload together. A strong result on AppWorld could be relevant to workflows involving several tools and changing information across apps. It is not proof that CUGA will outperform Hermes or Deep Agents in personal assistance, research or other tasks, and it says nothing about cost, speed or operational reliability.
The scenario scores add a measure of consistency within this particular test, but they do not resolve whether the same pattern would appear with different tasks, models or configurations. The benchmark did not test policy enforcement, even though access rules and approvals are important to many enterprise deployments.
Products Worth Considering
EcoFlow Delta 3 Ultra 3072Wh Portable Power Station – 3600W AC Output
Jackery HomePower 3600 Plus Portable Power Station, 3584Wh, 3600W
Samsung 27-Inch Odyssey OLED G5 QHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
Three Harnesses, Different Aims
The source describes CUGA, short for ConfigUrable Generalist Agent, as IBM’s open-source harness for enterprise-oriented work across systems such as customer relationship management and payments. It presents Hermes from Nous Research as designed for always-on personal assistants and Deep Agents from LangChain as suited to longer tasks such as research. These descriptions help explain why the comparison is best read as a test of particular configurations, not a contest that settles which product is best overall.
AppWorld tests multi-step tasks across nine consumer applications and 457 APIs, including email, payments and file storage. IBM characterizes the benchmark as a proxy for some capabilities relevant to enterprise automation, such as reading API documentation, paging through results and changing state across applications. But it does not directly establish performance on real enterprise systems.
The source also cites a separate directory analysis in which GLM-5.2’s SWE-bench Pro result reportedly changed from 23% to 52% when the harness changed. That is a distinct comparison, not part of the AppWorld experiment, and should not be treated as corroboration of its specific results.
“This is not a verdict on Hermes or Deep Agents.”
— IBM
Limits of the AppWorld Test
The comparison covers one model, one 90-task sample and one attempt per task. IBM says the harnesses were configured differently, so the results do not isolate every implementation difference. The supplied material does not show how scores would change with other models, repeated runs, alternative task sets or production workloads.
It is also unclear whether the reported gap would hold on workflows beyond AppWorld’s consumer-app tasks. The experiment did not use policies, so it provides no evidence about whether or how policy rules are enforced. IBM separately describes CUGA features for applying rules through the harness, but their effectiveness is not demonstrated by these benchmark scores. Comparative figures for cost, latency, large-scale reliability and deployment outcomes are also absent.
Evaluate With Real Workflows
The reported harbor-eval setup gives teams a starting point for examining the experiment, according to the source. Organizations considering these systems would need to test representative workflows with the models, tools and configurations they expect to use, and track both completion and consistency across repeated or varied tasks.
Enterprise evaluation would also need to cover matters AppWorld did not test, including access controls, approval steps, data boundaries and performance in the organization’s own environment. The supplied material gives no timeline for a broader comparison or independent replication, so whether the ranking persists beyond this test remains open.
Key Questions
What did the AppWorld comparison find?
IBM reported that CUGA completed 73 of 90 tasks, Hermes 42 and Deep Agents 27, with all three using GLM-5.3-flash.
Does the result prove CUGA is the best agent harness?
No. The results cover one benchmark, one model and one attempt per task. They do not establish an overall ranking across different tasks or deployments.
What is AppWorld testing?
AppWorld tests multi-step tasks across nine consumer apps and 457 APIs. IBM describes it as a proxy for some enterprise-automation capabilities, not a direct test of real enterprise systems.
Did the experiment test enterprise policy enforcement?
No. The experiment did not use policies, so its scores do not show whether CUGA or the other harnesses enforce rules, approvals or access controls.
What should organizations do with these results?
Use them as a reason to test the intended model-and-harness combination on representative workflows. Also measure consistency and evaluate controls and deployment needs not covered by AppWorld.
Primary source: Hugging Face · via ThorstenMeyerAI.com
