Introducing Harness Adapters: Adapt Your Favorite Model To Your Harness To Improve Performance And Cost
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Introducing Harness Adapters: Adapt Your Favorite Model To Your Harness To Improve Performance And Cost on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

TL;DR

Hugging Face says reinforcement-learning LoRA adapters tailored to Pi, OpenCode and Hermes Agent improved Qwen3.8-27B performance and efficiency in the team’s evaluations. For Pi on Terminal-Bench 2.1, the reported pass@1 score rose from 39% to 55%, while average token use fell from 2.9K to 1.5K per task. Independent replication and detailed results for the other harnesses are not provided in the supplied account.

Hugging Face says it trained reinforcement-learning LoRA adapters for the Pi, OpenCode and Hermes Agent AI harnesses, reporting that each adapter outperformed the base Qwen3.8-27B model in its matching environment, as described in the original analysis. The clearest result supplied is for Pi: on Terminal-Bench 2.1, reported pass@1 rose from 39% to 55%, while average token use fell from 2.9K to 1.5K per task.

The reported Pi comparison used Qwen3.8-27B with and without a custom LoRA adapter trained through reinforcement learning. Hugging Face says the base model scored 39% pass@1 and used an average of 2.9K tokens per task. With the adapter, the score reached 55% and average use was 1.5K tokens. Pass@1 measures whether a system completes a task on its first attempt.

The project team attributes the change largely to the model reasoning less, making fewer tool calls and succeeding more often on its first attempt, a consideration when choosing an AI agent harness. The team also trained separate adapters for OpenCode and Hermes Agent and says all three harness-specific versions beat the base model on performance and efficiency across two benchmarks. The supplied account gives no numerical results for those other harnesses and does not identify both benchmarks in its summary.

Hugging Face says the model weights remained frozen while the adapters were trained. Training and agent rollouts ran on its Hub infrastructure using Jobs, TRL’s AsyncGRPOTrainer, OpenEnv and Harbor. The project says it used the harnesses in their shipped form, rather than substituting a training framework’s recreation of each agent loop, and used a proxy to record token use.

At a glance
announcementWhen: Reported in the supplied project accoun…
The developmentHugging Face has reported that harness-specific reinforcement-learning LoRA adapters improved Qwen3.8-27B coding-task results across three AI agent harnesses.
AI Agents · Briefing

Harness adapters tune models to the way agents work

Hugging Face reports that harness-specific LoRA adapters improved Qwen3.8-27B coding results while reducing token use. The detailed comparison provided is for Pi.

55%
Pi pass@1, up from 39%
1.5K
average tokens per task, down from 2.9K
3
harnesses: Pi, OpenCode, Hermes Agent

One adapter for each workflow

The team trained adapters inside the harnesses where agents use tools and complete tasks.

Pi

Detailed result reported

On Terminal-Bench 2.1, pass@1 rose from 39% to 55% with the adapter.

OpenCode

Harness-specific tuning

The project says its adapter beat the base model, but supplied figures do not quantify the result.

Hermes Agent

Same reported approach

A separate adapter was trained for this harness; detailed scores were not provided.

What changed in the Pi test

The team attributes the gains to less reasoning, fewer tool calls and more successful first attempts.

Base model
  • 39% pass@1
  • 2.9K average tokens per task
With LoRA adapter
  • 55% pass@1
  • 1.5K average tokens per task

A recipe others can inspect

The project describes training in the shipped agent environments and says artifacts are available on Hugging Face Hub.

1
Prepare coding tasks

Tasks came from existing sources and generated examples, with validation checks described by the team.

2
Train in each harness

The model weights stayed frozen in the reported Pi comparison while a custom LoRA adapter was added.

3
Share artifacts

Adapters, GGUF files, tasks and a SafeAgents plugin are reported as available for inspection.

What the evidence leaves open

The reported gains are tied to the project’s experiments and evaluation setup.

  • Numerical comparisons for OpenCode and Hermes Agent are not included.
  • The supplied account does not provide uncertainty ranges or independent replication.
  • More results are needed to assess the merged “super adapter” and performance on other task sets.
CaveatLower token use may matter for inference costs, but the material does not report costs or show whether the gains generalize.
The takeaway
Harness-specific tuning is promising: the Pi result pairs higher first-attempt success with lower token use, while broader claims need full results and independent replication.

Why Harness-Specific Training Matters

The report addresses a practical challenge in deploying AI agents: a model’s general coding ability may not translate directly into success inside a particular agent harness. A harness shapes how a model uses tools and proceeds through a task. Training in that environment could tune its behavior to the workflow developers actually run, rather than evaluating the model only in isolation.

The reported reduction in token use may also matter to teams running agents repeatedly. Lower token consumption can reduce one component of inference expense, while fewer tool calls and more first-attempt successes could make task execution more efficient. But the figures are project-team results from selected evaluations, not evidence that every model, task set or deployment will see similar gains. The account does not give financial costs or evaluation uncertainty.

The team also reports merging the three harness-specific LoRAs into one “super adapter” that runs unchanged across Pi, OpenCode and Hermes Agent. If it performs reliably, a shared adapter could simplify distribution and use across environments. The supplied material does not explain the merging method or provide comparisons that show how this combined version performs against the separate adapters.

Products Worth Considering

Bose QuietComfort Ultra Earbuds (2nd Gen), Wireless Noise-Cancelling Earbuds, White Smoke

Bose QuietComfort Ultra Earbuds (2nd Gen), Wireless Noise-Cancelling Earbuds, White Smoke

Logitech MX Keys S Combo Wireless Keyboard and Mouse, Graphite

Logitech MX Keys S Combo Wireless Keyboard and Mouse, Graphite

Logitech MX Mechanical Mini for Mac Wireless Keyboard, Space Gray

Logitech MX Mechanical Mini for Mac Wireless Keyboard, Space Gray

As an affiliate, we earn on qualifying purchases.

Tasks and Training Setup

Hugging Face describes the work as an extension of earlier experiments in multi-harness reinforcement learning and harness adapters, which it says covered data-analysis and code-building tasks. This project applies the approach to a 27-billion-parameter model and software-engineering work across three harnesses, with the stated aim of publishing a reusable training recipe.

The team says it assembled coding tasks in Harbor format from existing sources and newly generated examples. Those sources included SWE-smith, which introduces bugs into Python and Go repositories, and SWE-Gym tasks packaged in TaskTrove, based on real GitHub issues and merged fixes. It also used Tasksmith to create tasks from merged pull requests. The account says 24 tasks drawn from 30 pull requests passed checks across 10 Python repositories.

According to the project, tasks were screened through OpenEnv validation and checked for difficulty against the base model. Such checks can affect reinforcement-learning results because training relies on tasks and verifiers that reward correct solutions. The supplied account describes efforts to identify faulty tests but does not provide the full dataset composition or complete validation results.

“Most of this is down to the model reasoning less, calling less tools, and performing better first time.”

— Hugging Face project team

Evidence Still Missing

The detailed numerical evidence in the supplied account is limited to one Pi result on Terminal-Bench 2.1. It does not include the scores for OpenCode or Hermes Agent, the name of the second benchmark in its summary, sample sizes, uncertainty ranges or a full breakdown of task-level outcomes. That limits how well readers can compare the results across harnesses or judge their statistical reliability.

The account also does not establish whether the reported gains have been independently replicated or hold on tasks outside the selected evaluations. It does not provide an independent audit of the task verifiers, explain how the merged adapter was produced, or show how that combined adapter compares with the separate versions. The findings should be understood as the project team’s reported results, not a general guarantee of performance.

Inspecting and Reproducing the Adapters

Hugging Face says the adapters, GGUF files, reinforcement-learning tasks and SafeAgents plugin are available through a collection on its Hub. Those artifacts give researchers and developers a way to inspect the materials and attempt the training recipe. The supplied account does not specify a date for further results or an independent evaluation.

Useful next evidence would include complete benchmark results for all three harnesses, clearer information about task counts and evaluation uncertainty, and comparisons of the combined adapter with the individual LoRAs. Independent reproduction on additional tasks would help establish whether the reported performance and token-use changes extend beyond the project’s own evaluations.

Key Questions

What did Hugging Face report?

Hugging Face says it trained reinforcement-learning LoRA adapters for Pi, OpenCode and Hermes Agent and that each improved Qwen3.8-27B results and efficiency under its matching harness in the project’s evaluations.

What was the reported Pi result?

On Terminal-Bench 2.1, the team reports that Pi’s pass@1 rose from 39% with the base model to 55% with the adapter. Average token use fell from 2.9K to 1.5K per task.

Are detailed results available for all three harnesses?

Not in the supplied account. It says the adapters for Pi, OpenCode and Hermes Agent beat the base model on two benchmarks, but provides a numerical comparison only for Pi.

Has the result been independently verified?

The supplied material does not report independent replication. It describes findings from the project team’s experiments, so broader performance claims remain unconfirmed.

Where can developers find the project materials?

Hugging Face says its Hub collection includes the adapters, GGUF files, training tasks and SafeAgents plugin. The account says these can be used to inspect the artifacts and try the recipe.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

Basketball Highlights And Stats Through A Computer Vision Lens

A proposed youth basketball app would turn phone footage into highlights and game stats. Its accuracy, demand and pricing remain untested.

Why OpenAI’s Compute Campuses Could Put Pressure On Local Utilities

OpenAI confirmed it may call data centers “compute campuses.” Grid costs, household bills and water use remain key questions for host communities.

Search as Code: Perplexity Is Right About the Future — Just Not First to It

Perplexity introduces Search as Code, enabling AI models to dynamically assemble search pipelines, improving accuracy and efficiency, with early promising results.

Anthropic Brings Mythos 5 To Its Claude Security Vulnerability Scanner – The New Stack

Anthropic has announced the addition of Mythos 5 to its Claude Security vulnerability scanner, but details on performance, deployment, and scope remain unclear.