🔍 Read the full analysis: We Tried Weight-Space Programming At Home: Claude Edited A Local Model, GPT-6 Astra Checked The Work, And The Legal Score Stayed At Zero on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A home experiment used Claude Opus 5.5-written code to remove a looping direction from a local model’s weights. Ornith 1.5 went from answering 4 of 10 published legal benchmark tasks to answering all 10, but its legal score stayed at zero; a separate sampling adjustment without editing also improved completion.
A home experiment reported November 5 found that editing the weights of a local AI model helped it finish more legal benchmark answers, but did not improve its legal score. The test used code written during a Claude Opus 5.5 session to modify Ornith 1.5 35B-A3B; on the 10 published Lexassay tasks, the edited model answered all 10 rather than 4 in the comparison run, while its legal score remained zero.
The experiment ran from Wednesday evening to Thursday morning and was inspired by MIT researchers’ work on what they call weight-space programming. It was not a replication of their method: Claude wrote code that ran Ornith, recorded its internal signals, identified a direction associated with looping, and removed that direction at half strength from the output weights in 27 of the model’s 40 layers. The report says there was no additional training or legal data, and no AI read the weights as text.
On the 10 published Lexassay tasks, the edited model completed 10 of 10, compared with 4 of 10 in the same-setup comparison. But the report says the answers met only 7 of 50 checked criteria, up from 3, and included 55 wrong or invented citations. Lexassay checks cited statutes and court decisions against primary sources. The reported legal score was zero both before and after the edit.
The report also describes a separate test of the model’s recommended sampling settings, with no weight edit. In that test, the unchanged model answered 9 of 10 tasks on its original setup. The figures therefore distinguish the edit’s result from a simpler settings change; the report cautions that the two local setups are not directly comparable.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Completion Improved, Legal Accuracy Did Not
The result highlights a practical distinction for users of local AI systems: finishing an answer is not the same as answering correctly. A model that stops repeating itself may produce a complete response, yet still lack the subject knowledge needed to support its conclusions or citations. Here, the completion count rose, while the legal score remained at zero.
That gap matters in legal research, where a fluent answer with fabricated or incorrect references could mislead a reader. The report presents the test as an experiment, not legal advice, and its citation findings argue against treating improved completion as evidence of reliability. The result also suggests that before editing a model’s weights, practitioners should test ordinary configuration changes: the sampling adjustment improved completion without a weight edit in the reported setup.
The test offers a limited example of a broader research proposition: an intervention may remove a behavior that obstructs a model’s output without adding knowledge the model does not have. That is a useful distinction for evaluating model repairs, but this small experiment does not establish how often such edits work or whether they transfer to other tasks and systems.
Products Worth Considering
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
DJI Mini 4 Pro Fly More Combo with DJI RC 2
Logitech MX Master 4 Ergonomic Wireless Mouse with Haptic Feedback, Graphite
As an affiliate, we earn on qualifying purchases.
From MIT’s Edit to a Home Test
The report was prompted by a study from four MIT CSAIL researchers: Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. Their account describes using GPT-6 Astra to reason directly about another model’s weights. In a Qwen example, they traced looping to a direction in the model’s internal signals and subtracted it from a small portion of the weights. The report says that model’s maths score rose from 7.9% to 48.1% after the intervention.
The home test used a different, established technique called directional ablation: run the model, measure internal activations, calculate a direction, then project it out of selected weights. Its authors frame it as a simpler test of the MIT work’s central prediction, rather than the same method. The distinction matters because the home result cannot be presented as evidence that an AI directly inspected and edited another network’s weights.
Ornith was selected because the report found it was the local model in the benchmark that showed the relevant looping behavior and could be tested on the available machine. Its publisher describes the model as designed for coding and agent tasks, not law. The report says two other local models failed mainly through lack of knowledge rather than looping, while DeepSeek V4 Flash was too large for the test machine and had a different token budget.
“Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked.”
— The MIT researchers, as quoted in the report
Limits of the Ten-Task Test
The report describes one run per setting, a small development set and a loop detector that changed during the experiment. The benchmark has 15 questions across German, US and EU law, but only 10 tasks are published; five remain hidden. The reported figures refer to the 10 published tasks, so the results do not establish performance on the hidden questions or on a broader legal workload.
The author also disclosed that Claude carried out the edit and Claude agents performed blind grading. OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, figures and claims and found faults, according to the report. That review is not a second experiment. The report also notes that the unchanged model answered 8 of 10 tasks in a published run but only 4 of 10 on the other setup, making direct comparisons across setups unreliable.
It remains unclear whether the completion improvement would repeat across multiple runs, other prompts, models or benchmark domains. The available results do not show that the edit improves legal reasoning, citation accuracy or performance on the five unpublished Lexassay tasks.
Public Runs and Further Testing
The report says the experiment’s runs, answers, grades and citation checks are public on Lexassay.com, alongside a method page. Readers can examine the edited run, the edit control, the model-card sampling test and the earlier run. Those records provide a way to inspect the specific results, but they do not resolve the limits of a single run or the difference between local setups.
The next useful test would be repeated runs under matched conditions, with a stable loop detector and independent grading, followed by evaluation beyond the published 10 tasks. Until those results exist, the supported conclusion is narrow: the edit helped this model complete more answers in one comparison, while its legal score remained zero. No future evaluation schedule or additional findings are specified in the report.
Key Questions
What changed in the local model?
Code written during a Claude Opus 5.5 session removed a measured looping direction at half strength from the output weights in 27 of Ornith 1.5’s 40 layers. The report says the model was not retrained and received no new legal data.
Did the edit make the model better at law?
Not according to the reported legal score. It remained at zero, despite the model answering more tasks. Its edited answers met 7 of 50 checked criteria and contained 55 wrong or invented citations, the report says.
Was this the same method used by MIT?
No. The home test used directional ablation: run the model, measure internal signals and remove a direction from selected weights. The MIT work described in the report had GPT-6 Astra reason directly about another network’s weights.
How reliable are the results?
The report describes one run per setting, a small development set and a changing loop detector. It also identifies differences between local setups. The findings are limited to this experiment and do not establish repeatable gains or better performance on legal tasks generally.
Source: ThorstenMeyerAI.com
