Beyond the Model's Ceiling: How to Design Open-Ended AI Systems That Keep Finding Solutions
Review copy. The full designed version, with diagrams, charts and FAQ schema, is in open-ended-systems.html and should be pasted in through the admin console before publishing.
"If the model cannot solve it, it cannot be solved" describes a design choice, not a law of nature. A system that asks one model once has a ceiling exactly as high as that model. A system that can check answers, try again differently, use tools, break problems down, call a stronger model, remember what worked and hand off to a human has no single ceiling. With automatic checking, a mid-sized open model went from 15.9% to 56% on real GitHub issues. A self-improving coding agent climbed from 20% to 50% on SWE-bench. An evolutionary system built on Gemini recovers 0.7% of Google's global compute with a scheduling rule it found.
- 15.9% → 56% SWE-bench Lite solved by one model, from 1 sample to 250 checked samples
- 20% → 50% SWE-bench score of the Darwin Gödel Machine after rewriting its own code
- 48 multiplications for 4×4 complex matrices found by AlphaEvolve, beating Strassen's 49 from 1969
Closed systems versus open-ended systems
A closed system sends one prompt to one model and keeps whatever comes back, so its ceiling is the model. An open-ended system treats the first answer as a draft, checks it against something outside the model, and when it fails, changes strategy. Its ceiling is set by its budget and the quality of its verifier.
The principle underneath: checking is easier than solving
A compiler, a test suite, a proof checker or a database state check does not need to be as smart as the model, but it lets a system discard bad answers and keep searching. Brown et al.'s Large Language Monkeys found that coverage, the share of problems solved by at least one sample, rose over four orders of magnitude of samples, and that DeepSeek-Coder-V2 beat the single-attempt state of the art of 43% by reaching 56% with 250 checked samples. Where answers could not be checked automatically, majority voting and reward models plateaued after a few hundred samples. The verifier set the limit. Our cheap-LLM agent benchmark shows the same: best-of-five success spanned only 85% to 92% across 19 models.
Six levers that lift a system above its model
| Lever | Key evidence | Works when | Main cost |
|---|---|---|---|
| 1. Resample and verify | 15.9% → 56% with 250 checked samples | A cheap checker exists | Linear in attempts |
| 2. Execute, do not guess | Up to 12% higher accuracy with test feedback | Code, maths, data, APIs | Sandbox, latency |
| 3. Decompose and plan | Agents fail most on silent tool errors | Long multi-step tasks | Error compounding |
| 4. Escalate up a model ladder | 45.8% cheaper cascade | Difficulty varies | Router mistakes |
| 5. Remember and accumulate | 3.3× more items with a skill library | Tasks repeat | Memory curation |
| 6. Evolve a population | New maths and production results | A numeric score exists | Many evaluations |
Lever 1: resample and verify
Try again with some diversity and check each attempt. It widens what the model can reach, but does not extend it.
Lever 2: execute, do not guess
Run code, query data and read real errors. Our context engineering review covers Self-Debugging (up to 12% higher accuracy with test feedback) and SWE-agent (11% to 18% from better tool output alone).
Lever 3: decompose and plan
Smaller checked steps help only if each step fails loudly. PlanBench-XL, in our 2026 research review, found agents struggled most with silent tool failures.
Lever 4: escalate up a model ladder
PayPal's RLM-Cascade sent simple turns to DeepSeek and hard ones to Claude Opus 4.8, cutting API cost 45.8% and median latency 1.83×, passing 20 of 20 test tasks against 19 for Opus alone. Escalation should carry the full history forward.
Lever 5: remember and accumulate
Voyager stored verified Minecraft skills as code in a library, found 3.3× more unique items and reached milestones up to 15.3× faster, and its library transferred to a new world. ACE's growing playbook beat baselines by 10.6% on agent benchmarks.
Lever 6: evolve a population
Keep an archive of candidates, mutate promising ones with a model, score every child with an evaluator, and repeat.
| System | Result | Verifier |
|---|---|---|
| FunSearch (Nature 2023) | New cap set constructions, including size 512 in dimension 8, and better bin-packing heuristics | Program that checks constructions |
| AlphaEvolve (2025) | 48-multiplication 4×4 complex matrix algorithm, 14 matrix algorithms improved, beat the best known answer on 20% of 50+ open problems | Automated evaluators |
| AlphaEvolve in production | Borg heuristic recovering 0.7% of Google's compute, a Gemini kernel 23% faster | Simulators and benchmarks |
| Darwin Gödel Machine (ICLR 2026) | SWE-bench 20% → 50%, Polyglot 14.2% → 30.7% | Coding benchmarks |
The last rung: a human, with a good handoff
When the budget is spent or the specification is unclear, hand off with every approach tried and the latest failure, so a person starts where the system stopped.
A reference architecture
- Task arrives with a budget in rupees and steps.
- Define the verifier first: tests, schema or state check, read-only to the model.
- Attempt with a cheap model and tools, with lessons from memory in the prompt.
- If the verifier passes, deliver and save the lesson.
- If not, diagnose: wrong approach, too big, too hard or bad specification.
- Then retry differently, decompose, escalate with history, or hand off once the budget is spent.
Where open-ended systems break
- No verifier: more attempts mostly cost more.
- Verifier hacking: SWE-Marathon found exploit attempts cost only 0.05 to 0.45 times the tokens of honest work, and an audit found reward-hackable verifiers in over 15% of tasks across five terminal benchmarks. Read-only tests blocked test edits while keeping legitimate success.
- Runaway cost: Anthropic's multi-agent research system used about 15 times the tokens of a chat.
- Truly out of reach: search cannot create absent skills.
The economics: cost per solved task in rupees
One attempt = 12 steps × (8,000 input + 400 output tokens), Velona prices on 24 October 2026
DeepSeek V4 Flash: ≈ ₹2.11 per attempt
Claude Opus 4.8: ≈ ₹62.16 per attempt
Assumed Flash success: 69.1% after 1 try, 80% after 2, 85% after 3
Expected Flash attempts = 1.51 → ₹3.18
Escalate remaining 15% to Opus: ₹9.32
Expected cost per task ≈ ₹12.51, overall success ≈ 91%
Cost per solved task ≈ ₹13.75| Approach | Solved | Cost per solved task |
|---|---|---|
| Opus 4.8, single attempt | 69.1% | ₹89.96 |
| Open-ended ladder | ~91% | ₹13.75 |
| Flash, single attempt | 69.1% | ₹3.05 |
Single-attempt rates come from APIFlow-Bench 20-step chains. Ladder rates after the first attempt are illustrative. See live prices for DeepSeek V4 Flash and Claude Opus 4.8, and our model router tutorial.
Build it: an open-ended loop in Python
"""open_loop.py: generate, verify, escalate, remember, hand off."""
import json
import os
import pathlib
import subprocess
import tempfile
from openai import OpenAI
client = OpenAI(api_key=os.environ["VELONA_API_KEY"], base_url="https://velona.in/v1")
LADDER = [ # (model, attempts): cheap first, strong last
("deepseek/deepseek-v4-flash", 3),
("anthropic/claude-opus-4.8", 1),
]
LESSONS = pathlib.Path("lessons.md")
def verify(code: str, test_file: str) -> tuple[bool, str]:
with tempfile.TemporaryDirectory() as folder:
pathlib.Path(folder, "solution.py").write_text(code)
pathlib.Path(folder, "test_solution.py").write_text(pathlib.Path(test_file).read_text())
run = subprocess.run(["python", "-m", "pytest", "-q", "-x"], cwd=folder,
capture_output=True, text=True, timeout=120)
return run.returncode == 0, (run.stdout + run.stderr)[-1500:]
def attempt(model: str, task: str, history: list) -> str:
lessons = LESSONS.read_text() if LESSONS.exists() else "none yet"
messages = [
{"role": "system", "content": "Write solution.py in one python code block. "
"Never special-case test inputs.\nLessons from past runs:\n" + lessons},
{"role": "user", "content": task},
]
for code, log in history[-2:]:
messages.append({"role": "assistant", "content": f"```python\n{code}\n```"})
messages.append({"role": "user", "content": f"Tests failed:\n{log}\nTry a different approach."})
reply = client.chat.completions.create(model=model, messages=messages, temperature=0.8)
text = reply.choices[0].message.content
return text.split("```python")[-1].split("```")[0].strip()
def solve(task: str, test_file: str) -> dict:
history = []
for model, tries in LADDER:
for _ in range(tries):
code = attempt(model, task, history)
passed, log = verify(code, test_file)
if passed:
if history:
first_error = history[0][1].strip().splitlines()[-1:] or ["unknown"]
with LESSONS.open("a") as notes:
notes.write(f"- {task[:60]}: first attempt failed with {first_error[0]}\n")
return {"status": "solved", "model": model, "attempts": len(history) + 1}
history.append((code, log))
packet = {"status": "needs_human", "task": task, "attempts": len(history),
"last_failure": history[-1][1] if history else ""}
pathlib.Path("handoff.json").write_text(json.dumps(packet, indent=2))
return packetA checklist for open-ended design
- Write the verifier first.
- Protect it: read-only tests, held-out cases, grade final state.
- Make tool failures loud.
- Diagnose, then pick a lever.
- Carry history up the ladder.
- Keep a lessons file and append to it.
- Set a budget per task.
- Measure cost per solved task.
Limits of this analysis
Evidence comes mostly from domains with strong verifiers. AlphaEvolve's production gains are reported by Google. The rupee example uses illustrative later-attempt rates. Self-modifying agents need sandboxes, budgets and human oversight.
Frequently asked questions
What is an open-ended AI system?
A system that checks answers against a verifier and, when they fail, retries differently, decomposes, escalates, reuses lessons or hands off to a human.
Can an AI system solve problems its model cannot?
Often, if answers can be verified, as Large Language Monkeys, FunSearch and AlphaEvolve show.
Why is a verifier so important?
Checking is easier than solving, so a verifier lets a system discard wrong attempts and keep searching.
What is verifier or reward hacking?
Satisfying the checker without solving the task. Read-only tests and grading on final state reduce it.
Is an open-ended system more expensive?
Per call yes, per solved task often no: about ₹13.75 against about ₹90 in our example.
When should an AI system hand off to a human?
When the budget is spent, the verifier cannot judge, or the specification is unclear.