Can Five 27B Agents Really Take Down Fable 5?

Opening Dialogue

Shiwen: Today, I heard on the overseas forum that a GitHub project is claiming five Qwen3.8-27B agents formed a team and defeated Fable 5 in a single invocation on the top ten problems of LiveCodeBench Hard. The source of this information is a workflow example from Memdoor, and the author has shared the complete record of 50 runs. Don’t judge by the title; does this matter enough to discuss?

Yongliang: It matters, but not in the way of a “small model beats a large model” story. What’s really interesting is that they turned a proven multi-agent approach into a YAML workflow, allowing anyone to replace models, temperatures, and add approval steps. This is like breaking down the academic implementation of GVS5H into components that an engineering team can directly integrate into CI. Whether Fable 5 was truly defeated needs to be evaluated carefully.

Q1: How did this “defeat” happen?

Shiwen: Memdoor’s README is quite clear: the first five passes averaged 9.8/10, while the original author’s same model with the same prompts got 9.6, and Fable 5’s single invocation was 8.0. How should we interpret this data?

Yongliang: Let’s clarify the scope. They ran Qwen3.8-27B through a multi-agent cycle: the planner generates solutions, two ideators submit proposals, the solver writes code, the manager reviews, and a repair worker fixes bugs. Each role operates in a new context, communicating via files on disk. After five passes, the average was 9.8, slightly higher than the original author’s 9.6 but much better than Fable 5’s single invocation score of 8.0.

Shiwen: So, it’s not that the model itself became stronger, but the way it’s used?

Yongliang: Exactly. Fable 5’s 8.0 came from single invocation data in the GVS5H author’s files, not from Memdoor rerunning it under the same conditions. Strictly speaking, this is “the same set of problems, under the same conditions, a 27B multi-agent outperforms a single invocation of Fable 5,” not “a 27B model comprehensively surpasses Fable 5.” But even so, this gap shows that organizing multiple small models can indeed help them exceed frontier models in specific tasks. More importantly, Memdoor’s workflow in YAML means you can replace the planner, ideator, and manager with your own models and temperatures, inserting approval steps at any step. This is more useful for engineering teams than just the score itself.

Q2: Isn’t model size no longer a hard criterion?

Shiwen: Over the years, the industry has默认 “the larger the model, the better.” This project seems to say that orchestration is the new lever. Do you agree?

Yongliang: I agree half. The model size still determines the upper limit, but the architecture determines how much of that limit can be utilized. In the Memdoor example, the real leverage comes from four things: first, division of labor, where each agent is responsible for only a small portion of the task, keeping the context cleaner; second, persistent state, with plans, notes, and current solutions all stored in files, eliminating the need for roles to cram all history into prompts; third, multi-round verification, where managers consider not only code but also test results; fourth, recoverability, with YAML workflows featuring approval gates and resume from failed steps.

Shiwen: Memdoor also mentioned a “decision model” that saves tokens upfront. How should this be understood?

Yongliang: This is a very practical engineering point. Regular chat models have to process the entire history of the conversation, file content, and tool results each time, leading to linear increases in token costs. The decision model first reviews the input to determine which tools, files, and contexts are needed for this step, then only sends the necessary reduced information to the large model. Memdoor stated in its tests that some tasks can save 74% of input tokens. This is especially crucial for multi-agent loops, as each agent repeatedly reads and writes files. Without filtering, the context would quickly expand.

Shiwen: Does this mean that buying frontier APIs will become less cost-effective in the future?

Yongliang: It depends on the scenario. For short tasks with shallow contexts and sensitivity to latency, a single frontier call is still the most convenient. If the task can be broken into multiple sub-tasks, requires multiple iterations, or needs auditing or local running, “decision model + multiple workers + persistent workflow” becomes more cost-effective. In the future, engineering teams will focus more on designing pipelines rather than constantly dealing with model prompts.

Q3: How is the cost accounting calculated? Can local models really save money?

Shiwen: Memdoor emphasizes BYOK and local models, which Ollama, llama.cpp, and vLLM can handle. The original GVS5H paper also states that costs can be reduced to one-ninth of frontier’s. Is this reliable?

Yongliang: Memdoor’s reproduction actually runs on OpenRouter, not on local machines. 50 valid runs cost $35.79, plus 16 truncated or repaired attempts totaling $38.77. The median cost per run is $0.39, with a median runtime of 40 minutes and the longest at 4 hours. If running on a local vLLM FP8, the authors said it could run smoothly on a single GPU. So, local models can indeed save API call costs, but you have to pay for hardware, electricity, and maintenance.

Shiwen: Does this mean that individual developers can also run locally?

Yongliang: Theoretically, it’s possible. Memdoor supports Ollama, llama.cpp, vLLM, LM Studio, as well as its own janis gateway. Someone mentioned on the forum that after quantization with Bonsai 2, Qwen3.8-27B can run at 50 token/s on an ordinary gaming machine. A comment on HN said, “Fable 5-level coding performance can be achieved locally if the harness is well-designed.” But there are two issues here: first, whether quantization will reduce the quality of inference; second, the latency in multi-agent loops. Running an agent locally might be fast, but when five agents run together, plus file reading and testing, the overall time consumed increases.

Shiwen: How does it compare with frontier API?

Yongliang: Frontier API charges per token, which offers flexibility, speed, and no need to worry about hardware. Local models require a one-time investment in hardware, with low marginal costs. For example, after quantization with Bonsai 2, Qwen3.8-27B can run at 50 token/s on an ordinary gaming machine, which is very attractive for individual developers and small teams. However, note that in Memdoor’s reproduction, the longest multi-agent loop lasted 4 hours. If the local GPU is slow, the experience will be greatly reduced. Costs go beyond just the bill; it also depends on time, complexity, and failure rate.

Q4: How much discount should be applied to “beats” in the title?

Shiwen: You kept reminding us to apply a discount. What are the specific warnings?

Yongliang: First, only the top ten questions are used as samples. The GVS5H paper uses 100 questions, but Memdoor only used the top ten, so the representativeness is limited. Second, contamination cannot be ruled out clearly. These ten questions come from competitions from late March to early April 2025, and Qwen3.8-27B’s training data likely saw these before, and the authors explicitly said contamination cannot be ruled out. Third, Fable 5 is not run in the same environment; the data comes from the GVS5H author’s files, so harness differences cannot be ruled out. Fourth, the time cost for multi-agents is very high, with a median of 40 minutes and a maximum of 4 hours, which is completely different from single calls. Fifth, the accuracy across different hosts on OpenRouter is not transparent; the authors used FP8 vLLM locally, and it might be different on the cloud.

Shiwen: What do you think of the comments on the forum?

Yongliang: Some people on Hacker News also pointed out that the top ten questions might be saturated, and LiveCodeBench Hard is quite old. Others hope to see tests for Qwen3.8-flash-next because flash-next is faster and stronger in many coding tasks than 27B. The authors themselves admit that by adjusting prompts and temperatures, they might get a perfect score of 10. This shows that the results are sensitive to parameters, and not all scenarios can be replicated with the same formula. Another comment asked a good question: “Can five 27B agents beat five Fable 5 agents?” This question hits the nail on the head because a truly fair comparison should be between multi-agents and multi-agents, not between small multi-agent models and single call large models.

Shiwen: Does that mean the title “Five 27B Models Beat Fable 5” is just clickbait?

Yongliang: Under certain circumstances, it’s true, but it’s still far from a general conclusion. It’s more like an engineering reproduction and method validation, not a conclusion that can be written into a purchasing guide. As a reader, I would interpret “beats” in the title as “showing comparability under limited conditions,” rather than “completely surpassing.”

Q5: What is the biggest lesson for frontline engineers from this?

Shiwen: Last question, if this is true and scalable, what does it mean for ordinary engineers’ work?

Yongliang: It means our focus will continue to shift from “writing code” to “designing workflows and acceptance thresholds.” Over the past two years, everyone has been competing in prompt engineering; next, it’s about how to break tasks into verifiable steps, design collaboration protocols between agents, and set approval gates and rollback points. There are several valuable lessons in Memdoor’s workflow: using YAML to define steps, running independent steps in parallel, allowing failed steps to be retried, adding manual approval for critical steps, and writing results back to disk instead of stuffing them all into context.

Shiwen: What impact does this shift have on team structure?

Yongliang: The most direct impact is that “writing code with AI” is no longer a scarce skill; “designing agent processes” is. In the past, the value of a senior engineer lay in writing complex algorithms, but now it lies in judging which problems are suitable for single agents, which require multiple agents, and which steps must be manually approved. New roles like workflow designer and agent reviewer will emerge in teams. For managers, evaluation metrics also need to change: don’t just look at the number of lines of code, but at the stability, reproducibility, and error rollback capability of the process.

Shiwen: Will engineers no longer need to write code in the future?

Yongliang: No. But “writing code” will become a smaller part, with greater value in defining problems, breaking down processes, and designing acceptance standards. Just as DevOps connected operations and development, now agent workflows string together requirements analysis, code generation, testing validation, and manual approval into a continuous pipeline. In the future, an engineer’s core ability might be translating business problems into flowcharts that agents can execute, rather than simply competing to write more beautiful functions.

Shiwen: What advice would you give to teams wanting to try this?

Yongliang: Start with your own real code tasks. Don’t use benchmark numbers to guide selection. Take a specific bug fix or redesign task, run it using Memdoor or GVS5H methods, and record time, token cost, and accuracy rate. Also, set acceptance thresholds clearly, such as must-pass unit tests and manual reviews; otherwise, multi-agent systems can easily get stuck. Finally, don’t treat local models as a free lunch. Consider hardware, quantization accuracy, and inference latency. If the team has no experience with local GPU operations, starting with BYOK solutions like OpenRouter is more practical.

---

Signature: Yongliang