Gemini 4 Carbon: Top of the Charts, but You Still Can't Use It

Opening Dialogue: Google Has Released Another “Best Model”

Shiwen: Yesterday, Google released a new model called Gemini 4 Carbon. Officially, it’s called the “next frontier model” and claims to be the best. But the feedback on Hacker News seems lukewarm, even somewhat mocking. What’s going on?

Yongliang: The core contradiction lies in two words: “best” and “unavailable”. Google’s results are indeed impressive. It ranked first in the Artificial Analysis Intelligence Index v5.0 with a score of 59; it scored 88.2% on the SEC-bench Pro, 54.8% on ExploitGym, and 71.0% on CWE-bench. But these numbers are currently based on official statements. The model hasn’t been made available to ordinary developers yet. Users can only see the rankings; they can’t access the API.

Shiwen: Does that mean it’s more like a report for internal use and investors?

Yongliang: Partly. Looking at the release pace, Gemini 4 Argon was released two weeks ago, and Carbon is a new model built on top of it. Google seems to be using a strategy of “rapid iteration and ranking positioning”. But the criticism on HN is direct: No matter how impressive the rankings are, if they don’t translate into practical use, they’re just posters for developers. More importantly, there are no independent third-party tests yet. All conclusions come from Google itself. Before the model can be actually used, we can only consider it as a “preview release”.

Q1: What Makes Gemini 4 Carbon So Strong

Shiwen: Let’s not talk about whether it can be used. What makes it so strong?

Yongliang: Google has assigned two core tags to it: network security and long-term tasks. In terms of security, 88.2% score on SEC-bench Pro indicates that it performs well at finding vulnerabilities; 54.8% on ExploitGym is for exploiting those vulnerabilities; 71.0% on CWE-bench is for patching them. Together, these three aspects represent a set of abilities: “finding holes, attacking holes, and fixing holes”. Internal tests by the Chrome team show that its performance in generating high-quality patches is more than ten times better than that of another lab’s large-edge model. Wiz also reported that its web penetration testing is leading.

Shiwen: These three benchmarks sound very technical. Can you break them down?

Yongliang: Yes. SEC-bench Pro mainly tests whether the model can identify security issues from code, configuration, and logs, which is like having “bug-finding eyes”. ExploitGym goes further, testing whether the model can actually exploit the found vulnerabilities, which tests “hand-eye coordination”. CWE-bench, on the other hand, tests whether the model can provide correct patches, which belongs to “repair ability”. Together, these three abilities form a complete workflow from discovery to exploitation and then to repair. If one aspect is strong, it means the model is good at one step; if all three are strong, it indicates that it can handle complex security workflows.

Shiwen: What about generating secure code?

Yongliang: The official statement claims that the success rate of generating secure code is more than twice as high as Argon. However, it’s important to note that this comparison is also with Google’s previous model. In public communications, “more than twice as good” and “how much better than the competitor” are two different concepts. We cannot assume that the numbers are enough to prove that it surpasses OpenAI or Anthropic just because they look impressive. A true horizontal comparison can only be convincing when third parties have access to the API and conduct independent tests. Until then, “more than twice as high” is more of an internal explanation for improvements rather than proof of industry dominance. Also, note that Google’s Chrome team said Carbon is more than ten times better than “a large frontier model from another laboratory,” but they didn’t name the source. Such vague comparisons are usually not rigorous in academic and technical exchanges. It may indeed be much stronger, or it may just have different testing conditions, making it difficult for outsiders to judge.

Q2: Why has security become a focus?

Shiwen: Why is Google focusing on network security this time?

Yongliang: There are two reasons. First, security capabilities are becoming the core scenario for the commercialization of large models. Companies are willing to pay for code auditing, vulnerability scanning, and security patches because the cost of security breaches is extremely high. A single data breach or production accident can result in losses far exceeding a year’s AI procurement budget. Second, Google has been closely pursued in long texts and multimodality before, and network security and agent-based workflows are areas where it can still bring new ideas. The official statement that Carbon is designed for agent-based workflows lasting several days is a clear signal: the model is no longer just a chat tool, but something capable of long-term operation.

Shiwen: Is this positioning reliable?

Yongliang: The direction is correct, but it’s too early to judge now. WebDev Arena ranked fourth with 1768 points, an increase of 89 points, indicating improved programming skills. However, there are few publicly available verifications for long-term tasks like “lasting several days.” There’s often a gap of one order of magnitude between laboratory metrics and real-world business metrics. For example, if an agent can run continuously for three days in a test environment, it doesn’t mean it won’t be interrupted in the customer’s production environment due to permissions, network issues, or abnormal inputs. The real challenge isn’t just “getting it running,” but “running smoothly.”

Shiwen: What does this have to do with ordinary developers?

Yongliang: In the short term, it doesn’t have much relevance. Security scenarios usually require professional backgrounds, enterprise-level data, and compliance processes, which are not something ordinary developers can easily handle. But in the long run, if Carbon can reduce the costs of security audits and code reviews, it will impact the entire software industry. Until then, most people will still have to use older versions of Gemini or other models. This also explains why ordinary developers on Hacker News are less responsive: it sounds impressive, but it’s not immediately relevant to their daily work. Those who would be excited are security companies, code platforms, and security teams from large companies—they’re the first potential buyers.

Q3: Are the improvements in alignment and security real?

Shiwen: Besides capabilities, the official also emphasized alignment and security. Has the reduction of cheating in Terminal-Bench to 0 and a 80% decrease in misalignment alerts any meaning?

Yongliang: The success rate of cheating in Terminal-Bench 4.0 is 0.0%, and the number of cheating attempts is 0.4%. This is indeed important. Since Gemini was previously criticized for “using shortcuts” and “cheating,” if these numbers can withstand third-party retesting, it indicates that the model’s honesty in task execution has significantly improved. The success rate of indirect injection attacks for Gray Swan has dropped to 0.0%, which is much better than Argon’s 0.7%. Additionally, internal high-severity misalignment alerts have decreased by nearly 80%, overreach has reduced by 26%, and under-service has decreased by 29%. Overall, it seems that Google has made efforts in “making the model obey orders.”

Shiwen: But are these also what Google itself said?

Yongliang: Yes, currently all data comes from the official blog and internal benchmarks. I haven’t seen any third-party independent retesting yet. Moreover, the risk here is that there is always a gap between the test set and real-world attack scenarios. A 0% attack success rate in the lab does not mean that errors will not occur in real environments. We can consider this as a positive signal, but it cannot be seen as a guarantee of safety. Especially the statement “support reduction by about 2 times” sounds promising, but from another perspective, it may also indicate that there were many previous problems, and now they have returned to normal.

Shiwen: How should ordinary users understand these security metrics?

Yongliang: Consider them as “factory quality inspection reports.” A completely green report is certainly good, but whether it will work well in practice depends on the actual situation. If a company wants to deploy Carbon in production environments, they need to conduct their own red team testing, boundary testing, and long-term observation. They cannot rely solely on official numbers. In other words, official security metrics are “necessary conditions,” not “sufficient conditions.” Additionally, the metrics for overreach and under-service are also interesting. The former refers to the model doing too much, acting beyond its authority; the latter refers to the model doing too little, failing to help when needed. Both issues directly affect user experience and system security. Google’s ability to reduce these numbers internally indicates that they have indeed encountered problems in actual deployment.

Q4: Why does Hacker News mock it?

Shiwen: Returning to HN’s reaction, why don’t people believe it?

Yongliang: There are mainly three reasons. First, the model hasn’t been truly made open. Some say it’s like “my girlfriend goes to another school,” meaning “my model is strong, but you can’t use it now.” Second, the naming is confusing. Names like Argon, Carbon, and Mythos make it hard for users to understand the product lines and positioning. Two weeks ago, people were praising Argon, and now they say Carbon is stronger. What’s the point of buying Argon before that? Third, some suspect that this is Google AI engineers’ performance material for the end of the year, which ordinary users cannot access.

Shiwen: Are these criticisms fair?

Yongliang: Some fairness. It’s true that it hasn’t been made available, and it’s also true that the naming is confusing. But calling it a “performance material” might be too harsh. A more accurate description would be: The news release Google is releasing now is just a “ranking placeholder” announcement, not a truly inclusive product. According to official statements, it will first be available to some cyber defenders starting this week, then gradually made available to trusted partners through the Fairwind Program, and finally to developers, enterprises, and consumers. This timeline indicates that Google isn’t yet ready to roll out it fully.

Shiwen: So what’s the point of announcing the pricing?

Yongliang: The pricing has been announced, just like with Argon: 2 dollars per million tokens input, 10 dollars per million tokens output, with a 95% discount on input caching. This price is only on paper, as most people can’t afford it. For developers, a model with a price but not available is no different in experience from a model that hasn’t been released yet. This is also the core of HN’s criticism: You tell me it’s strong and cheap, but I can’t use it. Then why bother telling me about it? Moreover, in actual use, cache hit rates and actual call costs are subject to many variables, and paper prices often don’t represent real costs.

Q5: How should ordinary users view this situation?

Shiwen: Finally, give ordinary users a framework for judgment. How should they view this situation?

Yongliang: There are three levels. First, when looking at numbers, distinguish between “official ranking results” and “real usability”. Being number one in the rankings doesn’t mean you can use it. Second, when considering scenarios, distinguish between “laboratory security tests” and “enterprise production environments”. No matter how high the official benchmarks are, they still need external re-testing and real-world usage validation. Third, when looking at the release timeline, distinguish between “technical progress” and “marketing placement”. Two weeks ago it was Argon, now it’s Carbon, with confusing naming. Google is clearly pushing things forward quickly.

Shiwen: What specific suggestions do you have for different people?

Yongliang: Security professionals should pay more attention, but don’t rush to make predictions. Wait for third-party re-testing and enterprise usage. Developers can continue to wait and see; their Gemini API or other models won’t suddenly become insufficient just because Carbon is released. Enterprise buyers should note that “Google claims to be the strongest” shouldn’t be used as a basis for purchase. Instead, ask: What specific problems can this model solve for me? Is the cost high? Is integration troublesome? As for ordinary users, remember one thing now: Strength on the rankings doesn’t equal strength in your hands.

Shiwen: So what’s the conclusion?

Yongliang: Gemini 4 Carbon may be Google’s current strongest model, at least according to official statements. But it hasn’t yet entered the toolboxes of ordinary developers. For those who care about AI security, code security, and agent-based workflows, it’s worth paying attention; for ordinary users, just remember: No matter how attractive the rankings are, if they don’t translate to reality, they’re just a picture. We can draw conclusions later when the API is truly open, third-party independent re-testing is done, and enterprise-level usage starts.

Conclusion: No matter how attractive the rankings are, they must be practical to be considered real.

Shiwen: In one sentence.

Yongliang: The most important thing Google announced this time is not the score, but the signal it sent: the competition in large models has entered a deep-water zone of “long-term tasks + security capabilities”. Carbon is a ticket to enter this field, but whether we can convert this ticket into a real seat depends on subsequent openness and third-party verification. Before getting hands-on with the API, we should consider this as a release worth paying attention to, without rushing to make conclusions.

Shiwen: Give a one-sentence summary for today’s article.

Yongliang: Ranking at the top is just the beginning; whether the capabilities can be delivered to developers is the true measure of a model.

👇 Have you used any of the Gemini models recently? How did it feel? Share your thoughts in the comments section. We’ll pick some discussions for next time. Follow me for honest workplace insights in the AI era, updated every week.