Gryphon SDLC Benchmark evaluates models on engineering tasks across the software development lifecycle. The chart compares quality with combined execution time and glyph usage. The table shows each metric separately. We use these results to select models for Gryphon modes.
Higher means better quality. Further left means a lower time–glyph product. Hover or tap a point to see the model and its metrics.
median · logarithmic scale
Manufacturers
Quality is a rounded engineering rubric index. “Lowest-scoring area” identifies the category with the lowest delivered-work score for that model, including mandatory checks; engineering rubric scores break ties.
We multiply execution time in minutes by recorded glyph usage for each task, then take the median. Equal horizontal distances represent equal ratios. A lower value can reflect faster execution, lower usage, or both; separate medians appear in the table.
Time and glyphs in the table are separate medians across the same task set, including unsuccessful solutions and excluding subsequent judging. Values are rounded to one decimal place.
Click a column heading to change the sort order.
| Model | |||
|---|---|---|---|
| GPT-6 AstraOpenAILowest-scoring area: testing | Quality97 | Time4.5 | Glyphs112.6 |
| Grok 4.6xAILowest-scoring area: development | Quality97 | Time6.7 | Glyphs32.8 |
| DeepSeek V4 Pro 0813DeepSeekLowest-scoring area: testing | Quality96 | Time9.5 | Glyphs13.8 |
| GPT-5.6 SolOpenAILowest-scoring area: security | Quality96 | Time2.6 | Glyphs16.4 |
| Muse Spark 1.3MetaLowest-scoring area: testing | Quality96 | Time2.8 | Glyphs25.7 |
| Claude Opus 5AnthropicLowest-scoring area: security | Quality95 | Time8.5 | Glyphs190.8 |
| Claude Fable 5.1AnthropicLowest-scoring area: security | Quality94 | Time6.7 | Glyphs223 |
| Gemini 3.8 FlashGoogleLowest-scoring area: development | Quality94 | Time4 | Glyphs44.1 |
| Claude Sonnet 5AnthropicLowest-scoring area: testing | Quality88 | Time9.1 | Glyphs99 |
| GPT-5.6 TerraOpenAILowest-scoring area: testing | Quality88 | Time1.6 | Glyphs16.7 |
| GPT-5.6 LunaOpenAILowest-scoring area: security | Quality83 | Time2 | Glyphs2.1 |
| Qwen 3.8 Max 0902AlibabaLowest-scoring area: testing | Quality81 | Time16.2 | Glyphs69.1 |
| Kimi K3Moonshot AILowest-scoring area: testing | Quality73 | Time18.2 | Glyphs97 |
| GLM 5.3Z.aiLowest-scoring area: security | Quality60 | Time5.6 | Glyphs1.5 |
| MiniMax M3MiniMaxLowest-scoring area: maintenance | Quality36 | Time9.8 | Glyphs3.2 |
Each model works through the CodeGryphon engine. It receives a task, source code and documentation, uses the available tools and saves its output in the workspace. The corpus consists of controlled projects that simulate engineering work, allowing models to be compared on the same task specifications and starting project state.
After execution, we examine changes and artifacts. Programmatic checks assess behavior, constraints and mandatory requirements. Model judges separately assess correctness, engineering safety, verification quality and explanation clarity. Candidate identity and execution time are hidden before this assessment.
The vertical axis shows an average engineering rubric score. Correctness carries the greatest weight, followed by safety, verification and explanation clarity. Categories have equal weight. “Lowest-scoring area” identifies the category with the lowest delivered-work score, including mandatory checks. Engineering rubric scores break ties. This is a relative comparison of categories within each model’s results.
For each task, we multiply execution time in minutes by recorded glyph usage. The median of those products defines the chart’s horizontal position. It is not the product of two separate medians. A logarithmic scale makes a wide range comparable: equal distances represent equal ratios. The table shows time and glyph usage separately. Unsuccessful solutions are included; subsequent judging is excluded.
We cover the full SDLC, from analysis and development to testing, safe changes and operations. These examples from the corpus show what each task asks for and how its result is checked.
Investigate an incident using source evidence and traces: reconstruct causal chains, estimate losses and distinguish supported conclusions from hypotheses.
What we checkImplement transfers in a ledger service, including amount validation, atomic operations and safe handling of retries and concurrent requests.
What we checkWrite regression tests for an asynchronous authorization cache and fix isolation, revocation and concurrent request handling.
What we checkSecure signed archive downloads: bind permissions to the tenant and content, reject unsafe paths and enforce decompression limits.
What we checkUpdate snapshot migration while preserving compatibility with older formats, operation history and replay.
What we checkProduce an executable event-processing recovery plan using incident evidence and an operations runbook.
What we checkWe use the benchmark to select models for Gryphon modes. We consider the kind of work, quality, mandatory requirements and execution time. Each mode defines a capability level; its model selection can change as new evaluations become available.
For simple, well-specified tasks: short explanations, text revisions and small edits with an obvious implementation.
For everyday engineering: development, debugging, tests and changes across multiple files.
For work with high uncertainty or serious consequences of error: architecture, security, complex migrations and concurrency.
Chooses a capability level from the request and context, considering reasoning depth, scope, uncertainty and risk.