The MLCommons Edge LLM Taskforce is excited to introduce a new Edge Agentic Inference benchmark for the MLPerf Inference v6.1 round. As agentic LLMs – coding copilots, robotics controllers, and private on-prem assistants – increasingly run on-device, measuring how well and how fast these models call tools under a real edge budget is more important than ever.
For this round we debut the Qwen3.6-27B model (released April 22, 2026), served as a Q4_K_M GGUF quantization (reference implementation’s format – submitters may use any allowed quantization that meets the accuracy threshold) on a single edge accelerator, paired with two complementary workloads: the Berkeley Function Calling Leaderboard v4 (BFCL v4) for a deterministic, judge-free accuracy gate, and a recorded agentic-coding replay for single-stream performance. Together they capture the messy reality of on-device agents: long, growing tool
calling conversations that must fit within a hard context wall and feel responsive to a single interactive user.
The submission deadline is July 31, 2026. We invite all hardware vendors, edge-device makers, and inference-system experts to submit results and help raise the bar for on-device agentic inference.
Why Agentic AI at the Edge
The shift from single-shot text generation to multi-turn agentic workloads is one of the most consequential trends in applied AI. A real agentic session is not one prompt and one answer – it is a trajectory where each turn appends tool outputs and model responses to a growing conversation history, the model alternates between short tool calls and longer reasoning, and every turn depends strictly on the one before it.
The datacenter MLPerf Agentic benchmark (coming up in September) measures this at scale – large mixture-of-experts models, long-horizon trajectories whose cumulative multi-turn context runs to 100K+ tokens, and a concurrency sweep that produces a per-GPU Pareto frontier. The edge regime is the opposite problem, and it is what this benchmark targets:
• A fixed memory and power budget: A single small accelerator (with a limited amount of memory) must hold weights, KV cache, and activations. Full-precision frontier models do not fit, so the model is quantized.
• A single in-flight request: Edge serving is dominated by one interactive user, not thousands of concurrent sessions.
• A fixed served context window. The reference fixes the served context at 32K tokens – a controlled benchmark parameter, not a device ceiling – so long trajectories can exhaust it. This makes context growth and truncation first-order, measured effects of the agentic workload rather than something that varies with per-device memory.
• Single-user latency, not aggregate throughput. On a single-slot edge device, throughput and latency are reciprocal to each other, so the metrics that matter are per-turn TTFT, TPOT, and end-to-end turn latency on one device – not the aggregate QPS/token throughput a multi-stream datacenter server optimizes for.
This benchmark inherits the multi-turn agentic methodology – terminology, deterministic replay, the JSONL dataset schema, and the inline accuracy check – from the datacenter agentic specification and specializes it to the edge: an edge model and quantization, a single-stream load pattern, latency-centric metrics, and a statistically robust accuracy gate.
Model Selection
The reference model is Qwen3.6-27B (https://qwen.ai/blog?id=qwen3.6-27b), served with llama.cpp (commit cfff1fc) as a Q4_K_M GGUF quantization (https://huggingface.co/unsloth/Qwen3.6-27B-GGUF). Qwen3.6 was selected because it is an open (Apache 2.0), capable tool-calling model with native MTP speculative-decoding head that punches above its size – a dense 27B that Alibaba reports (https://qwen.ai/blog?id=qwen3.6- 27b) beats the prior 397B-MoE flagship on major coding benchmarks (77.2% SWE-bench Verified), yet deploys easily: official weights on Hugging Face and ModelScope plus community GGUF builds (e.g. unsloth) that run directly under llama.cpp.
Quantization is the defining edge choice. Q4_K_M is a 4-bit K-quant “Medium” tier: weights are stored at 4 bits/value (a 4× memory reduction versus BF16), attention blocks are kept at higher precision than the feed-forward blocks, and importance-weighted (imatrix) calibration preserves the highest-impact weights. The result fits the 27B model in ~16.5 GB of VRAM (versus ~54 GB for BF16) at a ~2–5% accuracy cost – the difference between “runs on an edge GPU” and “does not.” The exact GGUF used for the reference runs is Qwen3.6-27B-Q4_K_M.gguf from unsloth/Qwen3.6-27B-GGUF.
Reference sampling parameters used are listed in the table below:
| Parameter | Value |
| temperature | 0 |
| top_k | 1 (Moot under temperature=0) |
| top_p | 1.0 (Moot under temperature=0) |
| seed | 42 |
max_new_tokens 1024
| repetition_penalty | 1 |
| reasoning | off |
| context size | 32768 (32K) |
| parallel slots | 1 |
Reasoning is deliberately disabled on this tool-calling workload; it lowers single-turn accuracy while costing far more time (see Accuracy Metric section). The benchmark has been validated across a range of edge accelerators from multiple vendors, confirming that the workload and methodology are portable.
The Benchmark Task
The benchmark uses two datasets with distinct roles.
Accuracy dataset: The Berkeley Function Calling Leaderboard v4 (BFCL v4) tests whether the model calls functions correctly, deterministically, and without an LLM judge. It spans single-turn requests (one prompt → one structured tool call) in three categories – non_live and live (graded by AST match against gold labels) and hallucination (a binary check that the model refuses to call a function when the available tools are irrelevant) – and optional multi-turn agentic conversations against in-process Python simulators. The categories are drawn from the public gorilla-llm/gorilla-eval-set (auto-downloaded at runtime). The scored gate is the single turn set only, per-category sampled to a stable ~995-sample point estimate: non_live 72% (~712 samples), live17% (~171), and hallucination11% (~112), with a subset_floor of 25 so any subset of ≤ 25 entries is taken in full.
Performance dataset. A subset of the MLPerf Agentic benchmark – recorded multi-turn agentic coding trajectories (SWE-bench-style tasks drawn from real repositories e.g., astropy), replayed deterministically as a performance workload, and scaled for single-device edge serving. The reference set comprises 20 conversations / 1,007 turns, sized so that no trajectory overflows the 32K-token served context (peak input length ~23.5K tokens). Because nothing overflows, every turn completes: the run is valid with zero dropped turns, and a single pass at concurrency 1 (one request in flight at a time) completes in a reasonable time on a single edge device. The dataset also serves as its own ground truth – each trajectory’s recorded tool calls drive a zero-cost inline accuracy check (multiset IOU of the executed calls) that runs during the latency measurement, so correctness and latency come from the same pass.
Example Request and Response
A single-turn function-calling request is sent to the model’s OpenAI-compatible /v1/chat/completions endpoint, carrying the available tool schemas:
},
{
"role": "user",
"content": "What is the current temperature in San Francisco, in Celsius?"
}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather for a given location.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City and state, e.g. 'San Francisco, CA'"
},
"unit": { "type": "string", "enum": ["celsius", "fahrenheit"] }
},
"required": ["location", "unit"]
}
}
}
]
} The model responds with a structured tool call:
{
"choices": [
{
"message": {
"role": "assistant",
"tool_calls": [
{
"type": "function",
"function": {
"name": "get_current_weather",
"arguments": "{\"location\": \"San Francisco, CA\", \"unit\": \"celsius\"}" }
}
]
},
"finish_reason": "tool_calls"
}
]
}BFCL v4’s AST checker compares the predicted call – function name and each argument – against the gold label, so scoring is exact and reproducible. In the agentic-coding performance workload, the same mechanism applies to multi-turn trajectories: the model emits bash tool calls, the runner records them, and the inline checker scores the multiset intersection-over union (IoU) of the canonicalized shell executables against the recorded ground-truth trajectory.
Performance Metrics
The edge benchmark reports single-stream, per-turn latency on a single accelerator rather than a per-GPU throughput frontier. A single simulated user replays trajectories closed-loop: send a turn, wait for the full streamed response, pause for the pre-recorded inter-turn delay, send the next turn – exactly one in-flight request at all times (target_concurrency = 1, one worker, one connection), matching the single-slot llama.cpp server. The reported metrics are TTFT (time to first token), TPOT (time per output token), end-to-end latency per turn, and ISL/OSL distributions, as p50/p90/p99/max.
Accuracy Metric
Accuracy is the scored BFCL v4 single-turn gate, chosen for statistical robustness: the full set of single-turn categories is sampled to ~995 samples, completing in a reasonable time on an edge device. Single-turn was chosen because its score is stable at large n and across devices, whereas a small multi-turn gate is brittle – an n=57 multi-turn gate carries a ~±11.6 pp 95% confidence interval, and Q4 quantization-kernel differences cause per-entry verdicts to flip across devices. Multi-turn is retained as an optional, non-scored reference artifact.
Reference single-turn accuracy (Qwen3.6-27B-Q4_K_M, seed 42, temperature 0; hardware independent, and identical across two seeded runs with a freshly restarted server each pass):
| Category | Accuracy | Notes |
|---|---|---|
| non_live (AST) | 82.59% | ~712 samples |
| live | 84.12% | ~171 samples |
| hallucination | 97.16% | ~112 samples |
| Overall (~995) | 86.23% | gated metric (sample weighted) |
| non_live-normalized | 87.96% | gated metric (mean of the above three categories) |
There are two gated metrics: Overall, the sample-weighted accuracy across all ~995 single-turn samples, and Normalized, the category-balanced mean of the three category scores (so each category counts equally regardless of size). The per-category rows are components, not gates.
The pass/fail criterion is a 3% one-sided band applied to both gated metrics: a submission passes only if its Overall and Normalized single-turn scores are each ≥ 0.97 × their respective references, with no upper bound (a higher score never fails).
| Gated metric | Reference | Pass threshold (0.97 ×) |
|---|---|---|
| Overall | 86.23% | ≥ 83.64% |
| non_live-normalized | 87.96% | ≥ 85.32% |
Reasoning off is the reference submission config. On this tool-calling workload, enabling server-side reasoning gives no accuracy benefit and costs far more wall-clock: on the accuracy gate, reasoning-off scores 86.23% versus reasoning-on’s 78.19% – an 8-point drop – and on the performance replay, reasoning-on moves inline IoU by only +0.004 (0.6374 vs 0.6335) while taking ~60% longer run-time.
Join the Benchmark
This is your opportunity to show how your edge hardware and software stack runs real multi-turn tool-calling workloads – on both accuracy and latency. Submissions for MLPerf Inference v6.1 are open to MLCommons members. Join MLCommons to submit your results by July 31, 2026.
• Model: Qwen/Qwen3.6-27B, served as a Q4_K_M GGUF – Qwen3.6-27B-Q4_K_M.gguf from unsloth/Qwen3.6-27B-GGUF.
• Accuracy dataset: gorilla-llm/gorilla-eval-set (BFCL v4 single-turn), auto-downloaded at runtime.
• Reference implementation: mlcommons/endpoints – examples/10_Edge_Agentic_Example/. A single mandated config (online_edge_full_run.yaml) runs performance and accuracy back-to-back against the same server; –accuracy-only runs the gate alone.
• Submission deadline: July 31, 2026.
The reference implementation drives any OpenAI-compatible Chat Completions endpoint, so you can benchmark arbitrary inference systems – not just the llama.cpp reference server. Anyone with an edge accelerator can reproduce the reference accuracy and latency numbers by following the example README. We look forward to your submissions!
Acknowledgements
We thank the MLCommons Edge LLM Taskforce and the MLPerf Inference Working Group for their feedback and guidance in developing this benchmark.