Issues

• [Bug]: AIME25 shared disk cache duplicates AIME2025-I and omits AIME2025-II

  mlcommons/endpoints#538 · Filter by author tianmu-li opened on Oct 14, 2026

• [Bug]: Agentic 10 h max_issue_duration_ms cannot complete the required full pass on bandwidth-limited systems for Atlas Inference, longest turn exceeds Qwen3.6's 262,144 context with max_new_tokens 8192

  status: needs-triagetype: bug
  mlcommons/endpoints#534 · Filter by author AzeezIsh opened on Oct 11, 2026
  2

• Streaming: sample.complete can be stamped before sample.recv_first (negative TPOT): COMPLETE uses decode-time completed_at, RECV_* use handling-time now_ns

  area: metricsstatus: needs-triage
  mlcommons/endpoints#533 · Filter by author kimseunghyun-kr opened on Oct 8, 2026
  1

• [Bug]: Agentic performance cutoff biases measured input lengths at high concurrency

  status: needs-triagetype: bug
  mlcommons/endpoints#528 · Filter by author tianmu-li opened on Oct 5, 2026
  10

• [Regression] Min duration under steady state

  priority: P0
  mlcommons/endpoints#525 · Filter by author attafosu opened on Oct 2, 2026
  2

  

• SWE-bench accuracy phase is silent in logs until completion: log phase transitions and progress

  mlcommons/endpoints#524 · Filter by author arekay-nv opened on Oct 2, 2026

• [Bug]: Pyxis environment for SWE bench is calling one srun per tool call, extremely long call time and crashes

  status: needs-triagetype: bug
  Bug

  mlcommons/endpoints#521 · Filter by author leopck opened on Sep 29, 2026
  

• Remove pip-audit ignore for nltk PYSEC-2026-3740 once a fixed release ships

  dependenciessecuritystatus: blockedtype: chore
  mlcommons/endpoints#520 · Filter by author arekay-nv opened on Sep 29, 2026

• [Feature]: move num_trajectories_to_issue from datasets.agentic_inference fields into setting.num_trajectories_to_issue

  status: needs-triagetype: feature
  mlcommons/endpoints#517 · Filter by author zihaok opened on Sep 22, 2026

• [Feature]: plug in steady state reporting

  status: needs-triagetype: feature
  mlcommons/endpoints#510 · Filter by author arekay-nv opened on Sep 18, 2026
  1

  

• [Bug]: Warning/Error "parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)" during running performance(online, offline), sglang 0.5.15.post1, 0.5.18

  status: needs-triagetype: bug
  mlcommons/endpoints#497 · Filter by author maria-18-git opened on Sep 8, 2026
  1

• Add an overall deadline for the complete compliance audit sequence

  mlcommons/endpoints#466 · Filter by author viraatc opened on Aug 25, 2026

• Parameterize run-timeout coverage across benchmark lifecycle stages

  mlcommons/endpoints#465 · Filter by author viraatc opened on Aug 25, 2026

• Add a deadline for accuracy scoring and finalization

  mlcommons/endpoints#464 · Filter by author viraatc opened on Aug 25, 2026

• [Feature]: populate session_started_ns correctly

  status: needs-triagetype: feature
  mlcommons/endpoints#463 · Filter by author arekay-nv opened on Aug 24, 2026
  1

• Consolidate run-outcome state flags into one abort/outcome model

  mlcommons/endpoints#459 · Filter by author viraatc opened on Aug 20, 2026

• schema.py: dedupe cyclopts help= vs Field(description=) across config fields

  mlcommons/endpoints#451 · Filter by author arekay-nv opened on Aug 18, 2026

• Promote warmup to a first-class phase type with per-phase config

  mlcommons/endpoints#449 · Filter by author viraatc opened on Aug 14, 2026
  1

• Optimize structured chat-template tokenization drain at scale

  mlcommons/endpoints#444 · Filter by author hvagadia opened on Aug 13, 2026
  2

  

• [Feature]: swebench pyxis x86 support

  area: evaluationpriority: P1type: enhancementtype: feature
  mlcommons/endpoints#440 · Filter by author arekay-nv opened on Aug 7, 2026
  

• [Bug]: VLM (qwen3-235B) scoring fails: inference_endpoint/evaluation/scoring.py msgspec.DecodeError: JSON is malformed: invalid character (byte 145)

  status: needs-triagetype: bug
  mlcommons/endpoints#427 · Filter by author ai-nikolai opened on Jul 23, 2026
  2

  

• [Bug]: output config.yaml should respect --mode flag

  status: needs-triagetype: bug
  mlcommons/endpoints#420 · Filter by author arekay-nv opened on Jul 16, 2026

• HTTP client: num_workers must be a multiple of the endpoint count for even load — auto-scale workers for multiple endpoints

  mlcommons/endpoints#419 · Filter by author viraatc opened on Jul 16, 2026

• Connection pool: treat EADDRNOTAVAIL as backpressure so the full-budget auto ceiling is safe under max_throughput bursts

  mlcommons/endpoints#418 · Filter by author viraatc opened on Jul 16, 2026

• Clarify BFCL / tool-call output token count (OSL is approximate for structured outputs)

  mlcommons/endpoints#410 · Filter by author arekay-nv opened on Jul 14, 2026