Most guardrail models classify prompts by reasoning toward one verdict. ARBITER introduces dual-hypothesis reasoning: before deciding, the guardrail explicitly constructs both the safe and unsafe interpretations of every prompt, then adjudicates. Combined with multi-component SFT (MC-SFT) that separately trains the safe-hypothesis, unsafe-hypothesis, and final-judgment components, ARBITER improves safety detection over single-hypothesis baselines. The adversarial framing — considering the best case for both sides — is a simple but underexplored idea in guardrail design.
Unslop scored 12,750 arXiv papers from 2021-2026, calibrating their detector to a 0.4% false-positive rate on pre-ChatGPT papers to build a rigorous baseline. Machine-written flags rose in two waves to ~32% in the most recent complete quarter, peaking near 39% in early 2026. Computer science leads at ~65%; mathematics is lowest at 0.7% (though that may reflect detector sensitivity to notation-heavy prose rather than true low adoption). The methodology is unusually careful about confounds and worth reading for practitioners who run or evaluate AI-assisted research pipelines.
Anthropic mathematician Levent Alpöge posted on July 19-20 that he and a colleague used Claude Fable 5 to find a counterexample to the Jacobian Conjecture — a problem open since 1939 and listed in Smale's 1998 Mathematical Problems for the Next Century. The counterexample is a concrete C³→C³ polynomial map with constant Jacobian determinant of -2 that is demonstrably non-injective; three distinct points all map to the same output, verifiable via Wolfram Alpha. The 2D case remains open. Wolfram MathWorld already updated its entry. HN thread (746 points) noted "Claude Fable was able to find a counterexample in degree 7" and called it a new era for AI in mathematics.
Proposes a competition for distinguishing robust from spurious reasoning in frontier mathematical LLMs using internal model mechanisms rather than final-answer accuracy. Motivated by the gap between strong benchmark scores and unreliable internal reasoning: a model can get the right answer for the wrong reasons. Focuses on math olympiad-level problems (AI Mathematical Olympiad context), providing a structured interpretability challenge tied to an existing competition with real stakes.
Framework for stress-testing multi-step search agents under poor-quality evidence conditions — a failure mode that is rare in standard benchmarks but catastrophic in production. DeepStress controls the frequency and severity of misleading/noisy evidence in retrieved documents and measures agent robustness, exposing brittleness invisible to standard QA metrics. Relevant for any practitioner deploying RAG-based or web-search agents in adversarial or noisy environments.
Empirical study of 25,000+ GitHub projects examining how agentic coding tools are adopted and managed at the project level — not just individual PR outcomes. Analyzes governance patterns, integration policies, and project-level responses to AI-generated PRs. Useful ground-truth data point on how open-source communities are adapting (or not) to autonomous code contributions, at a time when such tools are proliferating rapidly.
Hume AI published Real World VoiceEQ, a benchmark covering 40+ proprietary and open-source voice models across 15+ evaluation dimensions and 60+ metrics spanning ASR, TTS, Speech-to-Speech, and Speech Understanding — grounded in 1M+ human ratings (785k TTS, 48k STS). Key finding: no single model ranked in the top five across all eight TTS capability groups; speech models have become better at speaking than listening, often ignoring paralinguistic cues like tone, hesitation, and emphasis that humans rely on. The benchmark also found that LLM-based auto-evaluation degrades significantly for subjective voice quality judgments, suggesting SLMs can't yet replace human raters for open-ended voice assessment.
📑 arXiv 3w ago
⭐ Editor's Pick
As RLVR increasingly uses an LLM judge as the reward model, this paper asks: how capable must the judge be, and how biased is it? Studies this calibration question specifically for citation quality in deep-research systems — a practically important but understudied reward signal. Finds that frontier models are not always necessary as citation judges, and characterizes the bias profile of smaller rubric LLMs. Directly informs design of reward models for training web-grounded research agents.
OpenAI publishes an analysis of SWE-Bench Pro raising questions about the benchmark's reliability and accuracy for evaluating coding AI models. The post identifies specific issues in how SWE-Bench Pro tasks are constructed or scored that may affect the signal practitioners extract from results — notable given that SWE-Bench Pro is the benchmark where Claude Mythos 5 leads with 80.3% vs GPT-5.6 Sol's 64.6% in the concurrent model release. Practitioners using SWE-Bench Pro results to guide model selection should review the analysis before drawing conclusions.
OpenAI released LifeSciBench, a 750-task benchmark spanning 7 life-science workflows (evidence handling, analysis, design/optimization, reasoning, validation, translation, communication) authored by 173 PhD-level scientists with industry experience and reviewed by 453 experts. Tasks average 25 rubric criteria each (19,020 total), 79% require multi-step reasoning, and 53% require artifact interpretation (figures, PDFs, genomic sequences, structure files). GPT-Rosalind, a purpose-built life-sciences model, achieves 36.1% overall exact pass rate vs. 25.7% for GPT-5.5, with the largest gains in Translation (57.7% vs 36.8%) and Scientific Communication (71.1% vs 56.3%). Artifact-heavy tasks remain a clear gap: pass rates drop from ~45% (text-only) to 28% (artifact tasks) for GPT-Rosalind.
Alec Radford (GPT, CLIP, Whisper), Nick Levine, and David Duvenaud released Talkie, a 13B model trained only on text published before 1931. The model's worldview is frozen at December 31, 1930: it doesn't know about World War II or the internet. Claude Sonnet was used to help test and judge the model's output. The r/ClaudeAI post (786 upvotes) drew discussion about what the model reveals about the common ancestor problem — every major LLM today shares overlapping training data, making true behavioral diversity impossible without deliberate data partitioning. Notable for the eval methodology: using a modern LLM to judge a historically-constrained one.
Self-hostable, Apache 2.0-licensed platform covering the full LLM application observability and improvement loop: tracing, evals, simulations, datasets, gateway, and guardrails in one stack. Targets teams who want an integrated alternative to stitching together Langfuse, LangSmith, and separate guardrail layers. Open-source with enterprise-grade feature breadth.
Side-by-side comparison showing GPT Image 2 struggles with photorealistic nature scenes, producing a recognizable artifacting pattern absent in its predecessor. Three images from the same prompt illustrate the regression, flagging a quality tradeoff in the new model for natural/outdoor imagery.
Investigates how prompt optimization and judge choice interact in LLM-as-a-Judge evaluations for legal QA on the LEXam benchmark, using ProTeGi optimization with Qwen3-32B and DeepSeek-V3 as judges. Lenient judge feedback yields larger and more consistent gains than strict feedback, and prompts optimized with lenient judges transfer better across judge models. Results highlight that judge disposition is a significant, underappreciated variable in automated evaluation pipelines.
A practitioner's post-mortem on building fully autonomous multi-agent systems for clients: unpredictable recursive loops, runaway API costs ($200 in 2 hours), and zero client tolerance for black-box failures pushed the author toward human-in-the-loop, deterministic workflows instead. The core argument — autonomy is a liability for most business use cases — is grounded in specific failure modes rather than theory.
Open-source test harness for text-to-CAD generation, providing scaffolding to prompt LLMs and evaluate their CAD model outputs. Targets the emerging niche of AI-driven parametric and 3D design automation.
Identifies 'preference leakage': when the same LLM generates synthetic training data and serves as the judge, it systematically inflates scores for outputs matching its own generation style, biasing leaderboard rankings even when models perform similarly. Demonstrated empirically across several evaluation pipelines. A concrete warning against self-referential LLM-as-a-judge setups.
SkillLearnBench is the first benchmark for continual skill learning in LLM agents, covering 20 verified tasks across 15 sub-domains with evaluation at three levels: skill quality, execution trajectory, and task outcome. Tested methods include one-shot learning, self/teacher feedback, and skill-creator approaches; all improve over the no-skill baseline but none achieves consistent gains across domains. Highlights that automatic skill acquisition for agents remains an unsolved problem despite recent progress.
Two related papers: the first is a large-scale empirical study of 2,303 agent context files from 1,925 GitHub repos, characterizing how these 'READMEs for agents' evolve like configuration code across 16 instruction types. The second analyzes Claude Code's architecture, mapping five human values through thirteen design principles to concrete implementation choices including a core while-loop. Both reflect growing interest in formalizing agentic software engineering practices.
💬 Reddit Apr 16
⭐ Editor's Pick
User benchmarks show Claude Opus 4.7 scoring 59.2% vs Opus 4.6's 91.9% on the MRCR v2 8-needle 256K context benchmark — a sharp context retention regression. Compounding the issue, a tokenizer change reportedly causes Opus 4.7 to consume ~1.35x more tokens than Opus 4.6 and ~2x more than competing proprietary models, effectively raising costs ~50% for equivalent workloads. If the benchmark numbers hold, this is a meaningful quality-cost tradeoff moving in the wrong direction.
A 668-point HN post documents that swapping the evaluation harness — without changing any model — improved measured coding performance across 15 LLMs in an afternoon. Directly implicates harness sensitivity as a major confounder in coding benchmark results. High-signal for anyone designing or interpreting code evals.