🏃 Keep Up
← All topics
Inference 21 items

Everything Inference

🟧 Hacker News 1w ago

Cursor: Agent Swarms and the New Model Economics

Cursor ran old vs. new agent swarms on a task of implementing the full 835-page SQLite manual in Rust from scratch, graded against the sqllogictest suite. The new harness reached 73-85% at four hours across all model mixes; the new system peaks at ~1,000 commits/second vs. old system's 1,000/hour, and reduced merge conflicts from 70,000+ to under 1,000. Key cost finding: Opus 4.8 as planner + Composer 2.5 as worker achieved similar quality to GPT-5.5 solo ($1,339 vs. $10,565), because few moments in a large task require frontier intelligence. The post describes novel coordination mechanisms: shared design docs with compile-checked references, megafile decomposition, and a self-authored "Field Guide" for agent stigmergy.

🟧 Hacker News 1w ago
⭐ Editor's Pick

Kimi K3, Qwen 3.8, and Anthropic's structural disadvantage as a model-only provider

Wojciech Gryc argues that Kimi K3 (Moonshot Labs) and Qwen 3.8 (Alibaba) — both SOTA-competitive and soon open-weight — represent a deeper threat than the 2025 DeepSeek moment because they show a sustained pattern of catchup. The economics analysis is sharp: companies without owned data-center/power infrastructure (Anthropic, OpenAI, Knowledge Atlas) face variable costs that scale with revenue, while infrastructure owners (Meta, Alibaba, SpaceX/Grok) can grow margin. Fable 5 is ~3x more expensive per completed task than open alternatives; with no infrastructure moat and rising harness competition from OpenCode, OpenClaw, Hermes et al., Anthropic's position is uniquely precarious.

🟧 Hacker News 2w ago
⭐ Editor's Pick

Kimi K3: 2.8T-parameter open-weights MoE launches at Opus 4.8-class quality, Sonnet 5 pricing

Moonshot AI launched Kimi K3 on July 16 — a 2.8T-parameter MoE (16/896 experts active, 1M-token context) now live on API, with full weights promised by July 27. Architectural highlights: Kimi Delta Attention (6.3x faster decoding at 1M context), Attention Residuals, LatentMoE, MXFP4/MXFP8 quantization. Artificial Analysis scores it 57 on Intelligence Index (near Claude Opus 4.8, behind Fable 5/GPT-5.6 Sol); debuts #1 on LMArena Frontend Code Arena at 1679 Elo. Priced at $3/Mtok input, $15/Mtok output — well below Western frontier pricing. Community reaction is strong (399 activity score on Latent Space), with caveats: local inference is impractical for most consumer setups, and unverified distillation-from-Claude chatter is circulating.

🧠 DeepMind 2w ago

Gemini Spark launches: proactive cloud agent for macOS (Ultra subscribers)

Google launched Gemini Spark — a proactive agentic mode that works on tasks autonomously on the user's behalf (local file management, multi-step projects) — initially to trusted testers then rolling to Google AI Ultra subscribers ($100/month) in the US via the macOS Gemini app. Spark represents Google's first 24/7 cloud agent surface that works server-side without requiring the device to stay online, directly competing with Claude Cowork and ChatGPT Work.

✍️ Will Larson 2w ago

Make no assumptions

Larson extends his "code horizons" / soil-layers metaphor to LLM-assisted reasoning: when agents produce analysis that nobody reviews before passing it upstream, organizations accumulate "reasoning horizons" of flawed conclusions stacked on one another. The post gives a concrete incident — an MCP-pulled data analysis that omitted half the relevant incidents — and proposes five countermeasures: assume nothing, require creators to be first human in the loop, reclaim unreasonable software, separate play from production, and structure how you use LLMs for strategy. One of the sharpest practitioner takes on AI-in-the-workflow quality degradation this month.

🟧 Hacker News 3w ago
⭐ Editor's Pick

Frugon: Find which LLM calls a cheaper model could handle (local, MIT)

Show HN project appearing on the HN front page this week: Frugon is a local, MIT-licensed tool that analyzes LLM call logs and identifies which requests could have been handled by a cheaper or smaller model. Directly addresses the cost-routing problem that practitioners face when running mixed-model production workloads — complements the broader trend of tiered model families (GPT-5.6 Sol/Terra/Luna) by providing observability on where you're over-spending on capability.

🟢 OpenAI 3w ago
★ High Signal

GPT-5.6 (Sol, Terra, Luna) — General Availability

OpenAI released GPT-5.6 as a three-tier family on July 9: Sol (flagship, $5/$30 per M tokens), Terra (balanced, $2.50/$15), and Luna (fastest, $1/$6). Sol claims 54% more token efficiency on agentic coding vs. prior models, scores 80 on Artificial Analysis Coding Agent Index (2.8pp above Fable 5), and uses less than half the output tokens and one-third the cost per task. Available across ChatGPT, Codex, and the OpenAI API. The release was delayed briefly for US government safety review under OpenAI's Preparedness Framework, which classifies Sol as High capability in Cybersecurity and Bio/Chem risk.

🟧 Hacker News 3w ago
⭐ Editor's Pick ★ High Signal

GPT-5.6 (Sol/Terra/Luna) — general availability with new benchmark results and pricing

OpenAI launches GPT-5.6 as a three-tier family: Sol (flagship, $5/$30 per 1M tokens), Terra ($2.50/$15), and Luna ($1/$6). Sol scores 53.6 on Agents' Last Exam — 13.1 points above Claude Fable 5 — and hits 80 on the Artificial Analysis Coding Agent Index, just above Fable 5, while using less than half the output tokens. Key new API primitives: Programmatic Tool Calling (in-memory programs that coordinate tools without round-tripping) and a multi-agent beta enabling `ultra` mode (4 parallel agents by default). Prompt caching now has explicit breakpoints and a 30-minute guaranteed minimum cache lifetime; cache writes are billed at 1.25x uncached input rate. 1319 HN points.

🦜 LangChain 3w ago

[AINews] Lilian Weng summarizes 35 papers on Harness Engineering for RSI

Latent Space / AINews roundup for July 6–7, 2026. Lead item is Lilian Weng's harness engineering post (405K Twitter impressions, 3.89K likes). Additional signals: Claude Cowork launches on mobile and web as a "background agent" UX; Meta Superintelligence launches Muse Image (#2 on Image Arena) and Muse Video (#3 on Video Arena) with agentic generation loops and RL-emergent self-refinement; Liquid AI's Antidoom (FTPO) cuts doom-loop rates from 10.2%→1.4% on LFM2.5-2.6B and 22.9%→1% on Qwen3.5-4B; Tencent releases Hy3 (295B/21B active MoE) under Apache 2.0; Anthropic's J-space interpretability work draws both cross-model CKA universality findings and sharp criticism about consciousness framing.

🦜 LangChain 3w ago
⭐ Editor's Pick

Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

Deep interview with Modal CTO Akshat Bubna following the company's $355M Series C. Covers why Modal reoriented its SDK team from "developer experience" to "agent experience": agents can't read YAML or tolerate Kubernetes-style slow scaling, so the primitives—sandboxes, GPU snapshotting, elastic inference, RDMA multi-node, DeFlash speculative decoding, Auto Endpoints—need to be rebuilt from scratch for bursty agentic workloads. Highlights include: RL rollouts requiring 100,000 sandboxes, I6PN private IPv6 overlay network for networked sandboxes, and Modal Bench as an agent-graded eval for Modal's own tooling.

📝 Blog 3w ago

Grok 4.5 — SpaceXAI's coding-focused flagship on V9 1.5T foundation

SpaceXAI launched Grok 4.5 on July 8–9, built on a 1.5T-parameter V9 foundation model with supplemental training on real Cursor developer session data — the first model to benefit from SpaceX's $60B Cursor/Anysphere acquisition. Priced at $2/$6 per M tokens (over 60% cheaper than Opus 4.8), it supports configurable reasoning effort (low/medium/high) and slots into OpenAI-compatible APIs. Benchmarks: 83.3% on Terminal Bench 2.1, 64.7% on SWE Bench Pro, 29% on SWE Marathon pass@1. Uses highly asynchronous RL training across tens of thousands of NVIDIA GB300 GPUs. Available in Grok Build, Cursor (all plans), and SpaceXAI console; EU availability lagged ~1 week.

🐙 GitHub 3w ago
⭐ Editor's Pick

Pilotfish: multi-model orchestration layer for Claude Code (frontier plans, cheap models execute)

307-star GitHub project implementing a tiered orchestration layer on top of Claude Code: a frontier model (e.g., Claude Opus) handles planning and task decomposition, while cheaper models handle execution steps, with a verification guard that checks quality before accepting outputs. One-prompt install. The pattern directly mirrors what OpenAI is productizing with GPT-5.6 Sol/Terra/Luna tiering, but as an open-source wrapper for practitioners using Claude Code today.

💬 Reddit Apr 22

Qwen3 TTS is seriously underrated - I got it running locally in real-time and it's one of the most expressive open TTS models I've tried

Qwen3 TTS achieves real-time local inference with notably expressive output, integrated into the open-source Persona Engine project (ASR→LLM→TTS pipeline with lip-synced avatar). The author positions it as a meaningful step up from prior local TTS options like Sesame for latency-sensitive, fully offline deployments.

📑 arXiv Apr 22

Supplement Generation Training for Enhancing Agentic Task Performance

Supplement Generation Training (SGT) trains a small LLM to produce task-specific supplemental text prepended to the input of a larger frozen LLM, improving downstream task performance without modifying the large model. This decouples task-specific adaptation from expensive full model retraining, making it practical to update only the lightweight supplement generator as base models evolve. The approach is framed as an alternative to repeated post-training of frontier models for agentic tasks.

🟢 OpenAI Apr 22

Speeding up agentic workflows with WebSockets in the Responses API

OpenAI engineering post details how the Codex agent loop uses WebSockets in the Responses API to reduce per-request connection overhead and leverages connection-scoped caching to cut model latency in multi-turn agentic workflows. The post quantifies improvements but frames them around the specific Codex loop design. Practical reference for anyone building low-latency agents on top of the Responses API.

🤗 Hugging Face Apr 22

Qwen3.6-27B Released: 262K Context, Fits Single 24 GB GPU

Alibaba's Qwen3.6-27B dropped on r/LocalLLaMA (1,707 upvotes) with a 262,144-token context window and support for SGLang speculative decoding (NEXTN) and vLLM ≥0.19.0. The 27B dense model immediately drew community comparisons to Qwen's larger MoE variants; an r/LocalLLaMA thread questioning why the 27B outperforms the 397B MoE on some tasks reached 1,154 upvotes, surfacing discussion about MoE expert-utilization efficiency. A quantization shootout post (660 upvotes) showed Q4_K_M achieves 62.41% avg accuracy vs Q8_0's 63.67% and BF16's 63.78% across HumanEval, HellaSwag, and BFCL benchmarks—a near-negligible gap that makes Q4_K_M the practical default for 24 GB cards.

📝 Blog Apr 17

Practitioner post: Qwen3.6.35B-A3B MoE outperforms Claude Opus 4.7 locally on MacBook Pro at 20.9 GB quantized

Alibaba's Qwen3 6.35B-A3B MoE (35B total, 3B active parameters) reportedly matches or beats Claude Opus 4.7 on local tasks while fitting in 20.9 GB of quantized RAM on a MacBook Pro. If the benchmark methodology holds, this is a notable MoE-for-edge result: frontier-tier quality within consumer-RAM constraints. Practitioner claim; independent verification of benchmark methodology still needed.

💬 Reddit Apr 16
⭐ Editor's Pick

Opus 4.7 is 50% more expensive with context regression?!

User benchmarks show Claude Opus 4.7 scoring 59.2% vs Opus 4.6's 91.9% on the MRCR v2 8-needle 256K context benchmark — a sharp context retention regression. Compounding the issue, a tokenizer change reportedly causes Opus 4.7 to consume ~1.35x more tokens than Opus 4.6 and ~2x more than competing proprietary models, effectively raising costs ~50% for equivalent workloads. If the benchmark numbers hold, this is a meaningful quality-cost tradeoff moving in the wrong direction.

📝 Blog Mar 16

What Comes Next with Open Models

Lambert argues the open-closed performance gap will widen in 2026 because closed models are accumulating advantages on long-horizon, domain-specific tasks with non-public training data. Proposes a three-class taxonomy: true closed frontier, open frontier, and small specialized open models. Predicts the highest-impact open models will be narrow, fast, cheap sub-agents used as tools inside closed-model pipelines.