Engineering Fundamentals

Summary

A cluster of links on the bedrock of software engineering - the problems that don’t go away regardless of AI tooling. Idempotency, build caching, code reading, formal modeling, and now: structural correctness as a design philosophy for AI-generated code. W21 added a significant new thread via the backpressure piece - the argument that invariants belong in the substrate (types, compilers, tests, proof checkers), not in prompts. This directly connects to the formal methods work (TLA+) and distributed systems testing methodology. The theme emerging: the engineering fundamentals that make human-written systems reliable are the same ones that keep AI-generated systems correct. W22 adds two more pieces: platform capture of communication channels (push notifications are no longer developer-controlled delivery - Apple and Google have inserted AI intermediaries that summarize, reorder, and rewrite before delivery) and database-native durable workflows (DBOS argues that external orchestrators like Temporal are a needless abstraction - Postgres itself can coordinate workflows via standard locking and integrity constraints, making scalability and availability DB engineering problems with decades of prior art). W23 extends the durable workflow thread further: SQLite as the lightest-weight tier for bursty, agent-centric workloads (Litestream replication to S3, no separate DB service needed), a hands-on build-it-from-scratch workflow engine guide (Go + Postgres, Hatchet style), and a historical counterpoint in CASTOR - CERN’s decades-old hierarchical storage system for physics data, proof that purpose-built distributed storage at exabyte scale is a solved engineering problem when the domain is well-understood. W24 adds two new threads: Linear’s local-first architecture as a masterclass in inverting the traditional CRUD model (browser-as-database, async server sync, no spinners); and North Star values (Loris Cro) as a reminder that engineering fundamentals are always in service of a goal — user utility — not ends in themselves. W26 brings a cluster of fundamentals spanning multiple registers: ast.walk optimization (Reflex’s 220x speedup through generator-to-list replacement and C extension — a case study in hot-path profiling where the answer was simpler than the symptom suggested); the Bayer PRINCE architecture retrospective (harness and context engineering patterns for production agentic RAG, framed now that the vocabulary exists); a visual SSH tunnels reference (local and remote port forwarding — foundational networking primitives that outdate Cloud Native tools by decades); the Aalto sync taxonomy (systematic trade-off analysis of distributed sync architectures — theoretical complement to the practical durable workflow entries from W22/W23); a design document writing guide (the doc’s primary value is the thinking it forces, not the artifact it produces); and Data Compression Explained (Mahoney’s canonical reference from entropy coding through neural methods).

W28 adds a classic fundamentals cluster: software speed as a primary quality, B-tree indexes as database literacy, Apache Shiro 3.0 as security-framework maintenance, DSA fundamentals before LeetCode, and a surprising DIY bioinformatics workflow showing that rigorous data/science pipelines are increasingly accessible outside institutions.

W29 links three scales of engineering verification. Ploy demonstrates that eval harnesses can accidentally measure similarity to an incumbent rather than task success. Park argues that national mathematical capacity—the human ability to verify and challenge machine reasoning—is infrastructure, and proposes exposing consequential claims in machine-checkable form. Cloudflare’s nameserver history is a compact systems-design example: human-readable names hid a combinatorial verification code that resolved concurrent signups while discouraging invalid sequential guesses.

Key Sources

W29 2026 · 11-Jul-26 → 17-Jul-26

• Migrating a production AI agent to GPT-5.6 — production eval case study showing why test infrastructure must distinguish harness failures from model failures: tool-call budgets, batched reads, and implicit score thresholds initially penalized valid challenger behavior; trace triage exposed the hidden assumptions before the model comparison was trusted (engineering-blog · #engineering-fundamentals, #evals, #testing, #llm-infrastructure, #observability)
• Automation Without Understanding — Jun-Yong Park argues that theorem-producing AI and the erosion of US mathematics education/research funding form a strategic mismatch; mathematical capacity is intergenerational verification infrastructure, not a byproduct of theorem output, and consequential AI reasoning should expose decision-critical claims in formal, machine-checkable form (paper · #mathematics, #formal-verification, #ai-reasoning, #education, #strategic-capacity)
• What’s the story behind the names of CloudFlare’s name servers? — Cloudflare’s friendly nameserver pairs began as a conflict-resolution code: 51 male and 50 female names created 2,550 combinations that identify the authoritative signup when two users claim the same domain, while non-sequential names prevent users from inventing extra dnsN servers; each name still maps to the shared anycast pool (engineering-blog · #dns, #distributed-systems, #cloudflare, #system-design, #verification)

W28 2026 · 04-Jul-26 → 10-Jul-26

• Fast Software, the Best Software — Craig Mod: fastness is a fundamental product quality, not a polish layer; low-latency tools feel trustworthy and become part of users’ lives, while slow software accumulates reluctance even when functionally complete (opinion · #engineering-fundamentals, #performance, #ux, #software-quality, #product-design)
• How to sequence your own DNA at home — Bradley Woolf: end-to-end account of at-home genome sequencing using Oxford Nanopore MinION, from cheek swab collection through preparation, sequencing, and analysis with tools like VEP, ClinVar, gnomAD, and PharmCAT; a hands-on example of complex scientific workflows becoming technically accessible to individuals (engineering-blog · #bioinformatics, #diy-science, #genomics, #workflow, #data-analysis)
• Understanding B-Tree Indexes in PostgreSQL: Part 1 — Devli: practical PostgreSQL internals guide covering index types, B-tree logical schema, heap/page storage recap, pageinspect inspection, and why understanding index mechanics matters for query optimization (engineering-blog · #postgresql, #databases, #indexes, #btree, #performance)
• Apache Shiro 3.0.0 available — Apache Shiro major release: modern Java/Jakarta baseline, improved thread-safety, immutable principals, JDK Scoped Values for Subject/SecurityManager, and hardened default web filters; framework evolution as a fundamentals/security maintenance case study (news · #java, #security, #frameworks, #apache-shiro, #backend)
• Fernando Franco: 12 DSA fundamentals before LeetCode — LinkedIn post arguing engineers should master fundamentals before grinding LeetCode: recursion, time complexity, arrays, linked lists, hash tables, BSTs, priority queues/heaps, stacks/queues, graphs, sorting, dynamic programming, and problem-pattern recognition (other · #algorithms, #dsa, #interview-prep, #learning, #software-career)

W26 2026 · 20-Jun-26 → 26-Jun-26

• Making ast.walk 220x Faster — Reflex.dev: built a custom Python linter for AI-generated code; reflex compile stops at first error but AI generates many at once; optimized ast.walk from generator-based (285ns/node — yield suspend/resume overhead in hot path) to flat list accumulation + C extension — 220x speedup; profiling showed the bottleneck was simpler than expected: generators waste cycles when consuming the full list anyway (engineering-blog · #python, #performance, #ast, #linting, #optimization)
• Building Reliable Agentic AI Systems — Bayer PRINCE on martinfowler.com: retrospective on building an agentic RAG system for preclinical drug discovery before “context engineering” and “harness engineering” had names; every design decision maps to those concepts — what context each model received, and the scaffolding around models (orchestration, tool boundaries, state persistence, retries, fallbacks, validation, reflection loops, observability, human review); makes abstract principles concrete through a production case study in regulated science (engineering-blog · #harness-engineering, #context-engineering, #agentic-rag, #distributed-systems, #engineering-fundamentals)
• A Practical Guide to SSH Tunnels: Local and Remote Port Forwarding — iximiuz Labs: SSH port forwarding explained visually — local and remote forwarding, sshd configuration requirements, memory device for -L/-R flags; “learning a couple of SSH tricks is more profitable in the long run than mastering a dozen Cloud Native tools destined to be deprecated next quarter”; an interactive tutorial on a foundational networking primitive unchanged for decades (engineering-blog · #ssh, #networking, #linux, #port-forwarding, #sysadmin)
• The Limits of Generalized Sync: A Taxonomy of Architectures, Trade-offs, and Decision Factors — Aalto University research (PDF): systematic taxonomy of synchronization architectures for distributed systems; trade-off analysis and decision factors for choosing between sync strategies; scholarly theoretical complement to the practical durable workflow entries (DBOS, obeli.sk SQLite, hatchet-dev from W22/W23) — adds formal framework for reasoning about when and why different sync approaches break down (paper · #distributed-systems, #synchronization, #architecture, #database, #consistency)
• How to Write an Effective Software Design Document — Refactoring English: a design doc forces thinking through hard decisions before wasting time on wrong implementations; standard components (title, metadata, objective, background, goals, non-goals, implementation approaches, security/privacy); from Google, Microsoft, and personal company experience; the primary value is the thinking it forces — the artifact is secondary; coordinate design decisions across teams before any code exists (engineering-blog · #software-design, #documentation, #technical-writing, #engineering-fundamentals)
• Data Compression Explained — Matt Mahoney’s comprehensive reference on data compression theory and practice: entropy coding (Huffman, arithmetic), LZ family, BWT, PPM, neural methods; a canonical field reference that remains useful for decades; noted as foundational computer science background (other · #data-compression, #algorithms, #information-theory, #computer-science)

W24 2026 · 06-Jun-26 → 12-Jun-26

• How’s Linear so fast? A technical breakdown - Dennis Brotzky’s deep-dive: Linear inverts the traditional CRUD loop by making IndexedDB the browser’s primary database; mutations apply locally first, sync asynchronously via WebSocket delta broadcasts; UI re-renders off in-memory state (MobX observables) - no spinners, no network wait; the core principle: “hide all network requests from the user”; local-first architecture is the key, not backend optimization (engineering-blog · #engineering-fundamentals, #performance, #local-first, #frontend-architecture, #sync-engine)
• My Software North Star - Loris Cro’s values hierarchy: user utility (“software you can love”) > correctness > maintainability and efficiency; memory safety, abstractions, and DX only matter in service of user utility - a foundational articulation of what engineering fundamentals are actually for (opinion · #engineering-fundamentals, #software-quality, #software-career, #values)

W23 2026 · 30-May-26 → 05-Jun-26

• SQLite is All You Need for Durable Workflows - Obelisk blog: extends the DBOS/Postgres durable workflow argument downward; for bursty, experimental, agent-centric workloads, SQLite + Litestream (async replication to S3) is the right tier - no network hop, no separate control plane, each agent or tenant gets a self-contained SQLite file; Postgres remains the right choice for high-availability / high fan-out; the key insight: match infrastructure to the durability model you actually need on day one (engineering-blog · #durable-workflows, #sqlite, #distributed-systems, #orchestration, #ai-agents)
• hatchet-dev/durable-execution-the-hard-way - “Kubernetes the hard way” for workflow engines: builds a fully working durable execution engine from scratch in Go + Postgres; teaches how Hatchet and Temporal work internally - workers dequeue from a workflows table, checkpoint step outputs, detect duplicates; each lesson is self-contained from queue → checkpoint → full replay-capable engine; practical complement to the DBOS and obeli.sk arguments (repository · #durable-workflows, #go, #postgres, #distributed-systems, #learning)
• CASTOR: CERN Advanced STORage Manager - CERN’s hierarchical storage management system (disk + tape) for high-energy physics data archiving; component architecture supporting XROOT and GridFTP; operated from 1998 to 2022, superseded by CTA (CERN Tape Archive); a compelling historical case study in purpose-built distributed storage at science-grade scale, predating cloud-first orthodoxy by decades (tool · #distributed-systems, #storage, #cern, #hpc, #archiving)

W22 2026 · 23-May-26 → 29-May-26

• Postgres-backed Durable Workflow Execution | DBOS - external orchestrators (Temporal, Airflow, Step Functions) are structurally overcomplicated; Postgres itself can serve as orchestrator - workers dequeue from a workflows table, checkpoint step outputs, detect duplicates via DB integrity constraints; hard problems like scalability and availability become well-understood DB engineering problems; eliminates a separate orchestration tier (engineering-blog · #durable-workflows, #postgres, #distributed-systems, #orchestration)
• What Apple and Google are doing to your push notifications - deep history of how APNs and FCM became active intermediaries: from permissive delivery (2009-2017) to AI-powered on-device summarization, reordering, and rewriting; developers have lost direct control over their own communication channel - platform intermediation is now the default, not the exception (engineering-blog · #push-notifications, #platform-control, #mobile, #distributed-systems)
• What I’m Hearing About Cognitive Debt (So Far) - Margaret-Anne Storey synthesizes practitioner responses to her cognitive debt concept: the gap between a system’s evolving structure and the team’s shared understanding of how and why it works; key distinction - technical debt lives in code, cognitive debt lives in people; manifests as confidence loss, heavier review burden, debugging friction, slower onboarding, fatigue; Fowler agrees it must be repaid; repayment means restoring the distributed theory of the system across people, docs, tests, tooling (opinion · #cognitive-debt, #ai-coding, #code-comprehension, #developer-wellbeing)
• Staged publishing and new install-time controls for npm - npm CLI 11.15.0: staged publishing now GA - packages go to a queue requiring maintainer approval before becoming installable; new --allow-file / --allow-remote / --allow-directory install source flags; direct supply-chain security improvements for the npm ecosystem (news · #npm, #supply-chain-security, #package-management)
• How Shamir’s Secret Sharing Works - Ente’s visual explainer: threshold secret sharing using polynomial interpolation; two points fix a line, so two shares recover the secret; any k-of-n shares reconstruct; fewer than k shares reveal literally nothing (not just “hard to crack”); Shamir (the S in RSA) published this in 1979; excellent primer for practical use in key management, recovery flows, and distributed trust systems (engineering-blog · #cryptography, #secret-sharing, #security, #distributed-trust)
• How Shamir’s Secret Sharing Works - Hacker News - HN discussion; community adds real-world applications: hardware security modules, cold wallet recovery, team key custody; discussion of finite field arithmetic for exact (not floating-point) computation (hn-thread · #cryptography, #security)
• White Rabbit - CERN/OHWR open hardware project: sub-nanosecond accuracy and picosecond precision synchronization for large distributed systems over Ethernet (gigabit, deterministic delivery); connects thousands of nodes over typical 10km distances; fully open hardware, firmware, and software; commercially produced multi-vendor; a remarkable case of open hardware enabling science-grade distributed systems (tool · #open-hardware, #distributed-systems, #timing, #networking, #cern)
• A Circuit Prompt Programming Language (CPPL) - paper: compiler-mediated framework that turns LLM-assisted hardware generation into a statically checkable problem; extends the structural correctness principle into RTL/hardware design; another data point that structural gates (compilers, type systems) + LLMs is the correct architecture (paper · #formal-methods, #hardware-design, #llm-coding, #structural-correctness)

W21 2026 · 16-May-26 → 22-May-26

• Intro to TLA+ for the LLM Era - practical guide: LLMs now write TLA+ specs, removing the syntax barrier; the hard parts - understanding your system and defining temporal correctness - remain human work; includes beans puzzle walkthrough and Claude prompt template
• shenli/distributed-system-testing - SKILL.md skills for claim-driven distributed/stateful system testing; covers partial partitions, crash-recovery, idempotency under replay, concurrency; produces structured test plan + blame classification (SUT/harness/checker/environment); emphasizes coverage adequacy as a deliverable
• Structural Backpressure Beats Smarter Agents - key primitive: move invariants out of prompts into the substrate; structural gates (compiler, type checker, test runner, linter, proof checker) produce concrete answers the model must satisfy; the refusal is the point
• Remote Cache CDC: Reusing Bytes - BuildBuddy’s Content-Defined Chunking in Bazel remote cache; 40% less data uploaded, 40% smaller disk cache; moves build caching from “skip actions” toward “skip bytes”; merged into Bazel 8.7/9.1+
• Content-defined chunking added to Bazel - HN thread on the Bazel CDC merge; community discussion on implications for large monorepos
• Reading code instead of writing code: The underestimated senior discipline - code reading is the defining senior skill; LLMs tip the balance further: they generate code faster than humans can reason about it

W20 2026 · 09-May-26 → 15-May-26

• Can LLMs model real-world systems in TLA+? - ACM SIGOPS research on whether LLMs can produce useful TLA+ specs for distributed systems; explores the formal verification + language model intersection; empirical evaluation of correctness
• Idempotency Is Easy Until the Second Request Is Different - the “happy path” idempotency is not the hard part; the hard part: same key, different content, concurrent retry, partial side-effect; strong take: same key + different canonical command should be a hard error

W19 2026 · 02-May-26 → 08-May-26

• lsferreira42/nfsdiag - NFS diagnostic/doctor application; pragmatic tool for NFS troubleshooting

Open Questions / Tensions

• Verification capacity is infrastructure: Park’s argument generalizes beyond mathematics. If AI increases production while institutions stop training people who can audit the output, machine-checkable claims help but do not replace the human capacity to choose the right definitions, assumptions, and proof obligations.

• Evals must test outcomes, not behavioral resemblance: Ploy’s challenger initially looked worse because the harness encoded Opus’s sequential calling style. This is the testing analogue of overfitting: a suite can be internally consistent yet measure the wrong invariant.

• Fundamentals are resurfacing as AI raises output volume: DSA basics, B-tree internals, and framework security defaults are not displaced by AI tools. They become the shared substrate engineers need to evaluate generated work, troubleshoot performance, and understand systems below the abstraction layer.

• LLMs and formal reasoning: The ACM SIGOPS TLA+ piece and the emptysqua.re intro raise complementary questions - can LLMs produce syntactically valid TLA+ (yes) vs semantically correct TLA+ for real distributed systems (unclear). The backpressure piece implicitly argues that only machine-checkable structural gates close this gap reliably.

• Structural backpressure scope: Brooks’ argument is compelling for a specific class of invariants (access control, type safety). It’s less clear how it applies to emergent properties (performance, scalability, user experience) that don’t reduce to machine-checkable predicates.

• Distributed systems testing as an agent skill: shenli’s approach - encapsulating testing methodology in SKILL.md files for agents - is a new pattern. It suggests agents can carry methodology, not just execute code. The quality of the methodology matters enormously.

• Build caching and AI codegen: CDC in Bazel is motivated by “small changes, large artifacts.” AI-generated code may change artifacts in ways that disrupt chunking locality - an unexamined downstream effect.

• Code reading as a teachable skill: Connecting to Osmani’s “don’t outsource learning” - if AI writes more and reading remains undervalued, engineers may become increasingly unable to audit their own codebases.

• Postgres as orchestrator vs. purpose-built orchestration: The DBOS argument is elegant but has trade-offs - a Postgres-backed system may hit limits under very high workflow fan-out or when workflows need advanced scheduling semantics. The question is whether Postgres ergonomics are better than Temporal’s complexity - the answer likely depends on team maturity and existing DB expertise.

• SQLite vs. Postgres for agent workloads: The obeli.sk post adds a useful third tier below Postgres: SQLite is fine for bursty, single-tenant, recoverable-on-failure agent workflows. The trade-off is async replication risk (Litestream may miss the last few writes on a crash). For agents where replay is acceptable but loss isn’t, the durability model needs to be an explicit design choice, not a default.

• Platform capture of communication channels: The push notification piece documents a structural shift: Apple and Google now insert AI processing between developer intent and user experience. This pattern (platform as active intermediary) applies beyond push - email (Google/Yahoo filtering), search ranking, app store review. Engineers building on platform channels should assume the platform may rewrite, summarize, or suppress without notice.

• Domain-specific infrastructure vs. cloud primitives: CASTOR at CERN is a reminder that building purpose-specific distributed infrastructure from scratch is viable when the domain constraints are stable and the scale demands it. The cloud-first defaults of 2024-2026 make this seem anachronistic - but CERN’s hierarchical storage problem (petabytes to tape, fast disk cache for hot data) still has no off-the-shelf answer that beats a custom system.

• Local-first as the performance primitive: Linear’s architecture (browser DB, async sync) demonstrates that the traditional CRUD model is itself the bottleneck, not the network or the database. This is a more fundamental insight than optimization — it’s an architectural inversion. The question for other web apps: what would change if the browser were the source of truth and the server were the replication target?

• Design docs and AI-generated code: The Refactoring English design doc guide predates AI-generated code, but is more valuable in a world where AI can write the implementation but lacks the problem context a design doc captures. If implementation is cheap, the bottleneck shifts to correctly specifying what to implement — which is exactly what a design doc does. Design-first may become more critical, not less, as implementation cost approaches zero.

• Sync taxonomy as a decision framework for agent workloads: The Aalto paper provides theoretical grounding for the practical durable workflow debates (DBOS, obeli.sk, hatchet). As agents need durable, recoverable execution environments, the choice of sync architecture is a core reliability design decision. The W22/W23 practical guidance now has a formal taxonomy to reason against.

• Hot-path optimization as a second-order AI cost: Reflex’s ast.walk story suggests a pattern that will repeat across the ecosystem: tools designed for human-pace code generation will hit performance walls as AI code generation scales. The optimization work is not glamorous, but the cumulative cost of slow secondary tooling (linters, formatters, type checkers) at AI output speeds is non-trivial.