the machines are live
An essay on structure, observability, automation, delegation, and the compounding advantage of closing the machine decision loop.
In July 1969, about seven minutes before Apollo 11 touched the Moon, the lunar module’s guidance computer flashed a 1202 program alarm. Neil Armstrong asked Houston for a reading, and Charlie Duke answered within seconds: “We’re Go on that alarm.” The computer was overloaded, yet its Executive software remained capable of recovery. It restarted the machine, restored selected programs from protected restart points, and kept the guidance and display functions alive. Eagle continued toward the surface.[^1]
The remarkable part of that episode lies deeper than the computer itself. Mission Control had built a chain in which an alarm carried a defined meaning, the right context reached the right specialists, and the people with authority could act before the decision expired. Steve Bales, the guidance officer, called on Jack Garman in the back room; Garman had prepared a list of the computer alarms and recognized 1202 as a recoverable overload. The spacecraft produced a signal, the organization interpreted it, and the crew received a decision while that decision could still change the outcome. Together, those elements formed a closed decision loop. Hardware carried Eagle near the surface, but the quality of the decision loop allowed it to land.[^2]
Hard-tech advantage will come from closing the decision loop faster and more safely than competitors. The winning systems will structure evidence, observe state, infer meaning, assist judgment, automate bounded decisions, and delegate only after authority and failure modes are explicit. Better models and more telemetry matter, but neither creates operational advantage unless the organization can turn signal into action before the decision expires and turn each outcome into learning for the next cycle. The central constraint is the metabolism gap between machines that experience events in milliseconds and organizations that understand them in days. Closing that gap is the real path from data to autonomy.
Real time begins with a deadline.
We use “real time” casually to mean fast. In control engineering and real-time computing, the term is more exact: correctness depends on producing the right result within the time allowed by the system. A perfect answer delivered after the deadline can be functionally wrong. Jack Stankovic, whose work helped establish real-time computing as a distinct field, framed the problem as one of logical correctness and time correctness. The relevant deadline comes from the task and the physical process, rather than from a universal benchmark for speed.[^3]
A modern robot arm may exchange control commands and measurements at 1 kilohertz, completing a cycle every millisecond. A quadrotor configuration may update motor outputs at 400 hertz, or every 2.5 milliseconds. Those figures illustrate how the cadence follows the dynamics of the plant and the controller. A large spacecraft may use much lower control bandwidth because flexible structural modes can interact with attitude control and destabilize an aggressive loop. At interplanetary distances, the communications constraint becomes decisive: NASA gives a one-way Earth-to-Mars latency of roughly four to twenty-four minutes, which places time-critical control beyond the reach of an Earth-dependent loop.[^4]
The useful objective is sufficient lead time. A system must sense early enough, infer quickly enough, and act before it crosses into a state from which recovery is impossible or prohibitively expensive. Prediction can buy margin by moving recognition earlier, although the computation and intervention still face a deadline. This is why hard-tech systems run on nested clocks. A motor controller may operate in microseconds or milliseconds, a vehicle controller in milliseconds or seconds, a mission planner in seconds or minutes, and an engineering organization across tests, production runs, fleets, and years.
Each outer loop can move more slowly than the loop inside it because it governs a different class of consequence. It still has to move faster than the consequence it owns. A flight-control loop that arrives late loses stability, while a maintenance loop that arrives late may lose a component, a vehicle, or a season of operations. An organizational learning loop that arrives late repeats a defect across a production run or a fleet. Real time, in every case, is the amount of time the world gives you to remain effective.
Organizations have a metabolism.
Humans themselves are reasonably quick. In a calibrated study of 1,469 adults, mean simple visual reaction time was 231 milliseconds and 213 milliseconds after correction for measured hardware delay.[^5] Organizations operate on a different timescale. A test ends, an anomaly appears two hours into the data, and the evidence immediately crosses several boundaries of ownership. One team owns the sensor, another owns the vehicle configuration, a third owns the simulation, and the engineer who recognizes the signature may be asleep in another time zone.
Then the workflow begins to reveal its actual architecture. Someone exports a file, someone else writes a script, and a third person discovers that the clocks do not align. Units differ, metadata is incomplete, and the software version that matters lives in another system. A screenshot enters a group chat, a meeting appears on the calendar, and a report begins to take shape. Three days later, the organization reaches a decision about an event the machine experienced in milliseconds. The delay rarely comes from a single slow person; it emerges from the number of translations required to assemble a coherent state of the world.
I think of this difference as the metabolism gap: the distance between the rate at which a system produces signal and the rate at which the organization around it can turn that signal into action and learning. I have seen the same pattern across logistics, robotics, autonomy, navigation, telemetry, and mission-critical software. The machine changes, but the systems problem remains recognizable. Data is abundant while decisions still depend on manual reconstruction. Every handoff consumes time, drops context, and creates another place where the evidence can detach from the decision.
Early systems almost always begin this way. The team has not yet learned which distinctions matter, exceptions outnumber rules, and the workflow lives inside the heads of a few experienced people. Expert judgment is often the fastest and safest mechanism available under those conditions. The risk appears when a temporary discovery process hardens into permanent operating procedure. The work of maturation is to capture the distinctions experts repeatedly make, preserve the evidence behind them, and move each stable part of the loop into the system.
Machine intelligence advances one loop at a time.
I once described the progression as structure, accelerate, automate, and then introduce AI. The direction was useful, but acceleration is better understood as the result of maturity rather than a stage in itself. The sequence I use now is Structure, Observe, Infer, Assist, Automate, and Delegate. Learning forms the return path, carrying outcomes back into the next cycle. This ladder is a product-builder’s synthesis of ideas that have appeared for decades in human-factors research, autonomic computing, robotics, and hierarchical control.
Parasuraman, Sheridan, and Wickens separated automation into information acquisition, information analysis, decision and action selection, and action implementation. IBM’s autonomic-computing work organized self-managing systems around monitoring, analysis, planning, and execution over shared knowledge. NIST’s 4D/RCS architecture placed fast, high-resolution feedback at lower levels and longer-range planning at higher levels, while closing a feedback loop at every layer. These frameworks differ in purpose and vocabulary, but they converge on a practical point. The useful question is which part of the decision loop has become mature enough to entrust to the machine next.[^6]
Structure gives the problem a stable shape.
Structure begins by making the world explicit. The system needs durable identities for machines, components, configurations, tests, missions, sensors, requirements, states, anomalies, decisions, and owners. Time must be aligned, units normalized, provenance preserved, and authority scoped. Before this work, the organization owns files and tribal knowledge. Afterward, it owns a model of the operational world that can be queried, compared, and replayed.
Many attempts at AI transformation begin with a model while identity, state, and history remain implicit. That order produces fluent output with uncertain scope. A model may generate a plausible explanation while lacking a reliable answer to basic questions about which vehicle, calibration, operating state, software build, or requirement set the explanation concerns. Structure gives every claim an address in the world. It preserves ambiguity where ambiguity is real, while bounding the set of things that can reasonably be true.
Good structure also makes change legible. A configuration is more than a label attached to a file; it is a versioned statement about hardware, software, calibration, environment, and authority at a particular time. That statement allows two tests to be compared without quietly treating unlike systems as equivalent. It also makes later decisions auditable because the evidence can be reconstructed under the conditions that produced it. In hard tech, provenance is part of the engineering result.
Observation makes the system legible.
Once the world has a stable shape, the next task is to instrument it. The system should capture what happened, when and where it happened, and under which configuration it happened. It should preserve enough fidelity to reconstruct an event after the fact and make current state and historical state addressable through the same concepts. In control theory, observability has a precise meaning: the internal state of a system can be reconstructed from the outputs it exposes. Operational observability extends that instinct across telemetry, configuration, logs, commands, procedures, and human decisions.[^7]
A pile of measurements can still leave the system opaque. Sensor values without calibration, timestamps without clock provenance, and logs without software identity create the appearance of evidence while withholding the context needed to interpret it. Observation becomes useful when the organization can ask a question in the language of the system and retrieve a defensible answer. A task that once required three people, four tools, and two days can then become a query. That retrieval gain alone can collapse a large portion of the metabolism gap.
Inference turns output into meaning.
Observation tells us what the system emitted. Inference asks what those emissions imply about the state of the machine and the causes of change. Is a vibration expected in this operating regime, or does it indicate a developing fault? Did a temperature excursion come from the environment, the sensor, the control policy, or the hardware? Has the same signature appeared on another vehicle, and what changed immediately before it appeared?
Machine learning, probabilistic reasoning, simulation, and rules all become valuable here. The system can compare actual behavior with expected behavior, detect patterns across large histories, rank possible causes, and identify which missing evidence would most reduce uncertainty. The output of inference is a smaller and better ordered search space. Its quality should be measured through calibration, false-positive cost, sensitivity to missing data, and behavior outside the regimes represented in training. A credible inference layer knows how much uncertainty remains and makes that uncertainty visible to the people who must act.
Assistance improves judgment before it inherits authority.
The next stage is better judgment at human speed. An assisting system assembles the evidence, retrieves relevant history, explains why an event matters, and proposes what should happen next. It may draft a disposition, complete a report, generate an analysis, identify the governing procedure, or present several actions with their confidence, tradeoffs, and unresolved assumptions. This is the role of a copilot. The machine makes the human faster while the human retains decision authority.
That distinction is consequential in systems where actions can damage hardware or endanger people. Skill at generating explanations provides useful support, but it does not by itself justify permission to act. Recommendation is a product capability, while authority is a governance decision. A mature assistant should expose the evidence it used, the alternatives it considered, the conditions that would change its recommendation, and the gaps in its knowledge. It should also make rejection easy, because frictionless assent is a poor substitute for informed review.
Automation captures decisions whose ambiguity has been removed.
A decision path becomes a candidate for automation after it has been repeated, understood, bounded, and tested. A validation check can run whenever new test data arrives, a known fault can trigger a safe-mode transition, a production unit outside its envelope can be quarantined, and a software regression can block a release. Automation is captured judgment. The organization has seen the pattern often enough to specify the expected inputs, permitted actions, success criteria, and failure behavior. The best automations are boring because they are deterministic, inspectable, replayable, and easy to stop.
Reliable automation also requires attention to mechanics that disappear in a demo. Inputs need validation, repeated execution should be idempotent where possible, destructive actions need rate limits and scope limits, and the system needs a safe state when its assumptions fail. Google’s SRE literature offers a useful cautionary case in which a maintenance automation, combined with weak constraints, erased disks across a global class of machines; the remediation added sanity checks, idempotence, and mechanisms to reduce blast radius. The lesson is broader than software infrastructure. Automate the portion of a workflow whose ambiguity has been removed, and encode the limits as carefully as the happy path.[^8]
Delegation gives a bounded system a goal.
Automation follows a specified path, while delegation allows a system to choose and compose paths in pursuit of a goal. A delegated system can inspect the current state, gather missing information, form a plan, call tools, observe the result, revise the plan, and escalate at the edge of its authority. NASA defines autonomy in similar terms, as goal achievement while operating independently of external control, and distinguishes it from automation based on preplanned instructions. That distinction matters because planning changes the failure surface. Once a system can choose among actions, verification must cover the policy for choosing as well as the mechanics of each action.[^9]
In hard tech, agentic behavior should mean bounded goal pursuit with tools, memory, permissions, verification, and an explicit obligation to return control. NASA already describes a spectrum that includes advisors, advanced automation, and autonomous agents capable of adapting to changing conditions, knowledge, and constraints. Its Distributed Spacecraft Autonomy work provides a concrete example: spacecraft share what each can observe and what each prioritizes, integrate those perspectives, and select a plan for the group without waiting for step-by-step direction from Earth. The same program has demonstrated distributed planning and autonomous maintenance on the Starling spacecraft swarm. Agency becomes useful when it joins adaptive planning to an architecture that already knows what the system is, what state it occupies, and what consequences each action can create.[^10]
The progression into delegation should therefore be gradual. Let the machine retrieve before it infers, infer before it recommends, recommend before it executes known actions, and execute known actions before it plans across them. Each step should inherit identity, state, provenance, constraints, and authority from the layer beneath it. Otherwise, delegation expands uncertainty faster than it expands capability. In safety-critical systems, the decisive work lies as much in verification, bounded permissions, fallback behavior, and escalation as in planning intelligence.
Learning closes the return path.
The ladder remains incomplete until every recommendation, decision, action, and outcome becomes evidence for the next cycle. The record should preserve what the system observed, what it believed, what it recommended, who or what acted, which authority permitted the action, and what happened afterward. It should also preserve whether the diagnosis proved correct and whether the intervention produced its intended effect. When that history is absent, automation repeats without compounding. When it is present, every test can improve the next test, and every anomaly can make the next anomaly easier to diagnose.
This return path gives the organization two clocks to hold at once. The live clock exists to respond before the current decision expires. The memory clock exists to ensure that the next decision begins with what the system has already learned. A system serving only the live clock remains reactive, while a system serving only the memory clock becomes an archive. A frontier organization needs infrastructure that can support both without forcing engineers to reconstruct the bridge between them by hand.
Learning also requires more than storing the final report. The outcome must update the models, rules, priors, procedures, and thresholds that shape future decisions. A confirmed diagnosis should become a labeled case, a failed intervention should narrow the authority of the automation that proposed it, and an unexpected success should still trigger investigation into why the model was surprised. The return path is where operational experience becomes institutional capability. It is the mechanism by which one vehicle’s history begins to benefit the rest of the fleet.
The next hard-tech advantage is organizational speed.
Software spent several decades compressing its decision loops. Code became versioned, deployments became continuous, systems became observable, failures generated traces and postmortems, and repeated operational decisions became automation. Site reliability engineering turned incident records into shared learning and action items, while autonomic-computing research supplied architectures for self-configuration, self-optimization, self-healing, and self-protection. These practices made state more legible, actions more repeatable, and outcomes more reusable. Physical systems are now moving through a related transition, with a much less forgiving substrate.[^11]
A web service can often be restarted or rolled back with limited physical residue. A vehicle carries momentum, heat, pressure, fatigue, fuel, geography, weather, hardware configuration, and human safety into every decision. Its failures can alter the world before a human has time to open a dashboard. That makes the order of operations more important. Structure should precede inference, inference should precede authority, bounded automation should precede open-ended delegation, and provenance should survive every transition.
The winning companies and programs will distinguish themselves through the fraction of the decision loop they can close safely. They will operate inner loops at the speed required by physics, supervisory loops at the speed required by the mission, and learning loops fast enough to distribute experience across the organization. Better models and more telemetry will matter, but their value will depend on the system that connects evidence to authority and authority to action. The durable advantage comes from reducing the time between an event, a correct interpretation, an authorized response, and an updated body of knowledge.
A test will then produce more than a file. It will produce evidence that carries its identity, configuration, and provenance with it. That evidence will support a decision, the decision will produce an action, and the action will leave a record that improves the next decision. The result is a compounding operating system for physical work, one that learns while it acts and acts within the limits of what it knows. The machines are already live, and the systems around them now have to learn how to operate at the same tempo.
[^1]: NASA’s Apollo 11 Technical Air-to-Ground Voice Transcription records the exchange; the Apollo 11 landing transcript places the first alarm at 102:38:26; NASA’s mission overviewplaces landing at about 102:45; and the Lunar Surface Journal’s account of the program alarms explains the restart behavior of the Lunar Guidance Computer.
[^2]: NASA’s account of Jack Garman’s role during the landing describes the handoff from Steve Bales to Garman and Garman’s recognition that the overload was recoverable.
[^3]: Jennifer McManamay, “A Visionary Leader in Computer Science,” University of Virginia School of Engineering and Applied Science, January 31, 2023.
[^4]: See the Franka Control Interface documentation, the PX4 Matrice 100 documentation, NASA’s technical report on control of flexible spacecraft, and NASA’s overview of space-communications latency.
[^5]: David L. Woods et al., “Factors Influencing the Latency of Simple Reaction Time,” Frontiers in Human Neuroscience 9 (2015).
[^6]: Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens, “A Model for Types and Levels of Human Interaction with Automation,” IEEE Transactions on Systems, Man, and Cybernetics, Part A 30, no. 3 (2000); Joseph L. Hellerstein, “Self-Managing Systems: A Control Theory Foundation,” IBM Research, 2004; and James S. Albus, “4D/RCS: A Reference Model Architecture for Intelligent Unmanned Ground Vehicles,” NIST, 2002.
[^7]: Yang-Yu Liu, Jean-Jacques Slotine, and Albert-László Barabási, “Observability of Complex Systems,” Proceedings of the National Academy of Sciences 110, no. 7 (2013).
[^8]: Daniel Rogers et al., “Postmortem Culture: Learning from Failure,” in The Site Reliability Workbook.
[^9]: NASA Engineering and Safety Center, “Verification and Validation Challenges for Autonomous GNC Technology for NASA’s Next-Generation Missions,” March 3, 2022.
[^10]: NASA Ames Research Center, “Autonomous Systems & Robotics,” and “What Is NASA’s Distributed Spacecraft Autonomy?”.
[^11]: Google, “Postmortem Culture: Learning from Failure,”, and IBM Research, “Autonomic Computing: Architectural Approach and Prototype.”
the sensor substrate
The most consequential technology being built right now runs on sensor data. Figure is putting humanoid robots on factory floors. Tesla FSD is processing camera and radar telemetry across millions of miles. World Labs is building spatial intelligence from visual sensors. NVIDIA is running physics simulations that both consume and generate synthetic sensor data at industrial scale. K2 Space operates satellites that live or die by telemetry. Apple is building spatial computing on LiDAR and environmental sensing. Different companies. Different markets. Same substrate. High-fidelity data from physical systems is becoming the foundational layer of modern technology, and it is arriving at a volume and velocity that simply did not exist five years ago.
At Postmates, we created a network that was real-time and distributed, connecting millions of merchants, couriers, and customers. At Serve Robotics, I was building a system where telemetry was critical infrastructure and I watched very strong teams struggle to replay it, reason over it, learn, and train in sim-first environments at scale. Pendulum was a year of field research on neural net-based sensor fusion running on edge devices under electronic warfare conditions, where the signals themselves were contested and the models had to reason through degraded, adversarial inputs in real time. At Anduril, the stakes tightened to the centimeter in environments where a misread signal can cost a mission.
The pattern became obvious. The machines were getting dramatically more capable. The infrastructure to learn from what they generate was not keeping pace. What makes this moment different is that AI can now reason over physical system behavior in real time, on device. Not in a dashboard hours later. Not in a batch job someone runs on Thursday. At the point of operation, while the machine is running. That is a new paradigm. The gap between sensing and understanding starts to collapse. An anomaly that used to require a senior engineer to notice, investigate, and contextualize can now surface as structured evidence the moment it happens. I have seen the failure mode this replaces at every hardware company I have worked at. Physical systems fail because humans cannot process signal fast enough, cannot preserve institutional knowledge long enough, and cannot be everywhere at once across increasingly complex machines.
The senior validation engineer who knows why a specific anomaly matters on a specific vehicle is often the scarcest resource in the building. When that person leaves, the knowledge leaves with them. Real-time AI on sensor data does not replace those people. It multiplies the scarcest expertise in the industry.
Sift closes $42M Series B
Sift turns telemetry into ground truth for how machines interact with the physical world. What stands out in the announcement is not just the financing, but the specificity of the operating need: high-frequency sensor streams, multimodal telemetry, long-lived historical test data, and an infrastructure layer that makes all of it queryable and usable by both engineers and models. That is the missing layer for mission-critical hardware.
Sensor intelligence has been the consistent throughline in my career, increasing in importance, complexity, and consequence at every step. I watched sensor data quietly become mission-critical infrastructure for autonomous systems. More sensors, higher data rates, more complexity, and a tighter coupling between data and decision making.
What is different now is that this pattern is no longer isolated. It is becoming the default across robotics, autonomy, aerospace, manufacturing, and simulation. Physical AI systems are built on continuous perception and decision loops, and those loops are only as strong as the quality, context, and speed of the underlying sensor intelligence. This is where the leverage is, and it is pushing to the edge.
The bottleneck is converting raw data into intelligence that can be interrogated and acted on by humans and machines. Physical AI systems need infrastructure that turns sensor data into trusted decisions.
Cause Strife. Chew Glass. Tokenize Chaos.
My late mentor taught me the leverage of applying these concepts across company divisions to ship better products from the inside out. When strong leads embrace this mode of building, it becomes a team sport like the ones I grew up playing. Convert noise into signal to surface tension and catalyze the system to evolve from the top down. Identify what’s not working and fix it. Turn volatility into structure through insight and data. Force the right conversations. It’s not about constant harmony. The process is deliberate and incremental, like building strength through resistance.
Three quarters focused on 40% efficiency lift and millions saved in data spend. I don’t view alignment as the absence of tension but as the product of processing it as a team.
At Postmates I grew to see chaos as simply unstructured data that is often emotionally charged, tokenize it, and combine strongly held opinions with data. Every point of friction is a packet of truth about how a system behaves under stress. Structure it and extract the insights hidden in the noise. Pattern analysis and root cause analysis going upstream creating a measurable system. The faster you identify tension and loop it back into the system, the faster the system evolves. Technical support, design reviews, ops syncs, user research, and strategic planning turn hidden friction into shared understanding. chaos is telemetry. Big fan of vertical integration and platforms for it. Thanks James
Multimodal AI research 2023
Research project on a personal AI assistant with a constantly evolving prompt library based on engagement across platforms like Reddit. While the community doesn’t directly interact, they benefit from the incredible new network effects of community without the social. The project utilizes iOS and SMS using Xcode, Twilio, Figma, and the GPT3.5 API.
spacetime
/GET EarthThe next generation of machines are computing across multiple dimensions, processing fused spatial, temporal, and ephemeral data in real-time at the edge. These systems sense, localize, navigate, and compute without relying on any single network or signal.
Building geospatial infrastructure connecting space to the edge has demonstrated this across UAVs, UUVs, USVs, and HUDs. The potential for a vertically connected ecosystem linking intelligence from satellites to cloud to edge is clear, no GPS required.