Experimental Methodology

This page describes the methodology behind Jacquard’s maintained experiment corpus. It explains what an experiment is in this codebase, which variables the corpus varies deliberately versus measures as outcomes, why engines are tuned before comparative experiments, how tuned configurations flow into those experiments, and the pipeline from simulator run to final report.

Readers who want to run an experiment before reading the methodology should go directly to Running Experiments. For the simulator architecture the methodology sits on, see Simulator Architecture.

What An Experiment Is

An experiment in Jacquard is a maintained scenario family combined with a parameter sweep, a set of measured outputs, and a reduction into stable summary artifacts. A scenario family fixes the qualitative regime under test. A parameter sweep varies one or more engine or policy parameters inside that regime. A reduction aggregates per-run measurements into summary tables and boundary surfaces.

The unit of record is the scenario family, not a single scenario. Comparing two single runs invites noise from seed and regime idiosyncrasy. Comparing two families across multiple seeds averages over those idiosyncrasies and produces claims about engine or policy behavior rather than individual runs.

The simulator lanes described in Simulator Architecture are how these experiments execute. The full-stack lane drives the real router and engines for behavioral claims. The model lane runs pure planner and reducer fixtures for determinism checks that do not need host wiring.

Independent And Dependent Variables

A Jacquard experiment varies deliberate dimensions and measures separate outcomes. The dimensions and outcomes are kept disjoint so any correlation between them is a claim the experiment supports rather than a tautology.

The independent variables in the maintained corpus are topology and density, delivery pressure, medium pressure, directional mismatch, topology movement, local node pressure, and workload class. See Running Experiments for the exact bands each variable takes.

The dependent variables are run outcomes. The report surfaces activation success, route presence, first materialization, first loss, recovery timing, route churn, engine handoffs, stress boundary, and first breakdown family. Per-engine cost and replay metrics extend the outcome set where applicable.

Support Contracts And Measurement Eligibility

Route-visible artifact schema 2 separates route publication from usable connected support. Engines expose read-only RoutingForwardingSnapshot values through the shared trait boundary. The simulator captures actual installed paths and corridor alternatives. For next-hop engines it follows the installed first hop and the protocol forwarding choices at subsequent hosts. Validation checks the selected realization for missing nodes or directed links, suspended/faulted links, complete loss, and forwarding loops. It never substitutes an unrelated path through the ground-truth graph. These are modeled connectivity checks, not payload delivery or complete validation of every protection and capacity requirement.

Immediately before the first scheduled environment change, each objective is classified as supported and at risk, already absent/unsupported, deferred, or unobserved. Those counts remain in the artifacts and report. Unsupported connected records are counted across the entire post-change observation window, including episodes after recovery. This is cumulative exposure, not causal attribution to one disruption or a latency measured only among successful recoveries. An earlier loss does not exclude subsequent observations. Deferred hold and Scatter’s opaque claims do not require a simultaneous path and have no connected-staleness value. Missing forwarding evidence remains unobserved, rather than becoming success. Service/gateway endpoints without a complete evaluator are also unobserved.

Counters sum across objectives and seeds. Optional means include an explicit assessed-objective count. Do not combine schema-1 graph-proxy values with schema-2 realization checks as though they measured the same quantity. Missing values stay missing in tables, charts and rankings; zero is reserved for assessed zero exposure.

Common Application Outcomes

The contact-policy model exports target delivery, delivery by a workload deadline, unserved targets at observation end, per-run P95 delivery delay, transmission count, peak retained payload bytes and modeled energy expenditure. These outcomes belong to policy profiles, not execution of every actual engine’s data plane. The current full-stack corpus has no uniform application delivery instrumentation. It must not infer delivery from send acceptance or route presence.

CustomDiffusionScenarioSpec.delivery_deadline_round specifies an absolute scenario round independent of the forwarding policy’s retention horizon. With None, the benchmark uses the midpoint of the interval from creation to observation end, held constant across policies in the same scenario. A deadline beyond observation end is unresolved unless every target has already received the message. Such unresolved runs remain in the denominator, and aggregate on-time delivery is absent if any run is unresolved. Assessed/all run counts are exported. P95 is computed by nearest rank among delivered targets, so missing deliveries must be read alongside latency.

Generated reports include a provenance JSON artifact written by the provenance exporter. It records source revision, dirty status, source snapshot hash, seeds, artifact schema and raw artifact hashes. The PDF shows the source fingerprint and run count. Reproduce numerical claims from the matching source snapshot and artifacts.

Why Engines Are Tuned First

Every in-tree engine exposes parameters that strongly affect its behavior. Examples include the BATMAN decay window, the Pathway search budget, the Babel decay window, Scatter profile thresholds, and Mercator corridor scoring.

Running a comparative experiment before locking those parameters would conflate per-engine tuning choices with per-engine capability. A comparison that pits one engine against another at arbitrary parameter settings measures both the relative engine choice and the tuning gap between them.

The tuning sweep isolates parameter effects under one engine at a time. Its output is a representative parameter set per engine, chosen as a defensible operating point in the regimes the corpus targets. Subsequent comparative experiments hold those parameters fixed, so the contrast measures engines rather than tuning.

How Tuned Configurations Feed Into Experiments

Once the tuning phase selects a representative parameter set per engine, comparative and head-to-head experiments use those fixed settings. The corpus encodes the choice in the experiment suite definitions themselves. A reader auditing a comparison can trace which engines use which tuned parameters without a second lookup.

A 3rd party running an experiment that introduces a new engine should either contribute a tuning family for that engine first, or declare the fixed operating point it uses. The declared operating point must be defensible for the regimes in question. Skipping the tuning-first sequence is permitted only when the experiment is explicitly about tuning behavior itself.

The same discipline applies to policy parameters when a comparison covers policies that expose tuning knobs. A head-to-head that varies policy as well as engine should not simultaneously vary policy tuning unless tuning is the variable of interest.

The Pipeline From Experiment To Report

A simulator run produces a per-run log plus aggregate and breakdown JSON files under the run directory. Diffusion runs add their own per-run log plus diffusion aggregate and boundary summaries. Head-to-head reductions are exported into the generated report directory. Model-lane runs add model artifacts as validation companions rather than scoring inputs.

The analysis/ Python package reads route-visible router artifacts for the maintained routing report. data.py loads them into Polars frames. scoring.py derives per-run metrics. tables.py produces CSV tables. plots.py produces vector plots. sections.py and document.py compose report sections and lay them out. report.py is the entry that assembles the router PDF.

Report outputs are stable across releases subject to explicit schema versioning. A 3rd party can rely on the artifact shape to build custom reductions or alternate reports without waiting on changes to the included pipeline.

Actual Mercator Application Delivery

The local report also runs an actual-engine delivery matrix through reference-client host bridges. This track executes payload framing, hop acknowledgements, bounded retention, automatic relays, MTU fragmentation, retries, duplicate suppression and expiry. The terminal engine emits an application delivery record only after it accepts the complete payload. The experiment verifies its bytes and destination. Successful send calls and hop acknowledgements do not count as application delivery.

Eight four-node families cover connected operation, partition recovery, successive contacts without a simultaneous path, detours, relay storage pressure, seeded packet loss, link byte limits, and permanent disconnection. Each mode uses seeds 41, 43, 47, and 53, four 64-byte messages offered in rounds 1, 5, 9, and 13 (all four in round 1 for the congested burst), a 48-round observation window, a 12-round application deadline, and the same 32-tick retention policy. Carrier capacity and loss apply to actual frames, including acknowledgements. Hosts advance in canonical node order; reported latency is in logical rounds, not radio time.

The adaptive mode is compared with two ablations in the same engine. Connected-only forwarding disables partition holding. Fixed-path forwarding retains the initial path and allows holding, while disabling path reconsideration. These controls isolate mechanisms without substituting a separate contact-policy model. They are not an application-level comparison against the other routing engines.

mercator_delivery_runs.jsonl preserves offered, accepted, delivered, on-time and unserved counts, delivered-message P95 latency, wire bytes, lost frames, peak retained payload/reassembly bytes, held rounds, resumptions, and duplicate deliveries. Rejected sends stay in the offered denominator. All workload deadlines are observed within this matrix; undelivered messages have no latency. The report exports per-run and aggregate CSVs and fingerprints the raw artifact in its provenance.

These trials provide explicit topology observations to hosts. They do not establish the cost of distributed discovery, crash-safe retention, radio energy, or protection against malicious peers. Payload buffers and deduplication are volatile and bounded. Reboot recovery and delivery guarantees outside this observation and resource envelope require separate work.