GTFS-Madrid

Question: when one row of Madrid’s transit feed changes, how many values does .me recompute, and does that grow with the city?

Answer (measured, this.me 4.2.1): no. A write recomputed k = 1 value at the median, 18 at p95 and 27 at most, at scale 1 (2,364 stop_times) and at scale 10 (23,640); the scale-100 run on this.me 4.1.0 (236,400) gave the same 1 / 18 / 27. What grows with the data is a delete: 8.9 ms p50 at scale 1, 119 ms at scale 10.

The feed is GTFS-Madrid-Bench, changed with the benchmark’s own update packages. k is explain(probe).meta.k (me.explain(path)), the number of derived values one write recomputes; Extreme Fan-Out explains the wave it counts.

Live: AI decides, .me computes → runs on the same scale-1 / scale-10 feed in your browser, also on this.me 4.2.1. Its k is much larger, and that is expected: there one write changes policy.factor, a value every headway reads, so the wave is the whole headway subtree (every window, trip and route), the fan-out case of Extreme Fan-Out. Here each write is one feed row, so the wave is that row’s foreign-key neighbourhood. Its timings come from a browser Web Worker, these from Node.

Run: this.me 4.2.1 from npm (dist/me.es.js sha256 0bd2cc60d15dfc9ff8942d390c075bbc22b4ae3a24d3cbfe581511ea84478757), Node v22.23.3, shared KVM VM (Intel Xeon, 8 vCPU, 16 GiB), Linux 6.12, 2026-10-13 (UTC). History rows: this.me 4.2.0 (sha256 8cc94d5273b05728713e7c06bcba6ab0d88c85d2a748dc885e7bca745605a61a, same VM, 2026-10-10) and this.me 4.1.0. Every results file records its own run line.

Data and workload

The Van Assche et al. CHANGE packages (Zenodo 10.5281/zenodo.14038823): GTFS-Scale-1-CHANGE.tar.xz, GTFS-Scale-10-CHANGE.tar.xz, GTFS-Scale-100-CHANGE.tar.xz, generated with VIG from GTFS-Madrid-Bench (sizes and what to extract: Madrid GTFS dataset). Each run loads base, then applies every CREATE / UPDATE / DELETE from seed0 and measures each write. Write time is the write itself, including its propagation wave.

How GTFS maps onto .me

Rows become paths (gtfs.trips.T1.serviceId, written as me.gtfs.trips.T1.serviceId(value), see me.whatever(what)) and foreign keys become dependencies:

• services.S.active ← calendar.txt days and dates (adapter.ts L149)
• trips.T.live ← services.S.active, through service_id in trips.txt (L160)
• trips.T.validRoute ← routes.R.active, through route_id in routes.txt (L162)
• stopTimes.*.routeId ← trips.T.routeId, through trip_id in stop_times.txt (L174)
• one count per parent (tripCount, stopTimeCount), a sum over membership slots (L270–294)

k counts the derived values reached from the written row through those edges. This is not an RML/RDF materializer: it measures how a change propagates after the graph exists, not how a knowledge graph is built.

Setup

Node ≥ 22.6 (it runs the .ts files directly). All commands run from Typescript/. Full details in the GTFS-Madrid README.

git clone https://github.com/neurons-me/.me.git && cd .me/Typescript
npm install
npm pack this.me@4.2.1 && tar xzf this.me-4.2.1.tgz
rm -rf dist && mv package/dist dist
sha256sum dist/me.es.js

sha256sum should print 0bd2cc60…8478757 (full value above). Put the Zenodo CHANGE CSVs in tests/Benchmarks/GTFS-Madrid/data/scale{1,10}/{base,seed0}/. The scale-10 sweep takes about 1 minute, the skew sweep about 16 minutes.

1. How many values a write touches as the data grows

k is identical on 4.1.0, 4.2.0 and 4.2.1 for every write. The write p95 at scale 10 is the deletes (run 3). Scale 100 was not re-run on 4.2.1: the archive is 1.2 GB and, with each delete scanning the store (run 3), the sweep’s roughly 2,400 deletes would take on the order of an hour on this VM. It stays as 4.1.0 history.

• Script: sweep-seed0.ts
• Results: results/v4.2.1/explicit/ · history: v4.2.0, 4.1.0 scale 100

ME_VERSION=4.2.1 node tests/Benchmarks/GTFS-Madrid/sweep-seed0.ts --scale=1
ME_VERSION=4.2.1 node tests/Benchmarks/GTFS-Madrid/sweep-seed0.ts --scale=10

2. What a write costs when one parent is huge

VIG grows the data uniformly (every trip keeps about 18 stop_times), while real feeds are uneven. So the skew run reassigns trips.service_id and stop_times.trip_id with Zipf (a = 0 … 2) and hub profiles at scale 10, same row counts, 3 seeds each (22 runs), and replays the same workload plus a census of the widest parents.

All values this.me 4.2.1; in brackets this.me 4.2.0. Zipf 1 / 2: exponent a = 1 / 2; hub 50%: one service and one trip own half of their children. k is identical on both kernels.

• k follows the parent you touch. The median write is still k = 1; the tail is the widest parent (k max = its stop_times).
• UPDATE cost is linear in k: about 0.045 ms + 35 µs per recomputed value (r = 0.99, 1,100 writes; 4.2.0: 52 µs). 4.2.1 compiles each formula once per kernel instead of on every recompute (CHANGELOG). The largest wave (a service with 801 trips, k = 801) took 30 ms at worst (4.2.0: 52 ms), about 120× under a ~3.7 s rebuild. That is the worst case; for the ordinary seed0 UPDATEs, rebuild ÷ apply p50 is 7,900–28,000×.

• A CREATE on a giant parent costs more than k = 1 suggests. k is 1 (the parent’s count), but the count is a sum with one term per member, so re-evaluating it costs the parent’s width: 0.02 ms at the median trip, 95 ms at 14,378 members (4.2.0: 151 ms).

• Scripts: generate-skew.ts (makes the data), sweep-skew.ts (runs it), report-skew.ts (writes the tables)
• Results: results/skew/v4.2.1/ · history: v4.2.0

node tests/Benchmarks/GTFS-Madrid/skew/generate-skew.ts --scale=10
ME_VERSION=4.2.1 node --expose-gc tests/Benchmarks/GTFS-Madrid/skew/sweep-skew.ts --scale=10
node tests/Benchmarks/GTFS-Madrid/skew/report-skew.ts --scale=10

3. What a delete costs (known weakness)

UPDATE cost follows k; DELETE cost does not. Each delete, me["-"], scans the store: about 110–117 ms p50 at scale 10 whatever k is (run 2’s table: the same from k max 27 to 14,378), against 8.9 ms at scale 1. So a delete’s cost depends on n, the size of the data, not on k. On top of that it adds about 49 µs per k (4.2.0: 67 µs). Worst measured on 4.2.1: 842 ms to delete the 14,378-stop_time hub trip (4.2.0: 1.2 s), still under the rebuild.

Not fixed. The scan is still there on 4.2.1; deletes are about 20% faster than on 4.2.0 because each recomputed formula is cheaper, not because the scan changed. The likely cause: since 4.1.0, removing a subtree scans every subscribed path (Object.keys(refSubscribers)) to find what read the removed paths, and the same removal also walks every derivation and index key. Each of those is a full pass over the store, once per delete.

Deletes are a phase of the same skew run, no separate command.

• Scripts and command: the same as run 2
• Results: results/skew/v4.2.1/

4. The [] aggregate as the count

this.me 4.2.0 added collection aggregates (stopTimeMember[].on). Used for the counts on this.me 4.2.1, k is the same, but every CREATE re-scans the collection: 3.3 ms p50 at scale 1 and 42 ms at scale 10 (4.2.0: 3.5 and 50), instead of 0.20 ms with explicit sums, and loading scale 10 takes 63 s (4.2.0: 71 s) instead of 4.2 s. So it does not fix the giant-parent CREATE (run 2) yet, and the benchmark keeps explicit sums.

• Script: sweep-seed0.ts with GTFS_AGG=bracket
• Results: results/v4.2.1/bracket/ · history: v4.2.0

GTFS_AGG=bracket ME_VERSION=4.2.1 node tests/Benchmarks/GTFS-Madrid/sweep-seed0.ts --scale=1
GTFS_AGG=bracket ME_VERSION=4.2.1 node tests/Benchmarks/GTFS-Madrid/sweep-seed0.ts --scale=10

Across versions

One run per version on the same shared VM, on different days (4.1.0 skew 2026-10-07; 4.1.0 and 4.2.0 sweeps 2026-10-10; 4.2.1 2026-10-13): differences under ~10% (the load column) are within noise. k is the same on all three. Changelog: CHANGELOG.md.

Known limits

• explain() grows with k: about 1.9 KB + 56 bytes per recomputed value (unchanged in 4.2.1). On the trip with 14,378 stop_times (Zipf 2) it returns about 1.6 MB. There is no pagination or summary mode yet; it is an open item.
• Deletes scan the store (run 3): they depend on n, not k. Open.
• Max columns are noisy (GC pauses); p50 and bucket medians are the robust numbers.
• 4.2.0 and 4.2.1 ran on different days on the same shared VM; the speedups above are measured, but small ones (under ~10%) are within day-to-day noise.

More

Docsme.whatever(what)How to read and write a path: me.a.b(value) and me("a.b"). Docsme.explain(path)Why a value is what it is: inputs and the last wave, where k comes from. DocsExtreme Fan-OutThe concept behind k: one write, 100,000 dependents, every wave traceable with explain(). DemoAI decides, .me computesA policy decides headways on this data; one write, the wave replayed, explain() back to the decision. DemoGTFS Madrid universeA small subset live in the browser: click a node for explain(), mutate to see k. DemoMadrid faresThe same city as a human story: fares, abono, Sol → Atocha. DatasetGTFS on mlearning.studioThe feed format, the Madrid dataset, GTFS-Madrid-Bench links. SourceGTFS-Madrid READMEAdapter, workload, every table and results file, 4.2.1 with 4.2.0 and 4.1.0 history. DatasetMadrid GTFSZenodo CHANGE archives and sizes, what to extract, citations.