A SQL engine split across services, built so every stage is legible while it runs — the parse tree, the join-order DP table, the plan fragments fanning out to workers, and inside each worker the B+Tree descent, the buffer-pool frames, the lock waits and the WAL records.
Four algorithms carry this project. Each one is drawn on screen as it runs, which is only possible because span emission was built into layer 1 rather than retrofitted after.
The single public entry. Accepts a statement, opens a trace, and streams span events back to the browser while the query is still running — so the UI renders progress rather than a final result blob.
Lexer, recursive-descent parser, logical plan, then the physical plan and its fragment placement. The DP table is emitted as trace data, which is what makes the optimiser view exact rather than illustrative.
Where the database-internals work lives: heap and columnar page layout, a buffer pool with a real replacement policy, B+Tree indexes, the lock manager, and a Volcano iterator that later goes vectorised.
Its own gRPC service backed by Cloud SQL. Bad statistics are the usual reason a plan is bad, so the UI shows estimate against actual side by side for every operator.
The only strongly-consistent store in the system. Everything else may be stale; membership and shard ownership may not. Elections and log replication are drawn live in the distributed view.
Vector and graph workers federated behind the coordinator through the same table interface, so one query can span row, column, vector and graph storage without the planner special-casing any of them.
| FROM → TO | TRANSPORT | SHAPE | WHY THIS AND NOT THE OTHER |
|---|---|---|---|
| browser → gateway | SSE | server stream | spans must arrive while the query still runs; polling would miss the thing being visualised |
| gateway → coordinator | gRPC | unary | one statement, one plan — nothing worth streaming |
| coordinator → catalog | gRPC | unary + cache | read-mostly; stale stats are survivable, a planning stall is not |
| coordinator → worker | gRPC | server stream | result batches flow back as produced, with backpressure on the channel |
| worker ↔ worker | gRPC | bidi stream | exchange/shuffle is symmetric and needs backpressure in both directions |
| any → metadata | Raft RPC | quorum | the one place that must be linearizable — kept off the query hot path deliberately |
Two motions, not one sequence. Every layer gets a thin working MVP first; once all seven exist they wire into a walking skeleton on GCP, and only then does each layer get its own expansion track — picked in any order, re-integrated as it lands. There is no stage six where depth finally happens.
| # | LAYER | MVP SCOPE | EXPANSION TRACKS | STATE |
|---|---|---|---|---|
| 01 | SQL frontend | Hand-written lexer + recursive-descent parser for SELECT … WHERE → AST | Joins, subqueries, CTEs; semantic analysis against the catalog; type checking & coercion; parse-error recovery | MVP DONE |
| 02 | Query planner | AST → logical plan, one fixed rule-based physical plan | DP join-order optimiser over best[S] with a cost model; predicate/projection pushdown; GEQO for large N | BUILDING |
| 03 | Execution engine | Volcano iterator model: scan, filter, project | Hash & sort-merge join, aggregate, sort; vectorised batch execution; the exchange operator for cross-worker shuffle | QUEUED |
| 04 | Storage engine | Plain heap file on local disk, read-through, no buffer pool | Buffer pool with LRU-K/ARC; B+Tree indexes; WAL + ARIES recovery; MVCC & lock manager; hand-rolled columnar pages; an LSM backend alternative | QUEUED |
| 05 | Distributed storage | Two workers, static hash partitioning, no replication | Consistent hashing; real exchange/shuffle for distributed joins; hand-rolled Raft for coordinator HA; 2PC across shards; locality-aware scheduling | QUEUED |
| 06 | Service layer | Gateway → coordinator → workers over gRPC; catalog as its own service on Cloud SQL | Streaming result sets; coordinator↔worker backpressure; API versioning; auth/authz at the gateway | QUEUED |
| 07 | Cloud & infra | Containers on Cloud Run or GKE, Terraform for GCP resources, one Cloud Build pipeline | Full CI/CD on merge; VPC + private networking; autoscaling; Cloud Monitoring/Logging/Trace on the trace contract; chaos testing — kill a worker mid-query | QUEUED |
Every screen has an owning layer and a status. Live means it renders real data from a running service; mockup means it is a design reference for a layer that does not exist yet. If a screen cannot be placed in this table, either the layer list is short or the screen is wrong.