What we do · expertise
Data you can query, trace and move.
We design and run open data platforms: open table formats, a query engine decoupled from storage, lineage and catalogue. Reversibility guides every choice — being able to leave without a rewrite.
Method
A data platform is judged by what it costs to leave it. As long as the table format, the catalogue and the engine belong to the same vendor, the question “what would a change cost” has no answer — it has a quote. We build the other way round: Iceberg tables sitting on object storage you own, an engine decoupled from that storage, a catalogue readable without a licence.
What we can do beyond the market average starts one floor lower. We do not merely use Trino: we write the Trino client. trino-rust-client is a fork of prusto taken over in December 2024 — Presto support removed, advanced Trino features added — that is forty-five Rust files and 6,374 lines, published under the MIT licence.
The spooling protocol is the example that makes the difference concrete. In the historical protocol, every page of results travels back through the coordinator, serialised as JSON: on a large export, the coordinator carries everyone's bytes, memory and CPU, and other teams' interactive queries suffer for it. With spooling, the coordinator no longer returns rows — it returns segment descriptors: an address on object storage, a row offset, a row count, a size.
The client does the rest, and that is the code opposite. It fetches segments straight from storage, with bounded concurrency, decompresses them — zstd or lz4, depending on what the coordinator announces — then acknowledges each segment so the storage is reclaimed instead of waiting for expiry. Going down to that level is not a library refinement: it is the difference between a bulk export that coexists with the rest of the company's queries and one that takes them down.
This kind of platform handles 100s To (1) at peak on one platform we operate. That figure is an observed peak, not a guaranteed ceiling: it appears here with its scope, and nowhere without it. What the open format and the client do not settle, on the other hand, is the quality of what comes in and the cost of what computes. ▪
The repository this code comes from
trino-rust-client
Trino client in Rust. Fork of prusto taken over in December 2024: Presto support removed, advanced Trino features added, including the spooling protocol.
github.com/nudibranches-tech/trino-rust-client
Measured over src/, excluding tests and examples.
// Segment is a part of a query result when using the spooling protocol
#[derive(Debug, PartialEq, Deserialize, Serialize)]
#[serde(rename_all = "camelCase", untagged)]
pub enum Segment {
// Inlined segment
Inlined {
#[serde(rename = "type")]
segment_type: String,
data: String,
metadata: DataAttributes,
},
// Spooled segment
Spooled {
#[serde(rename = "type")]
segment_type: String,
uri: String,
#[serde(rename = "ackUri")]
#[serde(skip_serializing_if = "Option::is_none")]
ack_uri: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
headers: Option<HashMap<String, Vec<String>>>,
metadata: DataAttributes,
},
} /// Fetch a single segment and return the decoded data
pub async fn fetch_segment(&self, segment: &Segment) -> Result<Vec<u8>> {
match segment {
Inlined { data, .. } => self.fetch_inline_segment(data).await,
Segment::Spooled {
uri,
ack_uri,
headers,
..
} => {
let data = self.fetch_spooled_segment(uri, headers.as_ref()).await?;
// Acknowledge the segment if ackUri is provided
if let Some(ack) = ack_uri {
if let Err(e) = self.acknowledge_segment(ack, headers.as_ref()).await {
tracing::warn!("Failed to acknowledge segment {}: {}", ack, e);
}
}
Ok(data)
}
}
}Scope
The deciding criterion is the cost of leaving, calculated on the day you arrive.
Engagements
Described by the setup that was put in place and by what it produced.
Embedded engagement · ongoing, more than 12 months
Engineers embedded in the client's teams, on their repository and their delivery chain. What was done by hand is now described, versioned and reviewed.
Advisory and implementation · ongoing
A Kubernetes base on an immutable OS, described end to end in a GitOps repository that carries fourteen applications. That repository is the only way in: it holds the infrastructure as well as the services running on it.
Get in touch
Thirty minutes are enough to place a data stack on a single axis: what it would cost to move off it. We will also say so when the answer is that nothing should change — it has happened, and it does not get invoiced.