Skip to content

caliper.data.v1

SQL over the host's embedded analytical store — datasets as named, shared, queryable resources, with results crossing the ABI as Arrow C streams. Service id: caliper.data.v1.

#pragma once
/* caliper.data.v1 — SQL over the host's embedded analytical store, results
 * out as Arrow C streams (PLATFORM.md §7.7). Datasets become named, shared,
 * queryable resources instead of per-applet private downloads. IMMUTABLE once
 * published. Callable from applet job threads; the host serializes
 * internally.
 *
 * Stream ownership: on success the host fills *out with a live
 * ArrowArrayStream; the CALLER drains it (get_schema/get_next) and MUST call
 * out->release(out) exactly once when done. On failure (false), *out is
 * untouched and last_error() describes why. last_error() returns a host-owned
 * string valid until the next data.v1 call on the same thread. */
#include <stdint.h>
#include <stdbool.h>
#include <caliper/arrow_c.h>

#define CALIPER_DATA_V1 "caliper.data.v1"

#ifdef __cplusplus
extern "C" {
#endif

typedef struct CaliperDataV1 {
    uint32_t struct_size;
    /* Run SQL against the host store; results stream out as Arrow. */
    bool (*query)(const char* sql_utf8, struct ArrowArrayStream* out);
    /* Name a dataset: uri may be a parquet/csv path or an existing table
       name. Re-registering a name replaces it. */
    bool (*register_dataset)(const char* name, const char* uri);
    /* Open a registered dataset as a full-table stream. */
    bool (*open_dataset)(const char* name, struct ArrowArrayStream* out);
    /* Why the last call on this thread returned false; never NULL. */
    const char* (*last_error)(void);
} CaliperDataV1;

#ifdef __cplusplus
}
#endif

Semantics

Datasets become named, shared, queryable resources instead of per-applet private downloads. The query engine is the host's embedded DuckDB; results cross the ABI as Arrow C streams — no DuckDB or Arrow C++ type ever appears in the frozen header. Four entry points, all callable from a job thread.

  • query(sql_utf8, out) runs a SQL statement against the host store. On success it fills *out with a live ArrowArrayStream and returns true; on failure it returns false, leaves *out untouched, and sets last_error(). DDL/INSERT statements that produce no rows still succeed with an empty stream.
  • register_dataset(name, uri) names a dataset. The uri may be a .parquet path, a .csv path, or the name of an existing table; the host wraps it in a view. Re-registering a name replaces it. The name must be a SQL identifier — [A-Za-z_][A-Za-z0-9_]* — because it is spliced into SQL as an identifier; anything else is rejected (false + last_error), which is also the injection guard.
  • open_dataset(name, out) streams a registered dataset back as a full-table ArrowArrayStream — the SELECT * convenience over query.
  • last_error() returns why this thread's last call returned false. It is never NULL ("" when nothing has failed yet).

Stream ownership protocol

This is the sharp edge — read it before you drain a stream.

  • On success the host hands back a live ArrowArrayStream. The caller drains it (get_schema, then get_next until end) and MUST call out->release(out) exactly once when done. Releasing sets the callback to NULL; a stream whose release is NULL is already empty.
  • End of stream is signalled by get_next yielding an array whose own release is NULL (an empty array) — not by an error. Stop there.
  • Materialized-stream independence. query/open_dataset fully materialize the result before returning; the stream's producer state owns those rows. Draining the stream therefore never touches the DuckDB connection — one job thread can drain a result while another thread issues a new query, with no lock contention between them.

Threading and errors

query/register_dataset/open_dataset serialize on one internal mutex (the artifacts/metrics model). last_error() is thread-local: each thread sees only the error of its last failing call, so one thread's failure never clobbers another's diagnostic. If the store failed to open at start-up the service is still vended but inert (every call false + a last_error).

C++ sugar

The caliper::Data wrapper is falsy-inert when absent and exposes the raw stream API plus one helper — drain_numeric — that drains an all-numeric-columns result into column-major doubles and releases the stream, so simple consumers never touch Arrow buffers:

caliper::Data data(host);                       // falsy if the host doesn't vend it
// DDL/INSERT still hand back a (empty) stream — release it (EmbedScope's
// data_exec helper wraps exactly this):
ArrowArrayStream ddl{};
if (data.query("CREATE OR REPLACE TABLE embed_points(label INT, x REAL, y REAL, z REAL)", &ddl)
    && ddl.release)
    ddl.release(&ddl);
ArrowArrayStream s{};
std::vector<std::string> names;
std::vector<std::vector<double>> cols;
if (data.query("SELECT label, AVG(x), AVG(y), AVG(z) FROM embed_points GROUP BY label", &s))
    caliper::Data::drain_numeric(&s, &names, &cols);   // releases s for you

The reference consumer

EmbedScope registers its published test-set embeddings as a table each eval tick, then runs live SQL over that genuinely tabular data: per-class centroids (AVG(x), AVG(y), AVG(z) GROUP BY label, drawn as 3-D diamonds) and a misclassified count (SUM(CASE ...)), both drained through drain_numeric. It is demonstrative but honest — real learned data, real aggregation — and degrades gracefully: absent, the panels say so and the applet still runs.