--- name: extension-oql description: Make a canister's data queryable by the Caffeine Data Intelligence agent. Use whenever an app stores structured data (Maps/Lists/arrays of records) that should be answerable in natural language — "top customers", "revenue by region", "active projects". Adds a discoverable `schema()` and a JSON `execute()` query endpoint via the `caffeineai-oql` mops package's `Expose` mixin. version: 0.6.2 compatibility: mops: caffeineai-oql: "~0.6.1" caffeineai-subscription: [none] --- # OQL — Object Query Layer Go over the actor's fields (non-transient) and, for each collection worth querying, consider how its data maps to a **table** in a database (an *entity*). You only declare one entity per table — the `Expose` mixin makes them queryable. # Backend Each entity carries an authorization level; the default `.controllerOnly()` is safe (private to users, still readable by the Data Intelligence agent). Model your entities first, then pick a level per entity — see `## Auth`. ## Setup Run `mops add caffeineai-oql@0.6.1` in the **same write batch** as your first `mo:caffeineai-oql/...` import. Auto-derivation requires `moc >= 1.11` (the generated-app template already satisfies this). ### Build flags `--default-persistent-actors` is mandatory. `--implicit-package=core` is optional convenience; every snippet and source file must import the `mo:core` modules it uses. If the app uses `OQL.Table` and needs more than 4 GiB of `Region`, add `--max-stable-pages 1638400` as well; a dependency's own flags are not applied to the project that depends on it, so it has to be set in the app's own build. ### Imports — one per resolver module `.toEntity`, the builder chain (`.sample` / `.build` / `.public_` / …), and record `_toRow` derivation are resolved from modules **imported top-level in the file that declares entities** — the resolver does not walk submodules, so importing only `mo:caffeineai-oql` is not enough. Import exactly the resolver modules your code uses: - `mo:caffeineai-oql/Entity` — always (the `.sample` / `.build` / `.edge` / `.ownedBy` / auth-level builder chain, and `.payload` / `.flatten` in manual mode). - the collection module for each `.toEntity` / `.toEntityManual` receiver — `MapEntity`, `SetEntity`, `ListEntity`, `ArrayEntity`, or `VarArrayEntity`. - for each **auto-derived** (`.toEntity`) record: `RecordValue`, plus one `Value` per primitive field type present — `NatValue`, `TextValue`, `PrincipalValue`, `BoolValue`, `IntValue`, `FloatValue`, the sized `Nat`/`Int` widths, `BlobValue`. Manual `.payload` return types need their `Value` too; manual-only entities need no `RecordValue`. When a collection module is missing the compiler names it — *"field toEntity does not exist … Did you mean to import mo:caffeineai-oql/MapEntity?"* — add the named import. A missing `Entity` import gets no such hint: it surfaces as a bare *"field payload does not exist in type Builder<…>"* (M0072). Treat any `field does not exist` error as a missing top-level import from this list, never as a wrong package version. ## Declare entities and install `.toEntity(name, typeName, primaryKey)` turns a collection of records into a queryable entity; the compiler auto-derives the fields. Each entity sets its own authorization level (see `## Auth`); the example below shows one table per level. `Expose` adds only the OQL query methods (`schema` / `execute`) — your existing state, types, and `shared` methods are untouched. - Always call `.sample({...})` on every `.toEntity` / `Entity.manual` chain; dummy values are fine. Empty collection + no sample → empty schema (`fields: []` / `"record { }"`). ```motoko filepath=src/backend/sample_required.mo include Expose({ entities = [tasks.toEntity("task", "Task", "id").sample({ id = 0; title = "" }).public_().build()] }) ``` ```motoko filepath=src/backend/main.mo import Map "mo:core/Map"; import Nat "mo:core/Nat"; import Principal "mo:core/Principal"; import OQL "mo:caffeineai-oql"; import Expose "mo:caffeineai-oql/Expose"; // Resolver modules, imported top-level (see "Imports" above). This app derives // Map entities over records of Nat / Text / Principal fields: import MapEntity "mo:caffeineai-oql/MapEntity"; import Entity "mo:caffeineai-oql/Entity"; import RecordValue "mo:caffeineai-oql/RecordValue"; import NatValue "mo:caffeineai-oql/NatValue"; import TextValue "mo:caffeineai-oql/TextValue"; import PrincipalValue "mo:caffeineai-oql/PrincipalValue"; actor { type Product = { id : Nat; name : Text; priceUsd : Nat }; type Vendor = { id : Nat; name : Text }; type AuditLog = { id : Nat; action : Text; atNs : Nat }; type Note = { id : Nat; user : Principal; body : Text }; type Document = { id : Nat; owner : Principal; title : Text; ciphertext : Text }; type User = { id : Principal; isAdmin : Bool }; let products : Map.Map; let vendors : Map.Map; let supplies : Map.Map; let auditLogs : Map.Map; let notes : Map.Map; let documents : Map.Map; // not all collections need to be exposed if there is no need — `users` backs // auth only, so it is intentionally never turned into an entity below let users : Map.Map; transient let anyP = Principal.fromText("aaaaa-aa"); // sample owner; the value is ignored // Look up whether a caller is an admin. func isAdmin(p : Principal) : Bool = switch (users.get(p)) { case (?u) u.isAdmin; case null false }; // A custom .ownedByWith rule: admins see every document, everyone else only // their own. `owner` is the field's Value — a Principal column arrives as #text. func canSeeDocument(caller : Principal, owner : OQL.Value) : Bool = isAdmin(caller) or owner == #text(caller.toText()); include Expose({ entities = [ // #public_ — anyone, incl. anonymous, reads the whole catalogue products.toEntity("product", "Product", "id") .sample({ id = 0; name = ""; priceUsd = 0 }) .public_() .build(), vendors.toEntity("vendor", "Vendor", "id") .sample({ id = 0; name = "" }) .public_() .build(), // `supplies : Map` — a map between two non-primitive types. // The identity lives in the key/value records, not a field, so iterate // .entries() in manual mode, promote each side's id, and .edge both — a // query can then traverse "product.name" and "vendor.name". OQL.Entity.manual<(Product, Vendor)>("supply", func () = supplies.entries(), "Supply", "key") .sample(({ id = 0; name = ""; priceUsd = 0 }, { id = 0; name = "" })) .payload("key", func ((p, v)) = p.id.toText() # ":" # v.id.toText()) .payload("product", func ((p, _)) = p.id) .edge("product", "product") .payload("vendor", func ((_, v)) = v.id) .edge("vendor", "vendor") .controllerOnly() .build(), // #controllerOnly (the default, shown explicitly) — only the platform reads auditLogs.toEntity("auditLog", "AuditLog", "id") .sample({ id = 0; action = ""; atNs = 0 }) .controllerOnly() .build(), // #scopedPerUser — each signed-in user reads only their own rows notes.toEntity("note", "Note", "id") .sample({ id = 0; user = anyP; body = "" }) .ownedBy("user") .scopedPerUser() .build(), // #controllerOrScoped — controller reads all; scoped reads use canSeeDocument. // `.hidden` — opaque column absent from schema + default projection documents.toEntity("document", "Document", "id") .sample({ id = 0; owner = anyP; title = ""; ciphertext = "" }) .hidden("ciphertext") .ownedByWith("owner", canSeeDocument) .controllerOrScoped() .build(), ]; }); } ``` The migration chain head: ```motoko filepath=src/backend/migrations/00000000_000000.mo import Map "mo:core/Map"; module { type Product = { id : Nat; name : Text; priceUsd : Nat }; type Vendor = { id : Nat; name : Text }; type AuditLog = { id : Nat; action : Text; atNs : Nat }; type Note = { id : Nat; user : Principal; body : Text }; type Document = { id : Nat; owner : Principal; title : Text; ciphertext : Text }; type User = { id : Principal; isAdmin : Bool }; type NewActor = { products : Map.Map; vendors : Map.Map; supplies : Map.Map; auditLogs : Map.Map; notes : Map.Map; documents : Map.Map; users : Map.Map; }; public func migration(_old : {}) : NewActor { { products = Map.empty(); vendors = Map.empty(); supplies = Map.empty(); auditLogs = Map.empty(); notes = Map.empty(); documents = Map.empty(); users = Map.empty(); }; }; }; ``` ## Auth Authorization is **per entity** — each builder declares a level, and `schema()` and `execute()` both run the check against the live `caller`. No app-wide config, no tokens. The default when none is set is `#controllerOnly`. | Builder call | Who reads | Rows returned | |---|---|---| | `.public_()` | anyone (incl. anonymous) | all | | `.controllerOnly()` *(default)* | controllers only | all | | `.scopedPerUser()` | any signed-in caller | only the caller's own | | `.controllerOrScoped()` | controllers + signed-in callers | controller: all; user: own | ### Choosing a level Pick per entity by who should read its rows — when in doubt, keep the default. - **`.controllerOnly()`** *(default)* — private app data the agent should answer over, but no end user reads directly (orders, metrics, audit logs, config). The agent calls as the controller, so it reads everything while the data stays private to users. - **`.public_()`** — world-readable data, including logged-out visitors (public catalogue, published content, leaderboards). - **`.controllerOrScoped()`** — per-user data where each user reads only their own rows, but the agent must still answer aggregate questions (profiles, a user's orders). Requires an owner column. - **`.scopedPerUser()`** — strictly private per-user data: each user reads only their own, and the agent is scoped too, so it **cannot** answer over this table (DMs, private journals). Requires an owner column — prefer `.controllerOrScoped()` unless the agent must be blind to it. The user may override per entity; if a request implies per-user data but is ambiguous, ask. ### Per-user (row-level) scoping Scoped levels (`.scopedPerUser()`, `.controllerOrScoped()`) need a way to know which rows belong to the caller — an **owner column** or a subject-honouring source. `.build()` traps if a scoped entity has neither, and also traps if a `.public_()` entity declares an owner (the check would never run). This is the guardrail against the common data-leak footgun. **When to tag:** a `Principal` field is the signal. - `.ownedBy(field)` — the field *is* the owner; visibility is identity equality. - `.ownedByWith(field, canSee)` — custom visibility (teams, admins, sharing). `canSee : (caller : Principal, owner : Value) -> Bool` decides per row; `field` need not be a `Principal`, and the closure can read actor state. A scoped caller sees only its owned rows — both as the query target and through a join — so traversal can never leak another owner's rows. ```mo // Per-user notes: each signed-in user reads only their own rows. notes.toEntity("note", "Note", "id") .sample({ id = 0; owner = Principal.fromText("aaaaa-aa") /* any principal */; body = "" }) .ownedBy("owner") .scopedPerUser() .build() // .ownedByWith custom rule: the owner sees their own docs, listed admins see // everyone's, and the platform controller sees all (#controllerOrScoped). // `owner` is the field's Value — a Principal column arrives as #text(principal). docs.toEntity("doc", "Doc", "id") .sample({ id = 0; owner = Principal.fromText("aaaaa-aa"); title = "" }) .ownedByWith("owner", func (caller, owner) = admins.get(caller) != null or owner == #text(caller.toText())) .controllerOrScoped() .build() ``` Where ownership decides **which** rows a scoped caller sees, `.viewWith(view)` decides what **shape** it sees them in — a per-subject redaction that runs only on rows the ownership check already admitted: ```mo // Everyone sees their own bookings; exact amounts only on their own rows is // not needed here — but coarsen the contact field for non-owners of a shared // calendar, say: bookings.toEntity("booking", "Booking", "id") .sample({ id = 0; calendarId = 0; contact = "" }) .ownedByWith("calendarId", canSeeCalendar) .viewWith(func (subject, b) = if (isOwner(subject, b)) b else { b with contact = "" }) .scopedPerUser() .build() ``` `view : (subject : Principal, row : T) -> T` reshapes the typed row; the whole query pipeline (filters included) evaluates the VIEWED row, so a predicate can never probe a value the view hides. Views run for scoped subjects only — pairing `.viewWith` with `.public_()` traps at `.build()` (unrestricted reads always see raw rows). `.ownedBy(f)` is exactly `.ownedByWith(f, OQL.Entity.ownerIsCaller)`. At most one owner column; it must be a real field, not also `.edge` / `.hidden`. For owner-keyed storage (`Map>`) use `OQL.Entity.newScoped(name, scopedIter, typeName, primaryKey)` so the scan is O(user rows): `scopedIter(?p)` returns only `p`'s rows, `scopedIter(null)` all (schema seeding). ## Entity builder Two modes, picked by the row type `T`. ### Auto-derivation — `.toEntity` For records whose fields are all primitives with a built-in `_toRow` (`Nat`, `Int`, `Float`, `Text`, `Bool`, the sized `Nat`/`Int` widths, `Principal`): ```mo customers.toEntity(name, typeName, primaryKey) .sample(template) // REQUIRED — empty collection + no sample → empty schema .edge(field, targetEntity) // tag an existing field as a foreign key .ownedBy(field) // (or .ownedByWith(field, canSee)) per-user scoping .scopedPerUser() // auth level: .public_ / .controllerOnly (default) / .scopedPerUser / .controllerOrScoped .hidden(field) // opaque/sensitive — drop from schema + default projection .build() ``` - `.toEntity` is sugar for `OQL.Entity.new(name, func () = coll.values(), …)`; it exists on `Map`, `Set`, `List`, `[T]`, and `[var T]`. It iterates **values only** — if a row's identity (PK or owner) lives in the *Map key*, it is not a field: promote it via manual mode over `.entries()`, or `OQL.Entity.newScoped` when it's the owner. - `primaryKey`, and any `.edge` / `.ownedBy` field, must name a real, non-`.hidden` column of the row. - `.edge(name, target)` tags an **existing** field (it does not add one) as an FK, enabling dotted-path traversal `"name.targetField"` in queries. FK/PK types must be `Text`, `Nat`/`Int`, or `Bool` (`Float` keys are rejected), and the target's primary key must not be `.hidden`. - **The edge target must be an entity registered in THIS canister's `Expose`.** An edge to an absent entity (a typo, or an FK into another canister) silently drops the whole field from `schema()` — the column still stores and filters, but no schema-driven client can discover it. A cross-canister FK belongs as a plain payload field, not an `.edge`. - `.sample(template)` seeds schema discovery. Always call it; without it an empty collection yields an empty schema (`fields: []`). Only the shape matters, not the values. - `.hidden(name)` drops a derivable field from schema + default projection; it does **not** skip `_toRow` — unsupported field types still need manual mode or a `Value`. Schema fields are listed in **lexicographic order** (the `__record` combiner's canonical form); sort client-side if display order matters. ### Manual mode — `.toEntityManual` / `OQL.Entity.manual` For non-record `T`, computed fields, or records with nested / variant / option / collection fields: ```mo // REQUIRED, top-level in this file — `.payload` / `.flatten` are Entity // functions reached by receiver notation, not fields of the builder: import Entity "mo:caffeineai-oql/Entity"; authors.toEntityManual("author", "Author", "id") .sample({ id = 0; name = ""; address = { street = ""; city = "" }; tags = [] }) .payload("name", func a = a.name) // one field; extract returns a _toRow value .flatten(func a = a.address) // splice a nested record's fields as columns .payload("tagCount", func a = a.tags.size()) // .edge tags a declared column; .hidden drops one you already added via // `.payload` / `.flatten` — omit fields by not adding them .build() ``` **`field payload does not exist in type Builder<…>` (M0072) means the `Entity` import above is missing — it is never a package-version problem.** `.payload` and `.flatten` have been `Entity` functions since 0.1.0; `mops add`-ing a different `caffeineai-oql` version will not fix this error. Importing only `mo:caffeineai-oql` is not enough, and neither is reaching the module through the `OQL.Entity.…` re-export: receiver notation resolves against **top-level imports** only, so a file that calls `OQL.Entity.manual(...).payload(...)` still needs its own `import Entity "mo:caffeineai-oql/Entity";`. Unlike `.toEntity`, this failure carries no "Did you mean to import …?" hint. - `.payload(name, extract)` — `name` must not contain `.`. Prefer `func r = r.field` (let Motoko infer; avoid redundant annotations). For options/variants, return `Text`/`Nat` with a sentinel (see below). - `.flatten(extract : T -> S)` — `S` must be flat; each of its fields becomes a top-level column. Drop unwanted ones with `.hidden`. Name collisions get `__1`, `__2` suffixes (nothing is dropped). - `OQL.Entity.manual(name, iter, typeName, primaryKey)` for arbitrary row sources (custom flatteners, filtered iterators). Always chain `.sample(...)` with one dummy row of type `T`. The qualified call resolves through the `OQL` import, but any `.payload` / `.flatten` chained onto its result still needs the top-level `Entity` import. `OQL.Value` is `{ #null_; #bool; #nat; #int; #float; #text }`. Numeric variants compare across each other, so a JSON integer threshold matches a `Float` value. | Row type `T` | Mode | |---|---| | All-primitive record | `.toEntity` | | Record with `?` / variant / nested field | `.toEntity` once you ship `Value.mo` (below); else manual | | Record with a collection field | manual — `.size()` or `Text.join` into a payload | | Tuple / primitive / computed | manual | ## Converting non-primitive fields To keep a record on the auto-derive path, give each non-primitive field type a `_toRow : T -> OQL.Value`: one file per type named `Value.mo`, a single `public func _toRow`, imported **top-level** in the file that declares entities — the same top-level rule as the built-in value modules (see `## Setup` → Imports); the resolver does not walk submodules. Parent records then ride `.toEntity(...)` with no per-field `.payload`. ```mo // OptTextValue.mo — option → sentinel module { public func _toRow(self : ?Text) : OQL.Value = switch self { case null { #text("") }; case (?t) { #text(t) } }; }; // StatusValue.mo — variant → tag text module { public func _toRow(self : Status) : OQL.Value = #text(switch self { case (#draft) "draft"; case (#published) "published" }); }; // DepartmentValue.mo — nested record → child PK (then .edge the field) module { public func _toRow(self : Department) : OQL.Value = #text(self.name); }; ``` **Always return ONE