Storage

Storage

Generated from MemCP commit c42e19eba on 27 August 2026. See Full SCM API documentation.

The Storage module is the low-level data interface used by generated SQL plans. It manages databases, tables, columns, constraints, scans, indexes, RecSets, computed data, persistence, and schema lifecycle operations.

Most application code should use SQL instead of calling these functions directly. Planner-generated calls rely on exact transaction, locking, callback, ownership, and quoting contracts. Functions that accept table handles or transaction contexts must not be replaced with direct access to shard internals.

For the physical model see Columnar Storage, Shards, RecordIDs, Main Storage, Delta Storage, and Persistency and Performance Guarantees.

table

resolves a schema+table name pair into a table handle

Allowed number of parameters: 2–2

Parameters

• schema (string)
• table (string)

Returns

• value (table)

scan_estimate

estimate output row count for a table scan

Allowed number of parameters: 1–1

Parameters

• table (table)

Returns

• value (int)

table_planner_statistics

return the immutable O(1) planner-statistics snapshot for a table

Allowed number of parameters: 1–1

Parameters

• table (table)

Returns

• value (any)

scan_selectivity_estimate

bounded estimate of visible rows matching a table filter; stops at max_rows and does not log scan telemetry

Allowed number of parameters: 5–5

Parameters

• tx (any): transaction context to use for visibility; usually ((context "session") "__memcp_tx")
• table (table)
• condition_cols (list<string>): columns passed to the selectivity predicate
  • column (string): column name passed to the corresponding callback parameter
• condition (func): predicate sampled to estimate matching rows
  • Parameters
    • columns (any): one value for each entry in the matching column list, in the same order (variadic)
  • Returns
    • result (bool): true when the sampled row matches
• max_rows (int)

Returns

• value (list)

table_empty?

returns true if a table currently has no rows

Allowed number of parameters: 1–1

Parameters

• table (table)

Returns

• value (bool)

scan_recset

builds a query-local record-set handle from one table scan, or -- when given an existing recset instead of a table -- narrows that recset to the members which also satisfy filter, re-evaluating filter only over its existing membership. The latter is the cheap way to AND a further (possibly subscan-heavy) condition onto an already-narrowed recset without re-touching rows outside it (e.g. evaluating an expensive correlated check only over the rows a cheap selective filter already narrowed a table down to). The returned value is not persisted and can be scanned like a table

Allowed number of parameters: 4–4

Parameters

• tx (any): transaction context to use for visibility; usually ((context "session") "__memcp_tx")
• table (any): a table, or an existing recset to narrow further
• filterColumns (list<string>): physical columns passed to filter before map/reduce; $recset_contains supplies a row-bound RecSet membership closure
  • column (string): column name passed to the corresponding callback parameter
• filter (func): lambda function that decides whether a row enters the recset
  • Parameters
    • columns (any): one value for each entry in the matching column list, in the same order (variadic)
  • Returns
    • result (bool): true when the row belongs in the recset

Returns

• value (recset)

recset_count

returns the number of currently stored recids in a query-local recset

Allowed number of parameters: 1–1

Parameters

• recset (recset)

Returns

• value (int)

recset_project_join

projects a source recset through key columns into a query-local target-table recset

Allowed number of parameters: 5–5

Parameters

• tx (any): transaction context to use for visibility; usually ((context "session") "__memcp_tx")
• source_recset (recset)
• source_key_columns (list)
• target_table (table)
• target_key_columns (list)

Returns

• value (recset)

recset_key_index

builds an immutable lookup function for key columns of the rows contained in a query-local recset

Allowed number of parameters: 3–3

Parameters

• tx (any): transaction context used while reading source keys
• source_recset (recset)
• source_key_columns (list)

Returns

• lookup (func): tests whether the recset contains a row with the supplied composite key
  • Parameters
    • key (any): one value for each source key column, in the same order (variadic)
  • Returns
    • present (bool): whether the composite key occurs in the recset

recset_union

combines query-local recsets from the same table and removes duplicate record IDs

Allowed number of parameters: 1–1

Parameters

• recsets (list)

Returns

• value (recset)

recset_intersect

intersects query-local recsets from the same table

Allowed number of parameters: 1–1

Parameters

• recsets (list)

Returns

• value (recset)

recset_difference

returns the records from the first query-local recset which occur in none of the following same-table recsets

Allowed number of parameters: 1–1

Parameters

• recsets (list)

Returns

• value (recset)

recset_not

returns the complement of a query-local recset relative to the currently visible rows of its base table

Allowed number of parameters: 1–1

Parameters

• recset (recset)

Returns

• value (recset)

scan_exists

returns true if a table contains at least one visible row matching the given filter; uses scan boundary analysis without map/reduce setup

Allowed number of parameters: 4–4

Parameters

• tx (any): transaction context to use for visibility; usually ((context "session") "__memcp_tx")
• table (table|list|recset)
• filterColumns (list<string>): physical columns passed to filter before map/reduce; $recset_contains supplies a row-bound RecSet membership closure
  • column (string): column name passed to the corresponding callback parameter
• filter (func): lambda function that decides whether a row exists
  • Parameters
    • columns (any): one value for each entry in the matching column list, in the same order (variadic)
  • Returns
    • result (bool): true when the row satisfies the existence test

Returns

• value (bool)

scan

does an unordered parallel filter-map-reduce pass on a single table and returns the reduced result

Allowed number of parameters: 6–10

Parameters

• tx (any): transaction context to use for visibility and mutations; usually ((context "session") "__memcp_tx")
• table (table|list|recset): table handle, query-local recset, or a list for temporary data
• filterColumns (list<string>): physical columns passed to filter before map/reduce; $recset_contains supplies a row-bound RecSet membership closure
  • column (string): column name passed to the corresponding callback parameter
• filter (func): lambda function that decides whether a dataset is passed to the map phase. Equality and range comparisons may be translated into indexed scans
  • Parameters
    • columns (any): one value for each entry in the matching column list, in the same order (variadic)
  • Returns
    • result (bool): true when the row proceeds to map
• mapColumns (list<string>): physical columns passed to map after filtering; pseudo columns are $update (update/delete current row), $recset_contains (row-bound RecSet membership), $set:<column>, $increment:<column>, and $invalidate:<column> (computed-column maintenance), plus NEW.<column> in trigger plans
  • column (string): column name passed to the corresponding callback parameter
• map (func): lambda function that extracts or produces one value from the row; it may also use documented pseudo columns for mutations or result output
  • Parameters
    • columns (any): one value for each entry in the matching column list, in the same order (variadic)
  • Returns
    • result (any): value passed to reduce, or returned directly when no reducer is supplied
• reduce (func): optional aggregation function used first within shards and then to combine shard results (optional)
  • Parameters
    • accumulator (any): current aggregate, initially the neutral value
    • value (any): next mapped or partially reduced value
  • Returns
    • accumulator (any): aggregate passed to the next reducer call or returned by the scan
• neutral (any): (optional) neutral element for the reduce phase, otherwise nil is assumed (optional)
• reduce2 (func): optional final reducer that combines the neutral value with the result produced by reduce (optional)
  • Parameters
    • accumulator (any): current aggregate, initially the neutral value
    • value (any): next mapped or partially reduced value
  • Returns
    • accumulator (any): aggregate passed to the next reducer call or returned by the scan
• isOuter (bool): (optional) if true, in case of no hits, call map once anyway with NULL values (optional)

Returns

• value (any)

scan_batch

does an unordered parallel filter-map-reduce pass on a single table using batchdata-backed #N pseudo columns and returns the reduced result

Allowed number of parameters: 8–12

Parameters

• tx (any): transaction context to use for visibility and mutations; usually ((context "session") "__memcp_tx")
• table (table|list|recset): table handle, query-local recset, or a list for temporary data
• filterColumns (list<string>): columns passed to filter; #0, #1, ... address batchdata slots
  • column (string): column name passed to the corresponding callback parameter
• filter (func): lambda function that decides whether a dataset is passed to the map phase
  • Parameters
    • columns (any): one value for each entry in the matching column list, in the same order (variadic)
  • Returns
    • result (bool): true when this table row and batch row proceed to map
• mapColumns (list<string>): columns passed to map; #0, #1, ... address batchdata slots
  • column (string): column name passed to the corresponding callback parameter
• map (func): lambda function that extracts data from the table row and batch row
  • Parameters
    • columns (any): one value for each entry in the matching column list, in the same order (variadic)
  • Returns
    • result (any): value passed to reduce or returned directly
• stride (int): number of batchdata entries per batch row
• batchdata (list<any>): flat batch buffer accessed via #N pseudo columns
  • slot (any): one batch value; every stride consecutive slots form a batch row
• reduce (func): optional lambda function that aggregates mapped values (optional)
  • Parameters
    • accumulator (any): current aggregate, initially the neutral value
    • value (any): next mapped or partially reduced value
  • Returns
    • accumulator (any): aggregate passed to the next reducer call or returned by the scan
• neutral (any): (optional) neutral element for the reduce phase, otherwise nil is assumed (optional)
• reduce2 (func): optional final reducer that combines the neutral value with the result produced by reduce (optional)
  • Parameters
    • accumulator (any): current aggregate, initially the neutral value
    • value (any): next mapped or partially reduced value
  • Returns
    • accumulator (any): aggregate passed to the next reducer call or returned by the scan
• isOuter (bool): (optional) if true, in case of no hits, call map once anyway with NULL values (optional)

Returns

• value (any)

scan_order_batch_accept

incrementally scans a table or existing RecSet in scan_order order and applies a RecSet batch filter before OFFSET/LIMIT and map/reduce. The first candidate RecSet contains offset+limit rows; if too few rows are accepted, subsequent disjoint batches contain twice as many candidates until the accepted limit is satisfied or the input is exhausted. batchFilter is called as (batchFilter input_recset) and must return an exact subset RecSet of the same base table and transaction. A simple batchFilter may call (scan_recset tx input_recset filterColumns realFilter); complex filters may project input_recset to another table, apply search/ACL scans and project the result back to the input table. The returned RecSet is used only as a membership mask against the already ordered candidate vector, so output order is preserved without scanning the unordered RecSet again. For non-unique ORDER BY values, include an explicit unique tie-breaker. sortcols/sortdirs may both be empty; that path greedily collects candidates without sorting. limitPartitionCols is present for scan_order signature compatibility and currently must be 0

Allowed number of parameters: 10–14

Parameters

• tx (any): transaction context used consistently by the candidate scan and every batch filter operation; usually ((context "session") "__memcp_tx")
• table_or_recset (table|recset): base table or complete existing query-local RecSet from which ordered candidate batches are drawn
• batchFilter (func): function (lambda (input_recset) accepted_recset). It may naively narrow input_recset with scan_recset, or run arbitrary RecSet projections/search/ACL operations and project back. It must return a same-table, same-transaction subset of input_recset
  • Parameters
    • input_recset (recset)
  • Returns
    • value (recset)
• sortcols (list<string|func(columns:any...) -> any>): same as scan_order: columns or computed sort functions. Include a unique tie-breaker for a total repeatable order; use an empty list for greedy unsorted collection
  • sort column (string|func): a column name, or a function of row-column values that returns the sortable value
    • Parameters
      • columns (any): column values used to compute the sort key (variadic)
    • Returns
      • sort key (any): value compared at this sort position
• sortdirs (list<func(left:any, right:any) -> bool>): same as scan_order: one relation per sort column; must also be empty when sortcols is empty
  • direction (func): strict ordering relation such as <, >, or a collate relation
    • Parameters
      • left (any): left sort value
      • right (any): right sort value
    • Returns
      • ordered (bool): true when left belongs before right
• limitPartitionCols (number): reserved for scan_order signature compatibility; currently must be 0
• offset (number): number of batch-filter-accepted rows to skip; it is not the number of driver candidates already examined
• limit (number): finite maximum number of accepted rows passed to map; the initial candidate batch size is offset+limit and doubles for every subsequent batch
• mapColumns (list<string>): physical columns passed to map after filtering; pseudo columns are $update (update/delete current row), $recset_contains (row-bound RecSet membership), $set:<column>, $increment:<column>, and $invalidate:<column> (computed-column maintenance), plus NEW.<column> in trigger plans; $break is reserved for internal ORC convergence and must not implement SQL OFFSET/LIMIT, which belong in the native offset and limit arguments
  • column (string): column name passed to the corresponding callback parameter
• map (func): same map callback contract as scan_order; accepted record IDs are passed to its shard mapper in batches
  • Parameters
    • columns (any): one value for each entry in the matching column list, in the same order (variadic)
  • Returns
    • result (any): value passed to reduce or returned directly
• reduce (func): optional serial reducer over mapped accepted rows, with the same accumulator contract as scan_order (optional)
  • Parameters
    • accumulator (any): current aggregate, initially the neutral value
    • value (any): next mapped or partially reduced value
  • Returns
    • accumulator (any): aggregate passed to the next reducer call or returned by the scan
• neutral (any): optional neutral element for reduce; defaults to nil (optional)
• isOuter (bool): optional scan_order-compatible outer behavior: map one NULL row when no accepted row reaches map (optional)
• notFoundValue (any): optional result when no accepted row reaches map and isOuter is false; defaults to neutral (optional)

Returns

• value (any)

scan_order

does an ordered parallel filter and serial map-reduce pass on a single table and returns the reduced result

Allowed number of parameters: 11–17

Parameters

• tx (any): transaction context to use for visibility and mutations; usually ((context "session") "__memcp_tx")
• table (table|list|recset): table handle, query-local RecSet, or a list for temporary data
• filterColumns (list<string>): physical columns passed to filter before map/reduce; $recset_contains supplies a row-bound RecSet membership closure
  • column (string): column name passed to the corresponding callback parameter
• filter (func): lambda function that decides whether a dataset is passed to the map phase. Equality and range comparisons may be translated into indexed scans
  • Parameters
    • columns (any): one value for each entry in the matching column list, in the same order (variadic)
  • Returns
    • result (bool): true when the row proceeds to ordering and map
• sortcols (list<string|func(columns:any...) -> any>): columns used for ordering; each entry corresponds to one relation in sortdirs
  • sort column (string|func): a column name, or a function of row-column values that returns the sortable value
    • Parameters
      • columns (any): column values used to compute the sort key (variadic)
    • Returns
      • sort key (any): value compared at this sort position
• sortdirs (list<func(left:any, right:any) -> bool>): one ordering relation per entry in sortcols; < is ascending and > is descending
  • direction (func): strict ordering relation such as <, >, or a collate relation
    • Parameters
      • left (any): left sort value
      • right (any): right sort value
    • Returns
      • ordered (bool): true when left belongs before right
• limitPartitionCols (number): number of leading sort columns that form the partition key for per-partition offset/limit. 0 (default) means global offset/limit.
• offset (number): number of globally ordered, filter-accepted items to skip before map; apply SQL OFFSET here rather than in map
• limit (number): maximum globally ordered, filter-accepted items passed to map; -1 means unlimited; apply SQL LIMIT here so shard-local Top-K and the global merge can brake early
• mapColumns (list<string>): physical columns passed to map after filtering; pseudo columns are $update (update/delete current row), $recset_contains (row-bound RecSet membership), $set:<column>, $increment:<column>, and $invalidate:<column> (computed-column maintenance), plus NEW.<column> in trigger plans; $break is reserved for internal ORC convergence and must not implement SQL OFFSET/LIMIT, which belong in the native offset and limit arguments
  • column (string): column name passed to the corresponding callback parameter
• map (func): lambda function that extracts or produces one value from each accepted row
  • Parameters
    • columns (any): one value for each entry in the matching column list, in the same order (variadic)
  • Returns
    • result (any): value passed to reduce or returned directly
• reduce (func): optional serial aggregation function over mapped values (optional)
  • Parameters
    • accumulator (any): current aggregate, initially the neutral value
    • value (any): next mapped or partially reduced value
  • Returns
    • accumulator (any): aggregate passed to the next reducer call or returned by the scan
• neutral (any): (optional) neutral element for the reduce phase, otherwise nil is assumed (optional)
• isOuter (bool): (optional) if true, in case of no hits, call map once anyway with NULL values (optional)
• notFoundValue (any): (optional) result for no hits when isOuter is false; defaults to neutral (optional)
• postOrderFilterColumns (list<string>): optional columns for a predicate evaluated in global order before OFFSET/LIMIT are counted; use for expensive acceptance checks that cannot participate in index boundaries (optional)
  • column (string): column name passed to the corresponding callback parameter
• postOrderFilter (func): optional late acceptance predicate. Rejected rows do not count toward OFFSET/LIMIT and never reach map (optional)
  • Parameters
    • columns (any): one value for each entry in the matching column list, in the same order (variadic)
  • Returns
    • result (bool): true when the ordered row counts toward OFFSET/LIMIT and reaches map

Returns

• value (any)

scan_order_multi

does an ordered parallel filter and serial map-reduce pass across multiple tables simultaneously, merging results into a single sorted stream

Allowed number of parameters: 13–17

Parameters

• tx (any): transaction context
• tables (list<table|recset>): scan sources; all per-table lists must have this length
  • source (table|recset): base table or query-local record set for one input stream
• filterColumns (list<list<string>>): filter column lists, one per table
  • table filter columns (list<string>): columns supplied to the matching filterFns entry
    • column (string): column name in the corresponding table
• filterFns (list<func(columns:any...) -> bool>): filter lambdas, one per table
  • table filter (func): predicate for the corresponding table and filterColumns entry
    • Parameters
      • columns (any): one value for each entry in the matching column list, in the same order (variadic)
    • Returns
      • result (bool): true when the row enters that table's ordered stream
• sortcols (list<list<string|func(columns:any...) -> any>>): sort column lists, one per table; every inner list must match sortdirs in length and result domains
  • table sort columns (list<string|func(columns:any...) -> any>): sort expressions for the corresponding table
    • sort column (string|func): a column name, or a function of row-column values that returns the sortable value
      • Parameters
        • columns (any): column values used to compute the sort key (variadic)
      • Returns
        • sort key (any): value compared at this sort position
• sortdirs (list<func(left:any, right:any) -> bool>): shared ordering relations used for every table stream and for the outer merge
  • direction (func): strict ordering relation such as <, >, or a collate relation
    • Parameters
      • left (any): left sort value
      • right (any): right sort value
    • Returns
      • ordered (bool): true when left belongs before right
• perTableOffset (list<int>|nil): optional per-table offsets; nil disables all per-table offsets
  • offset (int): rows skipped in the corresponding table before the outer merge; -1 disables the offset
• perTableLimit (list<int>|nil): optional per-table limits; nil disables all per-table limits
  • limit (int): maximum rows retained from the corresponding table before the outer merge; -1 disables the limit
• limitPartitionCols (number): number of leading sort columns forming partition key
• offset (number): number of items to skip (global)
• limit (number): max number of items to read (global; -1 = unlimited)
• mapColumns (list<list<string>>): map column lists, one per table
  • table map columns (list<string>): columns supplied to the matching mapFns entry
    • column (string): column name in the corresponding table
• mapFns (list<func(columns:any...) -> any>): map lambdas, one per table
  • table map (func): mapper for the corresponding table and mapColumns entry
    • Parameters
      • columns (any): one value for each entry in the matching column list, in the same order (variadic)
    • Returns
      • result (any): value inserted into the merged stream and passed to reduce
• reduce (func): optional aggregation function over mapped values from the merged stream (optional)
  • Parameters
    • accumulator (any): current aggregate, initially the neutral value
    • value (any): next mapped or partially reduced value
  • Returns
    • accumulator (any): aggregate passed to the next reducer call or returned by the scan
• neutral (any): (optional) neutral element for reduce (optional)
• isOuter (bool): (optional) if true, emit null row when no hits (optional)
• notFoundValue (any): (optional) result for no hits when isOuter is false; defaults to neutral (optional)

Returns

• value (any)

createdatabase

creates a new database

Allowed number of parameters: 1–2

Parameters

• schema (string): name of the new database
• ignoreexists (bool): if true, return false instead of throwing an error (optional)

Returns

• value (bool)

dropdatabase

drops a database

Allowed number of parameters: 1–2

Parameters

• schema (string): name of the database
• ifexists (bool): if true, don't throw an error if it doesn't exist (optional)

Returns

• value (bool)

checktablemaintenance

checks whether a user-initiated maintenance operation is allowed for a table

Allowed number of parameters: 3–3

Parameters

• schema (string)
• table (string)
• operation (string)

Returns

• value (bool)

maintenance_capabilities

returns the server-side maintenance capabilities for a database or table

Allowed number of parameters: 1–2

Parameters

• schema (string)
• table (string) (optional)

Returns

• value (list)

createtable

creates a table, runs its oninit option and registered after-create-table lifecycle triggers synchronously, and returns only after initialization completes; concurrent if-not-exists callers wait for that same completion

Allowed number of parameters: 4–5

Parameters

• schema (string): name of the existing database that will contain the table
• table (string): name of the table to create
• cols (list<list>): column and constraint definitions
  • definition (list): one of ("column" name type dimensions typeparams), ("unique" name columns), or ("foreign" name local_columns referenced_table referenced_columns update_mode delete_mode). Column lists contain strings; foreign-key modes are restrict, cascade, or set null. A column definition's dimensions contains integers and its typeparams uses the same fields documented by createcolumn options
• options (list|assoc): table options as an alternating key/value list
  • auto_increment (int): first automatically assigned value; must be non-negative
  • charset (string): default character set name
  • collation (string): default collation name
  • comment (string): user-visible table comment
  • engine (string): storage engine: safe, logged, sloppy, memory, or cache
  • oninit (func): closed zero-argument initializer run synchronously once per data generation; concurrent if-not-exists callers wait for completion
    • Returns
      • result (any): ignored initializer result
• ifnotexists (bool): when true, return false instead of failing if the table exists; if another caller is still creating it, wait for that caller's after-create-table initialization before returning false (optional)

Returns

• value (bool): true when this call created and initialized the table, false when ifnotexists reused an initialized table

createcolumn

creates a new column in table

Allowed number of parameters: 5–7

Parameters

• table (table)
• colname (string): name of the new column
• type (string): name of the basetype
• dimensions (list<int>): dimensions of the type, for example precision and scale for decimal
  • dimension (int): one type-specific dimension
• options (list|assoc): column properties and computed-column configuration as an alternating key/value list
  • auto_increment (bool): assign increasing values automatically
  • collate (string): collation used for this column
  • comment (string): user-visible column comment
  • default (any): literal value used when an insert omits the column
  • default_expression (string): expression evaluated when an insert omits the column
  • filter (func): predicate limiting which rows are computed
    • Parameters
      • columns (any): one value for each entry in the matching column list, in the same order (variadic)
    • Returns
      • result (bool): true when the row should be computed
  • filtercols (list<string>): columns supplied to filter before computing a value
    • column (string): column name passed to the corresponding callback parameter
  • mapcols (list<string>): columns supplied to mapfn for ordered-reduce computation
    • column (string): column name passed to the corresponding callback parameter
  • mapfn (func): maps one source row into a value for reducefn
    • Parameters
      • columns (any): one value for each entry in the matching column list, in the same order (variadic)
    • Returns
      • result (any): value passed to reducefn
  • null (bool): whether the column accepts nil values
  • partitioncount (int): number of leading sort columns that define independent reducer partitions
  • primary (bool): whether this column belongs to the primary key
  • reducefn (func): combines ordered mapped values into the computed-column aggregate (optional)
    • Parameters
      • accumulator (any): current aggregate, initially the neutral value
      • value (any): next mapped or partially reduced value
    • Returns
      • accumulator (any): aggregate passed to the next reducer call or returned by the scan
  • reduceinit (any): initial accumulator supplied to reducefn
  • sortcols (list<string|func(columns:any...) -> any>): columns or expressions defining ordered-reduce input order
    • sort column (string|func): a column name, or a function of row-column values that returns the sortable value
      • Parameters
        • columns (any): column values used to compute the sort key (variadic)
      • Returns
        • sort key (any): value compared at this sort position
  • sortdirs (list<func(left:any, right:any) -> bool>): one ordering relation for every sortcols entry
    • direction (func): strict ordering relation such as <, >, or a collate relation
      • Parameters
        • left (any): left sort value
        • right (any): right sort value
      • Returns
        • ordered (bool): true when left belongs before right
  • temp (bool): whether this is a query-local temporary computed column
  • unique (bool): whether values must be unique
  • update (any): expression evaluated when a row is updated
• computorCols (list<string>): columns passed to computor in this order (optional)
  • column (string): column name passed to the corresponding callback parameter
• computor (func): lambda expression that computes this column from the values selected by computorCols (optional)
  • Parameters
    • columns (any): one value for each entry in the matching column list, in the same order (variadic)
  • Returns
    • result (any): computed column value

Returns

• value (bool)

createkey

creates a new key on a table

Allowed number of parameters: 4–4

Parameters

• table (table)
• keyname (string): name of the new key
• unique (bool): whether the key is unique
• columns (list): list of columns to include

Returns

• value (bool)

dropkey

drops a named unique key from a table

Allowed number of parameters: 2–2

Parameters

• table (table)
• keyname (string): name of the unique key

Returns

• value (bool)

createforeignkey

creates a new foreign key on a table

Allowed number of parameters: 7–7

Parameters

• table1 (table)
• keyname (string): name of the new key
• columns1 (list): list of columns to include
• table2 (table)
• columns2 (list): list of columns to include
• updatemode (string): restrict|cascade|set null
• deletemode (string): restrict|cascade|set null

Returns

• value (bool)

shardcolumn

tells us how it would partition a column according to their values. Returns a list of pivot elements.

Allowed number of parameters: 2–3

Parameters

• table (table)
• colname (string): name of the column
• numpartitions (number): number of partitions; optional. leave 0 if you want to detect the partiton number automatically or copy the partition schema of the table (optional)

Returns

• value (list)

partitiontable

suggests a partition scheme for a table. If the table has no partition scheme yet, it will immediately apply that scheme and return true. If the table already has a partition scheme, it will alter the partitioning score such that the partitioning scheme is considered in the next repartitioning and return false.

Allowed number of parameters: 2–2

Parameters

• table (table)
• columns (list): associative list of string -> list representing column name -> pivots. You can compute pivots by (shardcolumn ...)

Returns

• value (bool)

altertable

alters a table

Allowed number of parameters: 3–3

Parameters

• table (table)
• operation (string): one of owner|drop|engine|collation|auto_increment
• parameter (any): name of the column to drop or value of the parameter

Returns

• value (bool)

altercolumn

alters a column

Allowed number of parameters: 4–4

Parameters

• table (table)
• column (string): name of the column
• operation (string): one of drop|type|collation|auto_increment|comment
• parameter (any): name of the column to drop or value of the parameter

Returns

• value (bool)

droptable

removes a table

Allowed number of parameters: 2–3

Parameters

• schema (string)
• table (string)
• ifexists (bool): if true, don't throw an error if it already exists (optional)

Returns

• value (bool)

dropcolumn

drops a column from a table

Allowed number of parameters: 2–2

Parameters

• table (table)
• column (string): name of the column to drop

Returns

• value (bool)

migratedropcolumn

drops a legacy system column during startup migration

Allowed number of parameters: 2–2

Parameters

• table (table)
• column (string): legacy column name

Returns

• value (bool)

invalidatecolumn

marks all values of a computed column as stale

Allowed number of parameters: 2–2

Parameters

• table (table)
• column (string): name of the computed column

Returns

• value (bool)

invalidateorc

invalidates ORC column rows from a sort key onwards via validMask scan

Allowed number of parameters: 3–3

Parameters

• table (table)
• column (string): name of the ORC column
• sortkeys (list): composite sort key values from which to invalidate

Returns

• value (bool)

register_keytable_cleanup

registers triggers on a base table to maintain keytable entries (insert/delete group keys)

Allowed number of parameters: 4–4

Parameters

• base_table (table)
• kt_table (table)
• tblvar (string): table alias used in scan column prefixes
• key_pairs (list): list of (base_col kt_col) pairs

Returns

• value (bool)

initialize_cache_table

registers maintenance, locks source tables for a consistent snapshot, and runs a canonical planner-cache initializer exactly once

Allowed number of parameters: 5–6

Parameters

• transaction (any): explicit transaction context carrying query-session ownership
• table (table)
• source_tables (list)
• register_maintenance (func)
  • Returns
    • value (any)
• initializer (func)
  • Returns
    • value (any)
• finalizer (func): optional zero-argument finalizer run under the same source-table locks after initialization (optional)
  • Returns
    • value (any)

Returns

• value (bool)

touch_keytable

extends the lease on a keytable so CacheManager defers eviction

Allowed number of parameters: 1–1

Parameters

• table (table)

Returns

• value (bool)

locktables

acquires WRITE or READ user-level locks on a list of tables (LOCK TABLES); implicitly releases any previously held locks

Allowed number of parameters: 1–1

Parameters

• locks (list): flat list of schema, table, write? triples

Returns

• value (bool)

unlocktables

releases all user-level table locks held by this session

Allowed number of parameters: 0–0

Parameters

This function has no parameters.

Returns

• value (bool)

get_fk_target

returns (ref_table ref_column) if a single-column FK exists for the given column, nil otherwise

Allowed number of parameters: 2–2

Parameters

• table (table)
• column (string): column name

Returns

• value (any)

renametable

renames a table

Allowed number of parameters: 3–3

Parameters

• schema (string): name of the database
• oldname (string): current name of the table
• newname (string): new name of the table

Returns

• value (bool)

insert

inserts a new dataset into table and returns the number of successful items

Allowed number of parameters: 3–7

Parameters

• table (table)
• columns (list): list of column names, e.g. '("ID", "value")
• datasets (list): list of list of column values, e.g. '('(1 10) '(2 15))
• onCollisionCols (list): list of columns of the old dataset that have to be passed to onCollision. Can also request $update, $set:<computed-column>, or NEW.<insert-column>. (optional)
• onCollision (func): function called for each collision. Its positional parameters are the values requested by onCollisionCols, in the same order. If omitted, collisions raise an error. (optional)
  • Parameters
    • column values (any): one value for each onCollisionCols entry (variadic)
  • Returns
    • result (any)
• mergeNull (bool): if true, it will handle NULL values as equal according to SQL 2003's definition of DISTINCT (https://en.wikipedia.org/wiki/Null_(SQL)#When_two_nulls_are_equal:_grouping,_sorting,_and_some_set_operations) (optional)
• onInsertid (func): called once with the first auto_increment id assigned for this INSERT (optional)
  • Parameters
    • id (number): first assigned auto_increment id
  • Returns
    • result (any): ignored callback result

Returns

• value (number)

stat

return system statistics as assoc: mem_available, mem_total, process_memory, shard_memory, shard_budget, persisted_memory, persisted_budget, cache_entry_count, cache_entry_size. (stat schema) and (stat schema tbl) return a string with detailed memory usage.

Allowed number of parameters: 0–2

Parameters

• schema (string): (optional) database name for detailed string output (optional)
• table (string): (optional) table name for detailed string output (optional)

Returns

• value (any)

totalmem

Returns total physical memory in bytes (from /proc/meminfo)

Allowed number of parameters: 0–0

Parameters

This function has no parameters.

Returns

• value (number)

resolve_column_name

resolve a physical column name from immutable table metadata

Allowed number of parameters: 4–4

Parameters

• schema (string): database name
• table (string): table name
• column (string): column name
• ignorecase (bool): whether identifier case is ignored

Returns

• value (string|nil)

show

show databases/tables/columns/shards

(show) lists database names (show schema) lists table names (show table_handle) lists the memoized column defs (show table_handle true) returns table metadata (show table_handle "statistics") returns index statistics (show schema true) lists tables with full info: [{name,engine,row_count,size_bytes,collation,comment},...] (show schema tbl) lists column defs (show schema tbl true) returns assoc {columns,meta,shards} (show schema tbl N) returns shard N overview assoc {shard,state,main_count,delta,deletions,size_bytes} (show schema tbl N true) returns shard N full assoc adding columns and indexes (show schema tbl "statistics") returns INFORMATION_SCHEMA index statistics (show schema tbl "indexes") returns MySQL SHOW INDEX rows

Allowed number of parameters: 0–4

Parameters

• schema_or_table (string|table|recset): (optional) database name or resolved table/recset handle (optional)
• table_or_property (string|bool): (optional) table name, true for full info, or "statistics" for a handle (optional)
• property (int|bool|string): (optional) shard index (int), true for full table info, or "statistics" (optional)
• full (bool): (optional) true to include columns and indexes in shard detail (optional)

Returns

• value (any)

show_triggers

show triggers for a given table

Allowed number of parameters: 1–2

Parameters

• schema (string): database name
• table (string): (optional) table name, if omitted shows all triggers in schema (optional)

Returns

• value (any)

rebuild

rebuilds main storages and returns the amount of time it took; with a table handle, rebuilds only that table

Allowed number of parameters: 0–3

Parameters

• table_or_all (bool|table): table handle for a table-local rebuild; otherwise whether to rebuild unchanged shards globally (default: false) (optional)
• all_or_repartition (bool): with a table: whether to rebuild unchanged shards; globally: whether to repartition (default: true) (optional)
• repartition (bool): with a table handle, whether to repartition that table (default: true) (optional)

Returns

• value (string)

loadCSV

loads a CSV stream into a table and returns the amount of time it took. The first line of the file must be the headlines. The headlines must match the table's columns exactly.

Allowed number of parameters: 3–5

Parameters

• schema (string): name of the database
• table (string): name of the table
• stream (stream): CSV file, load with: (stream filename)
• delimiter (string): (optional) delimiter defaults to ";" (optional)
• firstline (bool): (optional) if the first line contains the column names (otherwise, the tables column order is used) (optional)

Returns

• value (string)

loadJSON

loads a .jsonl file from stream into a database and returns the amount of time it took. JSONL is a linebreak separated file of JSON objects. Each JSON object is one dataset in the database. Before you add rows, you must declare the table in a line '#table <tablename>'. All other lines starting with # are comments. Columns are created dynamically as soon as they occur in a json object.

Allowed number of parameters: 2–2

Parameters

• schema (string): name of the database where you want to put the tables in
• stream (stream): stream of the .jsonl file, read with: (stream filename)

Returns

• value (string)

settings

reads or writes a global settings value. This modifies your data/settings.json.

Allowed number of parameters: 0–2

Parameters

• key (string): name of the key to set or get (for reference, rts) (optional)
• value (any): new value of that setting (optional)

Returns

• value (any)

createcreatetabletrigger

registers a lifecycle trigger that fires synchronously after a future createtable for the given schema/table succeeds

Allowed number of parameters: 6–6

Parameters

• schema (string): name of the database
• table (string): name of the table to watch for creation
• name (string): name of the trigger
• source_sql (string): original SQL body text (for diagnostics)
• body (any): trigger body (Scheme procedure or deferred trigger expression)
• visible (bool): true = user trigger, false = internal trigger

Returns

• value (bool)

dropcreatetabletrigger

removes a registered create-table lifecycle trigger

Allowed number of parameters: 4–4

Parameters

• schema (string): name of the database
• table (string): name of the table watched for creation
• name (string): name of the trigger
• ifexists (bool): don't throw error if trigger doesn't exist

Returns

• value (bool)

createtrigger

creates a new trigger on a table

Allowed number of parameters: 6–6

Parameters

• table (table)
• name (string): name of the trigger
• timing (string): one of: before_insert, after_insert, before_update, after_update, before_delete, after_delete
• source_sql (string): original SQL body text (for SHOW TRIGGERS)
• body (any): trigger body (parsed Scheme expression)
• visible (bool): true = user trigger (shown in SHOW TRIGGERS), false = internal trigger (hidden)

Returns

• value (bool)

droptrigger

removes a trigger from a table

Allowed number of parameters: 3–3

Parameters

• schema (string): name of the database
• name (string): name of the trigger
• ifexists (bool): don't throw error if trigger doesn't exist

Returns

• value (bool)

mysql_import

imports schema+data from a MySQL server into MemCP

Allowed number of parameters: 4–8

Parameters

• host (string|nil): MySQL host (nil => 127.0.0.1)
• port (int|nil): MySQL port (nil => 3306)
• username (string): MySQL username
• password (string): MySQL password
• sourcedb (string|nil): source database (omit/nil => all non-system dbs) (optional)
• targetdb (string|nil): target database (omit/nil => sourcedb) (optional)
• sourcetable (string|nil): source table (omit/nil => all tables in sourcedb) (optional)
• targettable (string|nil): target table (omit/nil => sourcetable) (optional)

Returns

• value (bool)

psql_import

imports schema+data from a PostgreSQL server into MemCP

Allowed number of parameters: 4–9

Parameters

• host (string|nil): PostgreSQL host (nil => 127.0.0.1)
• port (int|nil): PostgreSQL port (nil => 5432)
• username (string): PostgreSQL username
• password (string): PostgreSQL password
• sourcedb (string|nil): source database (omit/nil => all non-system dbs) (optional)
• sourceschema (string|nil): source schema (omit/nil => all non-system schemas in sourcedb) (optional)
• targetdb (string|nil): target database (omit/nil => sourcedb) (optional)
• sourcetable (string|nil): source table (omit/nil => all tables in sourceschema) (optional)
• targettable (string|nil): target table (omit/nil => sourcetable) (optional)

Returns

• value (bool)