Storage - Revision history

Wikiservice: Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference

2026-08-28T10:14:04Z

Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference

Wikiservice: Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference

2026-08-28T09:59:37Z

Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference

Show changes

Carli at 06:20, 27 August 2026

2026-08-27T06:20:08Z

Show changes

Carli: Created page with "= Storage = The '''Storage''' module provides functions to manage databases, tables, columns, keys, partitions, and large data operations in SCM. ← Back to Full SCM API documentation == scan == Performs an unordered parallel filter-map-reduce on a table '''Allowed number of parameters:''' 6–10 '''Parameters:''' * '''schema''' (`string|nil`): database name * '''table''' (`string|list`): table name(s) * '''filterColumns''' (list 2025-08-25T18:10:16Z Created page with "= Storage = The '''Storage''' module provides functions to manage databases, tables, columns, keys, partitions, and large data operations in SCM. ← Back to Full SCM API documentation == scan == Performs an unordered parallel filter-map-reduce on a table '''Allowed number of parameters:''' 6–10 '''Parameters:''' * '''schema''' (<code>string|nil</code>): database name * '''table''' (<code>string|list</code>): table name(s) * '''filterColumns''' (<code>list</c..." New page = Storage = The '''Storage''' module provides functions to manage databases, tables, columns, keys, partitions, and large data operations in SCM. ← Back to [[Full SCM API documentation]] == scan == Performs an unordered parallel filter-map-reduce on a table '''Allowed number of parameters:''' 6–10 '''Parameters:''' * '''schema''' (<code>string|nil</code>): database name * '''table''' (<code>string|list</code>): table name(s) * '''filterColumns''' (<code>list</code>): filter columns * '''filter''' (<code>func</code>): lambda deciding dataset inclusion * '''mapColumns''' (<code>list</code>): map columns * '''map''' (<code>func</code>): lambda extracting/processing data * '''reduce''' (<code>func</code>): optional aggregator * '''neutral''' (<code>any</code>): neutral element * '''reduce2''' (<code>func</code>): optional second reducer * '''isOuter''' (<code>bool</code>): outer join–like behavior '''Returns:''' <code>any</code> == scan_order == Performs an ordered parallel filter with serial map-reduce '''Allowed number of parameters:''' 10–13 '''Parameters:''' * '''schema''' (<code>string</code>): database * '''table''' (<code>string</code>): table name * '''filterColumns''' (<code>list</code>): filter columns * '''filter''' (<code>func</code>): filter lambda * '''sortcols''' (<code>list</code>): sorting columns * '''sortdirs''' (<code>list</code>): sorting directions * '''offset''' (<code>number</code>): skip count * '''limit''' (<code>number</code>): max items * '''mapColumns''' (<code>list</code>): map columns * '''map''' (<code>func</code>): map lambda * '''reduce''' (<code>func</code>): optional reducer * '''neutral''' (<code>any</code>): neutral element * '''isOuter''' (<code>bool</code>): outer behavior '''Returns:''' <code>any</code> == createdatabase == Creates a new database '''Allowed number of parameters:''' 1–2 '''Parameters:''' * '''schema''' (<code>string</code>): database name * '''ignoreexists''' (<code>bool</code>): ignore if exists '''Returns:''' <code>bool</code> == dropdatabase == Drops a database '''Allowed number of parameters:''' 1–1 '''Parameters:''' * '''schema''' (<code>string</code>): database name '''Returns:''' <code>bool</code> == createtable == Creates a new table '''Allowed number of parameters:''' 4–5 '''Parameters:''' * '''schema''' (<code>string</code>): database * '''table''' (<code>string</code>): table name * '''cols''' (<code>list</code>): column definitions * '''options''' (<code>list</code>): table options * '''ifnotexists''' (<code>bool</code>): safe creation '''Returns:''' <code>bool</code> == createcolumn == Creates a new column in a table '''Allowed number of parameters:''' 6–8 '''Parameters:''' * '''schema''' (<code>string</code>): database * '''table''' (<code>string</code>): table name * '''colname''' (<code>string</code>): column name * '''type''' (<code>string</code>): base type * '''dimensions''' (<code>list</code>): type dimensions * '''options''' (<code>list</code>): column options * '''computorCols''' (<code>list</code>): columns for computed value * '''computor''' (<code>func</code>): compute lambda '''Returns:''' <code>bool</code> == createkey == Creates a new key '''Allowed number of parameters:''' 5–5 '''Parameters:''' * '''schema''' (<code>string</code>): database * '''table''' (<code>string</code>): table * '''keyname''' (<code>string</code>): key name * '''unique''' (<code>bool</code>): uniqueness flag * '''columns''' (<code>list</code>): key columns '''Returns:''' <code>bool</code> == createforeignkey == Creates a new foreign key '''Allowed number of parameters:''' 8–8 '''Parameters:''' * '''schema''' (<code>string</code>): database * '''keyname''' (<code>string</code>): key name * '''table1''' (<code>string</code>): first table * '''columns1''' (<code>list</code>): first columns * '''table2''' (<code>string</code>): second table * '''columns2''' (<code>list</code>): second columns * '''updatemode''' (<code>string</code>): action on update * '''deletemode''' (<code>string</code>): action on delete '''Returns:''' <code>bool</code> == shardcolumn == Suggests partitions for a column '''Allowed number of parameters:''' 3–4 '''Parameters:''' * '''schema''' (<code>string</code>): database * '''table''' (<code>string</code>): table * '''colname''' (<code>string</code>): column * '''numpartitions''' (<code>number</code>): partitions '''Returns:''' <code>list</code> == partitiontable == Applies or adjusts a partition scheme '''Allowed number of parameters:''' 3–3 '''Parameters:''' * '''schema''' (<code>string</code>): database * '''table''' (<code>string</code>): table * '''columns''' (<code>list</code>): column pivots '''Returns:''' <code>bool</code> == altertable == Alters a table '''Allowed number of parameters:''' 4–4 '''Parameters:''' * '''schema''' (<code>string</code>): database * '''table''' (<code>string</code>): table * '''operation''' (<code>string</code>): alteration * '''parameter''' (<code>any</code>): parameter '''Returns:''' <code>bool</code> == altercolumn == Alters a column '''Allowed number of parameters:''' 5–5 '''Parameters:''' * '''schema''' (<code>string</code>): database * '''table''' (<code>string</code>): table * '''column''' (<code>string</code>): column * '''operation''' (<code>string</code>): alteration * '''parameter''' (<code>any</code>): parameter '''Returns:''' <code>bool</code> == droptable == Removes a table '''Allowed number of parameters:''' 2–3 '''Parameters:''' * '''schema''' (<code>string</code>): database * '''table''' (<code>string</code>): table * '''ifexists''' (<code>bool</code>): safe drop '''Returns:''' <code>bool</code> == insert == Inserts a dataset into a table '''Allowed number of parameters:''' 4–7 '''Parameters:''' * '''schema''' (<code>string</code>): database * '''table''' (<code>string</code>): table * '''columns''' (<code>list</code>): column names * '''datasets''' (<code>list</code>): row values * '''onCollisionCols''' (<code>list</code>): collision columns * '''onCollision''' (<code>func</code>): collision handler * '''mergeNull''' (<code>bool</code>): handle null as equal '''Returns:''' <code>number</code> == stat == Returns memory statistics '''Allowed number of parameters:''' 0–2 '''Parameters:''' * '''schema''' (<code>string</code>): database (optional) * '''table''' (<code>string</code>): table (optional) '''Returns:''' <code>string</code> == show == Shows databases, tables, or columns '''Allowed number of parameters:''' 0–2 '''Parameters:''' * '''schema''' (<code>string</code>): optional database * '''table''' (<code>string</code>): optional table '''Returns:''' <code>any</code> == rebuild == Rebuilds storages '''Allowed number of parameters:''' 0–2 '''Parameters:''' * '''all''' (<code>bool</code>): rebuild all * '''repartition''' (<code>bool</code>): repartition '''Returns:''' <code>string</code> == loadCSV == Loads CSV into a table '''Allowed number of parameters:''' 3–5 '''Parameters:''' * '''schema''' (<code>string</code>): database * '''table''' (<code>string</code>): table * '''stream''' (<code>stream</code>): CSV stream * '''delimiter''' (<code>string</code>): delimiter * '''firstline''' (<code>bool</code>): use header '''Returns:''' <code>string</code> == loadJSON == Loads JSONL into a database '''Allowed number of parameters:''' 2–2 '''Parameters:''' * '''schema''' (<code>string</code>): database * '''stream''' (<code>stream</code>): JSONL stream '''Returns:''' <code>string</code> == settings == Reads or writes global settings '''Allowed number of parameters:''' 1–2 '''Parameters:''' * '''key''' (<code>string</code>): setting key * '''value''' (<code>any</code>): new value '''Returns:''' <code>any</code>

← Older revision		Revision as of 12:14, 28 August 2026
Line 293:		Line 293:
	** '''Returns'''		** '''Returns'''
	*** '''result''' (<code>bool</code>): true when the row proceeds to map		*** '''result''' (<code>bool</code>): true when the row proceeds to map
	* '''mapColumns''' (<code>list<string></code>): physical columns passed to map after filtering; pseudo columns are $update (update/delete current row), $recset_contains (row-bound RecSet membership), $set:<column>, $increment:<column>, and $invalidate:<column> (computed-column maintenance), plus NEW.<column> in trigger plans		* '''mapColumns''' (<code>list<string></code>): physical columns passed to map after filtering; pseudo columns are $update (update/delete current row), $recset_contains (row-bound RecSet membership), $set:<column>, $increment:<column>, and $invalidate:<column> (computed-column maintenance), plus NEW.<column> in trigger plans
	** '''column''' (<code>string</code>): column name passed to the corresponding callback parameter		** '''column''' (<code>string</code>): column name passed to the corresponding callback parameter
	* '''map''' (<code>func</code>): lambda function that extracts or produces one value from the row; it may also use documented pseudo columns for mutations or result output		* '''map''' (<code>func</code>): lambda function that extracts or produces one value from the row; it may also use documented pseudo columns for mutations or result output
Line 400:		Line 400:
	* '''offset''' (<code>number</code>): number of batch-filter-accepted rows to skip; it is not the number of driver candidates already examined		* '''offset''' (<code>number</code>): number of batch-filter-accepted rows to skip; it is not the number of driver candidates already examined
	* '''limit''' (<code>number</code>): finite maximum number of accepted rows passed to map; the initial candidate batch size is offset+limit and doubles for every subsequent batch		* '''limit''' (<code>number</code>): finite maximum number of accepted rows passed to map; the initial candidate batch size is offset+limit and doubles for every subsequent batch
	* '''mapColumns''' (<code>list<string></code>): physical columns passed to map after filtering; pseudo columns are $update (update/delete current row), $recset_contains (row-bound RecSet membership), $set:<column>, $increment:<column>, and $invalidate:<column> (computed-column maintenance), plus NEW.<column> in trigger plans; $break is reserved for internal ORC convergence and must not implement SQL OFFSET/LIMIT, which belong in the native offset and limit arguments		* '''mapColumns''' (<code>list<string></code>): physical columns passed to map after filtering; pseudo columns are $update (update/delete current row), $recset_contains (row-bound RecSet membership), $set:<column>, $increment:<column>, and $invalidate:<column> (computed-column maintenance), plus NEW.<column> in trigger plans; $break is reserved for internal ORC convergence and must not implement SQL OFFSET/LIMIT, which belong in the native offset and limit arguments
	** '''column''' (<code>string</code>): column name passed to the corresponding callback parameter		** '''column''' (<code>string</code>): column name passed to the corresponding callback parameter
	* '''map''' (<code>func</code>): same map callback contract as scan_order; accepted record IDs are passed to its shard mapper in batches		* '''map''' (<code>func</code>): same map callback contract as scan_order; accepted record IDs are passed to its shard mapper in batches
Line 456:		Line 456:
	* '''offset''' (<code>number</code>): number of globally ordered, filter-accepted items to skip before map; apply SQL OFFSET here rather than in map		* '''offset''' (<code>number</code>): number of globally ordered, filter-accepted items to skip before map; apply SQL OFFSET here rather than in map
	* '''limit''' (<code>number</code>): maximum globally ordered, filter-accepted items passed to map; -1 means unlimited; apply SQL LIMIT here so shard-local Top-K and the global merge can brake early		* '''limit''' (<code>number</code>): maximum globally ordered, filter-accepted items passed to map; -1 means unlimited; apply SQL LIMIT here so shard-local Top-K and the global merge can brake early
	* '''mapColumns''' (<code>list<string></code>): physical columns passed to map after filtering; pseudo columns are $update (update/delete current row), $recset_contains (row-bound RecSet membership), $set:<column>, $increment:<column>, and $invalidate:<column> (computed-column maintenance), plus NEW.<column> in trigger plans; $break is reserved for internal ORC convergence and must not implement SQL OFFSET/LIMIT, which belong in the native offset and limit arguments		* '''mapColumns''' (<code>list<string></code>): physical columns passed to map after filtering; pseudo columns are $update (update/delete current row), $recset_contains (row-bound RecSet membership), $set:<column>, $increment:<column>, and $invalidate:<column> (computed-column maintenance), plus NEW.<column> in trigger plans; $break is reserved for internal ORC convergence and must not implement SQL OFFSET/LIMIT, which belong in the native offset and limit arguments
	** '''column''' (<code>string</code>): column name passed to the corresponding callback parameter		** '''column''' (<code>string</code>): column name passed to the corresponding callback parameter
	* '''map''' (<code>func</code>): lambda function that extracts or produces one value from each accepted row		* '''map''' (<code>func</code>): lambda function that extracts or produces one value from each accepted row
Line 1,080:		Line 1,080:
	* '''columns''' (<code>list</code>): list of column names, e.g. '("ID", "value")		* '''columns''' (<code>list</code>): list of column names, e.g. '("ID", "value")
	* '''datasets''' (<code>list</code>): list of list of column values, e.g. '('(1 10) '(2 15))		* '''datasets''' (<code>list</code>): list of list of column values, e.g. '('(1 10) '(2 15))
	* '''onCollisionCols''' (<code>list</code>): list of columns of the old dataset that have to be passed to onCollision. Can also request $update, $set:<computed-column>, or NEW.<insert-column>. ''(optional)''		* '''onCollisionCols''' (<code>list</code>): list of columns of the old dataset that have to be passed to onCollision. Can also request $update, $set:<computed-column>, or NEW.<insert-column>. ''(optional)''
	* '''onCollision''' (<code>func</code>): function called for each collision. Its positional parameters are the values requested by onCollisionCols, in the same order. If omitted, collisions raise an error. ''(optional)''		* '''onCollision''' (<code>func</code>): function called for each collision. Its positional parameters are the values requested by onCollisionCols, in the same order. If omitted, collisions raise an error. ''(optional)''
	** '''Parameters'''		** '''Parameters'''
Line 1,230:		Line 1,230:
	== loadJSON ==		== loadJSON ==

	loads a .jsonl file from stream into a database and returns the amount of time it took. JSONL is a linebreak separated file of JSON objects. Each JSON object is one dataset in the database. Before you add rows, you must declare the table in a line '#table <tablename>'. All other lines starting with # are comments. Columns are created dynamically as soon as they occur in a json object.		loads a .jsonl file from stream into a database and returns the amount of time it took. JSONL is a linebreak separated file of JSON objects. Each JSON object is one dataset in the database. Before you add rows, you must declare the table in a line '#table <tablename>'. All other lines starting with # are comments. Columns are created dynamically as soon as they occur in a json object.

	'''Allowed number of parameters:''' 2–2		'''Allowed number of parameters:''' 2–2