Scan - Revision history

Wikiservice: Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference

2026-08-28T10:14:03Z

Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference

← Older revision		Revision as of 12:14, 28 August 2026
Line 15:		Line 15:
	For a query such as:		For a query such as:

	<~~syntaxhighlight lang="sql"~~>		<pre>
	SELECT customer_id, SUM(total)		SELECT customer_id, SUM(total)
	FROM orders		FROM orders
Line 22:		Line 22:
	ORDER BY SUM(total) DESC		ORDER BY SUM(total) DESC
	LIMIT 20;		LIMIT 20;
	</~~syntaxhighlight~~>		</pre>

	the logical plan establishes filtering, grouping, ordering, and LIMIT semantics. Physical lowering can then choose a bounded date range, read only <code>created_at</code>, <code>customer_id</code>, and <code>total</code>, fuse filter/aggregation, and apply top-k braking. Use <code>EXPLAIN PHYSICAL</code> to see what was actually selected; SQL spelling alone does not force an index or RecSet.		the logical plan establishes filtering, grouping, ordering, and LIMIT semantics. Physical lowering can then choose a bounded date range, read only <code>created_at</code>, <code>customer_id</code>, and <code>total</code>, fuse filter/aggregation, and apply top-k braking. Use <code>EXPLAIN PHYSICAL</code> to see what was actually selected; SQL spelling alone does not force an index or RecSet.
Line 46:		Line 46:
	These examples illustrate the callback roles; use the exact current signatures from [[Storage]] when writing low-level code.		These examples illustrate the callback roles; use the exact current signatures from [[Storage]] when writing low-level code.

	<~~syntaxhighlight lang="scheme"~~>		<pre>
	/* resolve the current transaction and table once */		/* resolve the current transaction and table once */
	(set tx ((context "session") "__memcp_tx"))		(set tx ((context "session") "__memcp_tx"))
Line 63:		Line 63:
	(scan tx tbl2 '() (lambda () true)		(scan tx tbl2 '() (lambda () true)
	'("weight") (lambda (weight) weight) + 0)		'("weight") (lambda (weight) weight) + 0)
	</~~syntaxhighlight~~>		</pre>

	An ordered scan additionally receives sort expressions/directions, OFFSET and LIMIT. Filtering and local ordering may run in parallel, but output order and order-sensitive reduction remain serial. Do not use side effects as a substitute for a reducer when result order matters. See [[RecSets]] for ordered membership intersection and progressively filtered candidate batches.		An ordered scan additionally receives sort expressions/directions, OFFSET and LIMIT. Filtering and local ordering may run in parallel, but output order and order-sensitive reduction remain serial. Do not use side effects as a substitute for a reducer when result order matters. See [[RecSets]] for ordered membership intersection and progressively filtered candidate batches.

Wikiservice: Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference

2026-08-28T09:59:36Z

Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference

Show changes

Carli at 05:07, 28 May 2024

2024-05-28T05:07:56Z

← Older revision		Revision as of 07:07, 28 May 2024
Line 1:		Line 1:
	Scan is the most important function in whole MemCP. It implements an optimized, indexed and parallelized <code>for</code> loop over items in a table.		Scan is the most important function in whole MemCP. It implements an optimized, indexed and parallelized <code>for</code> loop over items in a table.

			It represents a data local access operation where each shard can run on a different CPU core fully exploiting cache locality inside the shard. The result of each shard-local scan is then combined to a global result.

	There are two variants of scan: <code>scan</code> and <code>scan_order</code>.		There are two variants of scan: <code>scan</code> and <code>scan_order</code>.

	== (scan schema table filterColumns filter mapColumns map reduce neutral reduce2 isOuter) ==		== Unordered parallel scan + reduce: (scan schema table filterColumns filter mapColumns map reduce neutral reduce2 isOuter) ==
	Help for: scan		Help for: scan
	===		===
Line 39:		Line 41:
	/* equivalent to: */ (scan "schema" "tbl2" '() (lambda () true) '("weight") (lambda (a b) (+ a b)) 0)		/* equivalent to: */ (scan "schema" "tbl2" '() (lambda () true) '("weight") (lambda (a b) (+ a b)) 0)

	== (scan_order schema table filterColumn filter sortcols sortdirs offset limit mapColumns map reduce neutral isOuter) ==		== Ordered scan+reduce: (scan_order schema table filterColumn filter sortcols sortdirs offset limit mapColumns map reduce neutral isOuter) ==
	Help for: scan_order		Help for: scan_order
	===		===

Carli: /* (scan_order schema table filterColumn filter sortcols sortdirs offset limit mapColumns map reduce neutral isOuter) */

2024-05-20T11:04:41Z

(scan_order schema table filterColumn filter sortcols sortdirs offset limit mapColumns map reduce neutral isOuter)

← Older revision		Revision as of 13:04, 20 May 2024
Line 67:		Line 67:
	* Map and Reduce are executed sequentially		* Map and Reduce are executed sequentially
	* Map and Reduce can be pruned by <code>offset</code> and <code>limit</code>		* Map and Reduce can be pruned by <code>offset</code> and <code>limit</code>

			=== Examples ===
			Print all items from tbl3(id, weight) ordered by <code>weight DESC</code>
			(scan "schema" "tbl3" '() (lambda () true) '("weight") '(true) 0 -1 '("id" "weight") (lambda (id weight) (print id "," weight)))

Carli at 11:02, 20 May 2024

2024-05-20T11:02:13Z

@@ Line 21: / Line 21: @@
    - reduce2 (func): (optional) second stage reduce function that will apply a result of reduce to the neutral element/accumulator
    - isOuter (bool): (optional) if true, in case of no hits, call map once anyway with NULL values
 == (scan_order schema table filterColumn filter sortcols sortdirs offset limit mapColumns map reduce neutral isOuter) ==
@@ Line 43: / Line 60: @@
    - neutral (any): (optional) neutral element for the reduce phase, otherwise nil is assumed
    - isOuter (bool): (optional) if true, in case of no hits, call map once anyway with NULL values

Carli: Created page with "Scan is the most important function in whole MemCP. It implements an optimized, indexed and parallelized `for` loop over items in a table. There are two variants of scan: `scan` and `scan_order`. == (scan schema table filterColumns filter mapColumns map reduce neutral reduce2 isOuter) == Help for: scan === does an unordered parallel filter-map-reduce pass on a single table and returns the reduced result Allowed nø of parameters..."

2024-05-20T10:54:05Z

Created page with "Scan is the most important function in whole MemCP. It implements an optimized, indexed and parallelized <code>for</code> loop over items in a table. There are two variants of scan: <code>scan</code> and <code>scan_order</code>. == (scan schema table filterColumns filter mapColumns map reduce neutral reduce2 isOuter) == Help for: scan === does an unordered parallel filter-map-reduce pass on a single table and returns the reduced result Allowed nø of parameters..."

New page

Scan is the most important function in whole MemCP. It implements an optimized, indexed and parallelized <code>for</code> loop over items in a table.

There are two variants of scan: <code>scan</code> and <code>scan_order</code>.

== (scan schema table filterColumns filter mapColumns map reduce neutral reduce2 isOuter) ==
Help for: scan
===

does an unordered parallel filter-map-reduce pass on a single table and returns the reduced result

Allowed nø of parameters: 6 - 10

- schema (string|nil): database where the table is located
- table (string|list): name of the table to scan (or a list if you have temporary data)
- filterColumns (list): list of columns that are fed into filter
- filter (func): lambda function that decides whether a dataset is passed to the map phase. You can use any column of that table as lambda parameter. You should structure your lambda with an (and) at the root element. Every equal? < > <= >= will possibly translated to an indexed scan
- mapColumns (list): list of columns that are fed into map
- map (func): lambda function to extract data from the dataset. You can use any column of that table as lambda parameter. You can return a value you want to extract and pass to reduce, but you can also directly call insert, print or resultrow functions. If you declare a parameter named '$update', this variable will hold a function that you can use to delete or update a row. Call ($update) to delete the dataset, call ($update '("field1" value1 "field2" value2)) to update certain columns.
- reduce (func): (optional) lambda function to aggregate the map results. It takes two parameters (a b) where a is the accumulator and b the new value. The accumulator for the first reduce call is the neutral element. The return value will be the accumulator input for the next reduce call. There are two reduce phases: shard-local and shard-collect. In the shard-local phase, a starts with neutral and b is fed with the return values of each map call. In the shard-collect phase, a starts with neutral and b is fed with the result of each shard-local pass.
- neutral (any): (optional) neutral element for the reduce phase, otherwise nil is assumed
- reduce2 (func): (optional) second stage reduce function that will apply a result of reduce to the neutral element/accumulator
- isOuter (bool): (optional) if true, in case of no hits, call map once anyway with NULL values

== (scan_order schema table filterColumn filter sortcols sortdirs offset limit mapColumns map reduce neutral isOuter) ==
Help for: scan_order
===

does an ordered parallel filter and serial map-reduce pass on a single table and returns the reduced result

Allowed nø of parameters: 10 - 13

- schema (string): database where the table is located
- table (string): name of the table to scan
- filterColumns (list): list of columns that are fed into filter
- filter (func): lambda function that decides whether a dataset is passed to the map phase. You can use any column of that table as lambda parameter. You should structure your lambda with an (and) at the root element. Every equal? < > <= >= will possibly translated to an indexed scan
- sortcols (list): list of columns to sort. Each column is either a string to point to an existing column or a func(cols...)->any to compute a sortable value
- sortdirs (list): list of column directions to sort. Must be same length as sortcols. false means ASC, true means DESC
- offset (number): number of items to skip before the first one is fed into map
- limit (number): max number of items to read
- mapColumns (list): list of columns that are fed into map
- map (func): lambda function to extract data from the dataset. You can use any column of that table as lambda parameter. You can return a value you want to extract and pass to reduce, but you can also directly call insert, print or resultrow functions. If you declare a parameter named '$update', this variable will hold a function that you can use to delete or update a row. Call ($update) to delete the dataset, call ($update '("field1" value1 "field2" value2)) to update certain columns.
- reduce (func): (optional) lambda function to aggregate the map results. It takes two parameters (a b) where a is the accumulator and b the new value. The accumulator for the first reduce call is the neutral element. The return value will be the accumulator input for the next reduce call. There are two reduce phases: shard-local and shard-collect. In the shard-local phase, a starts with neutral and b is fed with the return values of each map call. In the shard-collect phase, a starts with neutral and b is fed with the result of each shard-local pass.
- neutral (any): (optional) neutral element for the reduce phase, otherwise nil is assumed
- isOuter (bool): (optional) if true, in case of no hits, call map once anyway with NULL values