Introduction to Scheme: Difference between revisions

From MemCP
Jump to navigation Jump to search
(Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference)
 
(2 intermediate revisions by 2 users not shown)
Line 1: Line 1:
When you run <code>./memcp</code>, you will be dropped at a scheme shell like this:
<!-- Copyright (C) 2026 Carl-Philip Haensch -->
memcp Copyright (C) 2023, 2024  Carl-Philip Hänsch
<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
    This program comes with ABSOLUTELY NO WARRANTY;
= Introduction to Scheme =
    This is free software, and you are welcome to redistribute it
    under certain conditions;
loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-id of type 10
loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-username of type 20
loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-password of type 20
Welcome to memcp
performing unit tests ...
finished unit tests
test result: 15/15
all tests succeeded.
Initializing SQL frontend
MySQL server listening on port 3307 (connect with `mysql -P 3307 -u root -p` using password 'admin')
listening on <nowiki>http://localhost:4321</nowiki>
    Type (help) to show help
Scheme is a functional programming language and a subset of LISP. Every expression is either a primitive value or a list.


To understand the semantics of Scheme, take a look at the following Scheme REPL session:
MemCP embeds a small Scheme dialect used by the SQL parsers, query planner, modules, and application endpoints. Start the interactive console with <code>./memcp lib/main.scm</code>; background services use <code>--no-repl</code>.
> 12
= 12
> "hi"
= "hi"
> (+ 1 2)
= 3
> (+ 1 2 3)
= 6
> (+ 2 (* 2 2))
= 6
> (+ 2 (* 2 4))
= 10
> (concat "Hello" "World")
= "HelloWorld"


== Syntax ==
Scheme represents calls and code with the same list syntax. That makes the dialect well suited to MemCP's query compiler: the SQL planner constructs Scheme programs as data, optimizes them, and evaluates or compiles them.


=== Function Call ===
The console prints each expression result, which makes the evaluation model easy to explore:
A function call has the following format:
(functionname param1 param2 param3 ...)
Function calls start with <code>(</code>, contain one function and zero or more parameters separated by space  .


Some basic functions are <code>+ - * / print concat</code>. For more consult the manual with <code>(help)</code>.
<pre>&gt; 12
= 12
&gt; (+ 1 2 3)
= 6
&gt; (+ 2 (* 2 4))
= 10
&gt; (concat "Hello " "World")
= "Hello World"</pre>


=== Literals ===
Startup also loads the configured data directory and Scheme modules before opening the SQL/HTTP listeners. The exact banner and number of startup tests change between releases, so scripts should not parse that text.
The following literals are allowed:


* Number Literals: <code>1 2 3 56 7.5 4.3e20 -6.123e-3</code>
== Values and calls ==
* String Literals: <code>"Hello World" "First Line\nSecond Line" "he said: \"what?\" and smiled"</code>
* Function names: <code>print</code>
* Symbol literals <code>'print</code>
* List literals <code>'(1 2 3) '("a" "b" "c")</code>
* Associative Array literals <code>'("key" "value" "size" 45 "name" "Peter" "sublist" '(1 2 3))</code> - associative arrays are lists with key value pairs flattened down to a one dimensional list of an even number of items.
* Lambdas: <code>(lambda (param1 param2 ...) body)</code>


=== Lambda Functions ===
<pre>
Scheme allows for creating lambda functions which enclose their own scope:
(+ 1 2)
(define decrease-by-one (lambda (number) (- number 1)))
(concat "hello " "world")
The function can now be called from scheme:
(map '(1 2 3) (lambda (x) (* x x)))
> (decrease-by-one 5)
</pre>
= 4
Lambda functions can be passed to other functions as values. They enclose your original scope, so you can use variables from the outside inside your function.


== Deeper Topics into Scheme ==
Numbers, booleans, strings, symbols, lists, functions, sessions, and storage objects are ordinary values. A list in expression position is a call: its first item is the procedure and the rest are arguments. Whitespace separates forms; parentheses define nesting.
You can deep-dive into the following topcis:


* [[Lists and Objects]]
{| class="wikitable"
* [[Pattern Matching]]
! Form !! Meaning
* [[Parsers]]
|-
| <code>42</code>, <code>3.5</code>, <code>true</code>, <code>"text"</code> || Literal values
|-
| <code>(+ 1 2)</code> || Call <code>+</code> with two arguments
|-
| <code>'name</code> || The symbol <code>name</code>, not the value bound to it
|-
| <code>'(1 2 3)</code> || Construct a list value instead of calling <code>1</code>
|-
| <code>'("name" "Ada" "active" true)</code> || Flat key/value list used as an associative object
|}


== Further Help and Documentation of all Functions ==
== Quoting and generated code ==
If you type <code>(help)</code> into the console, you will get the following overview:
 
> (help)
Use quoting to build code or data: <code>'(+ 4 5)</code> constructs delayed code, while <code>(eval '(+ 4 5))</code> evaluates it. Values that should be computed now can be embedded into a quoted outer form; procedure symbols whose execution is delayed remain quoted. This distinction matters throughout the query planner.
Available scm functions:
 
<pre>
-- SCM Builtins --
(set delayed '(+ 4 5))
  quote: returns a symbol or list without evaluating it
(eval delayed)                /* 9 */
  eval: executes the given scheme program in the current environment
(map '(1 2 3) (lambda (x) (* x x)))
  optimize: optimize the given scheme program
</pre>
  if: checks a condition and then conditionally evaluates code branches; there might be multiple condition+true-branch clauses
 
  and: returns true if all conditions evaluate to true
Do not add or remove parentheses by visual guesswork in large generated expressions. Repository Scheme files are formatted and checked with <code>python3 tools/lint_scm.py</code>.
  or: returns true if at least one condition evaluates to true
 
  coalesce: returns the first value that has a non-zero value
== Functions and scope ==
  coalesceNil: returns the first value that has a non-nil value
 
  define: defines or sets a variable in the current environment
Lambdas have exactly a parameter list and one body. Use <code>begin</code> when that body needs several forms:
  set: defines or sets a variable in the current environment
 
  error: halts the whole execution thread and throws an error message
<pre>
  try: tries to execute a function and returns its result. In case of a failure, the error is fed to the second function and its result value will be used
(define greet (lambda (name) (begin
  apply: runs the function with its arguments
(print "greeting " name)
  apply_assoc: runs the function with its arguments but arguments is a assoc list
(concat "Hello, " name))))
  symbol: returns a symbol built from that string
 
  list: returns a list containing the parameters as alements
(greet "Ada")
  string: converts the given value into string
</pre>
  match: takes a value evaluates the branch that first matches the given pattern
 
  lambda: returns a function (func) constructed from the given code
Functions are closures and may be passed to <code>map</code>, reducers, scanners, parsers, and handlers. <code>define</code> and <code>set</code> create a binding in the current scope in this dialect; <code>set</code> is not an imperative mutation of an outer lexical variable.
  begin: creates a own variable scope, evaluates all sub expressions and returns the result of the last one
 
  parallel: executes all parameters in parallel and returns nil if they are finished
== Pattern matching and modules ==
  source: annotates the node with filename and line information for better backtraces
 
<code>match</code> is the usual way to express alternatives and destructure lists, strings, or regular-expression results. Modules are loaded with <code>import</code>; application entrypoints typically import the required library files and then install handlers or start a server.
-- Arithmetic / Logic --
 
  number?: tells if the value is a number
<pre>
  +: adds two or more numbers
(match value
  -: subtracts two or more numbers from the first one
'(x y) (+ x y)
  *: multiplies two or more numbers
(regex "^/users/([0-9]+)$" path id) id
  /: divides two or more numbers from the first one
false)
  <=: compares two numbers
</pre>
  <: compares two numbers
 
  >: compares two numbers
See [[Parsers]], [[Lists and Objects]], and the matching examples in the repository for the exact pattern forms.
  >=: compares two numbers
 
  equal?: deep-compares two values of the same type
== State and concurrency ==
  equal??: performs a sloppy equality check on primitive values (number, string, bool. nil), strings are compared case insensitive
 
  !: negates the boolean value
The dialect is functional by default. <code>set</code> defines a binding in the current scope; it does not imperatively mutate an outer binding. Use <code>(newsession)</code> when code intentionally needs a thread-safe mutable key/value context shared across parallel work.
  not: negates the boolean value
 
  nil?: returns true if value is nil
<pre>
  min: returns the smallest value
(set request_state (newsession))
  max: returns the highest value
(request_state "user_id" 42)
  floor: rounds the number down
(request_state "user_id")       /* 42 */
  ceil: rounds the number up
</pre>
  round: rounds the number
 
Independent functional work may run in parallel. Supported hot procedures can be compiled by the native x86-64 JIT; unsupported shapes remain interpreted without changing language semantics. See [[Full SCM API documentation]], [[Parallel Computing]], and [[In-Database WebApps and REST Services]].
-- Strings --
 
  string?: tells if the value is a string
== Where to continue ==
  concat: concatenates stringable values and returns a string
 
  substr: returns a substring
* [[Full SCM API documentation]] lists generated chapters and signatures.
  simplify: turns a stringable input value in the easiest-most value (e.g. turn strings into numbers if they are numeric
* [[Lists and Objects]] explains lists, associative objects, mapping, and reduction.
  strlen: returns the length of a string
* [[IO]] covers files, HTTP request/response objects, and servers.
  strlike: matches the string against a wildcard pattern (SQL compliant)
* [[Storage]] documents the low-level table and scan interface; application code normally starts with SQL.
  toLower: turns a string into lower case
* [[JIT Compilation]] and [[Parallel Computing]] explain runtime optimization boundaries.
  toUpper: turns a string into upper case
  replace: replaces all occurances in a string with another string
  split: splits a string using a separator or space
  htmlentities: escapes the string for use in HTML
  urlencode: encodes a string according to URI coding schema
  urldecode: decodes a string according to URI coding schema
-- Lists --
  append: appends items to a list and return the extended list.
  append_unique: appends items to a list but only if they are new.
  cons: constructs a list from a head and a tail list
  car: extracts the head of a list
  cdr: extracts the tail of a list
  merge: flattens a list of lists into a list containing all the subitems. If one parameter is given, it is a list of lists that is flattened. If multiple parameters are given, they are treated as lists that will be merged into one
  merge_unique: flattens a list of lists into a list containing all the subitems. Duplicates are filtered out.
  has?: checks if a list has a certain item (equal?)
  filter: returns a list that only contains elements that pass the filter function
  map: returns a list that contains the results of a map function that is applied to the list
  mapIndex: returns a list that contains the results of a map function that is applied to the list
  reduce: returns a list that contains the result of a map function
  produce: returns a list that contains produced items - it works like for(state = startstate, condition(state), state = iterator(state)) {yield state}
  produceN: returns a list with numbers from 0..n-1
  list?: checks if a value is a list
  contains?: checks if a value is in a list; uses the equal?? operator
-- Associative Lists / Dictionaries --
  filter_assoc: returns a filtered dictionary according to a filter function
  map_assoc: returns a mapped dictionary according to a map function
  reduce_assoc: reduces a dictionary according to a reduce function
  has_assoc?: checks if a dictionary has a key present
  extract_assoc: applies a function (key value) on the dictionary and returns the results as a flat list
  set_assoc: returns a dictionary where a single value has been changed.
  merge_assoc: returns a dictionary where all keys from dict1 and all keys from dict2 are present.
-- Parsers --
  parser: creates a parser
-- Sync --
  newsession: Creates a new session which is a threadsafe key-value store represented as a function that can be either called as a getter (session key) or setter (session key value) or list all keys with (session)
  once: Creates a function wrapper that you can call multiple times but only gets executed once. The result value is cached and returned on a second call. You can add parameters to that resulting function that will be passed to the first run of the wrapped function.
  mutex: Creates a mutex. The return value is a function that takes one parameter which is a parameterless function. The mutex is guaranteed that all calls to that mutex get serialized.
-- IO --
  print: Prints values to stdout (only in IO environment)
  help: Lists all functions or print help for a specific function
  import: Imports a file .scm file into current namespace
  load: Loads a file and returns the string
  serve: Opens a HTTP server at a given port
  mysql: Imports a file .scm file into current namespace
  password: Hashes a password with sha1 (for mysql user authentication)
-- Storage --
  [[scan]]: does an unordered parallel filter-map-reduce pass on a single table and returns the reduced result
  [[Scan|scan_order]]: does an ordered parallel filter and serial map-reduce pass on a single table and returns the reduced result
  createdatabase: creates a new database
  dropdatabase: creates a new database
  createtable: creates a new database
  createcolumn: creates a new column in table
  shardcolumn: tells us how it would partition a column according to their values. Returns a list of pivot elements.
  altertable: alters a table
  droptable: removes a table
  insert: inserts a new dataset into table
  stat: return memory statistics
  show: show databases/tables/columns
  rebuild: rebuilds all main storages and returns the amount of time it took
  loadCSV: loads a CSV file into a table and returns the amount of time it took.
  loadJSON: loads a .jsonl file from disk into a database and returns the amount of time it took.
  settings: reads or writes a global settings value. This modifies your data/settings.json.
get further information by typing (help "functionname") to get more info

Latest revision as of 12:13, 28 August 2026

Introduction to Scheme

MemCP embeds a small Scheme dialect used by the SQL parsers, query planner, modules, and application endpoints. Start the interactive console with ./memcp lib/main.scm; background services use --no-repl.

Scheme represents calls and code with the same list syntax. That makes the dialect well suited to MemCP's query compiler: the SQL planner constructs Scheme programs as data, optimizes them, and evaluates or compiles them.

The console prints each expression result, which makes the evaluation model easy to explore:

> 12
= 12
> (+ 1 2 3)
= 6
> (+ 2 (* 2 4))
= 10
> (concat "Hello " "World")
= "Hello World"

Startup also loads the configured data directory and Scheme modules before opening the SQL/HTTP listeners. The exact banner and number of startup tests change between releases, so scripts should not parse that text.

Values and calls

(+ 1 2)
(concat "hello " "world")
(map '(1 2 3) (lambda (x) (* x x)))

Numbers, booleans, strings, symbols, lists, functions, sessions, and storage objects are ordinary values. A list in expression position is a call: its first item is the procedure and the rest are arguments. Whitespace separates forms; parentheses define nesting.

Form Meaning
42, 3.5, true, "text" Literal values
(+ 1 2) Call + with two arguments
'name The symbol name, not the value bound to it
'(1 2 3) Construct a list value instead of calling 1
'("name" "Ada" "active" true) Flat key/value list used as an associative object

Quoting and generated code

Use quoting to build code or data: '(+ 4 5) constructs delayed code, while (eval '(+ 4 5)) evaluates it. Values that should be computed now can be embedded into a quoted outer form; procedure symbols whose execution is delayed remain quoted. This distinction matters throughout the query planner.

(set delayed '(+ 4 5))
(eval delayed)                 /* 9 */
(map '(1 2 3) (lambda (x) (* x x)))

Do not add or remove parentheses by visual guesswork in large generated expressions. Repository Scheme files are formatted and checked with python3 tools/lint_scm.py.

Functions and scope

Lambdas have exactly a parameter list and one body. Use begin when that body needs several forms:

(define greet (lambda (name) (begin
	(print "greeting " name)
	(concat "Hello, " name))))

(greet "Ada")

Functions are closures and may be passed to map, reducers, scanners, parsers, and handlers. define and set create a binding in the current scope in this dialect; set is not an imperative mutation of an outer lexical variable.

Pattern matching and modules

match is the usual way to express alternatives and destructure lists, strings, or regular-expression results. Modules are loaded with import; application entrypoints typically import the required library files and then install handlers or start a server.

(match value
	'(x y) (+ x y)
	(regex "^/users/([0-9]+)$" path id) id
	false)

See Parsers, Lists and Objects, and the matching examples in the repository for the exact pattern forms.

State and concurrency

The dialect is functional by default. set defines a binding in the current scope; it does not imperatively mutate an outer binding. Use (newsession) when code intentionally needs a thread-safe mutable key/value context shared across parallel work.

(set request_state (newsession))
(request_state "user_id" 42)
(request_state "user_id")       /* 42 */

Independent functional work may run in parallel. Supported hot procedures can be compiled by the native x86-64 JIT; unsupported shapes remain interpreted without changing language semantics. See Full SCM API documentation, Parallel Computing, and In-Database WebApps and REST Services.

Where to continue