|
|
| (3 intermediate revisions by 2 users not shown) |
| Line 1: |
Line 1: |
| When you run <code>./memcp</code>, you will be dropped at a scheme shell like this:
| | <!-- Copyright (C) 2026 Carl-Philip Haensch --> |
| memcp Copyright (C) 2023, 2024 Carl-Philip Hänsch
| | <!-- SPDX-License-Identifier: GPL-3.0-or-later --> |
| This program comes with ABSOLUTELY NO WARRANTY;
| | = Introduction to Scheme = |
| This is free software, and you are welcome to redistribute it
| |
| under certain conditions;
| |
|
| |
| loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-id of type 10
| |
| loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-username of type 20
| |
| loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-password of type 20
| |
|
| |
| Welcome to memcp
| |
|
| |
| performing unit tests ...
| |
| finished unit tests
| |
| test result: 15/15
| |
| all tests succeeded.
| |
|
| |
| Initializing SQL frontend
| |
| MySQL server listening on port 3307 (connect with `mysql -P 3307 -u root -p` using password 'admin')
| |
| listening on <nowiki>http://localhost:4321</nowiki>
| |
|
| |
|
| |
| Type (help) to show help
| |
|
| |
| >
| |
| Scheme is a functional programming language and a subset of LISP. Every expression is either a primitive value or a list. | |
|
| |
|
| To understand the semantics of Scheme, take a look at the following Scheme REPL session:
| | MemCP embeds a small Scheme dialect used by the SQL parsers, query planner, modules, and application endpoints. Start the interactive console with <code>./memcp lib/main.scm</code>; background services use <code>--no-repl</code>. |
| > 12
| |
| = 12
| |
|
| |
| > "hi"
| |
| = "hi"
| |
|
| |
| > (+ 1 2)
| |
| = 3
| |
|
| |
| > (+ 1 2 3)
| |
| = 6
| |
|
| |
| > (+ 2 (* 2 2))
| |
| = 6
| |
|
| |
| > (+ 2 (* 2 4))
| |
| = 10
| |
|
| |
| > (concat "Hello" "World")
| |
| = "HelloWorld"
| |
|
| |
|
| == Syntax ==
| | Scheme represents calls and code with the same list syntax. That makes the dialect well suited to MemCP's query compiler: the SQL planner constructs Scheme programs as data, optimizes them, and evaluates or compiles them. |
|
| |
|
| === Function Call ===
| | The console prints each expression result, which makes the evaluation model easy to explore: |
| A function call has the following format:
| |
| (functionname param1 param2 param3 ...)
| |
| Function calls start with <code>(</code>, contain one function and zero or more parameters separated by space .
| |
|
| |
|
| Some basic functions are <code>+ - * / print concat</code>. For more consult the manual with <code>(help)</code>.
| | <pre>> 12 |
| | = 12 |
| | > (+ 1 2 3) |
| | = 6 |
| | > (+ 2 (* 2 4)) |
| | = 10 |
| | > (concat "Hello " "World") |
| | = "Hello World"</pre> |
|
| |
|
| === Literals ===
| | Startup also loads the configured data directory and Scheme modules before opening the SQL/HTTP listeners. The exact banner and number of startup tests change between releases, so scripts should not parse that text. |
| The following literals are allowed: | |
|
| |
|
| * Number Literals: <code>1 2 3 56 7.5 4.3e20 -6.123e-3</code>
| | == Values and calls == |
| * String Literals: <code>"Hello World" "First Line\nSecond Line" "he said: \"what?\" and smiled"</code>
| |
| * Function names: <code>print</code>
| |
| * Symbol literals <code>'print</code>
| |
| * List literals <code>'(1 2 3) '("a" "b" "c")</code>
| |
| * Associative Array literals <code>'("key" "value" "size" 45 "name" "Peter" "sublist" '(1 2 3))</code> - associative arrays are lists with key value pairs flattened down to a one dimensional list of an even number of items.
| |
| * Lambdas: <code>(lambda (param1 param2 ...) body)</code>
| |
|
| |
|
| === Lambda Functions ===
| | <pre> |
| Scheme allows for creating lambda functions which enclose their own scope:
| | (+ 1 2) |
| (define decrease-by-one (lambda (number) (- number 1)))
| | (concat "hello " "world") |
| The function can now be called from scheme:
| | (map '(1 2 3) (lambda (x) (* x x))) |
| > (decrease-by-one 5)
| | </pre> |
| = 4
| |
| Lambda functions can be passed to other functions as values. They enclose your original scope, so you can use variables from the outside inside your function.
| |
|
| |
|
| == Deeper Topics into Scheme ==
| | Numbers, booleans, strings, symbols, lists, functions, sessions, and storage objects are ordinary values. A list in expression position is a call: its first item is the procedure and the rest are arguments. Whitespace separates forms; parentheses define nesting. |
| You can deep-dive into the following topcis:
| |
|
| |
|
| * Lists and Objects
| | {| class="wikitable" |
| * Pattern Matching
| | ! Form !! Meaning |
| * Parsers
| | |- |
| | | <code>42</code>, <code>3.5</code>, <code>true</code>, <code>"text"</code> || Literal values |
| | |- |
| | | <code>(+ 1 2)</code> || Call <code>+</code> with two arguments |
| | |- |
| | | <code>'name</code> || The symbol <code>name</code>, not the value bound to it |
| | |- |
| | | <code>'(1 2 3)</code> || Construct a list value instead of calling <code>1</code> |
| | |- |
| | | <code>'("name" "Ada" "active" true)</code> || Flat key/value list used as an associative object |
| | |} |
|
| |
|
| == Further Help and Documentation of all Functions == | | == Quoting and generated code == |
| If you type <code>(help)</code> into the console, you will get the following overview:
| | |
| > (help)
| | Use quoting to build code or data: <code>'(+ 4 5)</code> constructs delayed code, while <code>(eval '(+ 4 5))</code> evaluates it. Values that should be computed now can be embedded into a quoted outer form; procedure symbols whose execution is delayed remain quoted. This distinction matters throughout the query planner. |
| Available scm functions:
| | |
|
| | <pre> |
| -- SCM Builtins --
| | (set delayed '(+ 4 5)) |
| quote: returns a symbol or list without evaluating it
| | (eval delayed) /* 9 */ |
| eval: executes the given scheme program in the current environment
| | (map '(1 2 3) (lambda (x) (* x x))) |
| optimize: optimize the given scheme program
| | </pre> |
| if: checks a condition and then conditionally evaluates code branches; there might be multiple condition+true-branch clauses
| | |
| and: returns true if all conditions evaluate to true
| | Do not add or remove parentheses by visual guesswork in large generated expressions. Repository Scheme files are formatted and checked with <code>python3 tools/lint_scm.py</code>. |
| or: returns true if at least one condition evaluates to true
| | |
| coalesce: returns the first value that has a non-zero value
| | == Functions and scope == |
| coalesceNil: returns the first value that has a non-nil value
| | |
| define: defines or sets a variable in the current environment
| | Lambdas have exactly a parameter list and one body. Use <code>begin</code> when that body needs several forms: |
| set: defines or sets a variable in the current environment
| | |
| error: halts the whole execution thread and throws an error message
| | <pre> |
| try: tries to execute a function and returns its result. In case of a failure, the error is fed to the second function and its result value will be used
| | (define greet (lambda (name) (begin |
| apply: runs the function with its arguments
| | (print "greeting " name) |
| apply_assoc: runs the function with its arguments but arguments is a assoc list
| | (concat "Hello, " name)))) |
| symbol: returns a symbol built from that string
| | |
| list: returns a list containing the parameters as alements
| | (greet "Ada") |
| string: converts the given value into string
| | </pre> |
| match: takes a value evaluates the branch that first matches the given pattern
| | |
| lambda: returns a function (func) constructed from the given code
| | Functions are closures and may be passed to <code>map</code>, reducers, scanners, parsers, and handlers. <code>define</code> and <code>set</code> create a binding in the current scope in this dialect; <code>set</code> is not an imperative mutation of an outer lexical variable. |
| begin: creates a own variable scope, evaluates all sub expressions and returns the result of the last one
| | |
| parallel: executes all parameters in parallel and returns nil if they are finished
| | == Pattern matching and modules == |
| source: annotates the node with filename and line information for better backtraces
| | |
|
| | <code>match</code> is the usual way to express alternatives and destructure lists, strings, or regular-expression results. Modules are loaded with <code>import</code>; application entrypoints typically import the required library files and then install handlers or start a server. |
| -- Arithmetic / Logic --
| | |
| number?: tells if the value is a number
| | <pre> |
| +: adds two or more numbers
| | (match value |
| -: subtracts two or more numbers from the first one
| | '(x y) (+ x y) |
| *: multiplies two or more numbers
| | (regex "^/users/([0-9]+)$" path id) id |
| /: divides two or more numbers from the first one
| | false) |
| <=: compares two numbers
| | </pre> |
| <: compares two numbers
| | |
| >: compares two numbers
| | See [[Parsers]], [[Lists and Objects]], and the matching examples in the repository for the exact pattern forms. |
| >=: compares two numbers
| | |
| equal?: deep-compares two values of the same type
| | == State and concurrency == |
| equal??: performs a sloppy equality check on primitive values (number, string, bool. nil), strings are compared case insensitive
| | |
| !: negates the boolean value
| | The dialect is functional by default. <code>set</code> defines a binding in the current scope; it does not imperatively mutate an outer binding. Use <code>(newsession)</code> when code intentionally needs a thread-safe mutable key/value context shared across parallel work. |
| not: negates the boolean value
| | |
| nil?: returns true if value is nil
| | <pre> |
| min: returns the smallest value
| | (set request_state (newsession)) |
| max: returns the highest value
| | (request_state "user_id" 42) |
| floor: rounds the number down
| | (request_state "user_id") /* 42 */ |
| ceil: rounds the number up
| | </pre> |
| round: rounds the number
| | |
|
| | Independent functional work may run in parallel. Supported hot procedures can be compiled by the native x86-64 JIT; unsupported shapes remain interpreted without changing language semantics. See [[Full SCM API documentation]], [[Parallel Computing]], and [[In-Database WebApps and REST Services]]. |
| -- Strings --
| | |
| string?: tells if the value is a string
| | == Where to continue == |
| concat: concatenates stringable values and returns a string
| | |
| substr: returns a substring
| | * [[Full SCM API documentation]] lists generated chapters and signatures. |
| simplify: turns a stringable input value in the easiest-most value (e.g. turn strings into numbers if they are numeric
| | * [[Lists and Objects]] explains lists, associative objects, mapping, and reduction. |
| strlen: returns the length of a string
| | * [[IO]] covers files, HTTP request/response objects, and servers. |
| strlike: matches the string against a wildcard pattern (SQL compliant)
| | * [[Storage]] documents the low-level table and scan interface; application code normally starts with SQL. |
| toLower: turns a string into lower case
| | * [[JIT Compilation]] and [[Parallel Computing]] explain runtime optimization boundaries. |
| toUpper: turns a string into upper case
| |
| replace: replaces all occurances in a string with another string
| |
| split: splits a string using a separator or space
| |
| htmlentities: escapes the string for use in HTML
| |
| urlencode: encodes a string according to URI coding schema
| |
| urldecode: decodes a string according to URI coding schema
| |
|
| |
| -- Lists --
| |
| append: appends items to a list and return the extended list.
| |
| append_unique: appends items to a list but only if they are new.
| |
| cons: constructs a list from a head and a tail list
| |
| car: extracts the head of a list
| |
| cdr: extracts the tail of a list
| |
| merge: flattens a list of lists into a list containing all the subitems. If one parameter is given, it is a list of lists that is flattened. If multiple parameters are given, they are treated as lists that will be merged into one
| |
| merge_unique: flattens a list of lists into a list containing all the subitems. Duplicates are filtered out.
| |
| has?: checks if a list has a certain item (equal?)
| |
| filter: returns a list that only contains elements that pass the filter function
| |
| map: returns a list that contains the results of a map function that is applied to the list
| |
| mapIndex: returns a list that contains the results of a map function that is applied to the list
| |
| reduce: returns a list that contains the result of a map function
| |
| produce: returns a list that contains produced items - it works like for(state = startstate, condition(state), state = iterator(state)) {yield state}
| |
| produceN: returns a list with numbers from 0..n-1
| |
| list?: checks if a value is a list
| |
| contains?: checks if a value is in a list; uses the equal?? operator
| |
|
| |
| -- Associative Lists / Dictionaries --
| |
| filter_assoc: returns a filtered dictionary according to a filter function
| |
| map_assoc: returns a mapped dictionary according to a map function
| |
| reduce_assoc: reduces a dictionary according to a reduce function
| |
| has_assoc?: checks if a dictionary has a key present
| |
| extract_assoc: applies a function (key value) on the dictionary and returns the results as a flat list
| |
| set_assoc: returns a dictionary where a single value has been changed.
| |
| merge_assoc: returns a dictionary where all keys from dict1 and all keys from dict2 are present.
| |
|
| |
| -- Parsers --
| |
| parser: creates a parser
| |
|
| |
| -- Sync --
| |
| newsession: Creates a new session which is a threadsafe key-value store represented as a function that can be either called as a getter (session key) or setter (session key value) or list all keys with (session)
| |
| once: Creates a function wrapper that you can call multiple times but only gets executed once. The result value is cached and returned on a second call. You can add parameters to that resulting function that will be passed to the first run of the wrapped function.
| |
| mutex: Creates a mutex. The return value is a function that takes one parameter which is a parameterless function. The mutex is guaranteed that all calls to that mutex get serialized.
| |
|
| |
| -- IO --
| |
| print: Prints values to stdout (only in IO environment)
| |
| help: Lists all functions or print help for a specific function
| |
| import: Imports a file .scm file into current namespace
| |
| load: Loads a file and returns the string
| |
| serve: Opens a HTTP server at a given port
| |
| mysql: Imports a file .scm file into current namespace
| |
| password: Hashes a password with sha1 (for mysql user authentication)
| |
|
| |
| -- Storage --
| |
| [[scan]]: does an unordered parallel filter-map-reduce pass on a single table and returns the reduced result
| |
| [[Scan|scan_order]]: does an ordered parallel filter and serial map-reduce pass on a single table and returns the reduced result
| |
| createdatabase: creates a new database
| |
| dropdatabase: creates a new database
| |
| createtable: creates a new database
| |
| createcolumn: creates a new column in table
| |
| shardcolumn: tells us how it would partition a column according to their values. Returns a list of pivot elements.
| |
| altertable: alters a table
| |
| droptable: removes a table
| |
| insert: inserts a new dataset into table
| |
| stat: return memory statistics
| |
| show: show databases/tables/columns
| |
| rebuild: rebuilds all main storages and returns the amount of time it took
| |
| loadCSV: loads a CSV file into a table and returns the amount of time it took.
| |
| loadJSON: loads a .jsonl file from disk into a database and returns the amount of time it took.
| |
| settings: reads or writes a global settings value. This modifies your data/settings.json.
| |
|
| |
| get further information by typing (help "functionname") to get more info
| |
Introduction to Scheme
MemCP embeds a small Scheme dialect used by the SQL parsers, query planner, modules, and application endpoints. Start the interactive console with ./memcp lib/main.scm; background services use --no-repl.
Scheme represents calls and code with the same list syntax. That makes the dialect well suited to MemCP's query compiler: the SQL planner constructs Scheme programs as data, optimizes them, and evaluates or compiles them.
The console prints each expression result, which makes the evaluation model easy to explore:
> 12
= 12
> (+ 1 2 3)
= 6
> (+ 2 (* 2 4))
= 10
> (concat "Hello " "World")
= "Hello World"
Startup also loads the configured data directory and Scheme modules before opening the SQL/HTTP listeners. The exact banner and number of startup tests change between releases, so scripts should not parse that text.
Values and calls
(+ 1 2)
(concat "hello " "world")
(map '(1 2 3) (lambda (x) (* x x)))
Numbers, booleans, strings, symbols, lists, functions, sessions, and storage objects are ordinary values. A list in expression position is a call: its first item is the procedure and the rest are arguments. Whitespace separates forms; parentheses define nesting.
| Form |
Meaning
|
42, 3.5, true, "text" |
Literal values
|
(+ 1 2) |
Call + with two arguments
|
'name |
The symbol name, not the value bound to it
|
'(1 2 3) |
Construct a list value instead of calling 1
|
'("name" "Ada" "active" true) |
Flat key/value list used as an associative object
|
Quoting and generated code
Use quoting to build code or data: '(+ 4 5) constructs delayed code, while (eval '(+ 4 5)) evaluates it. Values that should be computed now can be embedded into a quoted outer form; procedure symbols whose execution is delayed remain quoted. This distinction matters throughout the query planner.
(set delayed '(+ 4 5))
(eval delayed) /* 9 */
(map '(1 2 3) (lambda (x) (* x x)))
Do not add or remove parentheses by visual guesswork in large generated expressions. Repository Scheme files are formatted and checked with python3 tools/lint_scm.py.
Functions and scope
Lambdas have exactly a parameter list and one body. Use begin when that body needs several forms:
(define greet (lambda (name) (begin
(print "greeting " name)
(concat "Hello, " name))))
(greet "Ada")
Functions are closures and may be passed to map, reducers, scanners, parsers, and handlers. define and set create a binding in the current scope in this dialect; set is not an imperative mutation of an outer lexical variable.
Pattern matching and modules
match is the usual way to express alternatives and destructure lists, strings, or regular-expression results. Modules are loaded with import; application entrypoints typically import the required library files and then install handlers or start a server.
(match value
'(x y) (+ x y)
(regex "^/users/([0-9]+)$" path id) id
false)
See Parsers, Lists and Objects, and the matching examples in the repository for the exact pattern forms.
State and concurrency
The dialect is functional by default. set defines a binding in the current scope; it does not imperatively mutate an outer binding. Use (newsession) when code intentionally needs a thread-safe mutable key/value context shared across parallel work.
(set request_state (newsession))
(request_state "user_id" 42)
(request_state "user_id") /* 42 */
Independent functional work may run in parallel. Supported hot procedures can be compiled by the native x86-64 JIT; unsupported shapes remain interpreted without changing language semantics. See Full SCM API documentation, Parallel Computing, and In-Database WebApps and REST Services.
Where to continue