https://www.memcp.org/api.php?action=feedcontributions&feedformat=atom&user=CarliMemCP - User contributions [en]2026-05-04T15:46:20ZUser contributionsMediaWiki 1.39.1https://www.memcp.org/index.php?title=Persistency_and_Performance_Guarantees&diff=263Persistency and Performance Guarantees2026-03-09T12:20:19Z<p>Carli: /* Persistency Guarantees */</p>
<hr />
<div>MemCP gives the user several Guarantees for persistency and performance. These are the guarantees:<br />
<br />
== Performance Guarantees and Read Speed ==<br />
MemCP guarantees that:<br />
<br />
* All reading operations are from RAM and are fast<br />
* Insert/Update without any unique key check or foreign key check will scale over shards<br />
* Aggregate functions like SUM, AVG will scale perfectly with the amount of CPUs<br />
<br />
With these guarantees, you can add more CPU cores whenever you have more data to ensure the same query time even with growing data.<br />
<br />
Please consider that memory bandwith still might be a bottleneck. Even modern 128 core CPUs do not scale better than 8 cores for uncompressed data (usually 40 GiB/s bandwith on tested machines). '''MemCPs [[Columnar Storage]] scales beyond that limit.'''<br />
== Persistency Guarantees and Write Speed ==<br />
There are five persistency modes per table which are:<br />
<br />
* ENGINE = cache<br />
* ENGINE = memory<br />
* ENGINE = sloppy<br />
* ENGINE = logged<br />
* ENGINE = safe<br />
In comparison to other RDBMs, MemCP can do this setting per table. MySQL/MariaDB for instance only allows a global setting which is sometimes not what you want.<br />
<br />
In short, these engines guarantee different performance:<br />
<br />
* cache+memory -> RAM speed, no persistency<br />
* sloppy+logged -> HDD speed, persistency guarantees under circumstances<br />
* safe -> Sync speed (slow) and persistency guarantees even in case of power outage during queries<br />
<br />
=== ENGINE = cache ===<br />
<br />
* all data is held in memory and only in memory<br />
* in case of a crash, all data is gone<br />
* in case the server is running out of RAM, the data can be deleted, too<br />
* the schema is saved on disk<br />
* after a recovery, the table starts empty<br />
* fastest way to store data<br />
* use it for session data, observer handles, caches and other data that can be recreated by the software<br />
<br />
=== ENGINE = memory ===<br />
<br />
* all data is held in memory and only in memory<br />
* in case of a crash, all data is gone<br />
* while the server is running, the data will be kept alive<br />
* the schema is saved on disk<br />
* after a recovery, the table starts empty<br />
* fastest way to store data<br />
* use it for session data, observer handles, caches and other data that can be recreated by the software<br />
<br />
=== ENGINE = sloppy ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* the delta storage is RAM-only<br />
* a main storage rebuild is triggered every 15 minutes, so data older than 15 minutes are guaranteed to be persistent<br />
* in case of a crash, the delta storage is gone, the main storage is recovered<br />
* after a recovery, some datasets or deletions that happend in the last 15 minutes before the crash may be gone<br />
* extremely fast way to store data without the fear of losing them in normal operation<br />
* use it for frequently updated tables with unimportant data like usage statistics or sensor data<br />
<br />
=== ENGINE = logged ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* changes to the delta storage are logged on disk files<br />
* an operation succeeds even if data is not synced to disk permanently yet<br />
* in case of a crash, data might be recovered<br />
** in case of a crash of the process, all data will be recovered<br />
** in case of a power outage or kernel crash, data might be lost<br />
* allows buffering of the log file in return for the risk of data loss<br />
* use it for data where you need a high update performance but cannot afford losing the last few seconds of your work<br />
<br />
=== ENGINE = safe ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* changes to the delta storage are logged on disk files<br />
* an operation only succeeds after the data is synced to disk permanently<br />
* in case of a crash, all data will be recovered<br />
* introduces delays for each transaction since the system has to wait for a write fence<br />
* IO bound (limited to ~1,700 write operations per second)<br />
* use it for accounting data or any data that must not be lost</div>Carlihttps://www.memcp.org/index.php?title=Persistency_and_Performance_Guarantees&diff=262Persistency and Performance Guarantees2026-03-09T12:16:10Z<p>Carli: /* Persistency Guarantees */</p>
<hr />
<div>MemCP gives the user several Guarantees for persistency and performance. These are the guarantees:<br />
<br />
== Performance Guarantees ==<br />
MemCP guarantees that:<br />
<br />
* Insert/Update without any unique key check or foreign key check will scale over shards<br />
* Aggregate functions like SUM, AVG will scale perfectly with the amount of CPUs<br />
<br />
With these guarantees, you can add more CPU cores whenever you have more data to ensure the same query time even with growing data.<br />
<br />
Please consider that memory bandwith still might be a bottleneck. Even modern 128 core CPUs do not scale better than 8 cores for uncompressed data (usually 40 GiB/s bandwith on tested machines). '''MemCPs [[Columnar Storage]] scales beyond that limit.'''<br />
== Persistency Guarantees ==<br />
There are five persistency modes per table which are:<br />
<br />
* ENGINE = cache<br />
* ENGINE = memory<br />
* ENGINE = sloppy<br />
* ENGINE = logged<br />
* ENGINE = safe<br />
In comparison to other RDBMs, MemCP can do this setting per table. MySQL/MariaDB for instance only allows a global setting which is sometimes not what you want.<br />
<br />
In short, these engines guarantee different performance:<br />
<br />
* cache+memory -> RAM speed, no persistency<br />
* sloppy+logged -> HDD speed, persistency guarantees under circumstances<br />
* safe -> Sync speed (slow) and persistency guarantees even in case of power outage during queries<br />
<br />
=== ENGINE = cache ===<br />
<br />
* all data is held in memory and only in memory<br />
* in case of a crash, all data is gone<br />
* in case the server is running out of RAM, the data can be deleted, too<br />
* the schema is saved on disk<br />
* after a recovery, the table starts empty<br />
* fastest way to store data<br />
* use it for session data, observer handles, caches and other data that can be recreated by the software<br />
<br />
=== ENGINE = memory ===<br />
<br />
* all data is held in memory and only in memory<br />
* in case of a crash, all data is gone<br />
* while the server is running, the data will be kept alive<br />
* the schema is saved on disk<br />
* after a recovery, the table starts empty<br />
* fastest way to store data<br />
* use it for session data, observer handles, caches and other data that can be recreated by the software<br />
<br />
=== ENGINE = sloppy ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* the delta storage is RAM-only<br />
* a main storage rebuild is triggered every 15 minutes, so data older than 15 minutes are guaranteed to be persistent<br />
* in case of a crash, the delta storage is gone, the main storage is recovered<br />
* after a recovery, some datasets or deletions that happend in the last 15 minutes before the crash may be gone<br />
* extremely fast way to store data without the fear of losing them in normal operation<br />
* use it for frequently updated tables with unimportant data like usage statistics or sensor data<br />
<br />
=== ENGINE = logged ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* changes to the delta storage are logged on disk files<br />
* an operation succeeds even if data is not synced to disk permanently yet<br />
* in case of a crash, data might be recovered<br />
** in case of a crash of the process, all data will be recovered<br />
** in case of a power outage or kernel crash, data might be lost<br />
* allows buffering of the log file in return for the risk of data loss<br />
* use it for data where you need a high update performance but cannot afford losing the last few seconds of your work<br />
<br />
=== ENGINE = safe ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* changes to the delta storage are logged on disk files<br />
* an operation only succeeds after the data is synced to disk permanently<br />
* in case of a crash, all data will be recovered<br />
* introduces delays for each transaction since the system has to wait for a write fence<br />
* IO bound (limited to ~1,700 write operations per second)<br />
* use it for accounting data or any data that must not be lost</div>Carlihttps://www.memcp.org/index.php?title=Persistency_and_Performance_Guarantees&diff=261Persistency and Performance Guarantees2026-03-09T12:13:49Z<p>Carli: /* Persistency Guarantees */</p>
<hr />
<div>MemCP gives the user several Guarantees for persistency and performance. These are the guarantees:<br />
<br />
== Performance Guarantees ==<br />
MemCP guarantees that:<br />
<br />
* Insert/Update without any unique key check or foreign key check will scale over shards<br />
* Aggregate functions like SUM, AVG will scale perfectly with the amount of CPUs<br />
<br />
With these guarantees, you can add more CPU cores whenever you have more data to ensure the same query time even with growing data.<br />
<br />
Please consider that memory bandwith still might be a bottleneck. Even modern 128 core CPUs do not scale better than 8 cores for uncompressed data (usually 40 GiB/s bandwith on tested machines). '''MemCPs [[Columnar Storage]] scales beyond that limit.'''<br />
== Persistency Guarantees ==<br />
There are five persistency modes per table which are:<br />
<br />
* ENGINE = cache<br />
* ENGINE = memory<br />
* ENGINE = sloppy<br />
* ENGINE = logged<br />
* ENGINE = safe<br />
In comparison to other RDBMs, MemCP can do this setting per table. MySQL/MariaDB for instance only allows a global setting which is sometimes not what you want.<br />
<br />
=== ENGINE = cache ===<br />
<br />
* all data is held in memory and only in memory<br />
* in case of a crash, all data is gone<br />
* in case the server is running out of RAM, the data can be deleted, too<br />
* the schema is saved on disk<br />
* after a recovery, the table starts empty<br />
* fastest way to store data<br />
* use it for session data, observer handles, caches and other data that can be recreated by the software<br />
<br />
=== ENGINE = memory ===<br />
<br />
* all data is held in memory and only in memory<br />
* in case of a crash, all data is gone<br />
* while the server is running, the data will be kept alive<br />
* the schema is saved on disk<br />
* after a recovery, the table starts empty<br />
* fastest way to store data<br />
* use it for session data, observer handles, caches and other data that can be recreated by the software<br />
<br />
=== ENGINE = sloppy ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* the delta storage is RAM-only<br />
* a main storage rebuild is triggered every 15 minutes, so data older than 15 minutes are guaranteed to be persistent<br />
* in case of a crash, the delta storage is gone, the main storage is recovered<br />
* after a recovery, some datasets or deletions that happend in the last 15 minutes before the crash may be gone<br />
* extremely fast way to store data without the fear of losing them in normal operation<br />
* use it for frequently updated tables with unimportant data like usage statistics or sensor data<br />
<br />
=== ENGINE = logged ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* changes to the delta storage are logged on disk files<br />
* an operation succeeds even if data is not synced to disk permanently yet<br />
* in case of a crash, data might be recovered<br />
** in case of a crash of the process, all data will be recovered<br />
** in case of a power outage or kernel crash, data might be lost<br />
* allows buffering of the log file in return for the risk of data loss<br />
* use it for data where you need a high update performance but cannot afford losing the last few seconds of your work<br />
<br />
=== ENGINE = safe ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* changes to the delta storage are logged on disk files<br />
* an operation only succeeds after the data is synced to disk permanently<br />
* in case of a crash, all data will be recovered<br />
* introduces delays for each transaction since the system has to wait for a write fence<br />
* IO bound (limited to ~1,700 write operations per second)<br />
* use it for accounting data or any data that must not be lost</div>Carlihttps://www.memcp.org/index.php?title=Persistency_and_Performance_Guarantees&diff=260Persistency and Performance Guarantees2026-03-09T12:13:31Z<p>Carli: </p>
<hr />
<div>MemCP gives the user several Guarantees for persistency and performance. These are the guarantees:<br />
<br />
== Performance Guarantees ==<br />
MemCP guarantees that:<br />
<br />
* Insert/Update without any unique key check or foreign key check will scale over shards<br />
* Aggregate functions like SUM, AVG will scale perfectly with the amount of CPUs<br />
<br />
With these guarantees, you can add more CPU cores whenever you have more data to ensure the same query time even with growing data.<br />
<br />
Please consider that memory bandwith still might be a bottleneck. Even modern 128 core CPUs do not scale better than 8 cores for uncompressed data (usually 40 GiB/s bandwith on tested machines). '''MemCPs [[Columnar Storage]] scales beyond that limit.'''<br />
== Persistency Guarantees ==<br />
There are three persistency modes per table which are:<br />
<br />
* ENGINE = memory<br />
* ENGINE = sloppy<br />
* ENGINE = logged<br />
* ENGINE = safe<br />
In comparison to other RDBMs, MemCP can do this setting per table. MySQL/MariaDB for instance only allows a global setting which is sometimes not what you want.<br />
<br />
=== ENGINE = cache ===<br />
<br />
* all data is held in memory and only in memory<br />
* in case of a crash, all data is gone<br />
* in case the server is running out of RAM, the data can be deleted, too<br />
* the schema is saved on disk<br />
* after a recovery, the table starts empty<br />
* fastest way to store data<br />
* use it for session data, observer handles, caches and other data that can be recreated by the software<br />
<br />
=== ENGINE = memory ===<br />
<br />
* all data is held in memory and only in memory<br />
* in case of a crash, all data is gone<br />
* while the server is running, the data will be kept alive<br />
* the schema is saved on disk<br />
* after a recovery, the table starts empty<br />
* fastest way to store data<br />
* use it for session data, observer handles, caches and other data that can be recreated by the software<br />
<br />
=== ENGINE = sloppy ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* the delta storage is RAM-only<br />
* a main storage rebuild is triggered every 15 minutes, so data older than 15 minutes are guaranteed to be persistent<br />
* in case of a crash, the delta storage is gone, the main storage is recovered<br />
* after a recovery, some datasets or deletions that happend in the last 15 minutes before the crash may be gone<br />
* extremely fast way to store data without the fear of losing them in normal operation<br />
* use it for frequently updated tables with unimportant data like usage statistics or sensor data<br />
<br />
=== ENGINE = logged ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* changes to the delta storage are logged on disk files<br />
* an operation succeeds even if data is not synced to disk permanently yet<br />
* in case of a crash, data might be recovered<br />
** in case of a crash of the process, all data will be recovered<br />
** in case of a power outage or kernel crash, data might be lost<br />
* allows buffering of the log file in return for the risk of data loss<br />
* use it for data where you need a high update performance but cannot afford losing the last few seconds of your work<br />
<br />
=== ENGINE = safe ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* changes to the delta storage are logged on disk files<br />
* an operation only succeeds after the data is synced to disk permanently<br />
* in case of a crash, all data will be recovered<br />
* introduces delays for each transaction since the system has to wait for a write fence<br />
* IO bound (limited to ~1,700 write operations per second)<br />
* use it for accounting data or any data that must not be lost</div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=259Main Page2026-02-27T02:21:08Z<p>Carli: </p>
<hr />
<div>= MemCP – A Modern In-Memory Columnar Database =<br />
<br />
'''MemCP is a high-performance, in-memory, column-oriented database designed for modern workloads.''' <br />
It provides a lightweight, developer-friendly alternative to traditional relational databases such as MySQL, with a focus on speed, compression, and direct API integration.<br />
[[File:Memcp-Load.png|center|frameless|1600x1600px]]<br />
<br />
----<br />
<br />
== Key Features ==<br />
<br />
* '''High Performance''': NUMA-aware, parallelized query execution optimized for multicore CPUs, large caches, and NVMe SSDs. Handles both OLTP and OLAP workloads efficiently. <br />
* '''Columnar Storage''': Data is stored by column for improved compression, reduced memory footprint, and faster analytical queries. <br />
* '''In-Memory Operation''': Designed to keep data in memory, with configurable persistence backends for durability. <br />
* '''Built-in APIs''': Exposes RESTful endpoints directly from the database, reducing middleware overhead. <br />
* '''Compression''': Multiple strategies (bit-packing, dictionary encoding, sequence compression) reduce storage by up to 80% compared to MySQL/MariaDB. <br />
* '''Simple Deployment''': Start with a single <code>docker run</code> or <code>pm2 start</code> command. Lightweight footprint (~10MB). <br />
* '''Extensible''': Written in Go, with pluggable storage backends and custom frontend support (SQL, RDF, REST). <br />
<br />
----<br />
<br />
== Why MemCP? ==<br />
<br />
Traditional relational databases were designed decades ago, optimized for spinning disks and single-core CPUs. <br />
MemCP rethinks the core design for today’s hardware and workloads:<br />
<br />
* Real-time dashboards and analytics <br />
* Data-heavy SaaS platforms <br />
* Embedded systems with limited resources <br />
* High-throughput OLTP/OLAP hybrids <br />
<br />
----<br />
<br />
== Quick Start ==<br />
<br />
Clone and build MemCP from source:<br />
<br />
<pre><br />
git clone https://github.com/launix-de/memcp<br />
cd memcp<br />
go get<br />
make<br />
pm2 start ./memcp ./data/<br />
</pre><br />
<br />
Connect with MySQL tooling:<br />
<br />
<pre><br />
mysql -u root -p -P 3307<br />
Enter password: admin<br />
</pre><br />
<br />
----<br />
<br />
== MemCP vs. MySQL ==<br />
<br />
{| class="wikitable"<br />
! Feature<br />
! MySQL<br />
! MemCP<br />
|-<br />
| Storage Model<br />
| Row-based<br />
| Column-based (compressed)<br />
|-<br />
| Performance<br />
| Good<br />
| NUMA-optimized, in-memory<br />
|-<br />
| In-Memory Capable<br />
| Limited<br />
| Yes (default)<br />
|-<br />
| REST API Integration<br />
| External<br />
| Built-in<br />
|-<br />
| Installation Footprint<br />
| ~150MB+<br />
| ~10MB<br />
|-<br />
| Open Source<br />
| ✅<br />
| ✅<br />
|}<br />
<br />
----<br />
<br />
== Architecture Overview ==<br />
<br />
* '''Tables, Schemas, Columns''': Familiar SQL-style structures with a columnar physical layout. <br />
* '''Transaction Model''': Supports both OLTP and OLAP semantics with delta + main storage. <br />
* '''Persistence''': Configurable storage backends (filesystem, S3, Ceph). <br />
* '''Frontends''': Multiple query interfaces:<br />
- SQL frontend (MySQL wire protocol + SQL over REST) <br />
- RDF/graph query engine <br />
- Custom APIs via in-database web apps <br />
<br />
----<br />
<br />
== Documentation ==<br />
<br />
* [[What is OLTP and OLAP]] <br />
* [[History of the MemCP project]] <br />
* [[Hardware Requirements]] <br />
* [[Persistency and Performance Guarantees]] <br />
* [[Comparison: MemCP vs. MySQL]] <br />
* [[Install MemCP with Docker|Install with Docker]] <br />
* [[Compile MemCP from Source|Build from Source]] <br />
* [[Contributing]] <br />
* [[SQL over REST]] <br />
* [[In-Database WebApps|REST & Microservices]] <br />
<br />
----<br />
<br />
<br />
===Navigation===<br />
<br />
====Introduction==== <br />
*[[What is OLTP and OLAP]]<br />
*[[History of the MemCP project]]<br />
*[[Hardware Requirements]]<br />
*[[Persistency and Performance Guarantees]] <br />
*[[Current Status and Open Issues]]<br />
*[[Comparison: MemCP vs. MySQL]]<br />
<br />
====Getting Started====<br />
*[[Install MemCP with Docker|With Docker]]<br />
*[[With Singularity]]<br />
*[[Compile MemCP from Source|Build from Source]]<br />
*[[Contributing]] <br />
*[[Introduction to Scheme]]<br />
*[[Full SCM API documentation]]<br />
<br />
====Administration====<br />
<br />
* [[Deployment]]<br />
* [[Migration from MySQL and PostgreSQL]]<br />
* [[Settings]]<br />
*[[Process Hibernation]]<br />
*[[Performance Measurement]]<br />
*[[MemCP Console]]<br />
*[[Performance Measurement]]<br />
<br />
====Frontends====<br />
<br />
=====SQL Frontend===== <br />
*[[Supported SQL]]<br />
*[[Advanced SQL Tutorial]]<br />
*[[SQL over REST]]<br />
*[[Database Tools compatibility with MemCP|Supported Tooling]]<br />
*[[How SQL Operators are implemented on MemCP]]<br />
*[[Add custom SQL operators to MemCP]]<br />
<br />
=====RDF Frontend===== <br />
*[[Introduction to RDF]]<br />
*[[Advanced Graph Querying]]<br />
*[[RDF templating and model driven development]]<br />
<br />
=====Custom Frontends=====<br />
<br />
*[[In-Database WebApps|In-Database WebApps and REST Services]]<br />
*[[MemCP for Microservices]]<br />
*[[Websockets in MemCP]]<br />
<br />
==== Persistency Backends (= Storage) ====<br />
<br />
* [[File System]]<br />
* [[S3 Buckets]]<br />
* [[Ceph/Rados]]<br />
* [[Cluster Monitor]]<br />
<br />
====Internals====<br />
<br />
=====How things work in MemCP===== <br />
<br />
*[[Databases, Tables and Columns]]<br />
*[[Shards, RecordIDs, Main Storage, Delta Storage]]<br />
*[[Columnar Storage]]<br />
*[[Transactions]] <br />
*[[Full SCM API documentation]] <br />
<br />
===== SCM Documentation =====<br />
<br />
* [[SCM Builtins]]<br />
* [[Arithmetic / Logic]]<br />
* [[Strings]]<br />
* [[Streams]]<br />
* [[Lists]]<br />
* [[Associative Lists / Dictionaries]]<br />
* [[Date]]<br />
* [[Vectors]]<br />
* [[Parsers]]<br />
* [[Sync]]<br />
* [[IO]]<br />
* [[Storage]]<br />
<br />
=====Optimizations=====<br />
*[[In-Memory Compression, Columnar Compression Techniques]]<br />
*[[Temporary Columns]]<br />
*[[Data Auto Sharding and Auto Indexing]]<br />
* [[Parallel Computing]]<br />
<br />
<br />
----<br />
<br />
== Further Reading ==<br />
<br />
* [https://github.org/launix-de/memcp MemCP on GitHub] <br />
* [https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper] <br />
* [https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper] <br />
* [https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms] <br />
<br />
Additional blog posts on design decisions, compression techniques, and performance optimization are available on the [https://launix.de/launix/ Launix blog].<br />
<br />
----<br />
<br />
== Community ==<br />
<br />
MemCP is an open-source project maintained by developers for developers. <br />
Contributions are welcome — whether in the form of bug reports, feature requests, or pull requests. <br />
<br />
See: [[Contributing]]</div>Carlihttps://www.memcp.org/index.php?title=File:Memcp-Load.png&diff=258File:Memcp-Load.png2026-02-27T02:19:55Z<p>Carli: </p>
<hr />
<div>Performance Gauges of the DB</div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=256Main Page2026-01-26T06:39:56Z<p>Carli: </p>
<hr />
<div>= MemCP – A Modern In-Memory Columnar Database =<br />
<br />
'''MemCP is a high-performance, in-memory, column-oriented database designed for modern workloads.''' <br />
It provides a lightweight, developer-friendly alternative to traditional relational databases such as MySQL, with a focus on speed, compression, and direct API integration.<br />
<br />
----<br />
<br />
== Key Features ==<br />
<br />
* '''High Performance''': NUMA-aware, parallelized query execution optimized for multicore CPUs, large caches, and NVMe SSDs. Handles both OLTP and OLAP workloads efficiently. <br />
* '''Columnar Storage''': Data is stored by column for improved compression, reduced memory footprint, and faster analytical queries. <br />
* '''In-Memory Operation''': Designed to keep data in memory, with configurable persistence backends for durability. <br />
* '''Built-in APIs''': Exposes RESTful endpoints directly from the database, reducing middleware overhead. <br />
* '''Compression''': Multiple strategies (bit-packing, dictionary encoding, sequence compression) reduce storage by up to 80% compared to MySQL/MariaDB. <br />
* '''Simple Deployment''': Start with a single <code>docker run</code> or <code>pm2 start</code> command. Lightweight footprint (~10MB). <br />
* '''Extensible''': Written in Go, with pluggable storage backends and custom frontend support (SQL, RDF, REST). <br />
<br />
----<br />
<br />
== Why MemCP? ==<br />
<br />
Traditional relational databases were designed decades ago, optimized for spinning disks and single-core CPUs. <br />
MemCP rethinks the core design for today’s hardware and workloads:<br />
<br />
* Real-time dashboards and analytics <br />
* Data-heavy SaaS platforms <br />
* Embedded systems with limited resources <br />
* High-throughput OLTP/OLAP hybrids <br />
<br />
----<br />
<br />
== Quick Start ==<br />
<br />
Clone and build MemCP from source:<br />
<br />
<pre><br />
git clone https://github.com/launix-de/memcp<br />
cd memcp<br />
go get<br />
make<br />
pm2 start ./memcp ./data/<br />
</pre><br />
<br />
Connect with MySQL tooling:<br />
<br />
<pre><br />
mysql -u root -p -P 3307<br />
Enter password: admin<br />
</pre><br />
<br />
----<br />
<br />
== MemCP vs. MySQL ==<br />
<br />
{| class="wikitable"<br />
! Feature<br />
! MySQL<br />
! MemCP<br />
|-<br />
| Storage Model<br />
| Row-based<br />
| Column-based (compressed)<br />
|-<br />
| Performance<br />
| Good<br />
| NUMA-optimized, in-memory<br />
|-<br />
| In-Memory Capable<br />
| Limited<br />
| Yes (default)<br />
|-<br />
| REST API Integration<br />
| External<br />
| Built-in<br />
|-<br />
| Installation Footprint<br />
| ~150MB+<br />
| ~10MB<br />
|-<br />
| Open Source<br />
| ✅<br />
| ✅<br />
|}<br />
<br />
----<br />
<br />
== Architecture Overview ==<br />
<br />
* '''Tables, Schemas, Columns''': Familiar SQL-style structures with a columnar physical layout. <br />
* '''Transaction Model''': Supports both OLTP and OLAP semantics with delta + main storage. <br />
* '''Persistence''': Configurable storage backends (filesystem, S3, Ceph). <br />
* '''Frontends''': Multiple query interfaces:<br />
- SQL frontend (MySQL wire protocol + SQL over REST) <br />
- RDF/graph query engine <br />
- Custom APIs via in-database web apps <br />
<br />
----<br />
<br />
== Documentation ==<br />
<br />
* [[What is OLTP and OLAP]] <br />
* [[History of the MemCP project]] <br />
* [[Hardware Requirements]] <br />
* [[Persistency and Performance Guarantees]] <br />
* [[Comparison: MemCP vs. MySQL]] <br />
* [[Install MemCP with Docker|Install with Docker]] <br />
* [[Compile MemCP from Source|Build from Source]] <br />
* [[Contributing]] <br />
* [[SQL over REST]] <br />
* [[In-Database WebApps|REST & Microservices]] <br />
<br />
----<br />
<br />
<br />
===Navigation===<br />
<br />
====Introduction==== <br />
*[[What is OLTP and OLAP]]<br />
*[[History of the MemCP project]]<br />
*[[Hardware Requirements]]<br />
*[[Persistency and Performance Guarantees]] <br />
*[[Current Status and Open Issues]]<br />
*[[Comparison: MemCP vs. MySQL]]<br />
<br />
====Getting Started====<br />
*[[Install MemCP with Docker|With Docker]]<br />
*[[With Singularity]]<br />
*[[Compile MemCP from Source|Build from Source]]<br />
*[[Contributing]] <br />
*[[Introduction to Scheme]]<br />
*[[Full SCM API documentation]]<br />
<br />
====Administration====<br />
<br />
* [[Deployment]]<br />
* [[Migration from MySQL and PostgreSQL]]<br />
* [[Settings]]<br />
*[[Process Hibernation]]<br />
*[[Performance Measurement]]<br />
*[[MemCP Console]]<br />
*[[Performance Measurement]]<br />
<br />
====Frontends====<br />
<br />
=====SQL Frontend===== <br />
*[[Supported SQL]]<br />
*[[Advanced SQL Tutorial]]<br />
*[[SQL over REST]]<br />
*[[Database Tools compatibility with MemCP|Supported Tooling]]<br />
*[[How SQL Operators are implemented on MemCP]]<br />
*[[Add custom SQL operators to MemCP]]<br />
<br />
=====RDF Frontend===== <br />
*[[Introduction to RDF]]<br />
*[[Advanced Graph Querying]]<br />
*[[RDF templating and model driven development]]<br />
<br />
=====Custom Frontends=====<br />
<br />
*[[In-Database WebApps|In-Database WebApps and REST Services]]<br />
*[[MemCP for Microservices]]<br />
*[[Websockets in MemCP]]<br />
<br />
==== Persistency Backends (= Storage) ====<br />
<br />
* [[File System]]<br />
* [[S3 Buckets]]<br />
* [[Ceph/Rados]]<br />
* [[Cluster Monitor]]<br />
<br />
====Internals====<br />
<br />
=====How things work in MemCP===== <br />
<br />
*[[Databases, Tables and Columns]]<br />
*[[Shards, RecordIDs, Main Storage, Delta Storage]]<br />
*[[Columnar Storage]]<br />
*[[Transactions]] <br />
*[[Full SCM API documentation]] <br />
<br />
===== SCM Documentation =====<br />
<br />
* [[SCM Builtins]]<br />
* [[Arithmetic / Logic]]<br />
* [[Strings]]<br />
* [[Streams]]<br />
* [[Lists]]<br />
* [[Associative Lists / Dictionaries]]<br />
* [[Date]]<br />
* [[Vectors]]<br />
* [[Parsers]]<br />
* [[Sync]]<br />
* [[IO]]<br />
* [[Storage]]<br />
<br />
=====Optimizations=====<br />
*[[In-Memory Compression, Columnar Compression Techniques]]<br />
*[[Temporary Columns]]<br />
*[[Data Auto Sharding and Auto Indexing]]<br />
* [[Parallel Computing]]<br />
<br />
<br />
----<br />
<br />
== Further Reading ==<br />
<br />
* [https://github.org/launix-de/memcp MemCP on GitHub] <br />
* [https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper] <br />
* [https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper] <br />
* [https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms] <br />
<br />
Additional blog posts on design decisions, compression techniques, and performance optimization are available on the [https://launix.de/launix/ Launix blog].<br />
<br />
----<br />
<br />
== Community ==<br />
<br />
MemCP is an open-source project maintained by developers for developers. <br />
Contributions are welcome — whether in the form of bug reports, feature requests, or pull requests. <br />
<br />
See: [[Contributing]]</div>Carlihttps://www.memcp.org/index.php?title=Data_Auto_Sharding_and_Auto_Indexing&diff=255Data Auto Sharding and Auto Indexing2026-01-26T06:39:48Z<p>Carli: </p>
<hr />
<div>To understand the following explainations, you should read about [[Shards, RecordIDs, Main Storage, Delta Storage]] first. An understanding of [[Scan]] can also help.<br />
<br />
== Auto-Indexing ==<br />
Auto Indexing works the following way:<br />
<br />
* Whenever there is a [[scan]] over a table shard that has certain boundaries (e.g. <code>WHERE col1 = value1 AND col2 = value2 ORDER BY col3 ASC</code>), there exists a "'''desired index'''" (e.g. [col1 col2 col3])<br />
* Every desired index is created shard-locally but marked "inactive"<br />
* There is a cost model consisting of three components:<br />
** The cost of scanning without index<br />
** The cost of building the index<br />
** The cost of scanning with the index<br />
* After about two indexed scans, the cost of indexing will be ammortized<br />
** before the amortization threshold is met, do full scans<br />
** as soon as the amortization threshold is met, build the index<br />
<br />
Indexes are (currently) only built over the main storage. Delta storage items are currently appended to the scan in an unordered way.<br />
<br />
An index is a [[Integer Compression|integer-compressed]] list of recordIds, meaning: instead of storing the values in the index itself, the index is a cache-compressed pointerless structure. This list is sorted by the sort criterion. This makes storing indexes so cheap.<br />
<br />
A 60,000 item index would compress to 16 bits integer width, so the size of the index would be 120 KB.<br />
<br />
== Auto-Sharding ==<br />
Auto-Sharding is done similar to Auto-Indexing:<br />
<br />
* Whenever there is a [[scan]] over a table where a sharding scheme along a column <code>col</code> would be benefitial, the "partitioning score" for <code>col</code> is increased.<br />
* After a while (~15min), there is a reevaluation if the shards of a table should be repartitioned<br />
* The shard dimensions are chosen proportional to the "partitioning score" from the previous steps<br />
* The number of shard dimensions can be limited in the [[Settings]]<br />
* Then, data is reshuffled. This may take a while for bigger tables.<br />
<br />
Partitioning has the following positive effects:<br />
<br />
* A scan with boundaries matching the partitioning scheme will only touch those partitions with relevant data in it<br />
* INSERTs with unique keys will be able to scale whenever all unique keys are covered within the partitioning schema<br />
<br />
By using insert or scan very often, it will increase the partitioning score on the right dimensions, so auto-sharding in MemCP is self learning.<br />
<br />
== Parallel Sharding==<br />
MemCP automatically creates parallel shards for optimal query performance across multiple CPU cores.<br />
===Bulk Insert Optimization===<br />
When inserting large amounts of data, MemCP automatically:<br />
#'''Splits bulk inserts into chunks''' of 60,000 rows (configurable via ShardSize)<br />
#'''Creates new shards on-the-fly''' when the current shard fills up<br />
#'''Rebuilds full shards in parallel''' using background goroutines<br />
This enables insertion of millions of rows while maintaining parallel query execution.<br />
===Automatic Repartitioning===<br />
During <code>rebuild()</code>, if no partitioning hints exist but data exceeds ShardSize:*MemCP calculates the optimal number of shards based on data size<br />
*Creates at least <code>2 × NumCPU</code> shards for good parallelism<br />
*Uses the first column for round-robin distribution<br />
===Performance Characteristics===<br />
With parallel sharding enabled: - Query CPU utilization: 1500-1900% (15-19 cores on a 24-core machine) - Speedup: 3-6x compared to single-threaded execution - Throughput: ~0.04 µs/row for COUNT/SUM operations<br />
===Configuration===<br />
{| class="wikitable"<br />
!Setting<br />
! Default<br />
!Description<br />
|-<br />
|ShardSize<br />
|60,000<br />
| Rows per shard before splitting<br />
|-<br />
|PartitionMaxDimensions<br />
|10<br />
|Maximum partitioning dimensions <br />
|}</div>Carlihttps://www.memcp.org/index.php?title=Supported_SQL&diff=254Supported SQL2026-01-26T00:06:51Z<p>Carli: /* Statements */</p>
<hr />
<div>== Statements ==<br />
The following SQL statements are supported:<br />
<br />
* SELECT FROM GROUP BY HAVING ORDER BY LIMIT OFFSET<br />
* UPDATE SET WHERE<br />
* DELETE FROM WHERE<br />
* INSERT INTO VALUES ON DUPLICATE KEY UPDATE / INSERT ... ON CONFLICT (PostgreSQL)<br />
* INSERT IGNORE<br />
* CREATE TABLE<br />
* ALTER TABLE<br />
* CREATE DATABASE<br />
* CREATE USER<br />
* ALTER USER<br />
* SHOW DATABASES<br />
* SHOW TABLES<br />
* SHOW TABLE STATUS<br />
* SHOW VARIABLES<br />
* SET NAMES (no function)<br />
* DROP DATABASE<br />
* DROP TABLE<br />
* SET SESSION<br />
* LOCK TABLES (no function)<br />
* UNLOCK TABLES (no function)<br />
<br />
== Expressions ==<br />
The following functions are supported:<br />
* CASE WHEN THEN ELSE END<br />
* DATABASE<br />
* PASSWORD (hashes a password for user authentication)<br />
* FLOOR<br />
* CEIL<br />
* CEILING<br />
* ROUND<br />
* UPPER<br />
* LOWER<br />
* CAST AS UNSIGNED<br />
* CAST AS CHAR CHARACTER SET utf8<br />
* CONCAT<br />
* COALESCE<br />
* UNIX_TIMESTAMP<br />
* LIKE<br />
* ILIKE<br />
<br />
== Aggregates ==<br />
The following aggregates are available:<br />
<br />
* COUNT<br />
* SUM<br />
* AVG<br />
* MIN<br />
* MAX<br />
<br />
== Modifiers ==<br />
<br />
* COLLATE</div>Carlihttps://www.memcp.org/index.php?title=Auto_Sharding&diff=253Auto Sharding2026-01-26T00:06:40Z<p>Carli: Blanked the page</p>
<hr />
<div></div>Carlihttps://www.memcp.org/index.php?title=Advanced_SQL_Tutorial&diff=252Advanced SQL Tutorial2026-01-26T00:01:40Z<p>Carli: </p>
<hr />
<div>== Atomic Counters through Unique Key Checks ==<br />
Imagine the following table:<br />
CREATE TABLE u(id INT PRIMARY KEY, cnt INT);<br />
<br />
INSERT INTO u VALUES (1, 2), (2, 4);<br />
<br />
SELECT * FROM u;<br />
<br />
+------+------+<br />
| id | cnt |<br />
+------+------+<br />
| 1 | 2 |<br />
| 2 | 4 |<br />
+------+------+<br />
'''3 rows in set (0,001 sec)'''<br />
If you want to add to <code>cnt</code> whenever an event occurs, add a unique key over <code>id</code> and do the following:<br />
INSERT INTO u VALUES (2, 1) ON DUPLICATE KEY UPDATE cnt = cnt + VALUES(cnt);<br />
<br />
<br />
+------+------+<br />
| id | cnt |<br />
+------+------+<br />
| 1 | 2 |<br />
| 2 | 5 |<br />
+------+------+<br />
'''3 rows in set (0,001 sec)'''<br />
This kind of behaviour is atomic, so it is safer than splitting it into two query. This behaviour also scales over multicores via [[Data Auto Sharding and Auto Indexing]]<br />
<br />
== Subselects ==<br />
MemCP supports subselects in expressions including IN, EXISTS, and scalar subqueries.<br />
<br />
=== IN Subselect ===<br />
<code>SELECT * FROM orders WHERE customer_id IN (SELECT id FROM customers WHERE active = 1)</code><br />
<br />
=== EXISTS Subselect ===<br />
<code>SELECT * FROM products WHERE EXISTS (SELECT 1 FROM inventory WHERE product_id = products.id)</code><br />
<br />
=== Scalar Subselect in SELECT ===<br />
<code>SELECT id, (SELECT COUNT(*) FROM orders WHERE customer_id = customers.id) AS order_count<br />
FROM customers</code><br />
<br />
=== Scalar Subselect with Aggregation ===<br />
<code>SELECT (SELECT MAX(price) FROM products WHERE category = 'electronics') AS max_price</code><br />
<br />
=== Correlated Subselects ===<br />
Subselects can reference columns from the outer query:<br />
<code>SELECT id, name,<br />
(SELECT SUM(amount) FROM orders WHERE orders.customer_id = customers.id) AS total<br />
FROM customers</code><br />
<br />
=== Derived Tables ===<br />
<code>SELECT t.* FROM (<br />
SELECT id, COUNT(*) AS cnt FROM orders GROUP BY id<br />
) AS t<br />
WHERE t.cnt > 5</code><br />
<br />
=== Limitations ===<br />
<br />
* Scalar subselects must return exactly one row (error if multiple rows)<br />
* Scalar subselects return NULL if no rows match</div>Carlihttps://www.memcp.org/index.php?title=Performance_Measurement&diff=251Performance Measurement2026-01-26T00:00:10Z<p>Carli: Created page with " == Performance Testing Framework == MemCP includes an auto-calibrating performance test framework for regression detection and benchmarking. === Running Performance Tests === <code># Run performance tests (uses calibrated baselines) PERF_TEST=1 make test # Calibrate for your machine (run ~10 times to reach target) PERF_TEST=1 PERF_CALIBRATE=1 make test # Freeze row counts for bisecting regressions PERF_TEST=1 PERF_NORECALIBRATE=1 make test # Show query exe..."</p>
<hr />
<div><br />
== Performance Testing Framework ==<br />
MemCP includes an auto-calibrating performance test framework for regression detection and benchmarking.<br />
<br />
=== Running Performance Tests ===<br />
<code># Run performance tests (uses calibrated baselines)<br />
PERF_TEST=1 make test<br />
<br />
# Calibrate for your machine (run ~10 times to reach target)<br />
PERF_TEST=1 PERF_CALIBRATE=1 make test<br />
<br />
# Freeze row counts for bisecting regressions<br />
PERF_TEST=1 PERF_NORECALIBRATE=1 make test<br />
<br />
# Show query execution plans<br />
PERF_TEST=1 PERF_EXPLAIN=1 make test</code><br />
<br />
=== How Auto-Calibration Works ===<br />
<br />
# Target query time: '''10-20 seconds'''<br />
# Each calibration run scales row counts by '''30%''' up or down<br />
# Baselines are machine-specific, stored in <code>.perf_baseline.json</code><br />
# Tests include 2 warmup runs before the measured run<br />
<br />
=== Output Format ===<br />
<code>✅ Perf: COUNT (4.3s / 13s, 100,000,000 rows, 0.04µs/row, 25GB heap, 1522%/2400% CPU)<br />
│ │ │ │ │ └─ CPU utilization<br />
│ │ │ │ └─ Memory usage<br />
│ │ │ └─ Time per row<br />
│ │ └─ Calibrated row count<br />
│ └─ Threshold<br />
└─ Actual time</code><br />
<br />
=== Bisecting Performance Regressions ===<br />
<code># 1. Calibrate on known-good commit<br />
git checkout <good-commit><br />
PERF_TEST=1 PERF_CALIBRATE=1 make test # run 10x<br />
<br />
# 2. Save baseline<br />
cp .perf_baseline.json .perf_baseline_good.json<br />
<br />
# 3. Run git bisect<br />
git bisect start HEAD <good-commit><br />
git bisect run bash -c 'PERF_TEST=1 PERF_NORECALIBRATE=1 make test'</code><br />
<br />
=== Environment Variables ===<br />
{| class="wikitable"<br />
!Variable<br />
!Values<br />
!Description<br />
|-<br />
|<code>PERF_TEST</code><br />
|<code>0</code>/<code>1</code><br />
|Enable performance tests<br />
|-<br />
|<code>PERF_CALIBRATE</code><br />
|<code>0</code>/<code>1</code><br />
|Update baselines<br />
|-<br />
|<code>PERF_NORECALIBRATE</code><br />
|<code>0</code>/<code>1</code><br />
|Freeze row counts<br />
|-<br />
|<code>PERF_EXPLAIN</code><br />
|<code>0</code>/<code>1</code><br />
|Show query plans<br />
|}</div>Carlihttps://www.memcp.org/index.php?title=Auto_Sharding&diff=250Auto Sharding2026-01-25T23:58:26Z<p>Carli: </p>
<hr />
<div>== Parallel Sharding ==<br />
MemCP automatically creates parallel shards for optimal query performance across multiple CPU cores.<br />
<br />
=== Bulk Insert Optimization ===<br />
When inserting large amounts of data, MemCP automatically:<br />
<br />
# '''Splits bulk inserts into chunks''' of 60,000 rows (configurable via ShardSize)<br />
# '''Creates new shards on-the-fly''' when the current shard fills up<br />
# '''Rebuilds full shards in parallel''' using background goroutines<br />
<br />
This enables insertion of millions of rows while maintaining parallel query execution.<br />
<br />
=== Automatic Repartitioning ===<br />
During <code>rebuild()</code>, if no partitioning hints exist but data exceeds ShardSize:<br />
<br />
* MemCP calculates the optimal number of shards based on data size<br />
* Creates at least <code>2 × NumCPU</code> shards for good parallelism<br />
* Uses the first column for round-robin distribution<br />
<br />
=== Performance Characteristics ===<br />
With parallel sharding enabled: - Query CPU utilization: 1500-1900% (15-19 cores on a 24-core machine) - Speedup: 3-6x compared to single-threaded execution - Throughput: ~0.04 µs/row for COUNT/SUM operations<br />
<br />
=== Configuration ===<br />
{| class="wikitable"<br />
!Setting<br />
!Default<br />
!Description<br />
|-<br />
|ShardSize<br />
|60,000<br />
|Rows per shard before splitting<br />
|-<br />
|PartitionMaxDimensions<br />
|10<br />
|Maximum partitioning dimensions<br />
|}</div>Carlihttps://www.memcp.org/index.php?title=Auto_Sharding&diff=249Auto Sharding2026-01-25T23:57:40Z<p>Carli: Created page with "## Parallel Sharding MemCP automatically creates parallel shards for optimal query performance across multiple CPU cores. ### Bulk Insert Optimization When inserting large amounts of data, MemCP automatically: 1. **Splits bulk inserts into chunks** of 60,000 rows (configurable via ShardSize) 2. **Creates new shards on-the-fly** when the current shard fills up 3. **Rebuilds full shards in parallel** using background goroutines This enables insertion of millions of ro..."</p>
<hr />
<div>## Parallel Sharding<br />
<br />
MemCP automatically creates parallel shards for optimal query performance across multiple CPU cores.<br />
<br />
### Bulk Insert Optimization<br />
<br />
When inserting large amounts of data, MemCP automatically:<br />
<br />
1. **Splits bulk inserts into chunks** of 60,000 rows (configurable via ShardSize)<br />
2. **Creates new shards on-the-fly** when the current shard fills up<br />
3. **Rebuilds full shards in parallel** using background goroutines<br />
<br />
This enables insertion of millions of rows while maintaining parallel query execution.<br />
<br />
### Automatic Repartitioning<br />
<br />
During `rebuild()`, if no partitioning hints exist but data exceeds ShardSize:<br />
<br />
- MemCP calculates the optimal number of shards based on data size<br />
- Creates at least `2 × NumCPU` shards for good parallelism<br />
- Uses the first column for round-robin distribution<br />
<br />
### Performance Characteristics<br />
<br />
With parallel sharding enabled:<br />
- Query CPU utilization: 1500-1900% (15-19 cores on a 24-core machine)<br />
- Speedup: 3-6x compared to single-threaded execution<br />
- Throughput: ~0.04 µs/row for COUNT/SUM operations<br />
<br />
### Configuration<br />
<br />
| Setting | Default | Description |<br />
|---------|---------|-------------|<br />
| ShardSize | 60,000 | Rows per shard before splitting |<br />
| PartitionMaxDimensions | 10 | Maximum partitioning dimensions |</div>Carlihttps://www.memcp.org/index.php?title=Persistency_and_Performance_Guarantees&diff=248Persistency and Performance Guarantees2025-10-04T22:37:59Z<p>Carli: </p>
<hr />
<div>MemCP gives the user several Guarantees for persistency and performance. These are the guarantees:<br />
<br />
== Performance Guarantees ==<br />
MemCP guarantees that:<br />
<br />
* Insert/Update without any unique key check or foreign key check will scale over shards<br />
* Aggregate functions like SUM, AVG will scale perfectly with the amount of CPUs<br />
<br />
With these guarantees, you can add more CPU cores whenever you have more data to ensure the same query time even with growing data.<br />
<br />
Please consider that memory bandwith still might be a bottleneck. Even modern 128 core CPUs do not scale better than 8 cores for uncompressed data (usually 40 GiB/s bandwith on tested machines). '''MemCPs [[Columnar Storage]] scales beyond that limit.'''<br />
== Persistency Guarantees ==<br />
There are three persistency modes per table which are:<br />
<br />
* ENGINE = memory<br />
* ENGINE = sloppy<br />
* ENGINE = logged<br />
* ENGINE = safe<br />
In comparison to other RDBMs, MemCP can do this setting per table. MySQL/MariaDB for instance only allows a global setting which is sometimes not what you want.<br />
<br />
=== ENGINE = memory ===<br />
<br />
* all data is held in memory and only in memory<br />
* in case of a crash, all data is gone<br />
* the schema is saved on disk<br />
* after a recovery, the table starts empty<br />
* fastest way to store data<br />
* use it for session data, observer handles, caches and other data that can be recreated by the software<br />
<br />
=== ENGINE = sloppy ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* the delta storage is RAM-only<br />
* a main storage rebuild is triggered every 15 minutes, so data older than 15 minutes are guaranteed to be persistent<br />
* in case of a crash, the delta storage is gone, the main storage is recovered<br />
* after a recovery, some datasets or deletions that happend in the last 15 minutes before the crash may be gone<br />
* extremely fast way to store data without the fear of losing them in normal operation<br />
* use it for frequently updated tables with unimportant data like usage statistics or sensor data<br />
<br />
=== ENGINE = logged ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* changes to the delta storage are logged on disk files<br />
* an operation succeeds even if data is not synced to disk permanently yet<br />
* in case of a crash, data might be recovered<br />
** in case of a crash of the process, all data will be recovered<br />
** in case of a power outage or kernel crash, data might be lost<br />
* allows buffering of the log file in return for the risk of data loss<br />
* use it for data where you need a high update performance but cannot afford losing the last few seconds of your work<br />
<br />
=== ENGINE = safe ===<br />
<br />
* all data is held in memory<br />
* the main storage is mirrored on disk<br />
* changes to the delta storage are logged on disk files<br />
* an operation only succeeds after the data is synced to disk permanently<br />
* in case of a crash, all data will be recovered<br />
* introduces delays for each transaction since the system has to wait for a write fence<br />
* IO bound (limited to ~1,700 write operations per second)<br />
* use it for accounting data or any data that must not be lost</div>Carlihttps://www.memcp.org/index.php?title=Settings&diff=247Settings2025-10-04T22:34:55Z<p>Carli: </p>
<hr />
<div>Settings are changed by:<br />
(settings "key" "value")<br />
And read by:<br />
(settings "key")<br />
The following settings are available:<br />
<br />
* <code>"Backtrace"</code>: <code>true</code> or <code>false</code> whether to print backtraces from Scheme. Using true decreases the performance by ~10% but gives you better debugging capabilities.<br />
* <code>"PartitionMaxDimensions"</code>: number from 0 to 10 how many partitioning dimensions are allowed at max. 0 means, don't partition shards for values (use it to workaround bugs), 1 means only one-dimensional partitioning.<br />
* <code>"DefaultEngine"</code>: one of <code>"memory" "sloppy" "logging" "safe"</code> that defines which [[Persistency and Performance Guarantees|persistency level]] to choose if a CREATE TABLE statement is executed without the <code>ENGINE</code> parameter<br />
* <code>"ShardSize"</code>: default 60000, defines the size of a shard. Rule of thumb is that no shard scan should last longer than 100ms. For servers and workstations, a shard size of 60,000 is recommended. On Raspberry Pi, you should tend to 10,000 in order to get good use of parallelism.</div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=246Main Page2025-09-22T11:52:10Z<p>Carli: </p>
<hr />
<div>= MemCP – A Modern In-Memory Columnar Database =<br />
<br />
'''MemCP is a high-performance, in-memory, column-oriented database designed for modern workloads.''' <br />
It provides a lightweight, developer-friendly alternative to traditional relational databases such as MySQL, with a focus on speed, compression, and direct API integration.<br />
<br />
----<br />
<br />
== Key Features ==<br />
<br />
* '''High Performance''': NUMA-aware, parallelized query execution optimized for multicore CPUs, large caches, and NVMe SSDs. Handles both OLTP and OLAP workloads efficiently. <br />
* '''Columnar Storage''': Data is stored by column for improved compression, reduced memory footprint, and faster analytical queries. <br />
* '''In-Memory Operation''': Designed to keep data in memory, with configurable persistence backends for durability. <br />
* '''Built-in APIs''': Exposes RESTful endpoints directly from the database, reducing middleware overhead. <br />
* '''Compression''': Multiple strategies (bit-packing, dictionary encoding, sequence compression) reduce storage by up to 80% compared to MySQL/MariaDB. <br />
* '''Simple Deployment''': Start with a single <code>docker run</code> or <code>pm2 start</code> command. Lightweight footprint (~10MB). <br />
* '''Extensible''': Written in Go, with pluggable storage backends and custom frontend support (SQL, RDF, REST). <br />
<br />
----<br />
<br />
== Why MemCP? ==<br />
<br />
Traditional relational databases were designed decades ago, optimized for spinning disks and single-core CPUs. <br />
MemCP rethinks the core design for today’s hardware and workloads:<br />
<br />
* Real-time dashboards and analytics <br />
* Data-heavy SaaS platforms <br />
* Embedded systems with limited resources <br />
* High-throughput OLTP/OLAP hybrids <br />
<br />
----<br />
<br />
== Quick Start ==<br />
<br />
Clone and build MemCP from source:<br />
<br />
<pre><br />
git clone https://github.com/launix-de/memcp<br />
cd memcp<br />
go get<br />
make<br />
pm2 start ./memcp ./data/<br />
</pre><br />
<br />
Connect with MySQL tooling:<br />
<br />
<pre><br />
mysql -u root -p -P 3307<br />
Enter password: admin<br />
</pre><br />
<br />
----<br />
<br />
== MemCP vs. MySQL ==<br />
<br />
{| class="wikitable"<br />
! Feature<br />
! MySQL<br />
! MemCP<br />
|-<br />
| Storage Model<br />
| Row-based<br />
| Column-based (compressed)<br />
|-<br />
| Performance<br />
| Good<br />
| NUMA-optimized, in-memory<br />
|-<br />
| In-Memory Capable<br />
| Limited<br />
| Yes (default)<br />
|-<br />
| REST API Integration<br />
| External<br />
| Built-in<br />
|-<br />
| Installation Footprint<br />
| ~150MB+<br />
| ~10MB<br />
|-<br />
| Open Source<br />
| ✅<br />
| ✅<br />
|}<br />
<br />
----<br />
<br />
== Architecture Overview ==<br />
<br />
* '''Tables, Schemas, Columns''': Familiar SQL-style structures with a columnar physical layout. <br />
* '''Transaction Model''': Supports both OLTP and OLAP semantics with delta + main storage. <br />
* '''Persistence''': Configurable storage backends (filesystem, S3, Ceph). <br />
* '''Frontends''': Multiple query interfaces:<br />
- SQL frontend (MySQL wire protocol + SQL over REST) <br />
- RDF/graph query engine <br />
- Custom APIs via in-database web apps <br />
<br />
----<br />
<br />
== Documentation ==<br />
<br />
* [[What is OLTP and OLAP]] <br />
* [[History of the MemCP project]] <br />
* [[Hardware Requirements]] <br />
* [[Persistency and Performance Guarantees]] <br />
* [[Comparison: MemCP vs. MySQL]] <br />
* [[Install MemCP with Docker|Install with Docker]] <br />
* [[Compile MemCP from Source|Build from Source]] <br />
* [[Contributing]] <br />
* [[SQL over REST]] <br />
* [[In-Database WebApps|REST & Microservices]] <br />
<br />
----<br />
<br />
<br />
===Navigation===<br />
<br />
====Introduction==== <br />
*[[What is OLTP and OLAP]]<br />
*[[History of the MemCP project]]<br />
*[[Hardware Requirements]]<br />
*[[Persistency and Performance Guarantees]] <br />
*[[Current Status and Open Issues]]<br />
*[[Comparison: MemCP vs. MySQL]]<br />
<br />
====Getting Started====<br />
*[[Install MemCP with Docker|With Docker]]<br />
*[[With Singularity]]<br />
*[[Compile MemCP from Source|Build from Source]]<br />
*[[Contributing]] <br />
*[[Introduction to Scheme]]<br />
*[[Full SCM API documentation]]<br />
<br />
====Administration====<br />
<br />
* [[Deployment]]<br />
* [[Migration from MySQL and PostgreSQL]]<br />
* [[Settings]]<br />
*[[Process Hibernation]]<br />
*[[Performance Measurement]]<br />
*[[MemCP Console]]<br />
<br />
====Frontends====<br />
<br />
=====SQL Frontend===== <br />
*[[Supported SQL]]<br />
*[[Advanced SQL Tutorial]]<br />
*[[SQL over REST]]<br />
*[[Database Tools compatibility with MemCP|Supported Tooling]]<br />
*[[How SQL Operators are implemented on MemCP]]<br />
*[[Add custom SQL operators to MemCP]]<br />
<br />
=====RDF Frontend===== <br />
*[[Introduction to RDF]]<br />
*[[Advanced Graph Querying]]<br />
*[[RDF templating and model driven development]]<br />
<br />
=====Custom Frontends=====<br />
<br />
*[[In-Database WebApps|In-Database WebApps and REST Services]]<br />
*[[MemCP for Microservices]]<br />
*[[Websockets in MemCP]]<br />
<br />
==== Persistency Backends (= Storage) ====<br />
<br />
* [[File System]]<br />
* [[S3 Buckets]]<br />
* [[Ceph/Rados]]<br />
* [[Cluster Monitor]]<br />
<br />
====Internals====<br />
<br />
=====How things work in MemCP===== <br />
<br />
*[[Databases, Tables and Columns]]<br />
*[[Shards, RecordIDs, Main Storage, Delta Storage]]<br />
*[[Columnar Storage]]<br />
*[[Transactions]] <br />
*[[Full SCM API documentation]] <br />
<br />
===== SCM Documentation =====<br />
<br />
* [[SCM Builtins]]<br />
* [[Arithmetic / Logic]]<br />
* [[Strings]]<br />
* [[Streams]]<br />
* [[Lists]]<br />
* [[Associative Lists / Dictionaries]]<br />
* [[Date]]<br />
* [[Vectors]]<br />
* [[Parsers]]<br />
* [[Sync]]<br />
* [[IO]]<br />
* [[Storage]]<br />
<br />
=====Optimizations=====<br />
*[[In-Memory Compression, Columnar Compression Techniques]]<br />
*[[Temporary Columns]]<br />
*[[Data Auto Sharding and Auto Indexing]]<br />
* [[Parallel Computing]]<br />
<br />
<br />
----<br />
<br />
== Further Reading ==<br />
<br />
* [https://github.org/launix-de/memcp MemCP on GitHub] <br />
* [https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper] <br />
* [https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper] <br />
* [https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms] <br />
<br />
Additional blog posts on design decisions, compression techniques, and performance optimization are available on the [https://launix.de/launix/ Launix blog].<br />
<br />
----<br />
<br />
== Community ==<br />
<br />
MemCP is an open-source project maintained by developers for developers. <br />
Contributions are welcome — whether in the form of bug reports, feature requests, or pull requests. <br />
<br />
See: [[Contributing]]</div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=245Main Page2025-09-22T11:39:20Z<p>Carli: </p>
<hr />
<div>= MemCP – A Modern In-Memory Columnar Database =<br />
<br />
'''MemCP is a high-performance, in-memory, column-oriented database designed for modern workloads.''' <br />
It provides a lightweight, developer-friendly alternative to traditional relational databases such as MySQL, with a focus on speed, compression, and direct API integration.<br />
<br />
----<br />
<br />
== Key Features ==<br />
<br />
* '''High Performance''': NUMA-aware, parallelized query execution optimized for multicore CPUs, large caches, and NVMe SSDs. Handles both OLTP and OLAP workloads efficiently. <br />
* '''Columnar Storage''': Data is stored by column for improved compression, reduced memory footprint, and faster analytical queries. <br />
* '''In-Memory Operation''': Designed to keep data in memory, with configurable persistence backends for durability. <br />
* '''Built-in APIs''': Exposes RESTful endpoints directly from the database, reducing middleware overhead. <br />
* '''Compression''': Multiple strategies (bit-packing, dictionary encoding, sequence compression) reduce storage by up to 80% compared to MySQL/MariaDB. <br />
* '''Simple Deployment''': Start with a single <code>docker run</code> or <code>pm2 start</code> command. Lightweight footprint (~10MB). <br />
* '''Extensible''': Written in Go, with pluggable storage backends and custom frontend support (SQL, RDF, REST). <br />
<br />
----<br />
<br />
== Why MemCP? ==<br />
<br />
Traditional relational databases were designed decades ago, optimized for spinning disks and single-core CPUs. <br />
MemCP rethinks the core design for today’s hardware and workloads:<br />
<br />
* Real-time dashboards and analytics <br />
* Data-heavy SaaS platforms <br />
* Embedded systems with limited resources <br />
* High-throughput OLTP/OLAP hybrids <br />
<br />
----<br />
<br />
== Quick Start ==<br />
<br />
Clone and build MemCP from source:<br />
<br />
<pre><br />
git clone https://github.com/launix-de/memcp<br />
cd memcp<br />
go get<br />
make<br />
pm2 start ./memcp ./data/<br />
</pre><br />
<br />
Connect with MySQL tooling:<br />
<br />
<pre><br />
mysql -u root -p -P 3307<br />
Enter password: admin<br />
</pre><br />
<br />
----<br />
<br />
== MemCP vs. MySQL ==<br />
<br />
{| class="wikitable"<br />
! Feature<br />
! MySQL<br />
! MemCP<br />
|-<br />
| Storage Model<br />
| Row-based<br />
| Column-based (compressed)<br />
|-<br />
| Performance<br />
| Good<br />
| NUMA-optimized, in-memory<br />
|-<br />
| In-Memory Capable<br />
| Limited<br />
| Yes (default)<br />
|-<br />
| REST API Integration<br />
| External<br />
| Built-in<br />
|-<br />
| Installation Footprint<br />
| ~150MB+<br />
| ~10MB<br />
|-<br />
| Open Source<br />
| ✅<br />
| ✅<br />
|}<br />
<br />
----<br />
<br />
== Architecture Overview ==<br />
<br />
* '''Tables, Schemas, Columns''': Familiar SQL-style structures with a columnar physical layout. <br />
* '''Transaction Model''': Supports both OLTP and OLAP semantics with delta + main storage. <br />
* '''Persistence''': Configurable storage backends (filesystem, S3, Ceph). <br />
* '''Frontends''': Multiple query interfaces:<br />
- SQL frontend (MySQL wire protocol + SQL over REST) <br />
- RDF/graph query engine <br />
- Custom APIs via in-database web apps <br />
<br />
----<br />
<br />
== Documentation ==<br />
<br />
* [[What is OLTP and OLAP]] <br />
* [[History of the MemCP project]] <br />
* [[Hardware Requirements]] <br />
* [[Persistency and Performance Guarantees]] <br />
* [[Comparison: MemCP vs. MySQL]] <br />
* [[Install MemCP with Docker|Install with Docker]] <br />
* [[Compile MemCP from Source|Build from Source]] <br />
* [[Contributing]] <br />
* [[SQL over REST]] <br />
* [[In-Database WebApps|REST & Microservices]] <br />
<br />
----<br />
<br />
== Further Reading ==<br />
<br />
* [https://github.org/launix-de/memcp MemCP on GitHub] <br />
* [https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper] <br />
* [https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper] <br />
* [https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms] <br />
<br />
Additional blog posts on design decisions, compression techniques, and performance optimization are available on the [https://launix.de/launix/ Launix blog].<br />
<br />
----<br />
<br />
== Community ==<br />
<br />
MemCP is an open-source project maintained by developers for developers. <br />
Contributions are welcome — whether in the form of bug reports, feature requests, or pull requests. <br />
<br />
See: [[Contributing]]</div>Carlihttps://www.memcp.org/index.php?title=Comparison:_MemCP_vs._MySQL&diff=244Comparison: MemCP vs. MySQL2025-09-04T12:40:34Z<p>Carli: </p>
<hr />
<div>'''MemCP''' offers several distinct advantages compared to MySQL, making it a compelling choice for optimized database management. Read more under: [[MySQL is too slow]]<br />
<br />
=== '''No 64-Character Limit for Column Names''' ===<br />
MemCP removes the restrictive 64-character limit on column names found in MySQL, allowing for more descriptive and meaningful column naming conventions.<br />
<br />
=== '''Significantly Reduced Storage Requirements''' ===<br />
MemCP reduces '''storage reqirements by 80%''' compared to MySQL's InnoDB and MyISAM storage formats. This reduction in storage demand translates to lower infrastructure costs and better scalability for large datasets.<br />
<br />
=== '''Built-In Support for Natural Sorting''' ===<br />
MemCP supports '''natural sort order''', which arranges data logically as humans expect, such as <code>1, 2, 10, 11</code>, rather than lexicographically (<code>1, 10, 11, 2</code>). This feature works by adding collations, that support natural ordering.<br />
<br />
=== '''Dramatic Performance Improvements''' ===<br />
MemCP delivers up to '''10x faster query performance''' for aggregation operations compared to MySQL. This makes it ideal for data-intensive applications, business intelligence workloads, and real-time analytics.<br />
<br />
=== '''Unified Support for OLAP and OLTP''' ===<br />
MemCP combines '''Online Transaction Processing (OLTP)''' and '''Online Analytical Processing (OLAP)''' in the same database. This enables simultaneous handling of transactional and analytical workloads without the need for separate systems or data pipelines. Such flexibility enhances operational efficiency and simplifies database architecture.<br />
<br />
=== '''Support for Multiple Frontends''' ===<br />
MemCP provides support for diverse frontends, including '''SPARQL''', '''microservices''', and '''REST APIs'''. This versatility allows developers to use MemCP in various application architectures, ranging from semantic web applications to distributed microservices, ensuring seamless integration into modern development environments.<br />
<br />
By addressing common MySQL limitations and offering superior performance, MemCP positions itself as an efficient, scalable, and user-friendly alternative for modern database management needs.</div>Carlihttps://www.memcp.org/index.php?title=MySQL_is_too_slow&diff=243MySQL is too slow2025-09-04T12:40:13Z<p>Carli: Created page with " =MemCP — Faster alternative when "MySQL is too slow"= '''Short description:''' MemCP is an in-memory, column-oriented database optimized for high throughput OLAP and OLTP workloads. It is designed to deliver much faster aggregation and query performance than traditional row-based engines like MySQL. ==Overview== MemCP is a blazing-fast, in-memory, columnar database with built-in REST APIs and a lightweight SQL frontend. It is intended for modern workloads where MySQL..."</p>
<hr />
<div><br />
=MemCP — Faster alternative when "MySQL is too slow"=<br />
'''Short description:''' MemCP is an in-memory, column-oriented database optimized for high throughput OLAP and OLTP workloads. It is designed to deliver much faster aggregation and query performance than traditional row-based engines like MySQL.<br />
==Overview==<br />
MemCP is a blazing-fast, in-memory, columnar database with built-in REST APIs and a lightweight SQL frontend. It is intended for modern workloads where MySQL becomes a bottleneck for aggregation and analytic queries.<br />
==Key benefits==<br />
*'''In-memory, columnar storage''' — very fast for aggregations and analytics.<br />
*'''MySQL-compatible SQL frontend''' — easy to try alongside existing MySQL deployments.<br />
*'''Low memory footprint for base system''' — minimal system requirements for core system.<br />
*'''Open source''' — source code and history available on GitHub.<br />
==When to consider MemCP==<br />
*Your MySQL instance struggles with large aggregations or analytics queries.<br />
*You want an in-memory option that provides much faster group/aggregation performance.<br />
*You need a developer-friendly, REST-accessible datastore for microservices.<br />
==Quickstart / Installation==<br />
A basic example (high level):<pre><br />
# 1) Clone the repository<br />
git clone https://github.com/launix-de/memcp.git<br />
<br />
# 2) Build / run according to README (see GitHub repo for exact commands)<br />
cd memcp<br />
# follow build/run instructions from repository README<br />
</pre><br />
==Example usage==<br />
*MemCP runs a small management service and exposes a MySQL-compatible port for clients (see documentation and examples on the project pages).<br />
*For microservices, MemCP provides example code showing prepared statements and a simple key/value microservice using the built-in SQL frontend.<br />
==Benchmarks & comparisons==<br />
MemCP's documentation and comparison pages describe performance improvements versus MySQL for aggregation workloads (examples and benchmark summaries are available on the project site). Please consult the project's "Comparison: MemCP vs. MySQL" page and published benchmarks for details.<br />
==Links / Resources==<br />
*Official site: https://memcp.org<br />
*Project & source: https://github.com/launix-de/memcp<br />
==License==<br />
MemCP is published under GPL 3 in the GitHub repository<br />
==See also==<br />
*[[Comparison: MemCP vs. MySQL]]<br />
*[[Deployment]]<br />
*[[MemCP_for_Microservices]]</div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=242Main Page2025-09-04T12:38:50Z<p>Carli: /* 🚀 Meet MemCP – The MySQL Alternative You've Been Waiting For */</p>
<hr />
<div>== 🚀 Meet MemCP – The MySQL Alternative You've Been Waiting For ==<br />
'''Tired of outdated database engines slowing you down?'''<br />
<br />
MemCP is a blazing-fast, in-memory, column-oriented database designed for modern workloads. It delivers '''unmatched speed''', '''massive compression''', and '''built-in REST APIs''' – all with a developer-friendly open-source stack.<blockquote>'''Replace MySQL with something better.'''<br />
<br />
Something built for the 2020s – not stuck in 2005.</blockquote><br />
----<br />
<br />
=== ⚡ Why Developers Choose MemCP ===<br />
✅ '''Turbocharged Performance'''<br />
<br />
Designed for multicore CPUs, modern caches, and NVMe SSDs – MemCP handles analytical (OLAP) and transactional (OLTP) workloads at lightning speed. - read more under [[MySQL is too slow]]<br />
<br />
✅ '''80% Smaller Data Footprint'''<br />
<br />
Columnar storage means smarter compression. Typical use cases shrink data 5× smaller than MySQL – with no effort from your side.<br />
<br />
✅ '''Simple JSON APIs Built-in'''<br />
<br />
Query your data with RESTful endpoints – no middleware, no ORM overhead. Just fast results, directly from the DB.<br />
<br />
✅ '''Familiar SQL-like Structure – Zero Lock-In'''<br />
<br />
You don’t need to re-learn everything. Tables, schemas, and logic follow familiar patterns – just modernized and optimized.<br />
<br />
✅ '''Zero-Config Installation'''<br />
<br />
Spin up MemCP with a single <code>docker run</code> or <code>pm2 start</code>. Start building in minutes.<br />
----<br />
<br />
=== 🧑💻 Built for Developers, by Developers ===<br />
MemCP was made for the people who write real-world code, not just benchmark reports. Whether you're building a SaaS product, IoT platform, or data-heavy analytics dashboard, MemCP gives you the speed and flexibility you need to scale.<br />
<br />
# Ready to try it?<br />
git clone <nowiki>https://github.com/launix-de/memcp</nowiki><br />
cd memcp<br />
go get<br />
make<br />
pm2 start ./memcp ./data/<br />
<br />
mysql -u root -p -P 3307<br />
Enter password: admin<br />
<br />
=== 🆚 MemCP vs MySQL: The Key Differences ===<br />
{| class="wikitable"<br />
!Feature<br />
!MySQL<br />
!MemCP<br />
|-<br />
|Storage Model<br />
|Row-based<br />
|Column-based (compressed)<br />
|-<br />
|Performance<br />
|Good<br />
|Excellent (NUMA-optimized)<br />
|-<br />
|In-Memory Capable<br />
|Limited<br />
|Yes, by default<br />
|-<br />
|REST API Integration<br />
|External services<br />
|Built-in<br />
|-<br />
|Installation Footprint<br />
|~150MB+<br />
|~10MB<br />
|-<br />
|Open Source<br />
|✅<br />
|✅<br />
|}<br />
<br />
=== 💡 Ideal Use Cases ===<br />
<br />
* High-speed dashboards & real-time analytics<br />
* Embedded systems with minimal resources<br />
* Data-heavy SaaS backends<br />
* Replacing outdated MySQL deployments with modern performance<br />
<br />
<br />
=== 🧪 Try MemCP Today ===<br />
Explore the documentation, check out the GitHub repo, and join a growing community of developers building fast, efficient, modern systems with MemCP.<blockquote>Ditch the legacy. Embrace performance.<br />
<br />
'''Make the switch to MemCP.'''</blockquote><br />
----<br />
<br />
=== What is memcp? ===<br />
[[File:Webapps.svg|left|frameless]]<br />
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.<br />
<br />
memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.<br />
<br />
<br />
===Features===<br />
<br />
*'''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.<br />
*'''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB<br />
*'''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs<br />
* '''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime<br />
* Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.<br />
* In-memory database: Stores all data in memory, which allows for extremely fast query execution.<br />
*Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)<br />
*OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.<br />
*Compression: Lots of compression formats are supported like bit-packing and dictionary encoding<br />
* Scalability: Designed to scale on a single node with huge NUMA memory<br />
*Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time<br />
[[File:MemCP Port.png|frameless]]<br />
<br />
<youtube>g29FR4Jwius</youtube><br />
<br />
https://www.youtube.com/watch?v=g29FR4Jwius<br />
<br />
===Navigation===<br />
<br />
====Introduction==== <br />
*[[What is OLTP and OLAP]]<br />
*[[History of the MemCP project]]<br />
*[[Hardware Requirements]]<br />
*[[Persistency and Performance Guarantees]] <br />
*[[Current Status and Open Issues]]<br />
*[[Comparison: MemCP vs. MySQL]]<br />
<br />
====Getting Started====<br />
*[[Install MemCP with Docker|With Docker]]<br />
*[[With Singularity]]<br />
*[[Compile MemCP from Source|Build from Source]]<br />
*[[Contributing]] <br />
*[[Introduction to Scheme]]<br />
*[[Full SCM API documentation]]<br />
<br />
====Administration====<br />
<br />
* [[Deployment]]<br />
* [[Migration from MySQL and PostgreSQL]]<br />
* [[Settings]]<br />
*[[Process Hibernation]]<br />
*[[Performance Measurement]]<br />
*[[MemCP Console]]<br />
<br />
====Frontends====<br />
<br />
=====SQL Frontend===== <br />
*[[Supported SQL]]<br />
*[[Advanced SQL Tutorial]]<br />
*[[SQL over REST]]<br />
*[[Database Tools compatibility with MemCP|Supported Tooling]]<br />
*[[How SQL Operators are implemented on MemCP]]<br />
*[[Add custom SQL operators to MemCP]]<br />
<br />
=====RDF Frontend===== <br />
*[[Introduction to RDF]]<br />
*[[Advanced Graph Querying]]<br />
*[[RDF templating and model driven development]]<br />
<br />
=====Custom Frontends=====<br />
<br />
*[[In-Database WebApps|In-Database WebApps and REST Services]]<br />
*[[MemCP for Microservices]]<br />
*[[Websockets in MemCP]]<br />
<br />
==== Persistency Backends (= Storage) ====<br />
<br />
* [[File System]]<br />
* [[S3 Buckets]]<br />
* [[Ceph/Rados]]<br />
* [[Cluster Monitor]]<br />
<br />
====Internals====<br />
<br />
=====How things work in MemCP===== <br />
<br />
*[[Databases, Tables and Columns]]<br />
*[[Shards, RecordIDs, Main Storage, Delta Storage]]<br />
*[[Columnar Storage]]<br />
*[[Transactions]] <br />
*[[Full SCM API documentation]] <br />
<br />
===== SCM Documentation =====<br />
<br />
* [[SCM Builtins]]<br />
* [[Arithmetic / Logic]]<br />
* [[Strings]]<br />
* [[Streams]]<br />
* [[Lists]]<br />
* [[Associative Lists / Dictionaries]]<br />
* [[Date]]<br />
* [[Vectors]]<br />
* [[Parsers]]<br />
* [[Sync]]<br />
* [[IO]]<br />
* [[Storage]]<br />
<br />
=====Optimizations=====<br />
*[[In-Memory Compression, Columnar Compression Techniques]]<br />
*[[Temporary Columns]]<br />
*[[Data Auto Sharding and Auto Indexing]]<br />
* [[Parallel Computing]]<br />
<br />
<br />
[[File:Screenshot from htop.png|center|frameless|2490x2490px]]<br />
<br />
<br />
===Further Reading===<br />
[https://github.org/launix-de/memcp MemCP on Github]<br />
<br />
====Scientific====<br />
<br />
*[https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]<br />
*[https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]<br />
*[https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]<br />
*[https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]<br />
*https://wwwdb.inf.tu-dresden.de/research-projects/eris/ <br />
<br />
====How MemCP was built====<br />
<br />
*[https://launix.de/launix/how-to-balance-a-database-between-olap-and-oltp-workflows/ Balancing OLAP and OLTP Workflows]<br />
*[https://launix.de/launix/designing-a-programming-language-for-distributed-systems-and-highly-parallel-algorithms/ Designing Programming Languages for Distributed Systems]<br />
*[https://launix.de/launix/on-designing-an-interface-for-columnar-in-memory-storage-in-golang/ Columnar Storage Interface in Golang]<br />
*[https://launix.de/launix/how-in-memory-compression-affects-performance/ Impact of In-Memory Compression on Performance]<br />
*[https://launix.de/launix/memory-efficient-indices-for-in-memory-storages/ Memory-Efficient Indices for In-Memory Storages]<br />
*[https://launix.de/launix/on-compressing-null-values-in-bit-compressed-integer-storages/ Compressing Null Values in Bit-Compressed Integer Storages]<br />
*[https://launix.de/launix/when-the-benchmark-is-too-slow-golang-http-server-performance/ Improving Golang HTTP Server Performance]<br />
*[https://launix.de/launix/how-to-benchmark-a-sql-database/ Benchmarking SQL Databases]<br />
*[https://launix.de/launix/writing-a-sql-parser-in-scheme/ Writing a SQL Parser in Scheme]<br />
*[https://launix.de/launix/accessing-memcp-via-scheme/ Accessing memcp via Scheme]<br />
*[https://launix.de/launix/memcp-first-sql-query-is-correctly-executed/ First SQL Query in memcp]<br />
*[https://launix.de/launix/sequence-compression-in-in-memory-database-yields-99-memory-savings-and-a-total-of-13/ Sequence Compression in In-Memory Database]<br />
*[https://launix.de/launix/storing-a-bit-smaller-than-in-one-bit/ Storing Data Smaller Than One Bit]<br />
*[https://www.youtube.com/watch?v=DWg4nx4KVLo memcp Announcement Video]</div>Carlihttps://www.memcp.org/index.php?title=Cluster_Monitor&diff=241Cluster Monitor2025-08-31T21:57:10Z<p>Carli: /* How it works */</p>
<hr />
<div>The cluster monitor lets you scale out MemCP over multiple nodes. Here is what you can achieve:<br />
<br />
* '''Multiple users''': each user has his own database<br />
* '''One big user''': Big tables are spread over multiple nodes<br />
* A mixture of both variants<br />
<br />
Each database and each shard (the piece where a part of a table's data is stored) can be placed either:<br />
<br />
* '''COLD''': only in storage backend<br />
* '''SHARED''': read-only on multiple nodes (e.g. the access table)<br />
* '''WRITE''': exclusively on one node<br />
Cluster monitors can coordinate all kinds of storage:<br />
<br />
* [[File System]]<br />
<br />
* [[S3 Buckets]]<br />
* [[Ceph/Rados]]<br />
<br />
== How to set up ==<br />
TODO: describe how to edit (settings) to connect to other nodes and authenticate (add at least a common secret and optionally a list of nodes in the cluster)<br />
<br />
== How it works ==<br />
Each node in the network knows all other nodes.<br />
<br />
Also, there is a distributed key map that tracks which resource is claimed by whom: <br />
<br />
* COLD items are not tracked in the list at all to save space<br />
* SHARED and WRITE items are tracked together with their owners<br />
* when a user claims WRITE, all other owners must confirm that they give up their SHARED status<br />
* when a user claims SHARED, the WRITE owner must confirm that he gives up his SHARED status<br />
* after you claimed your access, you can use the storage backend to read/write the data<br />
* when a node dosen't own a database, the database is shared<br />
* when a node dosen't own a shard, the owner is asked to perform the computation<br />
* when a shard is owned by noone but the shardlist is very big, other nodes are asked to claim the nodes and perform the computation<br />
<br />
== ZooKeeper internals ==<br />
<br />
* /memcp/resources/<rid> -> information about the resource<br />
* /memcp/resources/<rid>/owner -> if a write lock exists<br />
* /memcp/resources/<rid>/readers/<reader-id> -> which nodes read it</div>Carlihttps://www.memcp.org/index.php?title=Cluster_Monitor&diff=240Cluster Monitor2025-08-31T20:27:30Z<p>Carli: </p>
<hr />
<div>The cluster monitor lets you scale out MemCP over multiple nodes. Here is what you can achieve:<br />
<br />
* '''Multiple users''': each user has his own database<br />
* '''One big user''': Big tables are spread over multiple nodes<br />
* A mixture of both variants<br />
<br />
Each database and each shard (the piece where a part of a table's data is stored) can be placed either:<br />
<br />
* '''COLD''': only in storage backend<br />
* '''SHARED''': read-only on multiple nodes (e.g. the access table)<br />
* '''WRITE''': exclusively on one node<br />
Cluster monitors can coordinate all kinds of storage:<br />
<br />
* [[File System]]<br />
<br />
* [[S3 Buckets]]<br />
* [[Ceph/Rados]]<br />
<br />
== How to set up ==<br />
TODO: describe how to edit (settings) to connect to other nodes and authenticate (add at least a common secret and optionally a list of nodes in the cluster)<br />
<br />
== How it works ==<br />
Each node in the network knows all other nodes.<br />
<br />
Also, there is a distributed key map that tracks which resource is claimed by whom: <br />
<br />
* COLD items are not tracked in the list at all to save space<br />
* SHARED and WRITE items are tracked together with their owners<br />
* when a user claims WRITE, all other owners must confirm that they give up their SHARED status<br />
* when a user claims SHARED, the WRITE owner must confirm that he gives up his SHARED status<br />
* after you claimed your access, you can use the storage backend to read/write the data<br />
<br />
== ZooKeeper internals ==<br />
<br />
* /memcp/resources/<rid> -> information about the resource<br />
* /memcp/resources/<rid>/owner -> if a write lock exists<br />
* /memcp/resources/<rid>/readers/<reader-id> -> which nodes read it</div>Carlihttps://www.memcp.org/index.php?title=File_System&diff=239File System2025-08-31T18:59:41Z<p>Carli: </p>
<hr />
<div>In the file system persistency layer, each subfolder of your data folder is a database.<br />
<br />
Each folder has the following files:<br />
<br />
* schema.json - list of all tables, columns, keys and shards; is updated every time a shard is added or removed or the schema changes<br />
* schema.json.old - fallback file for crash recovery<br />
* [uuid]-[columnname] - the content of a column (in case the [[Persistency and Performance Guarantees|ENGINE]] is not MEMORY)<br />
* [uuid].log - transaction log in case the [[Persistency and Performance Guarantees|ENGINE]] is LOGGED or SAFE<br />
You can also use a [[Cluster Monitor]] for the File System persistency backend. The only thing you have to assure is that each node of the cluster points with -data to the same shared directory.</div>Carlihttps://www.memcp.org/index.php?title=File_System&diff=238File System2025-08-31T18:58:36Z<p>Carli: Created page with "In the file system persistency layer, each subfolder of your data folder is a database. Each folder has the following files: * schema.json - list of all tables, columns, keys and shards; is updated every time a shard is added or removed or the schema changes * schema.json.old - fallback file for crash recovery * [uuid]-[columnname] - the content of a column (in case the ENGINE is not MEMORY) * [uuid].log - transaction log in c..."</p>
<hr />
<div>In the file system persistency layer, each subfolder of your data folder is a database.<br />
<br />
Each folder has the following files:<br />
<br />
* schema.json - list of all tables, columns, keys and shards; is updated every time a shard is added or removed or the schema changes<br />
* schema.json.old - fallback file for crash recovery<br />
* [uuid]-[columnname] - the content of a column (in case the [[Persistency and Performance Guarantees|ENGINE]] is not MEMORY)<br />
* [uuid].log - transaction log in case the [[Persistency and Performance Guarantees|ENGINE]] is LOGGED or SAFE</div>Carlihttps://www.memcp.org/index.php?title=Cluster_Monitor&diff=237Cluster Monitor2025-08-31T18:55:39Z<p>Carli: </p>
<hr />
<div>The cluster monitor lets you scale out MemCP over multiple nodes. Here is what you can achieve:<br />
<br />
* '''Multiple users''': each user has his own database<br />
* '''One big user''': Big tables are spread over multiple nodes<br />
* A mixture of both variants<br />
<br />
Each database and each shard (the piece where a part of a table's data is stored) can be placed either:<br />
<br />
* '''COLD''': only in storage backend<br />
* '''SHARED''': read-only on multiple nodes (e.g. the access table)<br />
* '''WRITE''': exclusively on one node<br />
Cluster monitors can coordinate all kinds of storage:<br />
<br />
* [[File System]]<br />
<br />
* [[S3 Buckets]]<br />
* [[Ceph/Rados]]<br />
<br />
== How to set up ==<br />
TODO: describe how to edit (settings) to connect to other nodes and authenticate (add at least a common secret and optionally a list of nodes in the cluster)<br />
<br />
== How it works ==<br />
Each node in the network knows all other nodes.<br />
<br />
Also, there is a distributed key map that tracks which resource is claimed by whom: <br />
<br />
* COLD items are not tracked in the list at all to save space<br />
* SHARED and WRITE items are tracked together with their owners<br />
* when a user claims WRITE, all other owners must confirm that they give up their SHARED status<br />
* when a user claims SHARED, the WRITE owner must confirm that he gives up his SHARED status<br />
* after you claimed your access, you can use the storage backend to read/write the data</div>Carlihttps://www.memcp.org/index.php?title=Cluster_Monitor&diff=236Cluster Monitor2025-08-31T18:51:09Z<p>Carli: </p>
<hr />
<div>The cluster monitor lets you scale out MemCP over multiple nodes. Here is what you can achieve:<br />
<br />
* '''Multiple users''': each user has his own database<br />
* '''One big user''': Big tables are spread over multiple nodes<br />
* A mixture of both variants<br />
<br />
Each database and each shard (the piece where a part of a table's data is stored) can be placed either:<br />
<br />
* '''COLD''': only in storage backend<br />
* '''SHARED''': read-only on multiple nodes (e.g. the access table)<br />
* '''WRITE''': exclusively on one node<br />
Cluster monitors can coordinate all kinds of storage:<br />
<br />
* [[File System]]<br />
<br />
* [[S3 Buckets]]<br />
* [[Ceph/Rados]]</div>Carlihttps://www.memcp.org/index.php?title=Cluster_Monitor&diff=235Cluster Monitor2025-08-31T18:50:01Z<p>Carli: Created page with "The cluster monitor lets you scale out MemCP over multiple nodes. Here is what you can achieve: * '''Multiple users''': each user has his own database * '''One big user''': Big tables are spread over multiple nodes * A mixture of both variants Each database and each shard (the piece where a part of a table's data is stored) can be placed either: * '''COLD''': only in storage backend * '''SHARED''': read-only on multiple nodes (e.g. the access table) * '''WRITE''': exc..."</p>
<hr />
<div>The cluster monitor lets you scale out MemCP over multiple nodes. Here is what you can achieve:<br />
<br />
* '''Multiple users''': each user has his own database<br />
* '''One big user''': Big tables are spread over multiple nodes<br />
* A mixture of both variants<br />
<br />
Each database and each shard (the piece where a part of a table's data is stored) can be placed either:<br />
<br />
* '''COLD''': only in storage backend<br />
* '''SHARED''': read-only on multiple nodes (e.g. the access table)<br />
* '''WRITE''': exclusively on one node</div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=234Main Page2025-08-31T18:37:49Z<p>Carli: /* Persistency Backends (= Storage) */</p>
<hr />
<div>== 🚀 Meet MemCP – The MySQL Alternative You've Been Waiting For ==<br />
'''Tired of outdated database engines slowing you down?'''<br />
<br />
MemCP is a blazing-fast, in-memory, column-oriented database designed for modern workloads. It delivers '''unmatched speed''', '''massive compression''', and '''built-in REST APIs''' – all with a developer-friendly open-source stack.<blockquote>'''Replace MySQL with something better.'''<br />
<br />
Something built for the 2020s – not stuck in 2005.</blockquote><br />
----<br />
<br />
=== ⚡ Why Developers Choose MemCP ===<br />
✅ '''Turbocharged Performance'''<br />
<br />
Designed for multicore CPUs, modern caches, and NVMe SSDs – MemCP handles analytical (OLAP) and transactional (OLTP) workloads at lightning speed.<br />
<br />
✅ '''80% Smaller Data Footprint'''<br />
<br />
Columnar storage means smarter compression. Typical use cases shrink data 5× smaller than MySQL – with no effort from your side.<br />
<br />
✅ '''Simple JSON APIs Built-in'''<br />
<br />
Query your data with RESTful endpoints – no middleware, no ORM overhead. Just fast results, directly from the DB.<br />
<br />
✅ '''Familiar SQL-like Structure – Zero Lock-In'''<br />
<br />
You don’t need to re-learn everything. Tables, schemas, and logic follow familiar patterns – just modernized and optimized.<br />
<br />
✅ '''Zero-Config Installation'''<br />
<br />
Spin up MemCP with a single <code>docker run</code> or <code>pm2 start</code>. Start building in minutes.<br />
----<br />
<br />
=== 🧑💻 Built for Developers, by Developers ===<br />
MemCP was made for the people who write real-world code, not just benchmark reports. Whether you're building a SaaS product, IoT platform, or data-heavy analytics dashboard, MemCP gives you the speed and flexibility you need to scale.<br />
<br />
# Ready to try it?<br />
git clone <nowiki>https://github.com/launix-de/memcp</nowiki><br />
cd memcp<br />
go get<br />
make<br />
pm2 start ./memcp ./data/<br />
<br />
mysql -u root -p -P 3307<br />
Enter password: admin<br />
<br />
=== 🆚 MemCP vs MySQL: The Key Differences ===<br />
{| class="wikitable"<br />
!Feature<br />
!MySQL<br />
!MemCP<br />
|-<br />
|Storage Model<br />
|Row-based<br />
|Column-based (compressed)<br />
|-<br />
|Performance<br />
|Good<br />
|Excellent (NUMA-optimized)<br />
|-<br />
|In-Memory Capable<br />
|Limited<br />
|Yes, by default<br />
|-<br />
|REST API Integration<br />
|External services<br />
|Built-in<br />
|-<br />
|Installation Footprint<br />
|~150MB+<br />
|~10MB<br />
|-<br />
|Open Source<br />
|✅<br />
|✅<br />
|}<br />
<br />
=== 💡 Ideal Use Cases ===<br />
<br />
* High-speed dashboards & real-time analytics<br />
* Embedded systems with minimal resources<br />
* Data-heavy SaaS backends<br />
* Replacing outdated MySQL deployments with modern performance<br />
<br />
<br />
=== 🧪 Try MemCP Today ===<br />
Explore the documentation, check out the GitHub repo, and join a growing community of developers building fast, efficient, modern systems with MemCP.<blockquote>Ditch the legacy. Embrace performance.<br />
<br />
'''Make the switch to MemCP.'''</blockquote><br />
----<br />
<br />
=== What is memcp? ===<br />
[[File:Webapps.svg|left|frameless]]<br />
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.<br />
<br />
memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.<br />
<br />
<br />
===Features===<br />
<br />
*'''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.<br />
*'''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB<br />
*'''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs<br />
* '''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime<br />
* Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.<br />
* In-memory database: Stores all data in memory, which allows for extremely fast query execution.<br />
*Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)<br />
*OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.<br />
*Compression: Lots of compression formats are supported like bit-packing and dictionary encoding<br />
* Scalability: Designed to scale on a single node with huge NUMA memory<br />
*Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time<br />
[[File:MemCP Port.png|frameless]]<br />
<br />
<youtube>g29FR4Jwius</youtube><br />
<br />
https://www.youtube.com/watch?v=g29FR4Jwius<br />
<br />
===Navigation===<br />
<br />
====Introduction==== <br />
*[[What is OLTP and OLAP]]<br />
*[[History of the MemCP project]]<br />
*[[Hardware Requirements]]<br />
*[[Persistency and Performance Guarantees]] <br />
*[[Current Status and Open Issues]]<br />
*[[Comparison: MemCP vs. MySQL]]<br />
<br />
====Getting Started====<br />
*[[Install MemCP with Docker|With Docker]]<br />
*[[With Singularity]]<br />
*[[Compile MemCP from Source|Build from Source]]<br />
*[[Contributing]] <br />
*[[Introduction to Scheme]]<br />
*[[Full SCM API documentation]]<br />
<br />
====Administration====<br />
<br />
* [[Deployment]]<br />
* [[Migration from MySQL and PostgreSQL]]<br />
* [[Settings]]<br />
*[[Process Hibernation]]<br />
*[[Performance Measurement]]<br />
*[[MemCP Console]]<br />
<br />
====Frontends====<br />
<br />
=====SQL Frontend===== <br />
*[[Supported SQL]]<br />
*[[Advanced SQL Tutorial]]<br />
*[[SQL over REST]]<br />
*[[Database Tools compatibility with MemCP|Supported Tooling]]<br />
*[[How SQL Operators are implemented on MemCP]]<br />
*[[Add custom SQL operators to MemCP]]<br />
<br />
=====RDF Frontend===== <br />
*[[Introduction to RDF]]<br />
*[[Advanced Graph Querying]]<br />
*[[RDF templating and model driven development]]<br />
<br />
=====Custom Frontends=====<br />
<br />
*[[In-Database WebApps|In-Database WebApps and REST Services]]<br />
*[[MemCP for Microservices]]<br />
*[[Websockets in MemCP]]<br />
<br />
==== Persistency Backends (= Storage) ====<br />
<br />
* [[File System]]<br />
* [[S3 Buckets]]<br />
* [[Ceph/Rados]]<br />
* [[Cluster Monitor]]<br />
<br />
====Internals====<br />
<br />
=====How things work in MemCP===== <br />
<br />
*[[Databases, Tables and Columns]]<br />
*[[Shards, RecordIDs, Main Storage, Delta Storage]]<br />
*[[Columnar Storage]]<br />
*[[Transactions]] <br />
*[[Full SCM API documentation]] <br />
<br />
===== SCM Documentation =====<br />
<br />
* [[SCM Builtins]]<br />
* [[Arithmetic / Logic]]<br />
* [[Strings]]<br />
* [[Streams]]<br />
* [[Lists]]<br />
* [[Associative Lists / Dictionaries]]<br />
* [[Date]]<br />
* [[Vectors]]<br />
* [[Parsers]]<br />
* [[Sync]]<br />
* [[IO]]<br />
* [[Storage]]<br />
<br />
=====Optimizations=====<br />
*[[In-Memory Compression, Columnar Compression Techniques]]<br />
*[[Temporary Columns]]<br />
*[[Data Auto Sharding and Auto Indexing]]<br />
* [[Parallel Computing]]<br />
<br />
<br />
[[File:Screenshot from htop.png|center|frameless|2490x2490px]]<br />
<br />
<br />
===Further Reading===<br />
[https://github.org/launix-de/memcp MemCP on Github]<br />
<br />
====Scientific====<br />
<br />
*[https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]<br />
*[https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]<br />
*[https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]<br />
*[https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]<br />
*https://wwwdb.inf.tu-dresden.de/research-projects/eris/ <br />
<br />
====How MemCP was built====<br />
<br />
*[https://launix.de/launix/how-to-balance-a-database-between-olap-and-oltp-workflows/ Balancing OLAP and OLTP Workflows]<br />
*[https://launix.de/launix/designing-a-programming-language-for-distributed-systems-and-highly-parallel-algorithms/ Designing Programming Languages for Distributed Systems]<br />
*[https://launix.de/launix/on-designing-an-interface-for-columnar-in-memory-storage-in-golang/ Columnar Storage Interface in Golang]<br />
*[https://launix.de/launix/how-in-memory-compression-affects-performance/ Impact of In-Memory Compression on Performance]<br />
*[https://launix.de/launix/memory-efficient-indices-for-in-memory-storages/ Memory-Efficient Indices for In-Memory Storages]<br />
*[https://launix.de/launix/on-compressing-null-values-in-bit-compressed-integer-storages/ Compressing Null Values in Bit-Compressed Integer Storages]<br />
*[https://launix.de/launix/when-the-benchmark-is-too-slow-golang-http-server-performance/ Improving Golang HTTP Server Performance]<br />
*[https://launix.de/launix/how-to-benchmark-a-sql-database/ Benchmarking SQL Databases]<br />
*[https://launix.de/launix/writing-a-sql-parser-in-scheme/ Writing a SQL Parser in Scheme]<br />
*[https://launix.de/launix/accessing-memcp-via-scheme/ Accessing memcp via Scheme]<br />
*[https://launix.de/launix/memcp-first-sql-query-is-correctly-executed/ First SQL Query in memcp]<br />
*[https://launix.de/launix/sequence-compression-in-in-memory-database-yields-99-memory-savings-and-a-total-of-13/ Sequence Compression in In-Memory Database]<br />
*[https://launix.de/launix/storing-a-bit-smaller-than-in-one-bit/ Storing Data Smaller Than One Bit]<br />
*[https://www.youtube.com/watch?v=DWg4nx4KVLo memcp Announcement Video]</div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=233Main Page2025-08-31T18:37:16Z<p>Carli: </p>
<hr />
<div>== 🚀 Meet MemCP – The MySQL Alternative You've Been Waiting For ==<br />
'''Tired of outdated database engines slowing you down?'''<br />
<br />
MemCP is a blazing-fast, in-memory, column-oriented database designed for modern workloads. It delivers '''unmatched speed''', '''massive compression''', and '''built-in REST APIs''' – all with a developer-friendly open-source stack.<blockquote>'''Replace MySQL with something better.'''<br />
<br />
Something built for the 2020s – not stuck in 2005.</blockquote><br />
----<br />
<br />
=== ⚡ Why Developers Choose MemCP ===<br />
✅ '''Turbocharged Performance'''<br />
<br />
Designed for multicore CPUs, modern caches, and NVMe SSDs – MemCP handles analytical (OLAP) and transactional (OLTP) workloads at lightning speed.<br />
<br />
✅ '''80% Smaller Data Footprint'''<br />
<br />
Columnar storage means smarter compression. Typical use cases shrink data 5× smaller than MySQL – with no effort from your side.<br />
<br />
✅ '''Simple JSON APIs Built-in'''<br />
<br />
Query your data with RESTful endpoints – no middleware, no ORM overhead. Just fast results, directly from the DB.<br />
<br />
✅ '''Familiar SQL-like Structure – Zero Lock-In'''<br />
<br />
You don’t need to re-learn everything. Tables, schemas, and logic follow familiar patterns – just modernized and optimized.<br />
<br />
✅ '''Zero-Config Installation'''<br />
<br />
Spin up MemCP with a single <code>docker run</code> or <code>pm2 start</code>. Start building in minutes.<br />
----<br />
<br />
=== 🧑💻 Built for Developers, by Developers ===<br />
MemCP was made for the people who write real-world code, not just benchmark reports. Whether you're building a SaaS product, IoT platform, or data-heavy analytics dashboard, MemCP gives you the speed and flexibility you need to scale.<br />
<br />
# Ready to try it?<br />
git clone <nowiki>https://github.com/launix-de/memcp</nowiki><br />
cd memcp<br />
go get<br />
make<br />
pm2 start ./memcp ./data/<br />
<br />
mysql -u root -p -P 3307<br />
Enter password: admin<br />
<br />
=== 🆚 MemCP vs MySQL: The Key Differences ===<br />
{| class="wikitable"<br />
!Feature<br />
!MySQL<br />
!MemCP<br />
|-<br />
|Storage Model<br />
|Row-based<br />
|Column-based (compressed)<br />
|-<br />
|Performance<br />
|Good<br />
|Excellent (NUMA-optimized)<br />
|-<br />
|In-Memory Capable<br />
|Limited<br />
|Yes, by default<br />
|-<br />
|REST API Integration<br />
|External services<br />
|Built-in<br />
|-<br />
|Installation Footprint<br />
|~150MB+<br />
|~10MB<br />
|-<br />
|Open Source<br />
|✅<br />
|✅<br />
|}<br />
<br />
=== 💡 Ideal Use Cases ===<br />
<br />
* High-speed dashboards & real-time analytics<br />
* Embedded systems with minimal resources<br />
* Data-heavy SaaS backends<br />
* Replacing outdated MySQL deployments with modern performance<br />
<br />
<br />
=== 🧪 Try MemCP Today ===<br />
Explore the documentation, check out the GitHub repo, and join a growing community of developers building fast, efficient, modern systems with MemCP.<blockquote>Ditch the legacy. Embrace performance.<br />
<br />
'''Make the switch to MemCP.'''</blockquote><br />
----<br />
<br />
=== What is memcp? ===<br />
[[File:Webapps.svg|left|frameless]]<br />
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.<br />
<br />
memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.<br />
<br />
<br />
===Features===<br />
<br />
*'''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.<br />
*'''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB<br />
*'''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs<br />
* '''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime<br />
* Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.<br />
* In-memory database: Stores all data in memory, which allows for extremely fast query execution.<br />
*Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)<br />
*OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.<br />
*Compression: Lots of compression formats are supported like bit-packing and dictionary encoding<br />
* Scalability: Designed to scale on a single node with huge NUMA memory<br />
*Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time<br />
[[File:MemCP Port.png|frameless]]<br />
<br />
<youtube>g29FR4Jwius</youtube><br />
<br />
https://www.youtube.com/watch?v=g29FR4Jwius<br />
<br />
===Navigation===<br />
<br />
====Introduction==== <br />
*[[What is OLTP and OLAP]]<br />
*[[History of the MemCP project]]<br />
*[[Hardware Requirements]]<br />
*[[Persistency and Performance Guarantees]] <br />
*[[Current Status and Open Issues]]<br />
*[[Comparison: MemCP vs. MySQL]]<br />
<br />
====Getting Started====<br />
*[[Install MemCP with Docker|With Docker]]<br />
*[[With Singularity]]<br />
*[[Compile MemCP from Source|Build from Source]]<br />
*[[Contributing]] <br />
*[[Introduction to Scheme]]<br />
*[[Full SCM API documentation]]<br />
<br />
====Administration====<br />
<br />
* [[Deployment]]<br />
* [[Migration from MySQL and PostgreSQL]]<br />
* [[Settings]]<br />
*[[Process Hibernation]]<br />
*[[Performance Measurement]]<br />
*[[MemCP Console]]<br />
<br />
====Frontends====<br />
<br />
=====SQL Frontend===== <br />
*[[Supported SQL]]<br />
*[[Advanced SQL Tutorial]]<br />
*[[SQL over REST]]<br />
*[[Database Tools compatibility with MemCP|Supported Tooling]]<br />
*[[How SQL Operators are implemented on MemCP]]<br />
*[[Add custom SQL operators to MemCP]]<br />
<br />
=====RDF Frontend===== <br />
*[[Introduction to RDF]]<br />
*[[Advanced Graph Querying]]<br />
*[[RDF templating and model driven development]]<br />
<br />
=====Custom Frontends=====<br />
<br />
*[[In-Database WebApps|In-Database WebApps and REST Services]]<br />
*[[MemCP for Microservices]]<br />
*[[Websockets in MemCP]]<br />
<br />
==== Persistency Backends (= Storage) ====<br />
<br />
* [[File System]]<br />
* [[S3 Buckets]]<br />
* [[Ceph/Rados]]<br />
<br />
====Internals====<br />
<br />
=====How things work in MemCP===== <br />
<br />
*[[Databases, Tables and Columns]]<br />
*[[Shards, RecordIDs, Main Storage, Delta Storage]]<br />
*[[Columnar Storage]]<br />
*[[Transactions]] <br />
*[[Full SCM API documentation]] <br />
<br />
===== SCM Documentation =====<br />
<br />
* [[SCM Builtins]]<br />
* [[Arithmetic / Logic]]<br />
* [[Strings]]<br />
* [[Streams]]<br />
* [[Lists]]<br />
* [[Associative Lists / Dictionaries]]<br />
* [[Date]]<br />
* [[Vectors]]<br />
* [[Parsers]]<br />
* [[Sync]]<br />
* [[IO]]<br />
* [[Storage]]<br />
<br />
=====Optimizations=====<br />
*[[In-Memory Compression, Columnar Compression Techniques]]<br />
*[[Temporary Columns]]<br />
*[[Data Auto Sharding and Auto Indexing]]<br />
* [[Parallel Computing]]<br />
<br />
<br />
[[File:Screenshot from htop.png|center|frameless|2490x2490px]]<br />
<br />
<br />
===Further Reading===<br />
[https://github.org/launix-de/memcp MemCP on Github]<br />
<br />
====Scientific====<br />
<br />
*[https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]<br />
*[https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]<br />
*[https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]<br />
*[https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]<br />
*https://wwwdb.inf.tu-dresden.de/research-projects/eris/ <br />
<br />
====How MemCP was built====<br />
<br />
*[https://launix.de/launix/how-to-balance-a-database-between-olap-and-oltp-workflows/ Balancing OLAP and OLTP Workflows]<br />
*[https://launix.de/launix/designing-a-programming-language-for-distributed-systems-and-highly-parallel-algorithms/ Designing Programming Languages for Distributed Systems]<br />
*[https://launix.de/launix/on-designing-an-interface-for-columnar-in-memory-storage-in-golang/ Columnar Storage Interface in Golang]<br />
*[https://launix.de/launix/how-in-memory-compression-affects-performance/ Impact of In-Memory Compression on Performance]<br />
*[https://launix.de/launix/memory-efficient-indices-for-in-memory-storages/ Memory-Efficient Indices for In-Memory Storages]<br />
*[https://launix.de/launix/on-compressing-null-values-in-bit-compressed-integer-storages/ Compressing Null Values in Bit-Compressed Integer Storages]<br />
*[https://launix.de/launix/when-the-benchmark-is-too-slow-golang-http-server-performance/ Improving Golang HTTP Server Performance]<br />
*[https://launix.de/launix/how-to-benchmark-a-sql-database/ Benchmarking SQL Databases]<br />
*[https://launix.de/launix/writing-a-sql-parser-in-scheme/ Writing a SQL Parser in Scheme]<br />
*[https://launix.de/launix/accessing-memcp-via-scheme/ Accessing memcp via Scheme]<br />
*[https://launix.de/launix/memcp-first-sql-query-is-correctly-executed/ First SQL Query in memcp]<br />
*[https://launix.de/launix/sequence-compression-in-in-memory-database-yields-99-memory-savings-and-a-total-of-13/ Sequence Compression in In-Memory Database]<br />
*[https://launix.de/launix/storing-a-bit-smaller-than-in-one-bit/ Storing Data Smaller Than One Bit]<br />
*[https://www.youtube.com/watch?v=DWg4nx4KVLo memcp Announcement Video]</div>Carlihttps://www.memcp.org/index.php?title=Storage&diff=232Storage2025-08-25T18:10:16Z<p>Carli: Created page with "= Storage = The '''Storage''' module provides functions to manage databases, tables, columns, keys, partitions, and large data operations in SCM. ← Back to Full SCM API documentation == scan == Performs an unordered parallel filter-map-reduce on a table '''Allowed number of parameters:''' 6–10 '''Parameters:''' * '''schema''' (<code>string|nil</code>): database name * '''table''' (<code>string|list</code>): table name(s) * '''filterColumns''' (<code>list</c..."</p>
<hr />
<div>= Storage =<br />
<br />
The '''Storage''' module provides functions to manage databases, tables, columns, keys, partitions, and large data operations in SCM.<br />
<br />
← Back to [[Full SCM API documentation]]<br />
<br />
== scan ==<br />
<br />
Performs an unordered parallel filter-map-reduce on a table<br />
<br />
'''Allowed number of parameters:''' 6–10<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string|nil</code>): database name<br />
* '''table''' (<code>string|list</code>): table name(s)<br />
* '''filterColumns''' (<code>list</code>): filter columns<br />
* '''filter''' (<code>func</code>): lambda deciding dataset inclusion<br />
* '''mapColumns''' (<code>list</code>): map columns<br />
* '''map''' (<code>func</code>): lambda extracting/processing data<br />
* '''reduce''' (<code>func</code>): optional aggregator<br />
* '''neutral''' (<code>any</code>): neutral element<br />
* '''reduce2''' (<code>func</code>): optional second reducer<br />
* '''isOuter''' (<code>bool</code>): outer join–like behavior<br />
<br />
'''Returns:''' <code>any</code><br />
<br />
== scan_order ==<br />
<br />
Performs an ordered parallel filter with serial map-reduce<br />
<br />
'''Allowed number of parameters:''' 10–13<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database<br />
* '''table''' (<code>string</code>): table name<br />
* '''filterColumns''' (<code>list</code>): filter columns<br />
* '''filter''' (<code>func</code>): filter lambda<br />
* '''sortcols''' (<code>list</code>): sorting columns<br />
* '''sortdirs''' (<code>list</code>): sorting directions<br />
* '''offset''' (<code>number</code>): skip count<br />
* '''limit''' (<code>number</code>): max items<br />
* '''mapColumns''' (<code>list</code>): map columns<br />
* '''map''' (<code>func</code>): map lambda<br />
* '''reduce''' (<code>func</code>): optional reducer<br />
* '''neutral''' (<code>any</code>): neutral element<br />
* '''isOuter''' (<code>bool</code>): outer behavior<br />
<br />
'''Returns:''' <code>any</code><br />
<br />
== createdatabase ==<br />
<br />
Creates a new database<br />
<br />
'''Allowed number of parameters:''' 1–2<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database name<br />
* '''ignoreexists''' (<code>bool</code>): ignore if exists<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== dropdatabase ==<br />
<br />
Drops a database<br />
<br />
'''Allowed number of parameters:''' 1–1<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database name<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== createtable ==<br />
<br />
Creates a new table<br />
<br />
'''Allowed number of parameters:''' 4–5<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database<br />
* '''table''' (<code>string</code>): table name<br />
* '''cols''' (<code>list</code>): column definitions<br />
* '''options''' (<code>list</code>): table options<br />
* '''ifnotexists''' (<code>bool</code>): safe creation<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== createcolumn ==<br />
<br />
Creates a new column in a table<br />
<br />
'''Allowed number of parameters:''' 6–8<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database<br />
* '''table''' (<code>string</code>): table name<br />
* '''colname''' (<code>string</code>): column name<br />
* '''type''' (<code>string</code>): base type<br />
* '''dimensions''' (<code>list</code>): type dimensions<br />
* '''options''' (<code>list</code>): column options<br />
* '''computorCols''' (<code>list</code>): columns for computed value<br />
* '''computor''' (<code>func</code>): compute lambda<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== createkey ==<br />
<br />
Creates a new key<br />
<br />
'''Allowed number of parameters:''' 5–5<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database<br />
* '''table''' (<code>string</code>): table<br />
* '''keyname''' (<code>string</code>): key name<br />
* '''unique''' (<code>bool</code>): uniqueness flag<br />
* '''columns''' (<code>list</code>): key columns<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== createforeignkey ==<br />
<br />
Creates a new foreign key<br />
<br />
'''Allowed number of parameters:''' 8–8<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database<br />
* '''keyname''' (<code>string</code>): key name<br />
* '''table1''' (<code>string</code>): first table<br />
* '''columns1''' (<code>list</code>): first columns<br />
* '''table2''' (<code>string</code>): second table<br />
* '''columns2''' (<code>list</code>): second columns<br />
* '''updatemode''' (<code>string</code>): action on update<br />
* '''deletemode''' (<code>string</code>): action on delete<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== shardcolumn ==<br />
<br />
Suggests partitions for a column<br />
<br />
'''Allowed number of parameters:''' 3–4<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database<br />
* '''table''' (<code>string</code>): table<br />
* '''colname''' (<code>string</code>): column<br />
* '''numpartitions''' (<code>number</code>): partitions<br />
<br />
'''Returns:''' <code>list</code><br />
<br />
== partitiontable ==<br />
<br />
Applies or adjusts a partition scheme<br />
<br />
'''Allowed number of parameters:''' 3–3<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database<br />
* '''table''' (<code>string</code>): table<br />
* '''columns''' (<code>list</code>): column pivots<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== altertable ==<br />
<br />
Alters a table<br />
<br />
'''Allowed number of parameters:''' 4–4<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database<br />
* '''table''' (<code>string</code>): table<br />
* '''operation''' (<code>string</code>): alteration<br />
* '''parameter''' (<code>any</code>): parameter<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== altercolumn ==<br />
<br />
Alters a column<br />
<br />
'''Allowed number of parameters:''' 5–5<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database<br />
* '''table''' (<code>string</code>): table<br />
* '''column''' (<code>string</code>): column<br />
* '''operation''' (<code>string</code>): alteration<br />
* '''parameter''' (<code>any</code>): parameter<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== droptable ==<br />
<br />
Removes a table<br />
<br />
'''Allowed number of parameters:''' 2–3<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database<br />
* '''table''' (<code>string</code>): table<br />
* '''ifexists''' (<code>bool</code>): safe drop<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== insert ==<br />
<br />
Inserts a dataset into a table<br />
<br />
'''Allowed number of parameters:''' 4–7<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database<br />
* '''table''' (<code>string</code>): table<br />
* '''columns''' (<code>list</code>): column names<br />
* '''datasets''' (<code>list</code>): row values<br />
* '''onCollisionCols''' (<code>list</code>): collision columns<br />
* '''onCollision''' (<code>func</code>): collision handler<br />
* '''mergeNull''' (<code>bool</code>): handle null as equal<br />
<br />
'''Returns:''' <code>number</code><br />
<br />
== stat ==<br />
<br />
Returns memory statistics<br />
<br />
'''Allowed number of parameters:''' 0–2<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database (optional)<br />
* '''table''' (<code>string</code>): table (optional)<br />
<br />
'''Returns:''' <code>string</code><br />
<br />
== show ==<br />
<br />
Shows databases, tables, or columns<br />
<br />
'''Allowed number of parameters:''' 0–2<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): optional database<br />
* '''table''' (<code>string</code>): optional table<br />
<br />
'''Returns:''' <code>any</code><br />
<br />
== rebuild ==<br />
<br />
Rebuilds storages<br />
<br />
'''Allowed number of parameters:''' 0–2<br />
<br />
'''Parameters:'''<br />
* '''all''' (<code>bool</code>): rebuild all<br />
* '''repartition''' (<code>bool</code>): repartition<br />
<br />
'''Returns:''' <code>string</code><br />
<br />
== loadCSV ==<br />
<br />
Loads CSV into a table<br />
<br />
'''Allowed number of parameters:''' 3–5<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database<br />
* '''table''' (<code>string</code>): table<br />
* '''stream''' (<code>stream</code>): CSV stream<br />
* '''delimiter''' (<code>string</code>): delimiter<br />
* '''firstline''' (<code>bool</code>): use header<br />
<br />
'''Returns:''' <code>string</code><br />
<br />
== loadJSON ==<br />
<br />
Loads JSONL into a database<br />
<br />
'''Allowed number of parameters:''' 2–2<br />
<br />
'''Parameters:'''<br />
* '''schema''' (<code>string</code>): database<br />
* '''stream''' (<code>stream</code>): JSONL stream<br />
<br />
'''Returns:''' <code>string</code><br />
<br />
== settings ==<br />
<br />
Reads or writes global settings<br />
<br />
'''Allowed number of parameters:''' 1–2<br />
<br />
'''Parameters:'''<br />
* '''key''' (<code>string</code>): setting key<br />
* '''value''' (<code>any</code>): new value<br />
<br />
'''Returns:''' <code>any</code></div>Carlihttps://www.memcp.org/index.php?title=IO&diff=231IO2025-08-25T18:09:49Z<p>Carli: Created page with "= IO = The '''IO''' module provides functions for input and output operations, environment handling, file streaming, server control, and argument parsing in SCM. ← Back to Full SCM API documentation == print == Prints values to stdout (only in IO environment) '''Allowed number of parameters:''' 1–1000 '''Parameters:''' * '''value...''' (<code>any</code>): values to print '''Returns:''' <code>bool</code> == env == returns the content of a environment vari..."</p>
<hr />
<div>= IO =<br />
<br />
The '''IO''' module provides functions for input and output operations, environment handling, file streaming, server control, and argument parsing in SCM.<br />
<br />
← Back to [[Full SCM API documentation]]<br />
<br />
== print ==<br />
<br />
Prints values to stdout (only in IO environment)<br />
<br />
'''Allowed number of parameters:''' 1–1000<br />
<br />
'''Parameters:'''<br />
* '''value...''' (<code>any</code>): values to print<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== env ==<br />
<br />
returns the content of a environment variable<br />
<br />
'''Allowed number of parameters:''' 1–2<br />
<br />
'''Parameters:'''<br />
* '''var''' (<code>string</code>): envvar<br />
* '''default''' (<code>string</code>): default if the env is not found<br />
<br />
'''Returns:''' <code>string</code><br />
<br />
== help ==<br />
<br />
Lists all functions or prints help for a specific function<br />
<br />
'''Allowed number of parameters:''' 0–1<br />
<br />
'''Parameters:'''<br />
* '''topic''' (<code>string</code>): function to print help about<br />
<br />
'''Returns:''' <code>nil</code><br />
<br />
== import ==<br />
<br />
Imports a .scm file into current namespace<br />
<br />
'''Allowed number of parameters:''' 1–1<br />
<br />
'''Parameters:'''<br />
* '''filename''' (<code>string</code>): filename relative to folder of source file<br />
<br />
'''Returns:''' <code>any</code><br />
<br />
== load ==<br />
<br />
Loads a file or stream and returns the string or iterates line-wise<br />
<br />
'''Allowed number of parameters:''' 1–3<br />
<br />
'''Parameters:'''<br />
* '''filenameOrStream''' (<code>string|stream</code>): filename or stream<br />
* '''linehandler''' (<code>func</code>): handler for each line<br />
* '''delimiter''' (<code>string</code>): delimiter for line extraction<br />
<br />
'''Returns:''' <code>string|bool</code><br />
<br />
== stream ==<br />
<br />
Opens a file readonly as stream<br />
<br />
'''Allowed number of parameters:''' 1–1<br />
<br />
'''Parameters:'''<br />
* '''filename''' (<code>string</code>): filename<br />
<br />
'''Returns:''' <code>stream</code><br />
<br />
== watch ==<br />
<br />
Loads a file and calls a callback whenever the file changes<br />
<br />
'''Allowed number of parameters:''' 2–2<br />
<br />
'''Parameters:'''<br />
* '''filename''' (<code>string</code>): filename<br />
* '''updatehandler''' (<code>func</code>): handler that receives content<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== serve ==<br />
<br />
Opens a HTTP server at a given port<br />
<br />
'''Allowed number of parameters:''' 2–2<br />
<br />
'''Parameters:'''<br />
* '''port''' (<code>number</code>): port number<br />
* '''handler''' (<code>func</code>): lambda(req res)<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== serveStatic ==<br />
<br />
Creates a static handler for use in (serve)<br />
<br />
'''Allowed number of parameters:''' 1–1<br />
<br />
'''Parameters:'''<br />
* '''directory''' (<code>string</code>): folder with files<br />
<br />
'''Returns:''' <code>func</code><br />
<br />
== mysql ==<br />
<br />
Opens a MySQL server with custom handlers<br />
<br />
'''Allowed number of parameters:''' 4–4<br />
<br />
'''Parameters:'''<br />
* '''port''' (<code>number</code>): port<br />
* '''getPassword''' (<code>func</code>): password resolver<br />
* '''schemacallback''' (<code>func</code>): schema access checker<br />
* '''handler''' (<code>func</code>): SQL handler<br />
<br />
'''Returns:''' <code>bool</code><br />
<br />
== password ==<br />
<br />
Hashes a password with sha1<br />
<br />
'''Allowed number of parameters:''' 1–1<br />
<br />
'''Parameters:'''<br />
* '''password''' (<code>string</code>): plain text password<br />
<br />
'''Returns:''' <code>string</code><br />
<br />
== args ==<br />
<br />
Returns command line arguments<br />
<br />
'''Allowed number of parameters:''' 0–0<br />
<br />
'''Parameters:''' _none_<br />
<br />
'''Returns:''' <code>list</code><br />
<br />
== arg ==<br />
<br />
Gets a command line argument value<br />
<br />
'''Allowed number of parameters:''' 2–3<br />
<br />
'''Parameters:'''<br />
* '''longname''' (<code>string</code>): long argument name<br />
* '''shortname''' (<code>string</code>): short argument or default<br />
* '''default''' (<code>any</code>): fallback value<br />
<br />
'''Returns:''' <code>any</code></div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=224Main Page2025-08-25T16:44:40Z<p>Carli: </p>
<hr />
<div>== 🚀 Meet MemCP – The MySQL Alternative You've Been Waiting For ==<br />
'''Tired of outdated database engines slowing you down?'''<br />
<br />
MemCP is a blazing-fast, in-memory, column-oriented database designed for modern workloads. It delivers '''unmatched speed''', '''massive compression''', and '''built-in REST APIs''' – all with a developer-friendly open-source stack.<blockquote>'''Replace MySQL with something better.'''<br />
<br />
Something built for the 2020s – not stuck in 2005.</blockquote><br />
----<br />
<br />
=== ⚡ Why Developers Choose MemCP ===<br />
✅ '''Turbocharged Performance'''<br />
<br />
Designed for multicore CPUs, modern caches, and NVMe SSDs – MemCP handles analytical (OLAP) and transactional (OLTP) workloads at lightning speed.<br />
<br />
✅ '''80% Smaller Data Footprint'''<br />
<br />
Columnar storage means smarter compression. Typical use cases shrink data 5× smaller than MySQL – with no effort from your side.<br />
<br />
✅ '''Simple JSON APIs Built-in'''<br />
<br />
Query your data with RESTful endpoints – no middleware, no ORM overhead. Just fast results, directly from the DB.<br />
<br />
✅ '''Familiar SQL-like Structure – Zero Lock-In'''<br />
<br />
You don’t need to re-learn everything. Tables, schemas, and logic follow familiar patterns – just modernized and optimized.<br />
<br />
✅ '''Zero-Config Installation'''<br />
<br />
Spin up MemCP with a single <code>docker run</code> or <code>pm2 start</code>. Start building in minutes.<br />
----<br />
<br />
=== 🧑💻 Built for Developers, by Developers ===<br />
MemCP was made for the people who write real-world code, not just benchmark reports. Whether you're building a SaaS product, IoT platform, or data-heavy analytics dashboard, MemCP gives you the speed and flexibility you need to scale.<br />
<br />
# Ready to try it?<br />
git clone <nowiki>https://github.com/launix-de/memcp</nowiki><br />
cd memcp<br />
go get<br />
make<br />
pm2 start ./memcp ./data/<br />
<br />
mysql -u root -p -P 3307<br />
Enter password: admin<br />
<br />
=== 🆚 MemCP vs MySQL: The Key Differences ===<br />
{| class="wikitable"<br />
!Feature<br />
!MySQL<br />
!MemCP<br />
|-<br />
|Storage Model<br />
|Row-based<br />
|Column-based (compressed)<br />
|-<br />
|Performance<br />
|Good<br />
|Excellent (NUMA-optimized)<br />
|-<br />
|In-Memory Capable<br />
|Limited<br />
|Yes, by default<br />
|-<br />
|REST API Integration<br />
|External services<br />
|Built-in<br />
|-<br />
|Installation Footprint<br />
|~150MB+<br />
|~10MB<br />
|-<br />
|Open Source<br />
|✅<br />
|✅<br />
|}<br />
<br />
=== 💡 Ideal Use Cases ===<br />
<br />
* High-speed dashboards & real-time analytics<br />
* Embedded systems with minimal resources<br />
* Data-heavy SaaS backends<br />
* Replacing outdated MySQL deployments with modern performance<br />
<br />
<br />
=== 🧪 Try MemCP Today ===<br />
Explore the documentation, check out the GitHub repo, and join a growing community of developers building fast, efficient, modern systems with MemCP.<blockquote>Ditch the legacy. Embrace performance.<br />
<br />
'''Make the switch to MemCP.'''</blockquote><br />
----<br />
<br />
=== What is memcp? ===<br />
[[File:Webapps.svg|left|frameless]]<br />
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.<br />
<br />
memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.<br />
<br />
<br />
===Features===<br />
<br />
*'''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.<br />
*'''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB<br />
*'''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs<br />
* '''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime<br />
* Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.<br />
* In-memory database: Stores all data in memory, which allows for extremely fast query execution.<br />
*Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)<br />
*OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.<br />
*Compression: Lots of compression formats are supported like bit-packing and dictionary encoding<br />
* Scalability: Designed to scale on a single node with huge NUMA memory<br />
*Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time<br />
[[File:MemCP Port.png|frameless]]<br />
<br />
<youtube>g29FR4Jwius</youtube><br />
<br />
https://www.youtube.com/watch?v=g29FR4Jwius<br />
<br />
===Navigation===<br />
<br />
====Introduction==== <br />
*[[What is OLTP and OLAP]]<br />
*[[History of the MemCP project]]<br />
*[[Hardware Requirements]]<br />
*[[Persistency and Performance Guarantees]] <br />
*[[Current Status and Open Issues]]<br />
*[[Comparison: MemCP vs. MySQL]]<br />
<br />
====Getting Started====<br />
*[[Install MemCP with Docker|With Docker]]<br />
*[[With Singularity]]<br />
*[[Compile MemCP from Source|Build from Source]]<br />
*[[Contributing]] <br />
*[[Introduction to Scheme]]<br />
*[[Full SCM API documentation]]<br />
<br />
====Administration====<br />
<br />
* [[Deployment]]<br />
* [[Migration from MySQL and PostgreSQL]]<br />
* [[Settings]]<br />
*[[Process Hibernation]]<br />
*[[Performance Measurement]]<br />
*[[MemCP Console]]<br />
<br />
====Frontends====<br />
<br />
=====SQL Frontend===== <br />
*[[Supported SQL]]<br />
*[[Advanced SQL Tutorial]]<br />
*[[SQL over REST]]<br />
*[[Database Tools compatibility with MemCP|Supported Tooling]]<br />
*[[How SQL Operators are implemented on MemCP]]<br />
*[[Add custom SQL operators to MemCP]]<br />
<br />
=====RDF Frontend===== <br />
*[[Introduction to RDF]]<br />
*[[Advanced Graph Querying]]<br />
*[[RDF templating and model driven development]]<br />
<br />
=====Custom Frontends=====<br />
<br />
*[[In-Database WebApps|In-Database WebApps and REST Services]]<br />
*[[MemCP for Microservices]]<br />
*[[Websockets in MemCP]]<br />
<br />
====Internals====<br />
<br />
=====How things work in MemCP===== <br />
<br />
*[[Databases, Tables and Columns]]<br />
*[[Shards, RecordIDs, Main Storage, Delta Storage]]<br />
*[[Columnar Storage]]<br />
*[[Transactions]] <br />
*[[Full SCM API documentation]] <br />
<br />
===== SCM Documentation =====<br />
<br />
* [[SCM Builtins]]<br />
* [[Arithmetic / Logic]]<br />
* [[Strings]]<br />
* [[Streams]]<br />
* [[Lists]]<br />
* [[Associative Lists / Dictionaries]]<br />
* [[Date]]<br />
* [[Vectors]]<br />
* [[Parsers]]<br />
* [[Sync]]<br />
* [[IO]]<br />
* [[Storage]]<br />
<br />
=====Optimizations=====<br />
*[[In-Memory Compression, Columnar Compression Techniques]]<br />
*[[Temporary Columns]]<br />
*[[Data Auto Sharding and Auto Indexing]]<br />
* [[Parallel Computing]]<br />
<br />
<br />
[[File:Screenshot from htop.png|center|frameless|2490x2490px]]<br />
<br />
<br />
===Further Reading===<br />
[https://github.org/launix-de/memcp MemCP on Github]<br />
<br />
====Scientific====<br />
<br />
*[https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]<br />
*[https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]<br />
*[https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]<br />
*[https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]<br />
*https://wwwdb.inf.tu-dresden.de/research-projects/eris/ <br />
<br />
====How MemCP was built====<br />
<br />
*[https://launix.de/launix/how-to-balance-a-database-between-olap-and-oltp-workflows/ Balancing OLAP and OLTP Workflows]<br />
*[https://launix.de/launix/designing-a-programming-language-for-distributed-systems-and-highly-parallel-algorithms/ Designing Programming Languages for Distributed Systems]<br />
*[https://launix.de/launix/on-designing-an-interface-for-columnar-in-memory-storage-in-golang/ Columnar Storage Interface in Golang]<br />
*[https://launix.de/launix/how-in-memory-compression-affects-performance/ Impact of In-Memory Compression on Performance]<br />
*[https://launix.de/launix/memory-efficient-indices-for-in-memory-storages/ Memory-Efficient Indices for In-Memory Storages]<br />
*[https://launix.de/launix/on-compressing-null-values-in-bit-compressed-integer-storages/ Compressing Null Values in Bit-Compressed Integer Storages]<br />
*[https://launix.de/launix/when-the-benchmark-is-too-slow-golang-http-server-performance/ Improving Golang HTTP Server Performance]<br />
*[https://launix.de/launix/how-to-benchmark-a-sql-database/ Benchmarking SQL Databases]<br />
*[https://launix.de/launix/writing-a-sql-parser-in-scheme/ Writing a SQL Parser in Scheme]<br />
*[https://launix.de/launix/accessing-memcp-via-scheme/ Accessing memcp via Scheme]<br />
*[https://launix.de/launix/memcp-first-sql-query-is-correctly-executed/ First SQL Query in memcp]<br />
*[https://launix.de/launix/sequence-compression-in-in-memory-database-yields-99-memory-savings-and-a-total-of-13/ Sequence Compression in In-Memory Database]<br />
*[https://launix.de/launix/storing-a-bit-smaller-than-in-one-bit/ Storing Data Smaller Than One Bit]<br />
*[https://www.youtube.com/watch?v=DWg4nx4KVLo memcp Announcement Video]</div>Carlihttps://www.memcp.org/index.php?title=Full_SCM_API_documentation&diff=216Full SCM API documentation2025-08-25T16:06:48Z<p>Carli: Created page with "= Documentation = * SCM Builtins * Arithmetic / Logic * Strings * Streams * Lists * Associative Lists / Dictionaries * Date * Vectors * Parsers * Sync * IO * Storage"</p>
<hr />
<div>= Documentation =<br />
<br />
* [[SCM Builtins]]<br />
* [[Arithmetic / Logic]]<br />
* [[Strings]]<br />
* [[Streams]]<br />
* [[Lists]]<br />
* [[Associative Lists / Dictionaries]]<br />
* [[Date]]<br />
* [[Vectors]]<br />
* [[Parsers]]<br />
* [[Sync]]<br />
* [[IO]]<br />
* [[Storage]]</div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=215Main Page2025-08-25T16:03:07Z<p>Carli: /* How things work in MemCP */</p>
<hr />
<div>== 🚀 Meet MemCP – The MySQL Alternative You've Been Waiting For ==<br />
'''Tired of outdated database engines slowing you down?'''<br />
<br />
MemCP is a blazing-fast, in-memory, column-oriented database designed for modern workloads. It delivers '''unmatched speed''', '''massive compression''', and '''built-in REST APIs''' – all with a developer-friendly open-source stack.<blockquote>'''Replace MySQL with something better.'''<br />
<br />
Something built for the 2020s – not stuck in 2005.</blockquote><br />
----<br />
<br />
=== ⚡ Why Developers Choose MemCP ===<br />
✅ '''Turbocharged Performance'''<br />
<br />
Designed for multicore CPUs, modern caches, and NVMe SSDs – MemCP handles analytical (OLAP) and transactional (OLTP) workloads at lightning speed.<br />
<br />
✅ '''80% Smaller Data Footprint'''<br />
<br />
Columnar storage means smarter compression. Typical use cases shrink data 5× smaller than MySQL – with no effort from your side.<br />
<br />
✅ '''Simple JSON APIs Built-in'''<br />
<br />
Query your data with RESTful endpoints – no middleware, no ORM overhead. Just fast results, directly from the DB.<br />
<br />
✅ '''Familiar SQL-like Structure – Zero Lock-In'''<br />
<br />
You don’t need to re-learn everything. Tables, schemas, and logic follow familiar patterns – just modernized and optimized.<br />
<br />
✅ '''Zero-Config Installation'''<br />
<br />
Spin up MemCP with a single <code>docker run</code> or <code>pm2 start</code>. Start building in minutes.<br />
----<br />
<br />
=== 🧑💻 Built for Developers, by Developers ===<br />
MemCP was made for the people who write real-world code, not just benchmark reports. Whether you're building a SaaS product, IoT platform, or data-heavy analytics dashboard, MemCP gives you the speed and flexibility you need to scale.<br />
<br />
# Ready to try it?<br />
git clone <nowiki>https://github.com/launix-de/memcp</nowiki><br />
cd memcp<br />
go get<br />
make<br />
pm2 start ./memcp ./data/<br />
<br />
mysql -u root -p -P 3307<br />
Enter password: admin<br />
<br />
=== 🆚 MemCP vs MySQL: The Key Differences ===<br />
{| class="wikitable"<br />
!Feature<br />
!MySQL<br />
!MemCP<br />
|-<br />
|Storage Model<br />
|Row-based<br />
|Column-based (compressed)<br />
|-<br />
|Performance<br />
|Good<br />
|Excellent (NUMA-optimized)<br />
|-<br />
|In-Memory Capable<br />
|Limited<br />
|Yes, by default<br />
|-<br />
|REST API Integration<br />
|External services<br />
|Built-in<br />
|-<br />
|Installation Footprint<br />
|~150MB+<br />
|~10MB<br />
|-<br />
|Open Source<br />
|✅<br />
|✅<br />
|}<br />
<br />
=== 💡 Ideal Use Cases ===<br />
<br />
* High-speed dashboards & real-time analytics<br />
* Embedded systems with minimal resources<br />
* Data-heavy SaaS backends<br />
* Replacing outdated MySQL deployments with modern performance<br />
<br />
<br />
=== 🧪 Try MemCP Today ===<br />
Explore the documentation, check out the GitHub repo, and join a growing community of developers building fast, efficient, modern systems with MemCP.<blockquote>Ditch the legacy. Embrace performance.<br />
<br />
'''Make the switch to MemCP.'''</blockquote><br />
----<br />
<br />
=== What is memcp? ===<br />
[[File:Webapps.svg|left|frameless]]<br />
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.<br />
<br />
memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.<br />
<br />
<br />
===Features===<br />
<br />
*'''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.<br />
*'''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB<br />
*'''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs<br />
* '''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime<br />
* Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.<br />
* In-memory database: Stores all data in memory, which allows for extremely fast query execution.<br />
*Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)<br />
*OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.<br />
*Compression: Lots of compression formats are supported like bit-packing and dictionary encoding<br />
* Scalability: Designed to scale on a single node with huge NUMA memory<br />
*Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time<br />
[[File:MemCP Port.png|frameless]]<br />
<br />
<youtube>g29FR4Jwius</youtube><br />
<br />
https://www.youtube.com/watch?v=g29FR4Jwius<br />
<br />
===Navigation===<br />
<br />
====Introduction==== <br />
*[[What is OLTP and OLAP]]<br />
*[[History of the MemCP project]]<br />
*[[Hardware Requirements]]<br />
*[[Persistency and Performance Guarantees]] <br />
*[[Current Status and Open Issues]]<br />
*[[Comparison: MemCP vs. MySQL]]<br />
<br />
====Getting Started====<br />
*[[Install MemCP with Docker|With Docker]]<br />
*[[With Singularity]]<br />
*[[Compile MemCP from Source|Build from Source]]<br />
*[[Contributing]] <br />
*[[Introduction to Scheme]]<br />
*[[Full SCM API documentation]]<br />
<br />
====Administration====<br />
<br />
* [[Deployment]]<br />
* [[Migration from MySQL and PostgreSQL]]<br />
* [[Settings]]<br />
*[[Process Hibernation]]<br />
*[[Performance Measurement]]<br />
*[[MemCP Console]]<br />
<br />
====Frontends====<br />
<br />
=====SQL Frontend===== <br />
*[[Supported SQL]]<br />
*[[Advanced SQL Tutorial]]<br />
*[[SQL over REST]]<br />
*[[Database Tools compatibility with MemCP|Supported Tooling]]<br />
*[[How SQL Operators are implemented on MemCP]]<br />
*[[Add custom SQL operators to MemCP]]<br />
<br />
=====RDF Frontend===== <br />
*[[Introduction to RDF]]<br />
*[[Advanced Graph Querying]]<br />
*[[RDF templating and model driven development]]<br />
<br />
=====Custom Frontends=====<br />
<br />
*[[In-Database WebApps|In-Database WebApps and REST Services]]<br />
*[[MemCP for Microservices]]<br />
*[[Websockets in MemCP]]<br />
<br />
====Internals====<br />
<br />
=====How things work in MemCP===== <br />
<br />
*[[Databases, Tables and Columns]]<br />
*[[Shards, RecordIDs, Main Storage, Delta Storage]]<br />
*[[Columnar Storage]]<br />
*[[Transactions]] <br />
*[[Full SCM API documentation]] <br />
<br />
=====Optimizations=====<br />
*[[In-Memory Compression, Columnar Compression Techniques]]<br />
*[[Temporary Columns]]<br />
*[[Data Auto Sharding and Auto Indexing]]<br />
* [[Parallel Computing]]<br />
<br />
<br />
[[File:Screenshot from htop.png|center|frameless|2490x2490px]]<br />
<br />
<br />
===Further Reading===<br />
[https://github.org/launix-de/memcp MemCP on Github]<br />
<br />
====Scientific====<br />
<br />
*[https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]<br />
*[https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]<br />
*[https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]<br />
*[https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]<br />
*https://wwwdb.inf.tu-dresden.de/research-projects/eris/ <br />
<br />
====How MemCP was built====<br />
<br />
*[https://launix.de/launix/how-to-balance-a-database-between-olap-and-oltp-workflows/ Balancing OLAP and OLTP Workflows]<br />
*[https://launix.de/launix/designing-a-programming-language-for-distributed-systems-and-highly-parallel-algorithms/ Designing Programming Languages for Distributed Systems]<br />
*[https://launix.de/launix/on-designing-an-interface-for-columnar-in-memory-storage-in-golang/ Columnar Storage Interface in Golang]<br />
*[https://launix.de/launix/how-in-memory-compression-affects-performance/ Impact of In-Memory Compression on Performance]<br />
*[https://launix.de/launix/memory-efficient-indices-for-in-memory-storages/ Memory-Efficient Indices for In-Memory Storages]<br />
*[https://launix.de/launix/on-compressing-null-values-in-bit-compressed-integer-storages/ Compressing Null Values in Bit-Compressed Integer Storages]<br />
*[https://launix.de/launix/when-the-benchmark-is-too-slow-golang-http-server-performance/ Improving Golang HTTP Server Performance]<br />
*[https://launix.de/launix/how-to-benchmark-a-sql-database/ Benchmarking SQL Databases]<br />
*[https://launix.de/launix/writing-a-sql-parser-in-scheme/ Writing a SQL Parser in Scheme]<br />
*[https://launix.de/launix/accessing-memcp-via-scheme/ Accessing memcp via Scheme]<br />
*[https://launix.de/launix/memcp-first-sql-query-is-correctly-executed/ First SQL Query in memcp]<br />
*[https://launix.de/launix/sequence-compression-in-in-memory-database-yields-99-memory-savings-and-a-total-of-13/ Sequence Compression in In-Memory Database]<br />
*[https://launix.de/launix/storing-a-bit-smaller-than-in-one-bit/ Storing Data Smaller Than One Bit]<br />
*[https://www.youtube.com/watch?v=DWg4nx4KVLo memcp Announcement Video]</div>Carlihttps://www.memcp.org/index.php?title=Introduction_to_Scheme&diff=214Introduction to Scheme2025-08-25T16:02:59Z<p>Carli: /* Deeper Topics into Scheme */</p>
<hr />
<div>When you run <code>./memcp</code>, you will be dropped at a scheme shell like this:<br />
memcp Copyright (C) 2023, 2024 Carl-Philip Hänsch<br />
This program comes with ABSOLUTELY NO WARRANTY;<br />
This is free software, and you are welcome to redistribute it<br />
under certain conditions;<br />
<br />
loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-id of type 10<br />
loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-username of type 20<br />
loading storage /tmp/x/system/bdf64a22-6315-463c-bbf7-329317ad50ed-password of type 20<br />
<br />
Welcome to memcp<br />
<br />
performing unit tests ...<br />
finished unit tests<br />
test result: 15/15<br />
all tests succeeded.<br />
<br />
Initializing SQL frontend<br />
MySQL server listening on port 3307 (connect with `mysql -P 3307 -u root -p` using password 'admin')<br />
listening on <nowiki>http://localhost:4321</nowiki><br />
<br />
<br />
Type (help) to show help<br />
<br />
> <br />
Scheme is a functional programming language and a subset of LISP. Every expression is either a primitive value or a list.<br />
<br />
To understand the semantics of Scheme, take a look at the following Scheme REPL session:<br />
> 12<br />
= 12<br />
<br />
> "hi"<br />
= "hi"<br />
<br />
> (+ 1 2)<br />
= 3<br />
<br />
> (+ 1 2 3)<br />
= 6<br />
<br />
> (+ 2 (* 2 2))<br />
= 6<br />
<br />
> (+ 2 (* 2 4))<br />
= 10<br />
<br />
> (concat "Hello" "World")<br />
= "HelloWorld"<br />
<br />
== Syntax ==<br />
<br />
=== Function Call ===<br />
A function call has the following format:<br />
(functionname param1 param2 param3 ...)<br />
Function calls start with <code>(</code>, contain one function and zero or more parameters separated by space .<br />
<br />
Some basic functions are <code>+ - * / print concat</code>. For more consult the manual with <code>(help)</code>.<br />
<br />
=== Literals ===<br />
The following literals are allowed:<br />
<br />
* Number Literals: <code>1 2 3 56 7.5 4.3e20 -6.123e-3</code><br />
* String Literals: <code>"Hello World" "First Line\nSecond Line" "he said: \"what?\" and smiled"</code><br />
* Function names: <code>print</code><br />
* Symbol literals <code>'print</code><br />
* List literals <code>'(1 2 3) '("a" "b" "c")</code><br />
* Associative Array literals <code>'("key" "value" "size" 45 "name" "Peter" "sublist" '(1 2 3))</code> - associative arrays are lists with key value pairs flattened down to a one dimensional list of an even number of items.<br />
* Lambdas: <code>(lambda (param1 param2 ...) body)</code><br />
<br />
=== Lambda Functions ===<br />
Scheme allows for creating lambda functions which enclose their own scope:<br />
(define decrease-by-one (lambda (number) (- number 1)))<br />
The function can now be called from scheme:<br />
> (decrease-by-one 5)<br />
= 4<br />
Lambda functions can be passed to other functions as values. They enclose your original scope, so you can use variables from the outside inside your function.<br />
<br />
== Deeper Topics into Scheme ==<br />
You can deep-dive into the following topcis:<br />
<br />
* [[Full SCM API documentation]]<br />
* [[Lists and Objects]]<br />
* [[Pattern Matching]]<br />
* [[Parsers]]</div>Carlihttps://www.memcp.org/index.php?title=Dictionary_Compression&diff=213Dictionary Compression2025-08-22T19:06:13Z<p>Carli: Created page with "Strings are the nightmare of every database. You never know their exact size and you have to either reference them in a separate storage or reserve enough character space to store them in-table. Columnar databases can do better. But that's not guaranteed, you have to put a bit of extra effort into it. Here's how: At first, memcp converts strings into dictionaries. If you have a large list consisting of [Male, Male, Male, Female, Male, Female, Male, Male], you only have..."</p>
<hr />
<div>Strings are the nightmare of every database. You never know their exact size and you have to either reference them in a separate storage or reserve enough character space to store them in-table.<br />
<br />
Columnar databases can do better. But that's not guaranteed, you have to put a bit of extra effort into it. Here's how:<br />
<br />
At first, memcp converts strings into dictionaries. If you have a large list consisting of [Male, Male, Male, Female, Male, Female, Male, Male], you only have two distinct values. So the dictionary can hold "Male,Female" and then you can reference them by integers (0,0,0,1,0,1,0,0).<br />
<br />
Then, these integers can be [[Integer Compression|Integer Compressed]]. And if that's not enough, you can also add [[Sequence Compression]].<br />
<br />
With dictionary compression, you can achieve compression ratios of 10x and higher.<br />
<br />
Here's an example from <code>(print (stat schema table))</code><br />
Shard 6254<br />
---<br />
main count: 61440, delta count: 0, deletions: 0<br />
mode: '''string-dict[1 entries; 5 bytes]''', size = 7.833KiB<br />
cntin: seq[176x int[20]/int[20]], size = 1.477KiB<br />
cntout: seq[187x int[6]/int[7]], size = 968B<br />
duration: int[15], size = 112.6KiB<br />
filters: seq[161x int[1]/int[2]], size = 680B<br />
append: seq[176x int[6]/int[7]], size = 928B<br />
p: '''string-dict[3 entries; 43 bytes]''', size = 15.36KiB<br />
---<br />
---<br />
+ insertions 40B<br />
+ deletions 48B<br />
---<br />
= total 140KiB<br />
As you can see, the dictionary can store massive amounts of data (61k items) in 15.4 KiB for a 3-way dictionary. That's 2 bits per item!</div>Carlihttps://www.memcp.org/index.php?title=Add_custom_SQL_operators_to_MemCP&diff=212Add custom SQL operators to MemCP2025-06-15T12:59:44Z<p>Carli: Created page with "Some special applications need user defined custom functions. You can define them in scheme like: (sql_builtins "ADD_1" (lambda (x) (+ x 1))) Now you can call the function by: SELECT ADD_1(4) AS result"</p>
<hr />
<div>Some special applications need user defined custom functions.<br />
<br />
You can define them in scheme like:<br />
(sql_builtins "ADD_1" (lambda (x) (+ x 1)))<br />
Now you can call the function by:<br />
SELECT ADD_1(4) AS result</div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=211Main Page2025-06-15T12:57:39Z<p>Carli: /* SQL Frontend */</p>
<hr />
<div>== 🚀 Meet MemCP – The MySQL Alternative You've Been Waiting For ==<br />
'''Tired of outdated database engines slowing you down?'''<br />
<br />
MemCP is a blazing-fast, in-memory, column-oriented database designed for modern workloads. It delivers '''unmatched speed''', '''massive compression''', and '''built-in REST APIs''' – all with a developer-friendly open-source stack.<blockquote>'''Replace MySQL with something better.'''<br />
<br />
Something built for the 2020s – not stuck in 2005.</blockquote><br />
----<br />
<br />
=== ⚡ Why Developers Choose MemCP ===<br />
✅ '''Turbocharged Performance'''<br />
<br />
Designed for multicore CPUs, modern caches, and NVMe SSDs – MemCP handles analytical (OLAP) and transactional (OLTP) workloads at lightning speed.<br />
<br />
✅ '''80% Smaller Data Footprint'''<br />
<br />
Columnar storage means smarter compression. Typical use cases shrink data 5× smaller than MySQL – with no effort from your side.<br />
<br />
✅ '''Simple JSON APIs Built-in'''<br />
<br />
Query your data with RESTful endpoints – no middleware, no ORM overhead. Just fast results, directly from the DB.<br />
<br />
✅ '''Familiar SQL-like Structure – Zero Lock-In'''<br />
<br />
You don’t need to re-learn everything. Tables, schemas, and logic follow familiar patterns – just modernized and optimized.<br />
<br />
✅ '''Zero-Config Installation'''<br />
<br />
Spin up MemCP with a single <code>docker run</code> or <code>pm2 start</code>. Start building in minutes.<br />
----<br />
<br />
=== 🧑💻 Built for Developers, by Developers ===<br />
MemCP was made for the people who write real-world code, not just benchmark reports. Whether you're building a SaaS product, IoT platform, or data-heavy analytics dashboard, MemCP gives you the speed and flexibility you need to scale.<br />
<br />
# Ready to try it?<br />
git clone <nowiki>https://github.com/launix-de/memcp</nowiki><br />
cd memcp<br />
go get<br />
make<br />
pm2 start ./memcp ./data/<br />
<br />
mysql -u root -p -P 3307<br />
Enter password: admin<br />
<br />
=== 🆚 MemCP vs MySQL: The Key Differences ===<br />
{| class="wikitable"<br />
!Feature<br />
!MySQL<br />
!MemCP<br />
|-<br />
|Storage Model<br />
|Row-based<br />
|Column-based (compressed)<br />
|-<br />
|Performance<br />
|Good<br />
|Excellent (NUMA-optimized)<br />
|-<br />
|In-Memory Capable<br />
|Limited<br />
|Yes, by default<br />
|-<br />
|REST API Integration<br />
|External services<br />
|Built-in<br />
|-<br />
|Installation Footprint<br />
|~150MB+<br />
|~10MB<br />
|-<br />
|Open Source<br />
|✅<br />
|✅<br />
|}<br />
<br />
=== 💡 Ideal Use Cases ===<br />
<br />
* High-speed dashboards & real-time analytics<br />
* Embedded systems with minimal resources<br />
* Data-heavy SaaS backends<br />
* Replacing outdated MySQL deployments with modern performance<br />
<br />
<br />
=== 🧪 Try MemCP Today ===<br />
Explore the documentation, check out the GitHub repo, and join a growing community of developers building fast, efficient, modern systems with MemCP.<blockquote>Ditch the legacy. Embrace performance.<br />
<br />
'''Make the switch to MemCP.'''</blockquote><br />
----<br />
<br />
=== What is memcp? ===<br />
[[File:Webapps.svg|left|frameless]]<br />
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.<br />
<br />
memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.<br />
<br />
<br />
===Features===<br />
<br />
*'''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.<br />
*'''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB<br />
*'''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs<br />
* '''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime<br />
* Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.<br />
* In-memory database: Stores all data in memory, which allows for extremely fast query execution.<br />
*Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)<br />
*OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.<br />
*Compression: Lots of compression formats are supported like bit-packing and dictionary encoding<br />
* Scalability: Designed to scale on a single node with huge NUMA memory<br />
*Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time<br />
[[File:MemCP Port.png|frameless]]<br />
<br />
<youtube>g29FR4Jwius</youtube><br />
<br />
https://www.youtube.com/watch?v=g29FR4Jwius<br />
<br />
===Navigation===<br />
<br />
====Introduction==== <br />
*[[What is OLTP and OLAP]]<br />
*[[History of the MemCP project]]<br />
*[[Hardware Requirements]]<br />
*[[Persistency and Performance Guarantees]] <br />
*[[Current Status and Open Issues]]<br />
*[[Comparison: MemCP vs. MySQL]]<br />
<br />
====Getting Started====<br />
*[[Install MemCP with Docker|With Docker]]<br />
*[[With Singularity]]<br />
*[[Compile MemCP from Source|Build from Source]]<br />
*[[Contributing]] <br />
*[[Introduction to Scheme]]<br />
<br />
====Administration====<br />
<br />
* [[Deployment]]<br />
* [[Migration from MySQL and PostgreSQL]]<br />
* [[Settings]]<br />
*[[Process Hibernation]]<br />
*[[Performance Measurement]]<br />
*[[MemCP Console]]<br />
<br />
====Frontends====<br />
<br />
=====SQL Frontend===== <br />
*[[Supported SQL]]<br />
*[[Advanced SQL Tutorial]]<br />
*[[SQL over REST]]<br />
*[[Database Tools compatibility with MemCP|Supported Tooling]]<br />
*[[How SQL Operators are implemented on MemCP]]<br />
*[[Add custom SQL operators to MemCP]]<br />
<br />
=====RDF Frontend===== <br />
*[[Introduction to RDF]]<br />
*[[Advanced Graph Querying]]<br />
*[[RDF templating and model driven development]]<br />
<br />
=====Custom Frontends=====<br />
<br />
*[[In-Database WebApps|In-Database WebApps and REST Services]]<br />
*[[MemCP for Microservices]]<br />
*[[Websockets in MemCP]]<br />
<br />
====Internals====<br />
<br />
=====How things work in MemCP===== <br />
<br />
*[[Databases, Tables and Columns]]<br />
*[[Shards, RecordIDs, Main Storage, Delta Storage]]<br />
*[[Columnar Storage]]<br />
*[[Transactions]] <br />
<br />
=====Optimizations=====<br />
*[[In-Memory Compression, Columnar Compression Techniques]]<br />
*[[Temporary Columns]]<br />
*[[Data Auto Sharding and Auto Indexing]]<br />
* [[Parallel Computing]]<br />
<br />
<br />
[[File:Screenshot from htop.png|center|frameless|2490x2490px]]<br />
<br />
<br />
===Further Reading===<br />
[https://github.org/launix-de/memcp MemCP on Github]<br />
<br />
====Scientific====<br />
<br />
*[https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]<br />
*[https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]<br />
*[https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]<br />
*[https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]<br />
*https://wwwdb.inf.tu-dresden.de/research-projects/eris/ <br />
<br />
====How MemCP was built====<br />
<br />
*[https://launix.de/launix/how-to-balance-a-database-between-olap-and-oltp-workflows/ Balancing OLAP and OLTP Workflows]<br />
*[https://launix.de/launix/designing-a-programming-language-for-distributed-systems-and-highly-parallel-algorithms/ Designing Programming Languages for Distributed Systems]<br />
*[https://launix.de/launix/on-designing-an-interface-for-columnar-in-memory-storage-in-golang/ Columnar Storage Interface in Golang]<br />
*[https://launix.de/launix/how-in-memory-compression-affects-performance/ Impact of In-Memory Compression on Performance]<br />
*[https://launix.de/launix/memory-efficient-indices-for-in-memory-storages/ Memory-Efficient Indices for In-Memory Storages]<br />
*[https://launix.de/launix/on-compressing-null-values-in-bit-compressed-integer-storages/ Compressing Null Values in Bit-Compressed Integer Storages]<br />
*[https://launix.de/launix/when-the-benchmark-is-too-slow-golang-http-server-performance/ Improving Golang HTTP Server Performance]<br />
*[https://launix.de/launix/how-to-benchmark-a-sql-database/ Benchmarking SQL Databases]<br />
*[https://launix.de/launix/writing-a-sql-parser-in-scheme/ Writing a SQL Parser in Scheme]<br />
*[https://launix.de/launix/accessing-memcp-via-scheme/ Accessing memcp via Scheme]<br />
*[https://launix.de/launix/memcp-first-sql-query-is-correctly-executed/ First SQL Query in memcp]<br />
*[https://launix.de/launix/sequence-compression-in-in-memory-database-yields-99-memory-savings-and-a-total-of-13/ Sequence Compression in In-Memory Database]<br />
*[https://launix.de/launix/storing-a-bit-smaller-than-in-one-bit/ Storing Data Smaller Than One Bit]<br />
*[https://www.youtube.com/watch?v=DWg4nx4KVLo memcp Announcement Video]</div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=210Main Page2025-05-30T09:43:09Z<p>Carli: /* 🧑💻 Built for Developers, by Developers */</p>
<hr />
<div>== 🚀 Meet MemCP – The MySQL Alternative You've Been Waiting For ==<br />
'''Tired of outdated database engines slowing you down?'''<br />
<br />
MemCP is a blazing-fast, in-memory, column-oriented database designed for modern workloads. It delivers '''unmatched speed''', '''massive compression''', and '''built-in REST APIs''' – all with a developer-friendly open-source stack.<blockquote>'''Replace MySQL with something better.'''<br />
<br />
Something built for the 2020s – not stuck in 2005.</blockquote><br />
----<br />
<br />
=== ⚡ Why Developers Choose MemCP ===<br />
✅ '''Turbocharged Performance'''<br />
<br />
Designed for multicore CPUs, modern caches, and NVMe SSDs – MemCP handles analytical (OLAP) and transactional (OLTP) workloads at lightning speed.<br />
<br />
✅ '''80% Smaller Data Footprint'''<br />
<br />
Columnar storage means smarter compression. Typical use cases shrink data 5× smaller than MySQL – with no effort from your side.<br />
<br />
✅ '''Simple JSON APIs Built-in'''<br />
<br />
Query your data with RESTful endpoints – no middleware, no ORM overhead. Just fast results, directly from the DB.<br />
<br />
✅ '''Familiar SQL-like Structure – Zero Lock-In'''<br />
<br />
You don’t need to re-learn everything. Tables, schemas, and logic follow familiar patterns – just modernized and optimized.<br />
<br />
✅ '''Zero-Config Installation'''<br />
<br />
Spin up MemCP with a single <code>docker run</code> or <code>pm2 start</code>. Start building in minutes.<br />
----<br />
<br />
=== 🧑💻 Built for Developers, by Developers ===<br />
MemCP was made for the people who write real-world code, not just benchmark reports. Whether you're building a SaaS product, IoT platform, or data-heavy analytics dashboard, MemCP gives you the speed and flexibility you need to scale.<br />
<br />
# Ready to try it?<br />
git clone <nowiki>https://github.com/launix-de/memcp</nowiki><br />
cd memcp<br />
go get<br />
make<br />
pm2 start ./memcp ./data/<br />
<br />
mysql -u root -p -P 3307<br />
Enter password: admin<br />
<br />
=== 🆚 MemCP vs MySQL: The Key Differences ===<br />
{| class="wikitable"<br />
!Feature<br />
!MySQL<br />
!MemCP<br />
|-<br />
|Storage Model<br />
|Row-based<br />
|Column-based (compressed)<br />
|-<br />
|Performance<br />
|Good<br />
|Excellent (NUMA-optimized)<br />
|-<br />
|In-Memory Capable<br />
|Limited<br />
|Yes, by default<br />
|-<br />
|REST API Integration<br />
|External services<br />
|Built-in<br />
|-<br />
|Installation Footprint<br />
|~150MB+<br />
|~10MB<br />
|-<br />
|Open Source<br />
|✅<br />
|✅<br />
|}<br />
<br />
=== 💡 Ideal Use Cases ===<br />
<br />
* High-speed dashboards & real-time analytics<br />
* Embedded systems with minimal resources<br />
* Data-heavy SaaS backends<br />
* Replacing outdated MySQL deployments with modern performance<br />
<br />
<br />
=== 🧪 Try MemCP Today ===<br />
Explore the documentation, check out the GitHub repo, and join a growing community of developers building fast, efficient, modern systems with MemCP.<blockquote>Ditch the legacy. Embrace performance.<br />
<br />
'''Make the switch to MemCP.'''</blockquote><br />
----<br />
<br />
=== What is memcp? ===<br />
[[File:Webapps.svg|left|frameless]]<br />
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.<br />
<br />
memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.<br />
<br />
<br />
===Features===<br />
<br />
*'''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.<br />
*'''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB<br />
*'''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs<br />
* '''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime<br />
* Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.<br />
* In-memory database: Stores all data in memory, which allows for extremely fast query execution.<br />
*Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)<br />
*OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.<br />
*Compression: Lots of compression formats are supported like bit-packing and dictionary encoding<br />
* Scalability: Designed to scale on a single node with huge NUMA memory<br />
*Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time<br />
[[File:MemCP Port.png|frameless]]<br />
<br />
<youtube>g29FR4Jwius</youtube><br />
<br />
https://www.youtube.com/watch?v=g29FR4Jwius<br />
<br />
===Navigation===<br />
<br />
====Introduction==== <br />
*[[What is OLTP and OLAP]]<br />
*[[History of the MemCP project]]<br />
*[[Hardware Requirements]]<br />
*[[Persistency and Performance Guarantees]] <br />
*[[Current Status and Open Issues]]<br />
*[[Comparison: MemCP vs. MySQL]]<br />
<br />
====Getting Started====<br />
*[[Install MemCP with Docker|With Docker]]<br />
*[[With Singularity]]<br />
*[[Compile MemCP from Source|Build from Source]]<br />
*[[Contributing]] <br />
*[[Introduction to Scheme]]<br />
<br />
====Administration====<br />
<br />
* [[Deployment]]<br />
* [[Migration from MySQL and PostgreSQL]]<br />
* [[Settings]]<br />
*[[Process Hibernation]]<br />
*[[Performance Measurement]]<br />
*[[MemCP Console]]<br />
<br />
====Frontends====<br />
<br />
=====SQL Frontend===== <br />
*[[Supported SQL]]<br />
*[[Advanced SQL Tutorial]]<br />
*[[SQL over REST]]<br />
*[[Database Tools compatibility with MemCP|Supported Tooling]]<br />
*[[How SQL Operators are implemented on MemCP]]<br />
<br />
=====RDF Frontend===== <br />
*[[Introduction to RDF]]<br />
*[[Advanced Graph Querying]]<br />
*[[RDF templating and model driven development]]<br />
<br />
=====Custom Frontends=====<br />
<br />
*[[In-Database WebApps|In-Database WebApps and REST Services]]<br />
*[[MemCP for Microservices]]<br />
*[[Websockets in MemCP]]<br />
<br />
====Internals====<br />
<br />
=====How things work in MemCP===== <br />
<br />
*[[Databases, Tables and Columns]]<br />
*[[Shards, RecordIDs, Main Storage, Delta Storage]]<br />
*[[Columnar Storage]]<br />
*[[Transactions]] <br />
<br />
=====Optimizations=====<br />
*[[In-Memory Compression, Columnar Compression Techniques]]<br />
*[[Temporary Columns]]<br />
*[[Data Auto Sharding and Auto Indexing]]<br />
* [[Parallel Computing]]<br />
<br />
<br />
[[File:Screenshot from htop.png|center|frameless|2490x2490px]]<br />
<br />
<br />
===Further Reading===<br />
[https://github.org/launix-de/memcp MemCP on Github]<br />
<br />
====Scientific====<br />
<br />
*[https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]<br />
*[https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]<br />
*[https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]<br />
*[https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]<br />
*https://wwwdb.inf.tu-dresden.de/research-projects/eris/ <br />
<br />
====How MemCP was built====<br />
<br />
*[https://launix.de/launix/how-to-balance-a-database-between-olap-and-oltp-workflows/ Balancing OLAP and OLTP Workflows]<br />
*[https://launix.de/launix/designing-a-programming-language-for-distributed-systems-and-highly-parallel-algorithms/ Designing Programming Languages for Distributed Systems]<br />
*[https://launix.de/launix/on-designing-an-interface-for-columnar-in-memory-storage-in-golang/ Columnar Storage Interface in Golang]<br />
*[https://launix.de/launix/how-in-memory-compression-affects-performance/ Impact of In-Memory Compression on Performance]<br />
*[https://launix.de/launix/memory-efficient-indices-for-in-memory-storages/ Memory-Efficient Indices for In-Memory Storages]<br />
*[https://launix.de/launix/on-compressing-null-values-in-bit-compressed-integer-storages/ Compressing Null Values in Bit-Compressed Integer Storages]<br />
*[https://launix.de/launix/when-the-benchmark-is-too-slow-golang-http-server-performance/ Improving Golang HTTP Server Performance]<br />
*[https://launix.de/launix/how-to-benchmark-a-sql-database/ Benchmarking SQL Databases]<br />
*[https://launix.de/launix/writing-a-sql-parser-in-scheme/ Writing a SQL Parser in Scheme]<br />
*[https://launix.de/launix/accessing-memcp-via-scheme/ Accessing memcp via Scheme]<br />
*[https://launix.de/launix/memcp-first-sql-query-is-correctly-executed/ First SQL Query in memcp]<br />
*[https://launix.de/launix/sequence-compression-in-in-memory-database-yields-99-memory-savings-and-a-total-of-13/ Sequence Compression in In-Memory Database]<br />
*[https://launix.de/launix/storing-a-bit-smaller-than-in-one-bit/ Storing Data Smaller Than One Bit]<br />
*[https://www.youtube.com/watch?v=DWg4nx4KVLo memcp Announcement Video]</div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=209Main Page2025-05-30T09:38:17Z<p>Carli: /* 🧑💻 Built for Developers, by Developers */</p>
<hr />
<div>== 🚀 Meet MemCP – The MySQL Alternative You've Been Waiting For ==<br />
'''Tired of outdated database engines slowing you down?'''<br />
<br />
MemCP is a blazing-fast, in-memory, column-oriented database designed for modern workloads. It delivers '''unmatched speed''', '''massive compression''', and '''built-in REST APIs''' – all with a developer-friendly open-source stack.<blockquote>'''Replace MySQL with something better.'''<br />
<br />
Something built for the 2020s – not stuck in 2005.</blockquote><br />
----<br />
<br />
=== ⚡ Why Developers Choose MemCP ===<br />
✅ '''Turbocharged Performance'''<br />
<br />
Designed for multicore CPUs, modern caches, and NVMe SSDs – MemCP handles analytical (OLAP) and transactional (OLTP) workloads at lightning speed.<br />
<br />
✅ '''80% Smaller Data Footprint'''<br />
<br />
Columnar storage means smarter compression. Typical use cases shrink data 5× smaller than MySQL – with no effort from your side.<br />
<br />
✅ '''Simple JSON APIs Built-in'''<br />
<br />
Query your data with RESTful endpoints – no middleware, no ORM overhead. Just fast results, directly from the DB.<br />
<br />
✅ '''Familiar SQL-like Structure – Zero Lock-In'''<br />
<br />
You don’t need to re-learn everything. Tables, schemas, and logic follow familiar patterns – just modernized and optimized.<br />
<br />
✅ '''Zero-Config Installation'''<br />
<br />
Spin up MemCP with a single <code>docker run</code> or <code>pm2 start</code>. Start building in minutes.<br />
----<br />
<br />
=== 🧑💻 Built for Developers, by Developers ===<br />
MemCP was made for the people who write real-world code, not just benchmark reports. Whether you're building a SaaS product, IoT platform, or data-heavy analytics dashboard, MemCP gives you the speed and flexibility you need to scale.<br />
<br />
# Ready to try it?<br />
git clone <nowiki>https://github.com/launix-de/memcp</nowiki><br />
cd memcp<br />
go get<br />
make<br />
pm2 start ./memcp -data ./data/<br />
<br />
mysql -u root -p -P 3307<br />
Enter password: admin<br />
<br />
=== 🆚 MemCP vs MySQL: The Key Differences ===<br />
{| class="wikitable"<br />
!Feature<br />
!MySQL<br />
!MemCP<br />
|-<br />
|Storage Model<br />
|Row-based<br />
|Column-based (compressed)<br />
|-<br />
|Performance<br />
|Good<br />
|Excellent (NUMA-optimized)<br />
|-<br />
|In-Memory Capable<br />
|Limited<br />
|Yes, by default<br />
|-<br />
|REST API Integration<br />
|External services<br />
|Built-in<br />
|-<br />
|Installation Footprint<br />
|~150MB+<br />
|~10MB<br />
|-<br />
|Open Source<br />
|✅<br />
|✅<br />
|}<br />
<br />
=== 💡 Ideal Use Cases ===<br />
<br />
* High-speed dashboards & real-time analytics<br />
* Embedded systems with minimal resources<br />
* Data-heavy SaaS backends<br />
* Replacing outdated MySQL deployments with modern performance<br />
<br />
<br />
=== 🧪 Try MemCP Today ===<br />
Explore the documentation, check out the GitHub repo, and join a growing community of developers building fast, efficient, modern systems with MemCP.<blockquote>Ditch the legacy. Embrace performance.<br />
<br />
'''Make the switch to MemCP.'''</blockquote><br />
----<br />
<br />
=== What is memcp? ===<br />
[[File:Webapps.svg|left|frameless]]<br />
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.<br />
<br />
memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.<br />
<br />
<br />
===Features===<br />
<br />
*'''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.<br />
*'''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB<br />
*'''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs<br />
* '''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime<br />
* Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.<br />
* In-memory database: Stores all data in memory, which allows for extremely fast query execution.<br />
*Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)<br />
*OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.<br />
*Compression: Lots of compression formats are supported like bit-packing and dictionary encoding<br />
* Scalability: Designed to scale on a single node with huge NUMA memory<br />
*Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time<br />
[[File:MemCP Port.png|frameless]]<br />
<br />
<youtube>g29FR4Jwius</youtube><br />
<br />
https://www.youtube.com/watch?v=g29FR4Jwius<br />
<br />
===Navigation===<br />
<br />
====Introduction==== <br />
*[[What is OLTP and OLAP]]<br />
*[[History of the MemCP project]]<br />
*[[Hardware Requirements]]<br />
*[[Persistency and Performance Guarantees]] <br />
*[[Current Status and Open Issues]]<br />
*[[Comparison: MemCP vs. MySQL]]<br />
<br />
====Getting Started====<br />
*[[Install MemCP with Docker|With Docker]]<br />
*[[With Singularity]]<br />
*[[Compile MemCP from Source|Build from Source]]<br />
*[[Contributing]] <br />
*[[Introduction to Scheme]]<br />
<br />
====Administration====<br />
<br />
* [[Deployment]]<br />
* [[Migration from MySQL and PostgreSQL]]<br />
* [[Settings]]<br />
*[[Process Hibernation]]<br />
*[[Performance Measurement]]<br />
*[[MemCP Console]]<br />
<br />
====Frontends====<br />
<br />
=====SQL Frontend===== <br />
*[[Supported SQL]]<br />
*[[Advanced SQL Tutorial]]<br />
*[[SQL over REST]]<br />
*[[Database Tools compatibility with MemCP|Supported Tooling]]<br />
*[[How SQL Operators are implemented on MemCP]]<br />
<br />
=====RDF Frontend===== <br />
*[[Introduction to RDF]]<br />
*[[Advanced Graph Querying]]<br />
*[[RDF templating and model driven development]]<br />
<br />
=====Custom Frontends=====<br />
<br />
*[[In-Database WebApps|In-Database WebApps and REST Services]]<br />
*[[MemCP for Microservices]]<br />
*[[Websockets in MemCP]]<br />
<br />
====Internals====<br />
<br />
=====How things work in MemCP===== <br />
<br />
*[[Databases, Tables and Columns]]<br />
*[[Shards, RecordIDs, Main Storage, Delta Storage]]<br />
*[[Columnar Storage]]<br />
*[[Transactions]] <br />
<br />
=====Optimizations=====<br />
*[[In-Memory Compression, Columnar Compression Techniques]]<br />
*[[Temporary Columns]]<br />
*[[Data Auto Sharding and Auto Indexing]]<br />
* [[Parallel Computing]]<br />
<br />
<br />
[[File:Screenshot from htop.png|center|frameless|2490x2490px]]<br />
<br />
<br />
===Further Reading===<br />
[https://github.org/launix-de/memcp MemCP on Github]<br />
<br />
====Scientific====<br />
<br />
*[https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]<br />
*[https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]<br />
*[https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]<br />
*[https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]<br />
*https://wwwdb.inf.tu-dresden.de/research-projects/eris/ <br />
<br />
====How MemCP was built====<br />
<br />
*[https://launix.de/launix/how-to-balance-a-database-between-olap-and-oltp-workflows/ Balancing OLAP and OLTP Workflows]<br />
*[https://launix.de/launix/designing-a-programming-language-for-distributed-systems-and-highly-parallel-algorithms/ Designing Programming Languages for Distributed Systems]<br />
*[https://launix.de/launix/on-designing-an-interface-for-columnar-in-memory-storage-in-golang/ Columnar Storage Interface in Golang]<br />
*[https://launix.de/launix/how-in-memory-compression-affects-performance/ Impact of In-Memory Compression on Performance]<br />
*[https://launix.de/launix/memory-efficient-indices-for-in-memory-storages/ Memory-Efficient Indices for In-Memory Storages]<br />
*[https://launix.de/launix/on-compressing-null-values-in-bit-compressed-integer-storages/ Compressing Null Values in Bit-Compressed Integer Storages]<br />
*[https://launix.de/launix/when-the-benchmark-is-too-slow-golang-http-server-performance/ Improving Golang HTTP Server Performance]<br />
*[https://launix.de/launix/how-to-benchmark-a-sql-database/ Benchmarking SQL Databases]<br />
*[https://launix.de/launix/writing-a-sql-parser-in-scheme/ Writing a SQL Parser in Scheme]<br />
*[https://launix.de/launix/accessing-memcp-via-scheme/ Accessing memcp via Scheme]<br />
*[https://launix.de/launix/memcp-first-sql-query-is-correctly-executed/ First SQL Query in memcp]<br />
*[https://launix.de/launix/sequence-compression-in-in-memory-database-yields-99-memory-savings-and-a-total-of-13/ Sequence Compression in In-Memory Database]<br />
*[https://launix.de/launix/storing-a-bit-smaller-than-in-one-bit/ Storing Data Smaller Than One Bit]<br />
*[https://www.youtube.com/watch?v=DWg4nx4KVLo memcp Announcement Video]</div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=208Main Page2025-05-30T09:38:00Z<p>Carli: /* 🧑💻 Built for Developers, by Developers */</p>
<hr />
<div>== 🚀 Meet MemCP – The MySQL Alternative You've Been Waiting For ==<br />
'''Tired of outdated database engines slowing you down?'''<br />
<br />
MemCP is a blazing-fast, in-memory, column-oriented database designed for modern workloads. It delivers '''unmatched speed''', '''massive compression''', and '''built-in REST APIs''' – all with a developer-friendly open-source stack.<blockquote>'''Replace MySQL with something better.'''<br />
<br />
Something built for the 2020s – not stuck in 2005.</blockquote><br />
----<br />
<br />
=== ⚡ Why Developers Choose MemCP ===<br />
✅ '''Turbocharged Performance'''<br />
<br />
Designed for multicore CPUs, modern caches, and NVMe SSDs – MemCP handles analytical (OLAP) and transactional (OLTP) workloads at lightning speed.<br />
<br />
✅ '''80% Smaller Data Footprint'''<br />
<br />
Columnar storage means smarter compression. Typical use cases shrink data 5× smaller than MySQL – with no effort from your side.<br />
<br />
✅ '''Simple JSON APIs Built-in'''<br />
<br />
Query your data with RESTful endpoints – no middleware, no ORM overhead. Just fast results, directly from the DB.<br />
<br />
✅ '''Familiar SQL-like Structure – Zero Lock-In'''<br />
<br />
You don’t need to re-learn everything. Tables, schemas, and logic follow familiar patterns – just modernized and optimized.<br />
<br />
✅ '''Zero-Config Installation'''<br />
<br />
Spin up MemCP with a single <code>docker run</code> or <code>pm2 start</code>. Start building in minutes.<br />
----<br />
<br />
=== 🧑💻 Built for Developers, by Developers ===<br />
MemCP was made for the people who write real-world code, not just benchmark reports. Whether you're building a SaaS product, IoT platform, or data-heavy analytics dashboard, MemCP gives you the speed and flexibility you need to scale.<br />
<br />
# Ready to try it?<br />
git clone <nowiki>https://github.com/launix-de/memcp</nowiki><br />
cd memcp<br />
make<br />
pm2 start ./memcp -data ./data/<br />
<br />
mysql -u root -p -P 3307<br />
Enter password: admin<br />
<br />
=== 🆚 MemCP vs MySQL: The Key Differences ===<br />
{| class="wikitable"<br />
!Feature<br />
!MySQL<br />
!MemCP<br />
|-<br />
|Storage Model<br />
|Row-based<br />
|Column-based (compressed)<br />
|-<br />
|Performance<br />
|Good<br />
|Excellent (NUMA-optimized)<br />
|-<br />
|In-Memory Capable<br />
|Limited<br />
|Yes, by default<br />
|-<br />
|REST API Integration<br />
|External services<br />
|Built-in<br />
|-<br />
|Installation Footprint<br />
|~150MB+<br />
|~10MB<br />
|-<br />
|Open Source<br />
|✅<br />
|✅<br />
|}<br />
<br />
=== 💡 Ideal Use Cases ===<br />
<br />
* High-speed dashboards & real-time analytics<br />
* Embedded systems with minimal resources<br />
* Data-heavy SaaS backends<br />
* Replacing outdated MySQL deployments with modern performance<br />
<br />
<br />
=== 🧪 Try MemCP Today ===<br />
Explore the documentation, check out the GitHub repo, and join a growing community of developers building fast, efficient, modern systems with MemCP.<blockquote>Ditch the legacy. Embrace performance.<br />
<br />
'''Make the switch to MemCP.'''</blockquote><br />
----<br />
<br />
=== What is memcp? ===<br />
[[File:Webapps.svg|left|frameless]]<br />
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.<br />
<br />
memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.<br />
<br />
<br />
===Features===<br />
<br />
*'''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.<br />
*'''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB<br />
*'''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs<br />
* '''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime<br />
* Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.<br />
* In-memory database: Stores all data in memory, which allows for extremely fast query execution.<br />
*Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)<br />
*OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.<br />
*Compression: Lots of compression formats are supported like bit-packing and dictionary encoding<br />
* Scalability: Designed to scale on a single node with huge NUMA memory<br />
*Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time<br />
[[File:MemCP Port.png|frameless]]<br />
<br />
<youtube>g29FR4Jwius</youtube><br />
<br />
https://www.youtube.com/watch?v=g29FR4Jwius<br />
<br />
===Navigation===<br />
<br />
====Introduction==== <br />
*[[What is OLTP and OLAP]]<br />
*[[History of the MemCP project]]<br />
*[[Hardware Requirements]]<br />
*[[Persistency and Performance Guarantees]] <br />
*[[Current Status and Open Issues]]<br />
*[[Comparison: MemCP vs. MySQL]]<br />
<br />
====Getting Started====<br />
*[[Install MemCP with Docker|With Docker]]<br />
*[[With Singularity]]<br />
*[[Compile MemCP from Source|Build from Source]]<br />
*[[Contributing]] <br />
*[[Introduction to Scheme]]<br />
<br />
====Administration====<br />
<br />
* [[Deployment]]<br />
* [[Migration from MySQL and PostgreSQL]]<br />
* [[Settings]]<br />
*[[Process Hibernation]]<br />
*[[Performance Measurement]]<br />
*[[MemCP Console]]<br />
<br />
====Frontends====<br />
<br />
=====SQL Frontend===== <br />
*[[Supported SQL]]<br />
*[[Advanced SQL Tutorial]]<br />
*[[SQL over REST]]<br />
*[[Database Tools compatibility with MemCP|Supported Tooling]]<br />
*[[How SQL Operators are implemented on MemCP]]<br />
<br />
=====RDF Frontend===== <br />
*[[Introduction to RDF]]<br />
*[[Advanced Graph Querying]]<br />
*[[RDF templating and model driven development]]<br />
<br />
=====Custom Frontends=====<br />
<br />
*[[In-Database WebApps|In-Database WebApps and REST Services]]<br />
*[[MemCP for Microservices]]<br />
*[[Websockets in MemCP]]<br />
<br />
====Internals====<br />
<br />
=====How things work in MemCP===== <br />
<br />
*[[Databases, Tables and Columns]]<br />
*[[Shards, RecordIDs, Main Storage, Delta Storage]]<br />
*[[Columnar Storage]]<br />
*[[Transactions]] <br />
<br />
=====Optimizations=====<br />
*[[In-Memory Compression, Columnar Compression Techniques]]<br />
*[[Temporary Columns]]<br />
*[[Data Auto Sharding and Auto Indexing]]<br />
* [[Parallel Computing]]<br />
<br />
<br />
[[File:Screenshot from htop.png|center|frameless|2490x2490px]]<br />
<br />
<br />
===Further Reading===<br />
[https://github.org/launix-de/memcp MemCP on Github]<br />
<br />
====Scientific====<br />
<br />
*[https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]<br />
*[https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]<br />
*[https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]<br />
*[https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]<br />
*https://wwwdb.inf.tu-dresden.de/research-projects/eris/ <br />
<br />
====How MemCP was built====<br />
<br />
*[https://launix.de/launix/how-to-balance-a-database-between-olap-and-oltp-workflows/ Balancing OLAP and OLTP Workflows]<br />
*[https://launix.de/launix/designing-a-programming-language-for-distributed-systems-and-highly-parallel-algorithms/ Designing Programming Languages for Distributed Systems]<br />
*[https://launix.de/launix/on-designing-an-interface-for-columnar-in-memory-storage-in-golang/ Columnar Storage Interface in Golang]<br />
*[https://launix.de/launix/how-in-memory-compression-affects-performance/ Impact of In-Memory Compression on Performance]<br />
*[https://launix.de/launix/memory-efficient-indices-for-in-memory-storages/ Memory-Efficient Indices for In-Memory Storages]<br />
*[https://launix.de/launix/on-compressing-null-values-in-bit-compressed-integer-storages/ Compressing Null Values in Bit-Compressed Integer Storages]<br />
*[https://launix.de/launix/when-the-benchmark-is-too-slow-golang-http-server-performance/ Improving Golang HTTP Server Performance]<br />
*[https://launix.de/launix/how-to-benchmark-a-sql-database/ Benchmarking SQL Databases]<br />
*[https://launix.de/launix/writing-a-sql-parser-in-scheme/ Writing a SQL Parser in Scheme]<br />
*[https://launix.de/launix/accessing-memcp-via-scheme/ Accessing memcp via Scheme]<br />
*[https://launix.de/launix/memcp-first-sql-query-is-correctly-executed/ First SQL Query in memcp]<br />
*[https://launix.de/launix/sequence-compression-in-in-memory-database-yields-99-memory-savings-and-a-total-of-13/ Sequence Compression in In-Memory Database]<br />
*[https://launix.de/launix/storing-a-bit-smaller-than-in-one-bit/ Storing Data Smaller Than One Bit]<br />
*[https://www.youtube.com/watch?v=DWg4nx4KVLo memcp Announcement Video]</div>Carlihttps://www.memcp.org/index.php?title=Main_Page&diff=207Main Page2025-05-29T11:03:17Z<p>Carli: </p>
<hr />
<div>== 🚀 Meet MemCP – The MySQL Alternative You've Been Waiting For ==<br />
'''Tired of outdated database engines slowing you down?'''<br />
<br />
MemCP is a blazing-fast, in-memory, column-oriented database designed for modern workloads. It delivers '''unmatched speed''', '''massive compression''', and '''built-in REST APIs''' – all with a developer-friendly open-source stack.<blockquote>'''Replace MySQL with something better.'''<br />
<br />
Something built for the 2020s – not stuck in 2005.</blockquote><br />
----<br />
<br />
=== ⚡ Why Developers Choose MemCP ===<br />
✅ '''Turbocharged Performance'''<br />
<br />
Designed for multicore CPUs, modern caches, and NVMe SSDs – MemCP handles analytical (OLAP) and transactional (OLTP) workloads at lightning speed.<br />
<br />
✅ '''80% Smaller Data Footprint'''<br />
<br />
Columnar storage means smarter compression. Typical use cases shrink data 5× smaller than MySQL – with no effort from your side.<br />
<br />
✅ '''Simple JSON APIs Built-in'''<br />
<br />
Query your data with RESTful endpoints – no middleware, no ORM overhead. Just fast results, directly from the DB.<br />
<br />
✅ '''Familiar SQL-like Structure – Zero Lock-In'''<br />
<br />
You don’t need to re-learn everything. Tables, schemas, and logic follow familiar patterns – just modernized and optimized.<br />
<br />
✅ '''Zero-Config Installation'''<br />
<br />
Spin up MemCP with a single <code>docker run</code> or <code>pm2 start</code>. Start building in minutes.<br />
----<br />
<br />
=== 🧑💻 Built for Developers, by Developers ===<br />
MemCP was made for the people who write real-world code, not just benchmark reports. Whether you're building a SaaS product, IoT platform, or data-heavy analytics dashboard, MemCP gives you the speed and flexibility you need to scale.<br />
<br />
# Ready to try it?<br />
git clone <nowiki>https://github.com/launix-de/memcp</nowiki><br />
cd memcp<br />
make<br />
pm2 start ./memcp -data ./data/<br />
<br />
=== 🆚 MemCP vs MySQL: The Key Differences ===<br />
{| class="wikitable"<br />
!Feature<br />
!MySQL<br />
!MemCP<br />
|-<br />
|Storage Model<br />
|Row-based<br />
|Column-based (compressed)<br />
|-<br />
|Performance<br />
|Good<br />
|Excellent (NUMA-optimized)<br />
|-<br />
|In-Memory Capable<br />
|Limited<br />
|Yes, by default<br />
|-<br />
|REST API Integration<br />
|External services<br />
|Built-in<br />
|-<br />
|Installation Footprint<br />
|~150MB+<br />
|~10MB<br />
|-<br />
|Open Source<br />
|✅<br />
|✅<br />
|}<br />
<br />
=== 💡 Ideal Use Cases ===<br />
<br />
* High-speed dashboards & real-time analytics<br />
* Embedded systems with minimal resources<br />
* Data-heavy SaaS backends<br />
* Replacing outdated MySQL deployments with modern performance<br />
<br />
<br />
=== 🧪 Try MemCP Today ===<br />
Explore the documentation, check out the GitHub repo, and join a growing community of developers building fast, efficient, modern systems with MemCP.<blockquote>Ditch the legacy. Embrace performance.<br />
<br />
'''Make the switch to MemCP.'''</blockquote><br />
----<br />
<br />
=== What is memcp? ===<br />
[[File:Webapps.svg|left|frameless]]<br />
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.<br />
<br />
memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.<br />
<br />
<br />
===Features===<br />
<br />
*'''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.<br />
*'''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB<br />
*'''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs<br />
* '''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime<br />
* Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.<br />
* In-memory database: Stores all data in memory, which allows for extremely fast query execution.<br />
*Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)<br />
*OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.<br />
*Compression: Lots of compression formats are supported like bit-packing and dictionary encoding<br />
* Scalability: Designed to scale on a single node with huge NUMA memory<br />
*Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time<br />
[[File:MemCP Port.png|frameless]]<br />
<br />
<youtube>g29FR4Jwius</youtube><br />
<br />
https://www.youtube.com/watch?v=g29FR4Jwius<br />
<br />
===Navigation===<br />
<br />
====Introduction==== <br />
*[[What is OLTP and OLAP]]<br />
*[[History of the MemCP project]]<br />
*[[Hardware Requirements]]<br />
*[[Persistency and Performance Guarantees]] <br />
*[[Current Status and Open Issues]]<br />
*[[Comparison: MemCP vs. MySQL]]<br />
<br />
====Getting Started====<br />
*[[Install MemCP with Docker|With Docker]]<br />
*[[With Singularity]]<br />
*[[Compile MemCP from Source|Build from Source]]<br />
*[[Contributing]] <br />
*[[Introduction to Scheme]]<br />
<br />
====Administration====<br />
<br />
* [[Deployment]]<br />
* [[Migration from MySQL and PostgreSQL]]<br />
* [[Settings]]<br />
*[[Process Hibernation]]<br />
*[[Performance Measurement]]<br />
*[[MemCP Console]]<br />
<br />
====Frontends====<br />
<br />
=====SQL Frontend===== <br />
*[[Supported SQL]]<br />
*[[Advanced SQL Tutorial]]<br />
*[[SQL over REST]]<br />
*[[Database Tools compatibility with MemCP|Supported Tooling]]<br />
*[[How SQL Operators are implemented on MemCP]]<br />
<br />
=====RDF Frontend===== <br />
*[[Introduction to RDF]]<br />
*[[Advanced Graph Querying]]<br />
*[[RDF templating and model driven development]]<br />
<br />
=====Custom Frontends=====<br />
<br />
*[[In-Database WebApps|In-Database WebApps and REST Services]]<br />
*[[MemCP for Microservices]]<br />
*[[Websockets in MemCP]]<br />
<br />
====Internals====<br />
<br />
=====How things work in MemCP===== <br />
<br />
*[[Databases, Tables and Columns]]<br />
*[[Shards, RecordIDs, Main Storage, Delta Storage]]<br />
*[[Columnar Storage]]<br />
*[[Transactions]] <br />
<br />
=====Optimizations=====<br />
*[[In-Memory Compression, Columnar Compression Techniques]]<br />
*[[Temporary Columns]]<br />
*[[Data Auto Sharding and Auto Indexing]]<br />
* [[Parallel Computing]]<br />
<br />
<br />
[[File:Screenshot from htop.png|center|frameless|2490x2490px]]<br />
<br />
<br />
===Further Reading===<br />
[https://github.org/launix-de/memcp MemCP on Github]<br />
<br />
====Scientific====<br />
<br />
*[https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]<br />
*[https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]<br />
*[https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]<br />
*[https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]<br />
*https://wwwdb.inf.tu-dresden.de/research-projects/eris/ <br />
<br />
====How MemCP was built====<br />
<br />
*[https://launix.de/launix/how-to-balance-a-database-between-olap-and-oltp-workflows/ Balancing OLAP and OLTP Workflows]<br />
*[https://launix.de/launix/designing-a-programming-language-for-distributed-systems-and-highly-parallel-algorithms/ Designing Programming Languages for Distributed Systems]<br />
*[https://launix.de/launix/on-designing-an-interface-for-columnar-in-memory-storage-in-golang/ Columnar Storage Interface in Golang]<br />
*[https://launix.de/launix/how-in-memory-compression-affects-performance/ Impact of In-Memory Compression on Performance]<br />
*[https://launix.de/launix/memory-efficient-indices-for-in-memory-storages/ Memory-Efficient Indices for In-Memory Storages]<br />
*[https://launix.de/launix/on-compressing-null-values-in-bit-compressed-integer-storages/ Compressing Null Values in Bit-Compressed Integer Storages]<br />
*[https://launix.de/launix/when-the-benchmark-is-too-slow-golang-http-server-performance/ Improving Golang HTTP Server Performance]<br />
*[https://launix.de/launix/how-to-benchmark-a-sql-database/ Benchmarking SQL Databases]<br />
*[https://launix.de/launix/writing-a-sql-parser-in-scheme/ Writing a SQL Parser in Scheme]<br />
*[https://launix.de/launix/accessing-memcp-via-scheme/ Accessing memcp via Scheme]<br />
*[https://launix.de/launix/memcp-first-sql-query-is-correctly-executed/ First SQL Query in memcp]<br />
*[https://launix.de/launix/sequence-compression-in-in-memory-database-yields-99-memory-savings-and-a-total-of-13/ Sequence Compression in In-Memory Database]<br />
*[https://launix.de/launix/storing-a-bit-smaller-than-in-one-bit/ Storing Data Smaller Than One Bit]<br />
*[https://www.youtube.com/watch?v=DWg4nx4KVLo memcp Announcement Video]</div>Carlihttps://www.memcp.org/index.php?title=Comparison:_MemCP_vs._MySQL&diff=206Comparison: MemCP vs. MySQL2024-12-20T19:47:32Z<p>Carli: </p>
<hr />
<div>'''MemCP''' offers several distinct advantages compared to MySQL, making it a compelling choice for optimized database management.<br />
<br />
=== '''No 64-Character Limit for Column Names''' ===<br />
MemCP removes the restrictive 64-character limit on column names found in MySQL, allowing for more descriptive and meaningful column naming conventions.<br />
<br />
=== '''Significantly Reduced Storage Requirements''' ===<br />
MemCP reduces '''storage reqirements by 80%''' compared to MySQL's InnoDB and MyISAM storage formats. This reduction in storage demand translates to lower infrastructure costs and better scalability for large datasets.<br />
<br />
=== '''Built-In Support for Natural Sorting''' ===<br />
MemCP supports '''natural sort order''', which arranges data logically as humans expect, such as <code>1, 2, 10, 11</code>, rather than lexicographically (<code>1, 10, 11, 2</code>). This feature works by adding collations, that support natural ordering.<br />
<br />
=== '''Dramatic Performance Improvements''' ===<br />
MemCP delivers up to '''10x faster query performance''' for aggregation operations compared to MySQL. This makes it ideal for data-intensive applications, business intelligence workloads, and real-time analytics.<br />
<br />
=== '''Unified Support for OLAP and OLTP''' ===<br />
MemCP combines '''Online Transaction Processing (OLTP)''' and '''Online Analytical Processing (OLAP)''' in the same database. This enables simultaneous handling of transactional and analytical workloads without the need for separate systems or data pipelines. Such flexibility enhances operational efficiency and simplifies database architecture.<br />
<br />
=== '''Support for Multiple Frontends''' ===<br />
MemCP provides support for diverse frontends, including '''SPARQL''', '''microservices''', and '''REST APIs'''. This versatility allows developers to use MemCP in various application architectures, ranging from semantic web applications to distributed microservices, ensuring seamless integration into modern development environments.<br />
<br />
By addressing common MySQL limitations and offering superior performance, MemCP positions itself as an efficient, scalable, and user-friendly alternative for modern database management needs.</div>Carlihttps://www.memcp.org/index.php?title=Comparison:_MemCP_vs._MySQL&diff=205Comparison: MemCP vs. MySQL2024-12-19T20:56:20Z<p>Carli: </p>
<hr />
<div>'''MemCP''' offers several distinct advantages compared to MySQL, making it a compelling choice for optimized database management.<br />
<br />
=== '''No 64-Character Limit for Column Names''' ===<br />
MemCP removes the restrictive 64-character limit on column names found in MySQL, allowing for more descriptive and meaningful column naming conventions.<br />
<br />
=== '''Significantly Reduced Storage Requirements''' ===<br />
MemCP uses only '''20% of the storage''' required by MySQL's InnoDB and MyISAM storage formats. This reduction in storage demand translates to lower infrastructure costs and better scalability for large datasets.<br />
<br />
=== '''Built-In Support for Natural Sorting''' ===<br />
MemCP supports '''natural sort order''', which arranges data logically as humans expect, such as <code>1, 2, 10, 11</code>, rather than lexicographically (<code>1, 10, 11, 2</code>). This feature works by adding collations, that support natural ordering.<br />
<br />
=== '''Dramatic Performance Improvements''' ===<br />
MemCP delivers up to '''10x faster query performance''' for aggregation operations compared to MySQL. This makes it ideal for data-intensive applications, business intelligence workloads, and real-time analytics.<br />
<br />
=== '''Unified Support for OLAP and OLTP''' ===<br />
MemCP combines '''Online Transaction Processing (OLTP)''' and '''Online Analytical Processing (OLAP)''' in the same database. This enables simultaneous handling of transactional and analytical workloads without the need for separate systems or data pipelines. Such flexibility enhances operational efficiency and simplifies database architecture.<br />
<br />
=== '''Support for Multiple Frontends''' ===<br />
MemCP provides support for diverse frontends, including '''SPARQL''', '''microservices''', and '''REST APIs'''. This versatility allows developers to use MemCP in various application architectures, ranging from semantic web applications to distributed microservices, ensuring seamless integration into modern development environments.<br />
<br />
By addressing common MySQL limitations and offering superior performance, MemCP positions itself as an efficient, scalable, and user-friendly alternative for modern database management needs.</div>Carlihttps://www.memcp.org/index.php?title=Comparison:_MemCP_vs._MySQL&diff=204Comparison: MemCP vs. MySQL2024-12-19T20:54:56Z<p>Carli: </p>
<hr />
<div>'''MemCP''' offers several distinct advantages compared to MySQL, making it a compelling choice for optimized database management.<br />
<br />
* '''No 64-Character Limit for Column Names''' MemCP removes the restrictive 64-character limit on column names found in MySQL, allowing for more descriptive and meaningful column naming conventions.<br />
* '''Significantly Reduced Storage Requirements''' MemCP uses only '''20% of the storage''' required by MySQL's InnoDB and MyISAM storage formats. This reduction in storage demand translates to lower infrastructure costs and better scalability for large datasets.<br />
* '''Built-In Support for Natural Sorting''' MemCP supports '''natural sort order''', which arranges data logically as humans expect, such as <code>1, 2, 10, 11</code>, rather than lexicographically (<code>1, 10, 11, 2</code>). This feature works by adding collations, that support natural ordering.<br />
* '''Dramatic Performance Improvements''' MemCP delivers up to '''10x faster query performance''' for aggregation operations compared to MySQL. This makes it ideal for data-intensive applications, business intelligence workloads, and real-time analytics.<br />
* '''Unified Support for OLAP and OLTP'''MemCP combines '''Online Transaction Processing (OLTP)''' and '''Online Analytical Processing (OLAP)''' in the same database. This enables simultaneous handling of transactional and analytical workloads without the need for separate systems or data pipelines. Such flexibility enhances operational efficiency and simplifies database architecture.<br />
* '''Support for Multiple Frontends'''MemCP provides support for diverse frontends, including '''SPARQL''', '''microservices''', and '''REST APIs'''. This versatility allows developers to use MemCP in various application architectures, ranging from semantic web applications to distributed microservices, ensuring seamless integration into modern development environments.<br />
<br />
By addressing common MySQL limitations and offering superior performance, MemCP positions itself as an efficient, scalable, and user-friendly alternative for modern database management needs.</div>Carlihttps://www.memcp.org/index.php?title=Comparison:_MemCP_vs._MySQL&diff=203Comparison: MemCP vs. MySQL2024-12-19T20:54:27Z<p>Carli: </p>
<hr />
<div>'''MemCP''' offers several distinct advantages compared to MySQL, making it a compelling choice for optimized database management.<br />
<br />
# '''No 64-Character Limit for Column Names''' MemCP removes the restrictive 64-character limit on column names found in MySQL, allowing for more descriptive and meaningful column naming conventions.<br />
# '''Significantly Reduced Storage Requirements''' MemCP uses only '''20% of the storage''' required by MySQL's InnoDB and MyISAM storage formats. This reduction in storage demand translates to lower infrastructure costs and better scalability for large datasets.<br />
# '''Built-In Support for Natural Sorting''' MemCP supports '''natural sort order''', which arranges data logically as humans expect, such as <code>1, 2, 10, 11</code>, rather than lexicographically (<code>1, 10, 11, 2</code>). This feature works by adding collations, that support natural ordering.<br />
# '''Dramatic Performance Improvements''' MemCP delivers up to '''10x faster query performance''' for aggregation operations compared to MySQL. This makes it ideal for data-intensive applications, business intelligence workloads, and real-time analytics.<br />
# '''Unified Support for OLAP and OLTP'''MemCP combines '''Online Transaction Processing (OLTP)''' and '''Online Analytical Processing (OLAP)''' in the same database. This enables simultaneous handling of transactional and analytical workloads without the need for separate systems or data pipelines. Such flexibility enhances operational efficiency and simplifies database architecture.<br />
# '''Support for Multiple Frontends'''MemCP provides support for diverse frontends, including '''SPARQL''', '''microservices''', and '''REST APIs'''. This versatility allows developers to use MemCP in various application architectures, ranging from semantic web applications to distributed microservices, ensuring seamless integration into modern development environments.<br />
<br />
By addressing common MySQL limitations and offering superior performance, MemCP positions itself as an efficient, scalable, and user-friendly alternative for modern database management needs.</div>Carlihttps://www.memcp.org/index.php?title=Comparison:_MemCP_vs._MySQL&diff=202Comparison: MemCP vs. MySQL2024-12-19T20:47:20Z<p>Carli: </p>
<hr />
<div>'''MemCP''' offers several distinct advantages compared to MySQL, making it a compelling choice for optimized database management.<br />
<br />
# '''No 64-Character Limit for Column Names''' MemCP removes the restrictive 64-character limit on column names found in MySQL, allowing for more descriptive and meaningful column naming conventions.<br />
# '''Significantly Reduced Storage Requirements''' MemCP uses only '''20% of the storage''' required by MySQL's InnoDB and MyISAM storage formats. This reduction in storage demand translates to lower infrastructure costs and better scalability for large datasets.<br />
# '''Built-In Support for Natural Sorting''' MemCP supports '''natural sort order''', which arranges data logically as humans expect, such as <code>1, 2, 10, 11</code>, rather than lexicographically (<code>1, 10, 11, 2</code>). This feature works by adding collations, that support natural ordering.<br />
# '''Dramatic Performance Improvements''' MemCP delivers up to '''10x faster query performance''' for aggregation operations compared to MySQL. This makes it ideal for data-intensive applications, business intelligence workloads, and real-time analytics.<br />
<br />
By addressing common MySQL limitations and offering superior performance, MemCP positions itself as an efficient, scalable, and user-friendly alternative for modern database management needs.</div>Carlihttps://www.memcp.org/index.php?title=Comparison:_MemCP_vs._MySQL&diff=201Comparison: MemCP vs. MySQL2024-12-19T20:46:04Z<p>Carli: </p>
<hr />
<div>'''MemCP''' offers several distinct advantages compared to MySQL, making it a compelling choice for optimized database management.<br />
<br />
# '''No 64-Character Limit for Column Names''' MemCP removes the restrictive 64-character limit on column names found in MySQL, allowing for more descriptive and meaningful column naming conventions.<br />
# '''Significantly Reduced Storage Requirements''' MemCP uses only '''20% of the storage''' required by MySQL's InnoDB and MyISAM storage formats. This reduction in storage demand translates to lower infrastructure costs and better scalability for large datasets.<br />
# '''Built-In Support for Natural Sorting''' MemCP supports '''natural sort order''', which arranges data logically as humans expect, such as <code>1, 2, 10, 11</code>, rather than lexicographically (<code>1, 10, 11, 2</code>). This feature works seamlessly with collations, enhancing both user experience and data presentation.<br />
# '''Dramatic Performance Improvements''' MemCP delivers up to '''10x faster query performance''' for aggregation operations compared to MySQL. This makes it ideal for data-intensive applications, business intelligence workloads, and real-time analytics.<br />
<br />
By addressing common MySQL limitations and offering superior performance, MemCP positions itself as an efficient, scalable, and user-friendly alternative for modern database management needs.</div>Carlihttps://www.memcp.org/index.php?title=Comparison:_MemCP_vs._MySQL&diff=200Comparison: MemCP vs. MySQL2024-12-19T20:45:02Z<p>Carli: Created page with "MemCP has the following advantages over MySQL: * No 64 character limit for column names * uses only 20% of storage compared to InnoDB and MyISAM storage format * supports natural sort (1, 2, 10, 11 instead of 1, 10, 11, 2) with collations * Up to 10x faster query performance for aggregations"</p>
<hr />
<div>MemCP has the following advantages over MySQL:<br />
<br />
* No 64 character limit for column names<br />
* uses only 20% of storage compared to InnoDB and MyISAM storage format<br />
* supports natural sort (1, 2, 10, 11 instead of 1, 10, 11, 2) with collations<br />
* Up to 10x faster query performance for aggregations</div>Carli