Main Page: Difference between revisions

From MemCP
Jump to navigation Jump to search
No edit summary
 
(54 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== 🚀 Meet MemCP – The MySQL Alternative You've Been Waiting For ==
'''Tired of outdated database engines slowing you down?'''
MemCP is a blazing-fast, in-memory, column-oriented database designed for modern workloads. It delivers '''unmatched speed''', '''massive compression''', and '''built-in REST APIs''' – all with a developer-friendly open-source stack.<blockquote>'''Replace MySQL with something better.'''
Something built for the 2020s – not stuck in 2005.</blockquote>
----
=== ⚡ Why Developers Choose MemCP ===
✅ '''Turbocharged Performance'''
Designed for multicore CPUs, modern caches, and NVMe SSDs – MemCP handles analytical (OLAP) and transactional (OLTP) workloads at lightning speed.
✅ '''80% Smaller Data Footprint'''
Columnar storage means smarter compression. Typical use cases shrink data 5× smaller than MySQL – with no effort from your side.
✅ '''Simple JSON APIs Built-in'''
Query your data with RESTful endpoints – no middleware, no ORM overhead. Just fast results, directly from the DB.
✅ '''Familiar SQL-like Structure – Zero Lock-In'''
You don’t need to re-learn everything. Tables, schemas, and logic follow familiar patterns – just modernized and optimized.
✅ '''Zero-Config Installation'''
Spin up MemCP with a single <code>docker run</code> or <code>pm2 start</code>. Start building in minutes.
----
=== 🧑‍💻 Built for Developers, by Developers ===
MemCP was made for the people who write real-world code, not just benchmark reports. Whether you're building a SaaS product, IoT platform, or data-heavy analytics dashboard, MemCP gives you the speed and flexibility you need to scale.
# Ready to try it?
git clone <nowiki>https://github.com/launix-de/memcp</nowiki>
cd memcp
go get
make
pm2 start ./memcp ./data/
mysql -u root -p -P 3307
Enter password: admin
=== 🆚 MemCP vs MySQL: The Key Differences ===
{| class="wikitable"
!Feature
!MySQL
!MemCP
|-
|Storage Model
|Row-based
|Column-based (compressed)
|-
|Performance
|Good
|Excellent (NUMA-optimized)
|-
|In-Memory Capable
|Limited
|Yes, by default
|-
|REST API Integration
|External services
|Built-in
|-
|Installation Footprint
|~150MB+
|~10MB
|-
|Open Source
|✅
|✅
|}
=== 💡 Ideal Use Cases ===
* High-speed dashboards & real-time analytics
* Embedded systems with minimal resources
* Data-heavy SaaS backends
* Replacing outdated MySQL deployments with modern performance
=== 🧪 Try MemCP Today ===
Explore the documentation, check out the GitHub repo, and join a growing community of developers building fast, efficient, modern systems with MemCP.<blockquote>Ditch the legacy. Embrace performance.
'''Make the switch to MemCP.'''</blockquote>
----


=== What is memcp? ===
=== What is memcp? ===
[[File:Webapps.svg|left|frameless]]
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.
memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.


memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.
memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.


=== Features ===


* '''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.
===Features===
* '''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB
 
* '''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs
*'''fast:''' MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.
*'''efficient:''' The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB
*'''modern:''' MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs
* '''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime
* '''versatile:''' Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime
* Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.
* Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.
* In-memory database: Stores all data in memory, which allows for extremely fast query execution.
* In-memory database: Stores all data in memory, which allows for extremely fast query execution.
* Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)
*Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)
* OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.
*OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.
* Compression: Lots of compression formats are supported like bit-packing and dictionary encoding
*Compression: Lots of compression formats are supported like bit-packing and dictionary encoding
* Scalability: Designed to scale on a single node with huge NUMA memory
* Scalability: Designed to scale on a single node with huge NUMA memory
* Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time
*Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time
[[File:MemCP Port.png|frameless]]
 
<youtube>g29FR4Jwius</youtube>
 
https://www.youtube.com/watch?v=g29FR4Jwius
 
===Navigation===
 
====Introduction====
*[[What is OLTP and OLAP]]
*[[History of the MemCP project]]
*[[Hardware Requirements]]
*[[Persistency and Performance Guarantees]]
*[[Current Status and Open Issues]]
*[[Comparison: MemCP vs. MySQL]]
 
====Getting Started====
*[[Install MemCP with Docker|With Docker]]
*[[With Singularity]]
*[[Compile MemCP from Source|Build from Source]]
*[[Contributing]]
*[[Introduction to Scheme]]
 
====Administration====
 
* [[Deployment]]
* [[Migration from MySQL and PostgreSQL]]
* [[Settings]]
*[[Process Hibernation]]
*[[Performance Measurement]]
*[[MemCP Console]]
 
====Frontends====
 
=====SQL Frontend=====
*[[Supported SQL]]
*[[Advanced SQL Tutorial]]
*[[SQL over REST]]
*[[Database Tools compatibility with MemCP|Supported Tooling]]
*[[How SQL Operators are implemented on MemCP]]
 
=====RDF Frontend=====
*[[Introduction to RDF]]
*[[Advanced Graph Querying]]
*[[RDF templating and model driven development]]
 
=====Custom Frontends=====
 
*[[In-Database WebApps|In-Database WebApps and REST Services]]
*[[MemCP for Microservices]]
*[[Websockets in MemCP]]
 
====Internals====
 
=====How things work in MemCP=====


=== Navigation ===
*[[Databases, Tables and Columns]]
*[[Shards, RecordIDs, Main Storage, Delta Storage]]
*[[Columnar Storage]]
*[[Transactions]]


* Introduction
=====Optimizations=====
** [[What is OLTP and OLAP]]
*[[In-Memory Compression, Columnar Compression Techniques]]
** [[History of the MemCP project]]
*[[Temporary Columns]]
** [[Hardware Requirements]]
*[[Data Auto Sharding and Auto Indexing]]
* Getting Started
* [[Parallel Computing]]
** [[Install MemCP with Docker|With Docker]]
** [[With Singularity]]
** [[Compile MemCP from Source|Build from Source]]
* SQL Frontend
** [[Replace MySQL with MemCP]]
** [[SQL over REST]]
** [[Database Tools compatibility with MemCP|Supported Tooling]]
* RDF Frontend
** [[Introduction to RDF]]
** [[RDF templating and model driven development]]
* Internals
** [[Databases, Tables and Columns]]
** [[Shards, RecordIDs, Main Storage, Delta Storage]]
** [[Columnar Storage]]
** [[In-Memory Compression, Columnar Compression Techniques]]
** [[Data Auto Sharding and Auto Indexing]]
** [[Parallel Computing]]
** [[Transactions]]




Line 49: Line 178:




=== Further Reading ===
===Further Reading===
[https://github.org/launix-de/memcp MemCP on Github]
[https://github.org/launix-de/memcp MemCP on Github]


==== Scientific ====
====Scientific====


* [https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]
*[https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf VLDB Research Paper]
* [https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]
*[https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdf LNI Proceedings Paper]
* [https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]
*[https://wwwdb.inf.tu-dresden.de/wp-content/uploads/T_2014_Master_Patrick_Damme.pdf TU Dresden Research Paper]
* [https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]
*[https://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf Large Graph Algorithms]
* https://wwwdb.inf.tu-dresden.de/research-projects/eris/
*https://wwwdb.inf.tu-dresden.de/research-projects/eris/  


==== How MemCP was built ====
====How MemCP was built====


* [https://launix.de/launix/how-to-balance-a-database-between-olap-and-oltp-workflows/ Balancing OLAP and OLTP Workflows]
*[https://launix.de/launix/how-to-balance-a-database-between-olap-and-oltp-workflows/ Balancing OLAP and OLTP Workflows]
* [https://launix.de/launix/designing-a-programming-language-for-distributed-systems-and-highly-parallel-algorithms/ Designing Programming Languages for Distributed Systems]
*[https://launix.de/launix/designing-a-programming-language-for-distributed-systems-and-highly-parallel-algorithms/ Designing Programming Languages for Distributed Systems]
* [https://launix.de/launix/on-designing-an-interface-for-columnar-in-memory-storage-in-golang/ Columnar Storage Interface in Golang]
*[https://launix.de/launix/on-designing-an-interface-for-columnar-in-memory-storage-in-golang/ Columnar Storage Interface in Golang]
* [https://launix.de/launix/how-in-memory-compression-affects-performance/ Impact of In-Memory Compression on Performance]
*[https://launix.de/launix/how-in-memory-compression-affects-performance/ Impact of In-Memory Compression on Performance]
* [https://launix.de/launix/memory-efficient-indices-for-in-memory-storages/ Memory-Efficient Indices for In-Memory Storages]
*[https://launix.de/launix/memory-efficient-indices-for-in-memory-storages/ Memory-Efficient Indices for In-Memory Storages]
* [https://launix.de/launix/on-compressing-null-values-in-bit-compressed-integer-storages/ Compressing Null Values in Bit-Compressed Integer Storages]
*[https://launix.de/launix/on-compressing-null-values-in-bit-compressed-integer-storages/ Compressing Null Values in Bit-Compressed Integer Storages]
* [https://launix.de/launix/when-the-benchmark-is-too-slow-golang-http-server-performance/ Improving Golang HTTP Server Performance]
*[https://launix.de/launix/when-the-benchmark-is-too-slow-golang-http-server-performance/ Improving Golang HTTP Server Performance]
* [https://launix.de/launix/how-to-benchmark-a-sql-database/ Benchmarking SQL Databases]
*[https://launix.de/launix/how-to-benchmark-a-sql-database/ Benchmarking SQL Databases]
* [https://launix.de/launix/writing-a-sql-parser-in-scheme/ Writing a SQL Parser in Scheme]
*[https://launix.de/launix/writing-a-sql-parser-in-scheme/ Writing a SQL Parser in Scheme]
* [https://launix.de/launix/accessing-memcp-via-scheme/ Accessing memcp via Scheme]
*[https://launix.de/launix/accessing-memcp-via-scheme/ Accessing memcp via Scheme]
* [https://launix.de/launix/memcp-first-sql-query-is-correctly-executed/ First SQL Query in memcp]
*[https://launix.de/launix/memcp-first-sql-query-is-correctly-executed/ First SQL Query in memcp]
* [https://launix.de/launix/sequence-compression-in-in-memory-database-yields-99-memory-savings-and-a-total-of-13/ Sequence Compression in In-Memory Database]
*[https://launix.de/launix/sequence-compression-in-in-memory-database-yields-99-memory-savings-and-a-total-of-13/ Sequence Compression in In-Memory Database]
* [https://launix.de/launix/storing-a-bit-smaller-than-in-one-bit/ Storing Data Smaller Than One Bit]
*[https://launix.de/launix/storing-a-bit-smaller-than-in-one-bit/ Storing Data Smaller Than One Bit]
* [https://www.youtube.com/watch?v=DWg4nx4KVLo memcp Announcement Video]
*[https://www.youtube.com/watch?v=DWg4nx4KVLo memcp Announcement Video]

Latest revision as of 11:43, 30 May 2025

🚀 Meet MemCP – The MySQL Alternative You've Been Waiting For

Tired of outdated database engines slowing you down?

MemCP is a blazing-fast, in-memory, column-oriented database designed for modern workloads. It delivers unmatched speed, massive compression, and built-in REST APIs – all with a developer-friendly open-source stack.

Replace MySQL with something better. Something built for the 2020s – not stuck in 2005.

⚡ Why Developers Choose MemCP

✅ Turbocharged Performance

Designed for multicore CPUs, modern caches, and NVMe SSDs – MemCP handles analytical (OLAP) and transactional (OLTP) workloads at lightning speed.

✅ 80% Smaller Data Footprint

Columnar storage means smarter compression. Typical use cases shrink data 5× smaller than MySQL – with no effort from your side.

✅ Simple JSON APIs Built-in

Query your data with RESTful endpoints – no middleware, no ORM overhead. Just fast results, directly from the DB.

✅ Familiar SQL-like Structure – Zero Lock-In

You don’t need to re-learn everything. Tables, schemas, and logic follow familiar patterns – just modernized and optimized.

✅ Zero-Config Installation

Spin up MemCP with a single docker run or pm2 start. Start building in minutes.

🧑‍💻 Built for Developers, by Developers

MemCP was made for the people who write real-world code, not just benchmark reports. Whether you're building a SaaS product, IoT platform, or data-heavy analytics dashboard, MemCP gives you the speed and flexibility you need to scale. 

# Ready to try it?
git clone https://github.com/launix-de/memcp
cd memcp
go get
make
pm2 start ./memcp ./data/

mysql -u root -p -P 3307
Enter password: admin

🆚 MemCP vs MySQL: The Key Differences

Feature MySQL MemCP
Storage Model Row-based Column-based (compressed)
Performance Good Excellent (NUMA-optimized)
In-Memory Capable Limited Yes, by default
REST API Integration External services Built-in
Installation Footprint ~150MB+ ~10MB
Open Source ✅ ✅

💡 Ideal Use Cases

  • High-speed dashboards & real-time analytics
  • Embedded systems with minimal resources
  • Data-heavy SaaS backends
  • Replacing outdated MySQL deployments with modern performance


🧪 Try MemCP Today

Explore the documentation, check out the GitHub repo, and join a growing community of developers building fast, efficient, modern systems with MemCP.

Ditch the legacy. Embrace performance. Make the switch to MemCP.

What is memcp?

Webapps.svg

memcp is an open-source, high-performance, columnar in-memory database that can handle both OLAP and OLTP workloads. It provides an alternative to proprietary analytical databases and aims to bring the benefits of columnar storage to the open-source world.

memcp is written in Golang and is designed to be portable and extensible, allowing developers to embed the database into their applications with ease. It is also designed with a focus on scalability and performance, making it a suitable choice for distributed applications.


Features

  • fast: MemCP is built with parallelization in mind. The parallelization pattern is made for minimal overhead.
  • efficient: The average compression ratio is 1:5 (80% memory saving) compared to MySQL/MariaDB
  • modern: MemCP is built for modern hardware with caches, NUMA memory, multicore CPUs, NVMe SSDs
  • versatile: Use it in big mainframes to gain analytical performance, use it in embedded systems to conserve flash lifetime
  • Columnar storage: Stores data column-wise instead of row-wise, which allows for better compression, faster query execution, and more efficient use of memory.
  • In-memory database: Stores all data in memory, which allows for extremely fast query execution.
  • Build fast REST APIs directly in the database (they are faster because there is no network connection / SQL layer in between)
  • OLAP and OLTP support: Can handle both online analytical processing (OLAP) and online transaction processing (OLTP) workloads.
  • Compression: Lots of compression formats are supported like bit-packing and dictionary encoding
  • Scalability: Designed to scale on a single node with huge NUMA memory
  • Adjustable persistency: Decide whether you want to persist a table or not or to just keep snapshots of a period of time

MemCP Port.png

https://www.youtube.com/watch?v=g29FR4Jwius

Navigation

Introduction

Getting Started

Administration

Frontends

SQL Frontend
RDF Frontend
Custom Frontends

Internals

How things work in MemCP
Optimizations


Screenshot from htop.png


Further Reading

MemCP on Github

Scientific

How MemCP was built

Retrieved from "https://www.memcp.org/index.php?title=Main_Page&oldid=210"