Main Page

MemCP – A Modern In-Memory Columnar Database

MemCP is a high-performance, in-memory, column-oriented database designed for modern workloads. It provides a lightweight, developer-friendly alternative to traditional relational databases such as MySQL, with a focus on speed, compression, and direct API integration.

---

Key Features

• High Performance: NUMA-aware, parallelized query execution optimized for multicore CPUs, large caches, and NVMe SSDs. Handles both OLTP and OLAP workloads efficiently.
• Columnar Storage: Data is stored by column for improved compression, reduced memory footprint, and faster analytical queries.
• In-Memory Operation: Designed to keep data in memory, with configurable persistence backends for durability.
• Built-in APIs: Exposes RESTful endpoints directly from the database, reducing middleware overhead.
• Compression: Multiple strategies (bit-packing, dictionary encoding, sequence compression) reduce storage by up to 80% compared to MySQL/MariaDB.
• Simple Deployment: Start with a single docker run or pm2 start command. Lightweight footprint (~10MB).
• Extensible: Written in Go, with pluggable storage backends and custom frontend support (SQL, RDF, REST).

---

Why MemCP?

Traditional relational databases were designed decades ago, optimized for spinning disks and single-core CPUs. MemCP rethinks the core design for today’s hardware and workloads:

• Real-time dashboards and analytics
• Data-heavy SaaS platforms
• Embedded systems with limited resources
• High-throughput OLTP/OLAP hybrids

---

Quick Start

Clone and build MemCP from source:

git clone https://github.com/launix-de/memcp
cd memcp
go get
make
pm2 start ./memcp ./data/

Connect with MySQL tooling:

mysql -u root -p -P 3307
Enter password: admin

---

MemCP vs. MySQL

Feature	MySQL	MemCP
Storage Model	Row-based	Column-based (compressed)
Performance	Good	NUMA-optimized, in-memory
In-Memory Capable	Limited	Yes (default)
REST API Integration	External	Built-in
Installation Footprint	~150MB+	~10MB
Open Source	✅	✅

---

Architecture Overview

• Tables, Schemas, Columns: Familiar SQL-style structures with a columnar physical layout.
• Transaction Model: Supports both OLTP and OLAP semantics with delta + main storage.
• Persistence: Configurable storage backends (filesystem, S3, Ceph).
• Frontends: Multiple query interfaces:

 - SQL frontend (MySQL wire protocol + SQL over REST)  
 - RDF/graph query engine  
 - Custom APIs via in-database web apps  

---

Documentation

• What is OLTP and OLAP
• History of the MemCP project
• Hardware Requirements
• Persistency and Performance Guarantees
• Comparison: MemCP vs. MySQL
• Install with Docker
• Build from Source
• Contributing
• SQL over REST
• REST & Microservices

---

Navigation

Introduction

• What is OLTP and OLAP
• History of the MemCP project
• Hardware Requirements
• Persistency and Performance Guarantees
• Current Status and Open Issues
• Comparison: MemCP vs. MySQL

Getting Started

• With Docker
• With Singularity
• Build from Source
• Contributing
• Introduction to Scheme
• Full SCM API documentation

Administration

• Deployment
• Migration from MySQL and PostgreSQL
• Settings
• Process Hibernation
• Performance Measurement
• MemCP Console

Frontends

SQL Frontend

• Supported SQL
• Advanced SQL Tutorial
• SQL over REST
• Supported Tooling
• How SQL Operators are implemented on MemCP
• Add custom SQL operators to MemCP

RDF Frontend

• Introduction to RDF
• Advanced Graph Querying
• RDF templating and model driven development

Custom Frontends

• In-Database WebApps and REST Services
• MemCP for Microservices
• Websockets in MemCP

Persistency Backends (= Storage)

• File System
• S3 Buckets
• Ceph/Rados
• Cluster Monitor

Internals

How things work in MemCP

• Databases, Tables and Columns
• Shards, RecordIDs, Main Storage, Delta Storage
• Columnar Storage
• Transactions
• Full SCM API documentation

SCM Documentation

• SCM Builtins
• Arithmetic / Logic
• Strings
• Streams
• Lists
• Associative Lists / Dictionaries
• Date
• Vectors
• Parsers
• Sync
• IO
• Storage

Optimizations

• In-Memory Compression, Columnar Compression Techniques
• Temporary Columns
• Data Auto Sharding and Auto Indexing
• Parallel Computing

---

Further Reading

• MemCP on GitHub
• VLDB Research Paper
• LNI Proceedings Paper
• Large Graph Algorithms

Additional blog posts on design decisions, compression techniques, and performance optimization are available on the Launix blog.

---

Community

MemCP is an open-source project maintained by developers for developers. Contributions are welcome — whether in the form of bug reports, feature requests, or pull requests.

See: Contributing