Current Status and Open Issues: Difference between revisions

From MemCP
Jump to navigation Jump to search
No edit summary
(Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference)
Line 1: Line 1:
<div style="padding:1rem 1.2rem; margin:0 0 1.5rem; border-left:5px solid #76b512; background:#f5f8f0; color:#17202a;">
<!-- Copyright (C) 2026 Carl-Philip Haensch -->
'''Status: Beta · Verified against commit <code>e90d6ce7fd</code> on 21 August 2026.''' MemCP is usable for tested workloads, but SQL compatibility and complex planner cases continue to evolve. Validate every application-critical query, durability requirement and client before a production migration.
</div>


This page describes the current implementation and the remaining publicly tracked work. It replaces the old static TODO list from November 2024. Active tasks and priorities belong in the [https://github.com/launix-de/memcp/issues GitHub issue tracker]; completed work belongs in the changelog and repository history.
<!-- SPDX-License-Identifier: GPL-3.0-or-later -->


== Current status ==
<span id="current-status-and-open-issues"></span>
= Current Status and Open Issues =


{| class="wikitable" style="width:100%;"
<blockquote>'''Verified against commit <code>c42e19eba</code> on 27 August 2026.''' Update this marker whenever the implementation-facing inventory is refreshed.
! Area
</blockquote>
! Status
MemCP is '''Beta'''. Core SQL, persistence, compression, parallel scans, REST, and MySQL-protocol access are usable, but compatibility and complex planner cases continue to evolve.
! Notes
 
MemCP's stated ambition remains a '''fast, compact database for mixed OLTP and OLAP'''. Measured OLAP and search-oriented workflows have reached '''up to 10× the performance of MariaDB/PostgreSQL''', particularly where RecSets and compressed column scans avoid wide intermediate rows. Isolated OLTP paths currently take about '''1.3–2.0× as long''', but in complete WordPress- and wiki-style page builds this has made '''no significant difference to overall page-loading time''' in the measured application workflows. A PostgreSQL filtered-list workflow over approximately one million documents, by contrast, took around '''30 seconds'''. The OLTP gap is tracked as JIT, compilation, point-path and compatibility work rather than used to dilute the search/analytics performance proposition.
 
For write-heavy workloads, project measurements have reached about '''10× <code>safe</code> throughput''' with <code>ENGINE=logged</code>. That result is inseparable from its durability trade-off: the WAL survives a process crash but is not synchronized against kernel crash or power loss. <code>sloppy</code> instead publishes compressed main storage on the normal 15-minute rebuild cycle, which can protect SD-card lifetime for reconstructible data while accepting loss of the unrebuilt delta.
 
== Readiness by area ==
 
{| class="wikitable"
! Area !! Current position !! What adopters should verify
|-
|-
| SQL frontend
| SQL applications || Broad tested relational subset, views, CTEs/subqueries, UNION, windows, DML/DDL, constraints, triggers, and JSON || Every production query, collation/type edge, error expectation, and metadata query
| Broad support, Beta compatibility
| DDL, DML, joins, transactions, views, triggers, window functions, UNION and extensively tested subqueries are available. MemCP does not claim complete MySQL or PostgreSQL compatibility. See [[Supported SQL]].
|-
|-
| Storage and durability
| Transactions/durability || Implicit/explicit transactions, cursor stability, snapshot/OCC infrastructure, per-table engines, WAL/replay || Required isolation/conflict patterns, power-loss behavior, backup and restore
| Operational
| Per-table <code>safe</code>, <code>logged</code>, <code>sloppy</code>, <code>memory</code> and <code>cache</code> engines are implemented. See [[Persistency and Performance Guarantees]] before choosing or changing an engine.
|-
|-
| Query planner
| Analytics || Compressed columns, batch/parallel scans, grouping/windows, adaptive indexes and reusable physical structures || Plans, memory, concurrent write latency, warm/cold performance
| New architecture implemented, actively optimized
| Logical decorrelation and join ordering are separated from cost-based physical lowering. RecSets, indexes, direct scans, group caches, ORC columns and pipeline variants are physical alternatives. See [[Query Planner and Physical Lowering]].
|-
|-
| Transactions and constraints
| Interfaces || MySQL TCP/socket, SQL/HTTP, PostgreSQL-syntax HTTP, RDF, Scheme/custom HTTP/WebSocket || Concrete connector/tool versions, authentication, protocol assumptions
| Implemented for tested semantics
| Cursor-stability transactions, an explicit ACID/OCC mode, rollback, unique and NOT NULL constraints, and enforced CREATE TABLE foreign keys are covered by tests. XA and complete MySQL isolation-level compatibility are not claimed.
|-
|-
| Operations
| Operations || Dashboard, processlist/cancellation, settings, logs, packaging, local/S3/Ceph storage || Monitoring, capacity, upgrades, restore drills, backend outages
| Implemented, still maturing
| Dashboard, metrics, processlist, query/connection cancellation, graceful shutdown, memory budgets and eviction are available.
|-
|-
| Storage backends
| Scale-out || Local sharding and parallel execution within one instance || Multi-node cluster/HA is not implemented as a production feature
| Implemented
| Local files, S3-compatible object storage and optional Ceph/RADOS are supported. Backend-specific outage, backup and latency behavior must be validated operationally.
|-
| RDF/SPARQL
| Supported subset
| Query, FILTER, OPTIONAL, update forms and Turtle loading are tested. This is not a claim of complete SPARQL standard coverage.
|-
| Scheme runtime
| Operational
| The functional runtime, concurrency primitives and native x86-64 JIT are integrated. Unsupported JIT shapes fall back to interpreted execution.
|}
|}


== What changed since the old roadmap ==
This page intentionally does not duplicate a static TODO list. Current bugs, planned work, and release blockers live in the [https://github.com/launix-de/memcp/issues GitHub issue tracker].


The previous page was a development checklist. The table below preserves every old item and records what happened to it.
== Product and engineering roadmap ==


=== Storage engine ===
The repository also contains design documents for longer-term work. They describe direction and invariants, not promised release dates:
 
* '''Broader JIT and vectorized execution:''' compile more scan filters, expressions, storage decoders and reducers into tight architecture-specific loops while retaining safe interpreter fallback.
* '''Lower compile time:''' bounded planner traversals, structural sharing and recipe-based physical emission for very large application-generated queries.
* '''Remote SQL tables:''' treat MySQL/PostgreSQL sources as scannable relations with explicit capability discovery, predicate/projection pushdown, transaction limits and safe fallback.
* '''Vector search:''' combine scalar partition boundaries with local vector “bubble” clusters and hierarchical top-k search.
* '''Cluster mode:''' leaderless CRUSH directory assignment, MOESI-inspired shard caches, RADOS persistence and remote partial scans; a later distributed catalog removes global registries for extremely large deployments.
* '''Specialized indexes and [[RecSets]]:''' choose ranges, sparse IDs, bitmaps, prefix/full-text and computed access paths according to observed density and cost.
 
See [[Cluster Monitor]], [[Query Planner and Physical Lowering]], [[RecSets]], [[JIT Compilation]], [[Vectors]] and [[Data Auto Sharding and Auto Indexing]].
 
<span id="implemented-since-the-old-status-page"></span>
== Implemented since the old status page ==
 
* per-table <code>ALTER TABLE … ENGINE</code> transitions;
* configurable LRU memory management and system-pressure eviction;
* AUTO_INCREMENT persistence and tested unique/not-null/foreign-key behavior;
* main/delta ordered index merging and LIKE/pattern access improvements;
* persistent SQL triggers and computed-column invalidation hooks;
* cursor-stability transactions plus ACID snapshot/OCC infrastructure;
* UNION, windows, views, broad subquery decorrelation, and join reordering;
* cost-based physical lowering, RecSets, group caches, ORC, and plan variants;
* native x86-64 Scheme JIT for supported procedures;
* processlist, query cancellation, dashboard, metrics, logs, and packaging;
* S3-compatible and optional Ceph/RADOS persistence;
* MySQL/PostgreSQL live import and PostgreSQL dump/archive import.
* native BSON-backed MySQL and PostgreSQL JSON support, including path access, mutation, constructors, aggregates, relational expansion, and computed-index acceleration.
 
== Outcome of the earlier roadmap ==
 
The former status page was also a design record. The compact inventory below preserves what those concrete proposals became instead of silently deleting them when they left the TODO list.


{| class="wikitable" style="width:100%;"
{| class="wikitable" style="width:100%;"
! Original roadmap item
! Earlier proposal !! Outcome
! Current result
|-
| <code>ALTER TABLE … ENGINE</code>
| '''Implemented.''' Tables can transition between the five MemCP engines. Some transitions change durability; persisted → <code>memory</code> permanently removes on-disk data.
|-
| LRU garbage collection for temporary columns
| '''Implemented and expanded.''' Global memory budgets cover persisted columns, indexes, temporary/computed columns, keytables, cache tables and dictionaries. Reloadable data can be evicted; memory-engine rows cannot.
|-
| Triggers and change hooks on computed columns
| '''Implemented.''' Persistent SQL triggers and hidden dependency/invalidation triggers maintain computed columns, group caches, prejoins and order-dependent caches.
|-
| Respect foreign keys
| '''Implemented for foreign keys declared with CREATE TABLE.''' RESTRICT, CASCADE and SET NULL paths are tested for inserts, deletes and key updates. Some ALTER TABLE foreign-key forms remain compatibility-only syntax.
|-
|-
| Serialize into memory-mapped database-sized key/value files
| <code>ALTER TABLE … ENGINE</code> || Implemented for all five engines; persisted → <code>memory</code> is deliberately destructive and documented as such.
| '''Not adopted as the persistence architecture.''' MemCP retains versioned per-column storage and WAL files and now also supports object backends. Selected storage/JIT implementations use mmap internally, but there is no single 100-GiB database blob contract.
|-
|-
| Sort index delta storage and merge it with the main index
| LRU cleanup for temporary columns || Expanded into budgets and eviction for reloadable columns, indexes, computed data, keytables, cache tables and dictionaries; memory-engine rows remain non-evictable.
| '''Implemented.''' Ordered iteration merges the compressed main permutation with an ordered index-local delta structure.
|-
|-
| Multi-table <code>scan_star</code>/merge-join operator
| Change hooks for computed columns || Implemented through SQL triggers and internal dependency/invalidation triggers for computed, grouped, prejoined and ordered data.
| '''Superseded by the current planner.''' Logical join trees are reordered and lowered into costed nested, parallel, indexed, ordered, RecSet or cached physical paths instead of one mandatory combined operator.
|-
|-
| Process IDs, request context and kill switch
| Enforced foreign keys || Implemented for tested CREATE TABLE RESTRICT, CASCADE and SET NULL paths; some ALTER compatibility forms remain non-operative.
| '''Implemented.''' <code>SHOW [FULL] PROCESSLIST</code>, dashboard process controls, <code>KILL QUERY</code>, <code>KILL CONNECTION</code> and cancellation-aware table waits are available.
|-
|-
| Transaction insertion/deletion overlays and conflict handling
| One memory-mapped database-sized key/value file || Not adopted. MemCP keeps versioned per-column files and WALs, with optional object backends; selected subsystems may still use mmap internally.
| '''Implemented and expanded.''' The storage engine provides cursor-stability undo handling plus snapshot/OCC transaction infrastructure, shard visibility, commit conflict checks, rollback and WAL synchronization at commit.
|-
|-
| Indexes for LIKE queries
| Merge sorted index delta data with the main index || Implemented with a compressed main permutation plus an ordered index-local delta structure.
| '''Implemented.''' LIKE-prefix and adaptive match-set boundaries participate in scan and physical cost decisions. Residual predicates remain when a boundary is not an exact proof.
|}
 
=== RDF frontend ===
 
The three former RDF items are implemented and covered by integration tests:
 
* SPARQL update with <code>DELETE { … } INSERT { … } WHERE { … }</code>;
* update templates driven by SELECT/WHERE bindings;
* <code>OPTIONAL { … }</code> with left-join behavior, including unmatched and multiple optional blocks.
 
RDF support remains a tested subset. Unsupported SPARQL syntax should not be inferred from the implementation of these constructs.
 
=== SQL frontend ===
 
{| class="wikitable" style="width:100%;"
! Original roadmap item
! Current result
|-
|-
| AUTO_INCREMENT
| One multi-table <code>scan_star</code>/merge-join operator || Superseded by reordered logical join trees and costed nested, indexed, ordered, RecSet, cached and parallel physical paths.
| '''Implemented and persisted''', including restart and ALTER TABLE behavior.
|-
|-
| Convert subqueries into joins
| Process IDs, request context and kill switch || Implemented through PROCESSLIST, dashboard controls, KILL QUERY/CONNECTION and cancellation-aware waits.
| '''Implemented far beyond the original item.''' The logical planner decorrelates scalar, IN/NOT IN and EXISTS/NOT EXISTS shapes using Neumann-style domains before join reordering and physical lowering.
|-
|-
| Shard group tables like their group keys
| Transaction insert/delete overlays and conflicts || Expanded into cursor-stability handling plus snapshot/OCC infrastructure, rollback, visibility and commit conflict checks.
| '''Implemented as a physical option.''' Group keytables can be partitioned from group-key information when the cost model selects reusable materialization.
|-
|-
| Prejoin complex plans
| LIKE indexes || Implemented as prefix/match boundaries with residual predicates whenever the boundary is only a candidate set.
| '''Implemented as one of several alternatives.''' Prejoins, FK/PK group reuse, group caches, direct scans and RecSets compete according to semantics and cost.
|-
|-
| Restrict users to databases
| SPARQL updates and OPTIONAL || Implemented for the tested DELETE/INSERT/WHERE, template and OPTIONAL subset; this is not complete SPARQL coverage.
| '''Implemented.''' Users, grants, revokes and database access policies are represented in system metadata and enforced by the SQL/API frontends.
|-
|-
| Test DBeaver, phpMyAdmin and metadata compatibility
| AUTO_INCREMENT persistence || Implemented, including restart and ALTER behavior.
| '''Ongoing compatibility work.''' MySQL protocol, prepared statements, SHOW metadata and selected INFORMATION_SCHEMA relations are implemented, but individual client releases still require dated compatibility tests.
|}
 
=== Scheme language and infrastructure ===
 
* '''Native JIT: implemented.''' Supported hot Scheme procedures can be compiled to x86-64 machine code, including an expanding set of calls, control flow and planner-generated lambdas. Unsupported procedures continue in the interpreter.
* '''HTTP/IPFS stream filenames: not generally implemented.''' Local files and supported virtual archive paths are available; arbitrary <code>http://</code> and <code>ipfs://</code> loading should not be documented as supported.
* '''General native plugin system: not implemented as a stable public API.''' MemCP has extension hooks such as storage boundary matchers and embedded Scheme modules, but no promised ABI for arbitrary C++/GPU plugins.
 
== Current roadmap ==
 
There is currently no published release milestone with a fixed feature schedule. As of the verification date, the public issue tracker contains one open feature issue:
 
* [https://github.com/launix-de/memcp/issues/9 #9 – Vector extension: helper functions]
 
=== Next SQL roadmap: JSON compatibility ===
 
JSON support is planned as a compatibility and integration feature, not as a replacement for relational modelling. Frequently queried fields should normally remain typed columns with constraints, statistics and indexes. JSON is useful for sparse external attributes, application payloads and migration compatibility; the SQL interface must still make extraction into relational rows and typed values straightforward.
 
MemCP already has JSONL import, JSON serialization for HTTP and Scheme, and type mapping for MySQL <code>json</code> and PostgreSQL <code>json</code>/<code>jsonb</code> imports. These facilities do not yet constitute MySQL, PostgreSQL or SQL/JSON query compatibility.
 
{| class="wikitable" style="width:100%;"
! Phase
! Planned scope
! Required semantic decisions
|-
|-
| 1. JSON value contract
| Convert subqueries into joins || Expanded into logical decorrelation of scalar, IN/NOT IN and EXISTS/NOT EXISTS forms before join ordering.
| A documented SQL JSON type/storage contract; validation and canonical serialization; <code>JSON_VALID</code>, <code>JSON_TYPE</code>, <code>JSON_QUOTE</code> and <code>JSON_UNQUOTE</code>; casts between text and JSON.
| Distinguish SQL NULL, JSON <code>null</code> and a missing path; preserve numeric precision; define duplicate-object-key behavior, invalid-input errors and equality/comparison rules before adding optimizer shortcuts.
|-
|-
| 2. Paths and read access
| Shard group tables and prejoin complex plans || Implemented as costed physical alternatives alongside direct scans, FK/PK reuse, group caches and RecSets.
| MySQL-compatible <code>JSON_EXTRACT</code>, <code>JSON_VALUE</code>, <code>JSON_QUERY</code>, <code>JSON_EXISTS</code>, <code>-></code> and <code>->></code>; <code>JSON_LENGTH</code>, <code>JSON_DEPTH</code>, <code>JSON_KEYS</code>, <code>JSON_CONTAINS</code>, <code>JSON_CONTAINS_PATH</code> and <code>JSON_SEARCH</code> where compatible semantics are defined.
| Define one parsed path representation shared by functions and operators. Constant paths should be parsed once, not once per row. Dynamic paths must have explicit cost and error behavior.
|-
|-
| 3. Construction and aggregation
| Restrict users to databases || Implemented through system users, grants, revokes and frontend enforcement.
| <code>JSON_OBJECT</code>, <code>JSON_ARRAY</code>, <code>JSON_OBJECTAGG</code> and <code>JSON_ARRAYAGG</code> with deterministic documented handling of NULLs, duplicate keys and aggregate ordering.
| Constructors must not silently conflate Scheme association lists, SQL rows and JSON objects. Aggregate results need stable semantics across shards and parallel execution.
|-
|-
| 4. Mutation
| DBeaver/phpMyAdmin metadata compatibility || Ongoing and version-specific; protocol, prepared statements, SHOW and selected INFORMATION_SCHEMA relations exist.
| <code>JSON_SET</code>, <code>JSON_INSERT</code>, <code>JSON_REPLACE</code>, <code>JSON_REMOVE</code>, array append/insert functions and merge-patch behavior.
| Functions remain pure expressions; an SQL UPDATE writes the resulting JSON value. Partial-update storage optimization may follow later but must not change transactional, trigger or durability semantics.
|-
|-
| 5. Relational projection
| Native Scheme JIT || Implemented for supported x86-64 procedures with interpreter fallback.
| SQL/JSON <code>JSON_TABLE</code> or an equivalent table-producing operator, with typed columns, nested paths, defaults and explicit error/empty behavior.
| JSON expansion belongs in the logical relational plan and must participate correctly in joins, correlation, NULL extension, cardinality estimation and cancellation. It must not be hidden as an uncosted per-row fallback.
|-
|-
| 6. Indexing and physical optimization
| Arbitrary HTTP/IPFS stream filenames || Not generally implemented; local files and supported archive paths are the documented contract.
| Reusable computed columns and adaptive indexes over stable JSON path expressions; statistics for extracted scalar values; batch extraction for scans.
| Optimize declared path expressions rather than pretending an opaque document is relationally indexed. Invalidating or updating a JSON document must invalidate every dependent extracted value safely.
|-
|-
| 7. Dialect compatibility
| Stable native C++/GPU plugin ABI || Not implemented; internal extension hooks are not a promised public plugin ABI.
| Map PostgreSQL <code>json</code>/<code>jsonb</code> extraction and containment operators such as <code>-></code>, <code>->></code>, <code>#></code>, <code>#>></code> and <code>@></code> onto the same semantic core where behavior is genuinely compatible.
| MySQL, PostgreSQL and SQL/JSON differ in path grammar, scalar return types, ordering, containment and errors. Dialect spellings may share internals, but must not be advertised as aliases when their observable semantics differ.
|}
|}


Definition of done for every JSON feature:
The earlier multi-phase JSON plan is now represented by the implemented capability inventory in [[JSON]] and [[Supported SQL]]. Keeping it labelled as future work here would be misleading.
 
* successful and must-fail SQL tests for SQL NULL, JSON null, missing paths, malformed JSON and malformed paths;
* MySQL and PostgreSQL dialect tests where the feature is advertised for both;
* persistence/restart, trigger, transaction and prepared-statement coverage;
* batch and allocation benchmarks before enabling path extraction in scan hot loops;
* EXPLAIN visibility for table expansion, computed extraction and index use;
* explicit documentation of deviations from MySQL, PostgreSQL or the SQL/JSON standard.
 
No phase above has a promised release date. Before implementation starts, the agreed compatibility slice should be represented by one or more GitHub issues and tests, rather than treating every vendor-specific JSON function as automatically in scope.
 
Recent development has concentrated on correctness and cost calibration of the planner, RecSet representations and bulk scans, JIT coverage, shard rebuild concurrency and query cancellation. Git history shows completed changes; it does not constitute a delivery promise for future work.
 
Areas that continue to need testing and incremental improvement include:
 
* MySQL/PostgreSQL syntax and metadata compatibility required by real applications;
* rare combinations of correlated subqueries, outer joins, UNION, grouping, windows and SQL three-valued logic;
* transaction contention, long-running rebuilds and crash recovery across engine/backends;
* ORM, administration-tool and connector compatibility by concrete version;
* non-x86-64 platform behavior where native JIT acceleration is unavailable;
* reproducible performance measurements across representative workloads.
 
These are quality areas, not commitments to a particular release date. Concrete work should be represented by a GitHub issue before this page calls it scheduled.


== Reporting an issue ==
<span id="areas-still-requiring-verification"></span>
== Areas still requiring verification ==


Please open a [https://github.com/launix-de/memcp/issues/new GitHub issue] with:
* complete MySQL/PostgreSQL grammar and metadata compatibility;
* rare correlated, outer-join, UNION, window, and three-valued-logic shapes;
* high-contention transaction behavior and long-running repartition/rebuild work;
* compatibility of individual ORMs and administration tools;
* platform coverage outside the main Linux/x86-64 development path;
* reproducible performance results across representative workloads.


* MemCP version or commit;
<span id="reporting-a-problem"></span>
* frontend used: MySQL protocol, <code>/sql</code>, <code>/psql</code>, RDF or embedded Scheme;
== Reporting a problem ==
* minimal schema, setup data and query;
* expected and actual result;
* table ENGINE and storage backend;
* whether the problem reproduces after a clean restart;
* for performance reports: row counts, EXPLAIN output, hardware, concurrency, cache state, successful response validation and raw samples.


See [[Contributing]] for the development and test workflow.
Include the MemCP commit/version, schema, minimal query, expected result, actual result, ENGINE mode, frontend, and whether the problem reproduces after restart. For performance reports include row counts, EXPLAIN output, cache state, hardware, concurrency, raw timings, and a current-master comparison.

Revision as of 11:59, 28 August 2026


Current Status and Open Issues

Verified against commit c42e19eba on 27 August 2026. Update this marker whenever the implementation-facing inventory is refreshed.

MemCP is Beta. Core SQL, persistence, compression, parallel scans, REST, and MySQL-protocol access are usable, but compatibility and complex planner cases continue to evolve.

MemCP's stated ambition remains a fast, compact database for mixed OLTP and OLAP. Measured OLAP and search-oriented workflows have reached up to 10× the performance of MariaDB/PostgreSQL, particularly where RecSets and compressed column scans avoid wide intermediate rows. Isolated OLTP paths currently take about 1.3–2.0× as long, but in complete WordPress- and wiki-style page builds this has made no significant difference to overall page-loading time in the measured application workflows. A PostgreSQL filtered-list workflow over approximately one million documents, by contrast, took around 30 seconds. The OLTP gap is tracked as JIT, compilation, point-path and compatibility work rather than used to dilute the search/analytics performance proposition.

For write-heavy workloads, project measurements have reached about 10× safe throughput with ENGINE=logged. That result is inseparable from its durability trade-off: the WAL survives a process crash but is not synchronized against kernel crash or power loss. sloppy instead publishes compressed main storage on the normal 15-minute rebuild cycle, which can protect SD-card lifetime for reconstructible data while accepting loss of the unrebuilt delta.

Readiness by area

Area Current position What adopters should verify
SQL applications Broad tested relational subset, views, CTEs/subqueries, UNION, windows, DML/DDL, constraints, triggers, and JSON Every production query, collation/type edge, error expectation, and metadata query
Transactions/durability Implicit/explicit transactions, cursor stability, snapshot/OCC infrastructure, per-table engines, WAL/replay Required isolation/conflict patterns, power-loss behavior, backup and restore
Analytics Compressed columns, batch/parallel scans, grouping/windows, adaptive indexes and reusable physical structures Plans, memory, concurrent write latency, warm/cold performance
Interfaces MySQL TCP/socket, SQL/HTTP, PostgreSQL-syntax HTTP, RDF, Scheme/custom HTTP/WebSocket Concrete connector/tool versions, authentication, protocol assumptions
Operations Dashboard, processlist/cancellation, settings, logs, packaging, local/S3/Ceph storage Monitoring, capacity, upgrades, restore drills, backend outages
Scale-out Local sharding and parallel execution within one instance Multi-node cluster/HA is not implemented as a production feature

This page intentionally does not duplicate a static TODO list. Current bugs, planned work, and release blockers live in the GitHub issue tracker.

Product and engineering roadmap

The repository also contains design documents for longer-term work. They describe direction and invariants, not promised release dates:

  • Broader JIT and vectorized execution: compile more scan filters, expressions, storage decoders and reducers into tight architecture-specific loops while retaining safe interpreter fallback.
  • Lower compile time: bounded planner traversals, structural sharing and recipe-based physical emission for very large application-generated queries.
  • Remote SQL tables: treat MySQL/PostgreSQL sources as scannable relations with explicit capability discovery, predicate/projection pushdown, transaction limits and safe fallback.
  • Vector search: combine scalar partition boundaries with local vector “bubble” clusters and hierarchical top-k search.
  • Cluster mode: leaderless CRUSH directory assignment, MOESI-inspired shard caches, RADOS persistence and remote partial scans; a later distributed catalog removes global registries for extremely large deployments.
  • Specialized indexes and RecSets: choose ranges, sparse IDs, bitmaps, prefix/full-text and computed access paths according to observed density and cost.

See Cluster Monitor, Query Planner and Physical Lowering, RecSets, JIT Compilation, Vectors and Data Auto Sharding and Auto Indexing.

Implemented since the old status page

  • per-table ALTER TABLE … ENGINE transitions;
  • configurable LRU memory management and system-pressure eviction;
  • AUTO_INCREMENT persistence and tested unique/not-null/foreign-key behavior;
  • main/delta ordered index merging and LIKE/pattern access improvements;
  • persistent SQL triggers and computed-column invalidation hooks;
  • cursor-stability transactions plus ACID snapshot/OCC infrastructure;
  • UNION, windows, views, broad subquery decorrelation, and join reordering;
  • cost-based physical lowering, RecSets, group caches, ORC, and plan variants;
  • native x86-64 Scheme JIT for supported procedures;
  • processlist, query cancellation, dashboard, metrics, logs, and packaging;
  • S3-compatible and optional Ceph/RADOS persistence;
  • MySQL/PostgreSQL live import and PostgreSQL dump/archive import.
  • native BSON-backed MySQL and PostgreSQL JSON support, including path access, mutation, constructors, aggregates, relational expansion, and computed-index acceleration.

Outcome of the earlier roadmap

The former status page was also a design record. The compact inventory below preserves what those concrete proposals became instead of silently deleting them when they left the TODO list.

Earlier proposal Outcome
ALTER TABLE … ENGINE Implemented for all five engines; persisted → memory is deliberately destructive and documented as such.
LRU cleanup for temporary columns Expanded into budgets and eviction for reloadable columns, indexes, computed data, keytables, cache tables and dictionaries; memory-engine rows remain non-evictable.
Change hooks for computed columns Implemented through SQL triggers and internal dependency/invalidation triggers for computed, grouped, prejoined and ordered data.
Enforced foreign keys Implemented for tested CREATE TABLE RESTRICT, CASCADE and SET NULL paths; some ALTER compatibility forms remain non-operative.
One memory-mapped database-sized key/value file Not adopted. MemCP keeps versioned per-column files and WALs, with optional object backends; selected subsystems may still use mmap internally.
Merge sorted index delta data with the main index Implemented with a compressed main permutation plus an ordered index-local delta structure.
One multi-table scan_star/merge-join operator Superseded by reordered logical join trees and costed nested, indexed, ordered, RecSet, cached and parallel physical paths.
Process IDs, request context and kill switch Implemented through PROCESSLIST, dashboard controls, KILL QUERY/CONNECTION and cancellation-aware waits.
Transaction insert/delete overlays and conflicts Expanded into cursor-stability handling plus snapshot/OCC infrastructure, rollback, visibility and commit conflict checks.
LIKE indexes Implemented as prefix/match boundaries with residual predicates whenever the boundary is only a candidate set.
SPARQL updates and OPTIONAL Implemented for the tested DELETE/INSERT/WHERE, template and OPTIONAL subset; this is not complete SPARQL coverage.
AUTO_INCREMENT persistence Implemented, including restart and ALTER behavior.
Convert subqueries into joins Expanded into logical decorrelation of scalar, IN/NOT IN and EXISTS/NOT EXISTS forms before join ordering.
Shard group tables and prejoin complex plans Implemented as costed physical alternatives alongside direct scans, FK/PK reuse, group caches and RecSets.
Restrict users to databases Implemented through system users, grants, revokes and frontend enforcement.
DBeaver/phpMyAdmin metadata compatibility Ongoing and version-specific; protocol, prepared statements, SHOW and selected INFORMATION_SCHEMA relations exist.
Native Scheme JIT Implemented for supported x86-64 procedures with interpreter fallback.
Arbitrary HTTP/IPFS stream filenames Not generally implemented; local files and supported archive paths are the documented contract.
Stable native C++/GPU plugin ABI Not implemented; internal extension hooks are not a promised public plugin ABI.

The earlier multi-phase JSON plan is now represented by the implemented capability inventory in JSON and Supported SQL. Keeping it labelled as future work here would be misleading.

Areas still requiring verification

  • complete MySQL/PostgreSQL grammar and metadata compatibility;
  • rare correlated, outer-join, UNION, window, and three-valued-logic shapes;
  • high-contention transaction behavior and long-running repartition/rebuild work;
  • compatibility of individual ORMs and administration tools;
  • platform coverage outside the main Linux/x86-64 development path;
  • reproducible performance results across representative workloads.

Reporting a problem

Include the MemCP commit/version, schema, minimal query, expected result, actual result, ENGINE mode, frontend, and whether the problem reproduces after restart. For performance reports include row counts, EXPLAIN output, cache state, hardware, concurrency, raw timings, and a current-master comparison.

Retrieved from "https://www.memcp.org/index.php?title=Current_Status_and_Open_Issues&oldid=322"