File System: Difference between revisions

From MemCP
Jump to navigation Jump to search
No edit summary
(Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference)
 
Line 1: Line 1:
In the file system persistency layer, each subfolder of your data folder is a database.
<!-- Copyright (C) 2026 Carl-Philip Haensch -->
<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
= File System Persistence =


Each folder has the following files:
The local backend stores every database in a subdirectory below <code>-data</code>. It is the simplest persistent backend and is appropriate when one MemCP process owns the writable data directory.


* schema.json - list of all tables, columns, keys and shards; is updated every time a shard is added or removed or the schema changes
== On-disk layout ==
* schema.json.old - fallback file for crash recovery
 
* [uuid]-[columnname] - the content of a column (in case the [[Persistency and Performance Guarantees|ENGINE]] is not MEMORY)
* <code>schema.json</code> is the authoritative database schema and names the published table/shard generation.
* [uuid].log - transaction log in case the [[Persistency and Performance Guarantees|ENGINE]] is LOGGED or SAFE
* <code>schema.json.old</code> is a rescue link to the previous committed schema generation and is read when the live schema is missing or empty.
You can also use a [[Cluster Monitor]] for the File System persistency backend. The only thing you have to assure is that each node of the cluster points with -data to the same shared directory.
* <code>&lt;shard-uuid&gt;-&lt;column&gt;</code> stores a serialized main column. Long column names are replaced in filenames by a stable hash.
* <code>&lt;shard-uuid&gt;.log</code> is the WAL for <code>safe</code> and <code>logged</code> shards.
* <code>blob/aa/bb/&lt;hash&gt;</code> stores content-addressed large values below fan-out directories.
 
Schema publication writes a same-directory temporary file, optionally fsyncs it, then atomically renames it into place. For durable writes, the directory is synchronized afterwards. A rebuild writes new immutable column files before <code>schema.json</code> makes that generation authoritative; obsolete files are removed only after publication.
 
== Backup and recovery ==
 
For a simple offline backup, stop MemCP cleanly and copy the complete data directory, including schema generations, column files, WALs and blobs. Do not copy only <code>schema.json</code>. A live filesystem snapshot must preserve a mutually consistent point in time; test restore and WAL replay with the actual filesystem and ENGINE modes.
 
Never let two independent MemCP processes write the same directory. A shared filesystem does not provide cluster ownership, invalidation or fencing. The planned multi-node design uses explicit coordination and RADOS; see [[Cluster Monitor]].
 
See [[Storage Backends]], [[Persistency and Performance Guarantees]] and [[Deployment]].

Latest revision as of 11:59, 28 August 2026

File System Persistence

The local backend stores every database in a subdirectory below -data. It is the simplest persistent backend and is appropriate when one MemCP process owns the writable data directory.

On-disk layout

  • schema.json is the authoritative database schema and names the published table/shard generation.
  • schema.json.old is a rescue link to the previous committed schema generation and is read when the live schema is missing or empty.
  • <shard-uuid>-<column> stores a serialized main column. Long column names are replaced in filenames by a stable hash.
  • <shard-uuid>.log is the WAL for safe and logged shards.
  • blob/aa/bb/<hash> stores content-addressed large values below fan-out directories.

Schema publication writes a same-directory temporary file, optionally fsyncs it, then atomically renames it into place. For durable writes, the directory is synchronized afterwards. A rebuild writes new immutable column files before schema.json makes that generation authoritative; obsolete files are removed only after publication.

Backup and recovery

For a simple offline backup, stop MemCP cleanly and copy the complete data directory, including schema generations, column files, WALs and blobs. Do not copy only schema.json. A live filesystem snapshot must preserve a mutually consistent point in time; test restore and WAL replay with the actual filesystem and ENGINE modes.

Never let two independent MemCP processes write the same directory. A shared filesystem does not provide cluster ownership, invalidation or fencing. The planned multi-node design uses explicit coordination and RADOS; see Cluster Monitor.

See Storage Backends, Persistency and Performance Guarantees and Deployment.

Retrieved from "https://www.memcp.org/index.php?title=File_System&oldid=329"