File System - Revision history

Wikiservice: Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference

2026-08-28T09:59:28Z

Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference

← Older revision		Revision as of 11:59, 28 August 2026
Line 1:		Line 1:
	~~In the file system persistency layer, each subfolder of your data folder is a database~~.		<!-- Copyright (C) 2026 Carl-Philip Haensch -->
			<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
			= File System Persistence =

	~~Each folder has~~ the ~~following files:~~		The local backend stores every database in a subdirectory below <code>-data</code>. It is the simplest persistent backend and is appropriate when one MemCP process owns the writable data directory.

	* schema.json ~~- list of all tables, columns, keys~~ and ~~shards;~~ is ~~updated every time~~ a ~~shard~~ is ~~added or removed or~~ the schema ~~changes~~		== On-disk layout ==
	* ~~schema~~.~~json~~.~~old~~ - ~~fallback file~~ for ~~crash recovery~~
	* ~~[uuid]~~-~~[columnname]~~ - the ~~content of~~ a column (in ~~case~~ the [[~~Persistency and Performance Guarantees\|ENGINE~~]] ~~is not MEMORY)~~		* <code>schema.json</code> is the authoritative database schema and names the published table/shard generation.
	* [~~uuid~~]~~.log - transaction log in case the~~ [[Persistency and Performance Guarantees~~\|ENGINE~~]] ~~is LOGGED or SAFE~~		* <code>schema.json.old</code> is a rescue link to the previous committed schema generation and is read when the live schema is missing or empty.
	~~You can also use a~~ [[~~Cluster Monitor~~]] ~~for the File System persistency backend. The only thing you have to assure is that each node of the cluster points with -data to the same shared directory~~.		* <code><shard-uuid>-<column></code> stores a serialized main column. Long column names are replaced in filenames by a stable hash.
			* <code><shard-uuid>.log</code> is the WAL for <code>safe</code> and <code>logged</code> shards.
			* <code>blob/aa/bb/<hash></code> stores content-addressed large values below fan-out directories.

			Schema publication writes a same-directory temporary file, optionally fsyncs it, then atomically renames it into place. For durable writes, the directory is synchronized afterwards. A rebuild writes new immutable column files before <code>schema.json</code> makes that generation authoritative; obsolete files are removed only after publication.

			== Backup and recovery ==

			For a simple offline backup, stop MemCP cleanly and copy the complete data directory, including schema generations, column files, WALs and blobs. Do not copy only <code>schema.json</code>. A live filesystem snapshot must preserve a mutually consistent point in time; test restore and WAL replay with the actual filesystem and ENGINE modes.

			Never let two independent MemCP processes write the same directory. A shared filesystem does not provide cluster ownership, invalidation or fencing. The planned multi-node design uses explicit coordination and RADOS; see [[Cluster Monitor]].

			See [[Storage Backends]], [[Persistency and Performance Guarantees]] and [[Deployment]].

Carli at 18:59, 31 August 2025

2025-08-31T18:59:41Z

← Older revision		Revision as of 20:59, 31 August 2025
Line 7:		Line 7:
	* [uuid]-[columnname] - the content of a column (in case the [[Persistency and Performance Guarantees\|ENGINE]] is not MEMORY)		* [uuid]-[columnname] - the content of a column (in case the [[Persistency and Performance Guarantees\|ENGINE]] is not MEMORY)
	* [uuid].log - transaction log in case the [[Persistency and Performance Guarantees\|ENGINE]] is LOGGED or SAFE		* [uuid].log - transaction log in case the [[Persistency and Performance Guarantees\|ENGINE]] is LOGGED or SAFE
			You can also use a [[Cluster Monitor]] for the File System persistency backend. The only thing you have to assure is that each node of the cluster points with -data to the same shared directory.

Carli: Created page with "In the file system persistency layer, each subfolder of your data folder is a database. Each folder has the following files: * schema.json - list of all tables, columns, keys and shards; is updated every time a shard is added or removed or the schema changes * schema.json.old - fallback file for crash recovery * [uuid]-[columnname] - the content of a column (in case the ENGINE is not MEMORY) * [uuid].log - transaction log in c..."

2025-08-31T18:58:36Z

Created page with "In the file system persistency layer, each subfolder of your data folder is a database. Each folder has the following files: * schema.json - list of all tables, columns, keys and shards; is updated every time a shard is added or removed or the schema changes * schema.json.old - fallback file for crash recovery * [uuid]-[columnname] - the content of a column (in case the ENGINE is not MEMORY) * [uuid].log - transaction log in c..."

New page

In the file system persistency layer, each subfolder of your data folder is a database.

Each folder has the following files:

* schema.json - list of all tables, columns, keys and shards; is updated every time a shard is added or removed or the schema changes
* schema.json.old - fallback file for crash recovery
* [uuid]-[columnname] - the content of a column (in case the [[Persistency and Performance Guarantees|ENGINE]] is not MEMORY)
* [uuid].log - transaction log in case the [[Persistency and Performance Guarantees|ENGINE]] is LOGGED or SAFE