Wikiservice: Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference

2026-08-28T09:59:28Z

Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference

← Older revision		Revision as of 11:59, 28 August 2026
Line 1:		Line 1:
	~~Strings are the nightmare of every database~~. ~~You never know their exact size and you have to either reference them in a separate storage~~ or ~~reserve enough character space to store them in~~-~~table.~~		<!-- Copyright (C) 2026 Carl-Philip Haensch -->
			<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
			= Dictionary Compression =

	~~Columnar databases can do better~~. ~~But that's not guaranteed~~, ~~you have to put a bit of extra effort into it~~. ~~Here's how:~~		Dictionary encoding stores each distinct string or value once and represents rows by compact identifiers. It is effective for repeated categories, status values, names, and shared prefixes; high-cardinality or mostly unique data may favor another representation.

	~~At first, memcp converts strings into dictionaries. If you have a large list consisting of [Male,~~ Male, Male, Female, Male, ~~Female, Male, Male], you only have two distinct values. So~~ the dictionary ~~can hold "Male,Female"~~ and ~~then you can reference them by integers (0,~~0,0,1,0~~,1,0,0)~~.		For <code>Male, Male, Female, Male</code>, the dictionary stores the two strings once and the row stream as compact IDs such as <code>0,0,1,0</code>. Those IDs can themselves be bit-packed or entropy-coded. A three-value dictionary needs only two raw identifier bits per row before metadata and alignment.

	~~Then~~, ~~these integers can be [[Integer Compression\|Integer Compressed]]~~. ~~And if that's not enough, you~~ can ~~also add [[Sequence Compression]]~~.		MemCP selects encodings from the observed physical column. String implementations can combine dictionaries, case-folded or encoded variants, and shared-prefix storage. Low-cardinality columns may use entropy-oriented coding. Scans decode batches and comparisons can often work with compact identifiers or boundaries.

	~~With dictionary compression~~, ~~you can achieve compression ratios of 10x~~ and ~~higher~~.		Compression ratios depend on cardinality, string length, frequency distribution, collation, ordering, and null density. Historical “10x” examples are not guarantees. Persistent formats retain permanent magic/version assignments and backward readers.

	~~Here's an example from~~ <code>~~(print (~~stat ~~schema table))~~</code>		Use <code>stat</code> to inspect the selected representation and its measured bytes per shard. Report dictionary payload, identifier stream, row count, NULL/default handling and process baseline rather than quoting only the smallest column from a dataset.
	~~Shard 6254~~
	---		An earlier 61,440-row shard illustrates how to read that output:
	main count: 61440, delta count: 0, deletions: 0
	mode: ~~'''~~string-dict[1 entries; 5 bytes]~~'''~~, size = 7.~~833KiB~~		<pre>main count: 61440, delta count: 0, deletions: 0
	cntin: seq[176x int[20]/int[20]], size = 1.~~477KiB~~		mode: string-dict[1 entries; 5 bytes], size = 7.833 KiB
	cntout: seq[187x int[6]/int[7]], size = ~~968B~~		cntin: seq[176x int[20]/int[20]], size = 1.477 KiB
	duration: int[15], size = 112.~~6KiB~~		cntout: seq[187x int[6]/int[7]], size = 968 B
	filters: seq[161x int[1]/int[2]], size = ~~680B~~		duration: int[15], size = 112.6 KiB
	append: seq[176x int[6]/int[7]], size = ~~928B~~		filters: seq[161x int[1]/int[2]], size = 680 B
	p: ~~'''~~string-dict[3 entries; 43 bytes]~~'''~~, size = 15.~~36KiB~~		append: seq[176x int[6]/int[7]], size = 928 B
	~~---~~		p: string-dict[3 entries; 43 bytes], size = 15.36 KiB
	~~---~~		+ insertions 40 B
	+ insertions ~~40B~~		+ deletions 48 B
	+ deletions ~~48B~~		= total 140 KiB</pre>
	~~---~~
	= total ~~140KiB~~		The three-value <code>p</code> column occupied 15.36 KiB in that historical shard—close to two identifier bits per row plus dictionary/alignment overhead. This is a concrete observation, not a promise that every three-value column has the same size.
	~~As you can see, the dictionary can store massive amounts of data (61k items) in~~ 15.4 KiB ~~for~~ a 3-~~way dictionary~~. ~~That's 2 bits per item!~~
			See [[In-Memory Compression, Columnar Compression Techniques]], [[Columnar Storage]], and [[Performance Measurement]].

Carli: Created page with "Strings are the nightmare of every database. You never know their exact size and you have to either reference them in a separate storage or reserve enough character space to store them in-table. Columnar databases can do better. But that's not guaranteed, you have to put a bit of extra effort into it. Here's how: At first, memcp converts strings into dictionaries. If you have a large list consisting of [Male, Male, Male, Female, Male, Female, Male, Male], you only have..."

2025-08-22T19:06:13Z

Created page with "Strings are the nightmare of every database. You never know their exact size and you have to either reference them in a separate storage or reserve enough character space to store them in-table. Columnar databases can do better. But that's not guaranteed, you have to put a bit of extra effort into it. Here's how: At first, memcp converts strings into dictionaries. If you have a large list consisting of [Male, Male, Male, Female, Male, Female, Male, Male], you only have..."

New page

Strings are the nightmare of every database. You never know their exact size and you have to either reference them in a separate storage or reserve enough character space to store them in-table.

Columnar databases can do better. But that's not guaranteed, you have to put a bit of extra effort into it. Here's how:

At first, memcp converts strings into dictionaries. If you have a large list consisting of [Male, Male, Male, Female, Male, Female, Male, Male], you only have two distinct values. So the dictionary can hold "Male,Female" and then you can reference them by integers (0,0,0,1,0,1,0,0).

Then, these integers can be [[Integer Compression|Integer Compressed]]. And if that's not enough, you can also add [[Sequence Compression]].

With dictionary compression, you can achieve compression ratios of 10x and higher.

Here's an example from <code>(print (stat schema table))</code>
Shard 6254
---
main count: 61440, delta count: 0, deletions: 0
mode: '''string-dict[1 entries; 5 bytes]''', size = 7.833KiB
cntin: seq[176x int[20]/int[20]], size = 1.477KiB
cntout: seq[187x int[6]/int[7]], size = 968B
duration: int[15], size = 112.6KiB
filters: seq[161x int[1]/int[2]], size = 680B
append: seq[176x int[6]/int[7]], size = 928B
p: '''string-dict[3 entries; 43 bytes]''', size = 15.36KiB
---
---
+ insertions 40B
+ deletions 48B
---
= total 140KiB
As you can see, the dictionary can store massive amounts of data (61k items) in 15.4 KiB for a 3-way dictionary. That's 2 bits per item!

Dictionary Compression - Revision history

Wikiservice: Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference