Dictionary Compression: Difference between revisions
(Created page with "Strings are the nightmare of every database. You never know their exact size and you have to either reference them in a separate storage or reserve enough character space to store them in-table. Columnar databases can do better. But that's not guaranteed, you have to put a bit of extra effort into it. Here's how: At first, memcp converts strings into dictionaries. If you have a large list consisting of [Male, Male, Male, Female, Male, Female, Male, Male], you only have...") |
Wikiservice (talk | contribs) (Refresh MemCP documentation: accuracy, operational guidance, performance profile and maintained API reference) |
||
| Line 1: | Line 1: | ||
<!-- Copyright (C) 2026 Carl-Philip Haensch --> | |||
<!-- SPDX-License-Identifier: GPL-3.0-or-later --> | |||
= Dictionary Compression = | |||
Dictionary encoding stores each distinct string or value once and represents rows by compact identifiers. It is effective for repeated categories, status values, names, and shared prefixes; high-cardinality or mostly unique data may favor another representation. | |||
For <code>Male, Male, Female, Male</code>, the dictionary stores the two strings once and the row stream as compact IDs such as <code>0,0,1,0</code>. Those IDs can themselves be bit-packed or entropy-coded. A three-value dictionary needs only two raw identifier bits per row before metadata and alignment. | |||
MemCP selects encodings from the observed physical column. String implementations can combine dictionaries, case-folded or encoded variants, and shared-prefix storage. Low-cardinality columns may use entropy-oriented coding. Scans decode batches and comparisons can often work with compact identifiers or boundaries. | |||
Compression ratios depend on cardinality, string length, frequency distribution, collation, ordering, and null density. Historical “10x” examples are not guarantees. Persistent formats retain permanent magic/version assignments and backward readers. | |||
Use <code>stat</code> to inspect the selected representation and its measured bytes per shard. Report dictionary payload, identifier stream, row count, NULL/default handling and process baseline rather than quoting only the smallest column from a dataset. | |||
An earlier 61,440-row shard illustrates how to read that output: | |||
<pre>main count: 61440, delta count: 0, deletions: 0 | |||
mode: string-dict[1 entries; 5 bytes], size = 7.833 KiB | |||
cntin: seq[176x int[20]/int[20]], size = 1.477 KiB | |||
cntout: seq[187x int[6]/int[7]], size = 968 B | |||
duration: int[15], size = 112.6 KiB | |||
filters: seq[161x int[1]/int[2]], size = 680 B | |||
append: seq[176x int[6]/int[7]], size = 928 B | |||
p: string-dict[3 entries; 43 bytes], size = 15.36 KiB | |||
+ insertions 40 B | |||
+ deletions 48 B | |||
= total 140 KiB</pre> | |||
The three-value <code>p</code> column occupied 15.36 KiB in that historical shard—close to two identifier bits per row plus dictionary/alignment overhead. This is a concrete observation, not a promise that every three-value column has the same size. | |||
See [[In-Memory Compression, Columnar Compression Techniques]], [[Columnar Storage]], and [[Performance Measurement]]. | |||
Latest revision as of 11:59, 28 August 2026
Dictionary Compression
Dictionary encoding stores each distinct string or value once and represents rows by compact identifiers. It is effective for repeated categories, status values, names, and shared prefixes; high-cardinality or mostly unique data may favor another representation.
For Male, Male, Female, Male, the dictionary stores the two strings once and the row stream as compact IDs such as 0,0,1,0. Those IDs can themselves be bit-packed or entropy-coded. A three-value dictionary needs only two raw identifier bits per row before metadata and alignment.
MemCP selects encodings from the observed physical column. String implementations can combine dictionaries, case-folded or encoded variants, and shared-prefix storage. Low-cardinality columns may use entropy-oriented coding. Scans decode batches and comparisons can often work with compact identifiers or boundaries.
Compression ratios depend on cardinality, string length, frequency distribution, collation, ordering, and null density. Historical “10x” examples are not guarantees. Persistent formats retain permanent magic/version assignments and backward readers.
Use stat to inspect the selected representation and its measured bytes per shard. Report dictionary payload, identifier stream, row count, NULL/default handling and process baseline rather than quoting only the smallest column from a dataset.
An earlier 61,440-row shard illustrates how to read that output:
main count: 61440, delta count: 0, deletions: 0 mode: string-dict[1 entries; 5 bytes], size = 7.833 KiB cntin: seq[176x int[20]/int[20]], size = 1.477 KiB cntout: seq[187x int[6]/int[7]], size = 968 B duration: int[15], size = 112.6 KiB filters: seq[161x int[1]/int[2]], size = 680 B append: seq[176x int[6]/int[7]], size = 928 B p: string-dict[3 entries; 43 bytes], size = 15.36 KiB + insertions 40 B + deletions 48 B = total 140 KiB
The three-value p column occupied 15.36 KiB in that historical shard—close to two identifier bits per row plus dictionary/alignment overhead. This is a concrete observation, not a promise that every three-value column has the same size.
See In-Memory Compression, Columnar Compression Techniques, Columnar Storage, and Performance Measurement.