Dictionary Compression
Dictionary Compression
Dictionary encoding stores each distinct string or value once and represents rows by compact identifiers. It is effective for repeated categories, status values, names, and shared prefixes; high-cardinality or mostly unique data may favor another representation.
For Male, Male, Female, Male, the dictionary stores the two strings once and the row stream as compact IDs such as 0,0,1,0. Those IDs can themselves be bit-packed or entropy-coded. A three-value dictionary needs only two raw identifier bits per row before metadata and alignment.
MemCP selects encodings from the observed physical column. String implementations can combine dictionaries, case-folded or encoded variants, and shared-prefix storage. Low-cardinality columns may use entropy-oriented coding. Scans decode batches and comparisons can often work with compact identifiers or boundaries.
Compression ratios depend on cardinality, string length, frequency distribution, collation, ordering, and null density. Historical “10x” examples are not guarantees. Persistent formats retain permanent magic/version assignments and backward readers.
Use stat to inspect the selected representation and its measured bytes per shard. Report dictionary payload, identifier stream, row count, NULL/default handling and process baseline rather than quoting only the smallest column from a dataset.
An earlier 61,440-row shard illustrates how to read that output:
main count: 61440, delta count: 0, deletions: 0 mode: string-dict[1 entries; 5 bytes], size = 7.833 KiB cntin: seq[176x int[20]/int[20]], size = 1.477 KiB cntout: seq[187x int[6]/int[7]], size = 968 B duration: int[15], size = 112.6 KiB filters: seq[161x int[1]/int[2]], size = 680 B append: seq[176x int[6]/int[7]], size = 928 B p: string-dict[3 entries; 43 bytes], size = 15.36 KiB + insertions 40 B + deletions 48 B = total 140 KiB
The three-value p column occupied 15.36 KiB in that historical shard—close to two identifier bits per row plus dictionary/alignment overhead. This is a concrete observation, not a promise that every three-value column has the same size.
See In-Memory Compression, Columnar Compression Techniques, Columnar Storage, and Performance Measurement.