Serialization Proposal 1

[iriswebb]

These additions provide a way to represent and render a parsed BDF font as a slice of u8s, and tooling for that conversion in the font converter and previewer.

The format for serialization attempts to be as close to the BdfFont struct as possible while saving space, using 17 bytes per character:

* Header (12 Bytes):
 -    Ascent  (pixels, u16 BE)
 -    Descent (pixels, u16 BE)
 -    Replacement Character (index into character table, u32 BE)
 -    Character Table Length (entries, u32 BE)

* Glyph Table (17 Bytes Per Entry):
 -    corresponding codepoint (u32 BE)
 -    top_left.x (i16 BE)
 -    top_left.y (i16 BE)
 -    size.width (u16 BE)
 -    size.height (u16 BE)
 -    device_width (pixels, u8)
 -    data index  (bytes from start of data, u32 BE)

[rfuest]

Thanks for the PR, I agree that the storage of the glyphs should be optimized to save space. But I would prefer not to implement manual serialization and deserialization if possible.

My idea was to store the glyphs as a struct of arrays instead of an array of BdfGlyph structs, which should prevent any padding/alignment of the glyph struct to waste space. Something like this (untested quick mockup):

struct BdfFont<'a, Coord, Size, Index> {
    top_left: &'a [Coord],
    size: &'a [Size],
    device_width: &'a [Size],
    data_index: &'a [Index],
}

const TOP_LEFT: [i8; 256 * 2] = ...;
const SIZE: [u8; 256 * 2] = ...;
const WIDTH: [i8; 256] = ...;
const DATA_INDEX: [u16; 256] = ...;

pub const MY_FONT: BdfFont<'static, i8, u8, u16> = {
    .top_left = &TOP_LEFT,
    .size = &SIZE,
    .width = &WIDTH,
    .data_index = &DATA_INDEX,
};

The advantage is that no manual serialization is necessary and that each font can use the minimum data size required, because of the type parameters in ´BdfFont. The eg-font-converter` would determine which is the smallest type that can be used to represent all values of one kind.

But my proposal only works for fonts embedded in the source code, a serialization format would still be useful for other applications which load fonts at runtime from some external storage. If you want to continue with this PR I would suggest to look into using a serialization format like https://github.com/jamesmunns/postcard.

[iriswebb]

Representing fonts in source code would make it harder to use those fonts in other languages. For example, generating custom fonts for u8g2_fonts requires compiling C source code since that's the only format u8g2's tooling generates.

Using postcard, would it be possible to index serialized data without de-serializing it? For a large font, creating a whole copy of it could use up the memory of a machine.

[iriswebb]

Although the serialization in this PR tries to replicate the BdfFont struct as close as possible, For fonts with large numbers of glyphs (like unifont, which has ~57000 in the BDF release), there could be a more optimized way of storing characters.

As of now, finding a glyph is O(n), and there are 17 bytes of metadata per glyph. This makes rendering slow for CJK characters and emojis, and means that the metadata of a font can get larger than the bitmap data.

Sorting the list of glyphs would allow for a binary search, but storing ranges of contiguous sections of glyphs would make the search functionally O(1), since fonts usually group glyphs together in large blocks

This would also remove the need to denote the corresponding character of each glyph, making the character struct 13 bytes per glyph, potentially saving up to 200kb for a font like unifont

I propose:

* ascent
* descent
* replacement
* section_count
* character_count

SECTION TABLE (sorted):
* starting char
* ending char
* starting index into glyph table

GLYPH TABLE (sorted):
* Bounding Box (8 bytes)
* Index from start of data block (4 bytes)
* Kerning (1 byte)

[iriswebb]

This recent commit refactored the SerializedBdfFont rendering into a new trait and struct ProportionalFont and PropotionalTextStyle

[rfuest]

In case this format is also used to load a font at runtime it would be a good idea to add some validation that it is a valid font file. A magic value, a version, and the file size should be enough.

[rfuest]

Using postcard, would it be possible to index serialized data without de-serializing it? For a large font, creating a whole copy of it could use up the memory of a machine.

You are correct that this would be an issue for a variable integer size format like postcard. It would need to be a hybrid approach, where at least the glyph index table would use fixed integer sizes for more efficient lookup. I'm OK with leaving the manual serialization/deserialization.

As of now, finding a glyph is O(n), and there are 17 bytes of metadata per glyph. This makes rendering slow for CJK characters and emojis, and means that the metadata of a font can get larger than the bitmap data.

Sorting the list of glyphs would allow for a binary search, but storing ranges of contiguous sections of glyphs would make the search functionally O(1), since fonts usually group glyphs together in large blocks

Using binary search for glpyh lookup was always the idea for the non serialized version. I would keep the non grouped version for this PR to keep things simpler.

[iriswebb]

Suggestions applied, I will push another commit to resolve the errors shortly

[iriswebb]

It is not possible to generate a reference to the bitmap data for each glyph using the function EgBdfOutput::new without changing the return value or the data structure back to an index

[rfuest]

It is not possible to generate a reference to the bitmap data for each glyph using the function EgBdfOutput::new without changing the return value or the data structure back to an index

My comment mentioned this, but I didn't explain this very well. The glyphs field inside the BdfFont cannot use the new version of BdfGlyph directly any longer. It will need to continue to use the old version of BdfGlyph with the start_index attribute, called something like GlyphData. The lookup function will convert between GlyphData and the new version of BdfGlyph (with the slice reference to the bitmap data).

[iriswebb]

BdfFont is becoming similar to the bdf::Font the parser generates

start_index indexes the bit the glyph data starts at, not the byte, which makes harder to generate a slice from

[iriswebb]

Padding each glyphs data to a byte boundary seems to work