Replace naive whitespace tokenizer with SentencePiece unigram (Viterbi + greedy longest-match)

[Copilot]

The T5 tokenizer was a whitespace-split + per-character fallback that mishandled subwords, multi-byte UTF-8, and any token not perfectly aligned with word boundaries. Replaces it with a correct SentencePiece unigram implementation and fixes two supporting gaps.

src/t5_encoder.hpp — tokenizer rewrite

• preprocess(): collapses whitespace, strips leading/trailing, prepends ▁, replaces spaces with ▁ — matching SentencePiece normalization
• viterbi(): Viterbi DP over byte positions maximizing sum of unigram log-probs; activated when tokenizer.ggml.scores is present in the GGUF
• greedy(): greedy longest-match scan; fallback when scores are absent — already far superior to the old approach
• utf8_char_len(): unk-fallback advances one full UTF-8 character (not one byte) to avoid corrupting subsequent lookups
• tok2id switched from std::map → std::unordered_map (O(log n) → O(1) lookup)

"A cinematic shot" (old) → ["▁A", "▁", "c", "i", "n", …]  ← per-char fallback
"A cinematic shot" (new) → ["▁A", "▁cinematic", "▁shot"]   ← greedy/Viterbi subword

src/ltx-quantize.cpp — array KV preservation

The manual KV-copy switch silently dropped all GGUF_TYPE_ARRAY entries (tokenizer vocab, scores). Replaced with gguf_set_kv(out_ctx, src.gguf_ctx) — copies all KV pairs including arrays in one call.

convert.py — scores round-trip

convert_t5 --tokenizer now writes tokenizer.ggml.scores via tok.sp_model.GetScore(i) using writer.add_token_scores(). GGUFs produced by the converter will carry scores, automatically enabling Viterbi mode in the C++ tokenizer.

DEV.md

Tokenizer and quantizer-metadata entries in the known-limitations table marked as resolved; §8 updated with Viterbi/greedy mode description and GGUF key table.

Original prompt

Inspired by llama.cpp, ltx.cpp is a Portable C++17 implementation of LTX 2.3 generation using GGML/GGUF.

Our references are:

• ComfyUI Unsloth LTX 2.3 workflow: https://gist.github.com/lmangani/e441d3b57e3fe623b67e4490b5a77c33
• Unsloth LTX 2.3 GGUF models: https://huggingface.co/unsloth/LTX-2.3-GGUF
• acestep.cpp an excellent reference for GGML: https://github.com/ServeurpersoCom/acestep.cpp

GOAL: ltx.cpp tools to run video inference using the GGUF models without python or other dependencies, replicatng what the reference ComfyUI workflow is capable of, but stand-alone.

---

📍 Connect Copilot coding agent with Jira, Azure Boards or Linear to delegate work to Copilot in one click without leaving your project management tool.