Python/Cython implementations and experimental data for the Pruned BPE and DH-BPE publications.
An educational Python project for learning tokenization step by step by building character-level, byte-level, and BPE tokenizers from scratch.
Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python
Autonomous Agent Byte-Pair Encoding (BPE) Subword Tokenizer Training & Compression Engine
Autonomous Agent Byte-Pair Encoding (BPE) Subword Tokenizer Training & Compression Engine
Production-grade genpark-bpe-byte-pair-encoding-tokenizer-skill skill for AI agents
Byte-Pair Encoding (BPE) subword tokenizer trainer and deterministic text encoder/decoder.
Production-grade genpark-bpe-byte-pair-encoding-tokenizer-skill skill for AI agents
Byte-Pair Encoding (BPE) subword tokenizer trainer and deterministic text encoder/decoder.
Controlled comparison of subword tokenizer vocabularies (BPE vs Unigram-LM) over chemistry SMILES
Pure Rust runtime for loading SentencePiece models and encoding/decoding text 🦀.
Early development repo of uzbek-bpe-16k: a byte-level BPE tokenizer for Uzbek (16k vocab), 1.749 tokens/word on held-out web text vs 2.578 for GPT-4o. Release: huggingface.co/IslombekT/uzbek-bpe-16k
A subword-level GRU language model inference engine.
A from-scratch implementation of a GPT-style byte-level BPE tokenizer in pure Python, demonstrating vocabulary learning, pair-frequency merging, token encoding, decoding, and lossless text reconstruction.
Byte pair encoding tokenizer for OpenAI models that encodes text into token IDs, decodes them back, and counts tokens.
LLM-inspired BiLSTM pipeline for real-time, multi-label toxicity inference across adversarial discourse modalities.
Akshara-aware tokenizer for six Brahmic scripts, with a trained SentencePiece model
Research framework and dataset evaluating subword tokenization exploits using Romanian morphology as an attack vector
Implementation of Deep Averaging Networks (DAN) for sentiment classification with experiments on GloVe embeddings and subword tokenization using Byte Pair Encoding (BPE).
Interactive laboratory for Myanmar Unicode NFC analysis, syllable and word segmentation, and subword tokenizers (BPE, Unigram, WordPiece).
To associate your repository with the subword-tokenization topic, visit your repo's landing page and select "manage topics."