tiktoken repositories

Trending repositories tagged tiktoken, ranked by trending score. Updated daily.

  1. 1
    tryAGI/Tiktoken

    High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

    GitHub · C# · 93 stars · 7 forks · 0 stars in 24 hours · Trending score 0.07

  2. 2
    CNSeniorious000/free-chat

    An elegant LLM chat UI forked from chatgpt-demo of @anse-app. Index site at https://free-chat.asia

    GitHub · TypeScript · 192 stars · 54 forks · 0 stars in 24 hours · Trending score 0.07

  3. 3
    georgemandis/tokenizer-mono

    A monospace color font that highlights LLM tokens as you type. 28 faces, one per tokenizer (cl100k, o200k, Llama, Gemma, Qwen, DeepSeek, Kimi K2 and more), built from OpenType substitution rules and a color table. No JavaScript.

    GitHub · Python · 10 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05

  4. 4
    behrouzbk/tokensaver

    Count tokens across AI models and get concrete, priced suggestions to cut your prompt costs. Runs offline.

    GitHub · Python · 7 stars · 2 forks · 0 stars in 24 hours · Trending score 0.01

  5. 5
    melonmath/microkimi

    The Kimi K3 and DeepSeek-V4-Flash-0731 architectures, reimplemented from scratch in pure Rust (zero dependencies) and verified 1:1 against the official code.

    GitHub · Rust · 23 stars · 2 forks · 0 stars in 24 hours · Trending score 0.00

  6. 6
    mandeal303/tiktokenizer

    An interactive playground for comparing how different language models tokenize the same text.

    GitHub · TypeScript · 50 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00