ZeroDriveX

ZDX Tools

Open Source

Open Source Tools

Tools from the ZDX Lab

Utilities built during real ZeroDriveX development. No paywalls, no accounts. Download, use, modify. Built for developers who want things that actually work.

AITraining DataBPEPythonFree

Available

Training Data Formatter

v1.0.0Python8 KB

Universal training data formatter for AI and language model pipelines. Feed it .txt, .json, .jsonl, .csv, .tsv, or .md files and it outputs clean, chunked text optimized for any model training pipeline.

Download formatter.py
AITrainingNLPPython

What it does

  • Auto-detects text fields in JSON and CSV — no manual mapping needed.
  • Strips HTML, markdown formatting, and encoding noise automatically.
  • Chunks text to model-sized pieces at sentence boundaries.
  • Deduplicates content so repeated data does not skew training.
  • Quality filter removes garbage chunks that would hurt model performance.
  • Compatible with HuggingFace, OpenAI, PyTorch, TensorFlow, and ZDX BAN pipelines.

Quick start

python3 formatter.py --input ./raw_data --output ./corpus/corpus.txt

ZDX Tokenizer Trainer

v1.0.0Python9 KB

BPE tokenizer training CLI. Train on any plain text corpus and export to every major format. Byte-level encoding handles any language with no unknown characters.

Download tokenizer.py
AINLPPythonBPE

What it does

  • Trains a Byte-level BPE tokenizer from scratch on your own data.
  • Exports to HuggingFace fast tokenizer, ZDX BAN native, vocab.txt, vocab.json, and GPT-2 style merges.txt.
  • Auto-detects text fields in JSON, JSONL, CSV, and plain text inputs.
  • Built-in smoke test verifies encode/decode round-trip after every training run.
  • Compression stats help you judge tokenizer quality before committing to a full training run.
  • zdx_tokenizer.json output is picked up automatically by the ZDX BAN train.py pipeline.

Quick start

python3 tokenizer.py --input ./corpus --output ./my_tokenizer --vocab_size 16000

Coming Soon

BAN Runtime

Soon

Disk-backed branched attractor inference engine. Runs on 256MB RAM.

AIInferencePython

Corpus Downloader

Soon

One-command download of public text corpora — Wikipedia, OpenWebText, and more.

AIDataCLI

Built in the open

These tools come directly from ZeroDriveX development. No stripped-down versions, no feature locks. If it works well enough to ship a product with, it is good enough to share. More tools drop as development continues.