RAG text chunker
Split a document into token-sized chunks with overlap for retrieval and embeddings. Choose the size, keep paragraphs or sentences whole, and download the result as JSON or JSONL.
How to use it
- Paste the document you want to split into the box above, or press Try an example.
- Choose the tokenizer that matches your embedding or chat model, a chunk size in tokens (500 is a common starting point) and an overlap (about 10 to 20 percent of the size).
- Choose what to keep together: paragraphs, sentences or lines, then press Split into chunks.
- Review the chunks and the token counts, then copy the JSON or download it as .jsonl or .json to feed your embedding or indexing step.
Read the guideHow to split documents into chunks for retrieval-augmented generation: choosing size and overlap, paragraph versus sentence splitting, token limits and common mistakes.
How to chunk text for RAG: sizes, overlap and examplesWhat chunking is and why it matters for RAG
Retrieval-augmented generation (RAG) answers questions by first searching your own documents and then giving the best-matching passages to a language model. Documents are usually too long to search or send whole, so they are cut into chunks. The size and the boundaries of those chunks affect everything that follows: a chunk that is too small loses context, a chunk that is too large mixes topics and wastes tokens, and a cut in the middle of a sentence can hide the very fact you need.
This tool cuts text by tokens, the unit models actually read and charge for, using OpenAI's real tokenizers. Every chunk is guaranteed to be no larger than the size you set, and chunks are packed from whole paragraphs, sentences or lines so that boundaries fall in sensible places.
Choosing a chunk size and overlap
There is no single right answer, and the figures below are common starting points, not rules. Chunks of a few hundred tokens (often 200 to 1,000) suit question answering over documentation, with an overlap of roughly 10 to 20 percent so that an idea at a boundary appears in both neighbouring chunks. Larger chunks keep more context but retrieve less precisely; smaller chunks retrieve precisely but may lack the surrounding explanation. Test with real questions from your users and adjust.
Also check the limits of the embedding model you use, because it can only embed a certain number of tokens at once. For example, OpenAI's text-embedding-3 models list a limit of 8,191 tokens per input at the time of writing; confirm the current figure in your provider's documentation.
How this tool splits text
The text is first divided into units (paragraphs, sentences or lines, as you choose). Units are added to a chunk until the next one would push it over the size. The next chunk then starts with whole units from the end of the previous one, up to the overlap you asked for, so the overlap is rounded down to a whole unit and never splits a sentence. A unit that is bigger than a whole chunk on its own is cut at the largest piece that fits, first at sentence boundaries and then by characters. Japanese, Chinese and other languages written without spaces work too.
Everything runs in your browser. Your document is never uploaded, and the tokenizer is downloaded once, the first time you split. Counts are exact for the OpenAI encodings; other providers use different tokenizers, so treat them as a close guide. The guide on chunking text for RAG explains the trade-offs in more detail, and the token calculator shows how any piece of text is split.
Frequently asked questions
What chunk size should I use for RAG?
A few hundred tokens is a common starting point, often 200 to 1,000, with 10 to 20 percent overlap. The best size depends on your documents, your questions and your embedding model, so test with real queries and compare retrieval quality.
Why use overlap between chunks?
Overlap repeats a little of the end of one chunk at the start of the next, so an idea that sits on the boundary is still found whole in at least one chunk. Too much overlap wastes tokens and storage.
Does it cut in the middle of a sentence?
Not unless a single sentence is larger than a whole chunk. Chunks are built from whole paragraphs, sentences or lines, and only an oversized unit is cut, first at sentence boundaries and finally by characters.
Which tokenizer should I choose?
Choose the one used by the model you will send the chunks to: o200k_base for GPT-4o and newer models, cl100k_base for GPT-4, GPT-3.5 and the text-embedding-3 models. For other providers the count is only a close guide.
Is my document uploaded?
No. The splitting runs in your browser. Only the tokenizer vocabulary is downloaded, once, and your text is never sent anywhere.
What do the JSON and JSONL files contain?
Each chunk has an id, its text and its token count. JSONL has one chunk per line, which is easy to stream into an indexing script; JSON wraps all chunks in one object together with the settings you used.