Split text by characters, words, approximate tokens, Markdown headings or paragraphs, with overlap. Copy the chunks or a JSON array. Nothing is uploaded.
To chunk text for embeddings, paste it, pick what to split by, set a chunk size and an overlap, and copy the chunks or the JSON. Characters cuts every N characters. Words cuts every N words, where a word is any run of characters without a space, so a,b counts as one word. Approx. tokens is an estimate: the size and overlap are multiplied by 4 to get characters, following the rough guide of 4 characters per token for English. Markdown headings starts a new chunk at each heading line up to the level you pick. Paragraphs packs whole paragraphs, separated by blank lines, into chunks of at most the size you set. A paragraph or heading section longer than that is never cut, and a notice says so.
Overlap repeats the end of one chunk at the start of the next, so a sentence on a boundary appears whole in at least one chunk. The next chunk starts at the end of the last one minus the overlap. Worked example: the 26 letters a to z with size 10 and overlap 2 give 3 chunks, abcdefghij, ijklmnopqr and qrstuvwxyz, which start at positions 0, 8 and 16. For words, 7 words with size 3 and overlap 1 give one two three, three four five and five six seven. The overlap must be smaller than the size, otherwise the chunks could never move forward, and the page shows an error instead. Under Where to cut, Keep words whole moves each cut back to the nearest space and starts the next chunk at a whole word. Prefer sentence ends cuts after a full stop, exclamation mark or question mark when one falls in the second half of the chunk. It cannot tell an abbreviation such as Dr. from the end of a sentence.
The first result is the chunks with a separator line between them, which you can change. The second is a JSON array where each chunk has an index counted from 0, its text, start and end positions and its length in characters. The start and end positions are offsets into your text with every line break counted as one character, and the end is not included. A plain emoji counts as 2 characters because browsers store it as two units, an emoji with a skin tone as 4 and a family as 8. A cut is never made inside a visible emoji, so a chunk can run a few characters past the size when one emoji would otherwise be split. The table lists each chunk with its characters and its approximate tokens, which is the character count divided by 4 and rounded up. For an exact OpenAI count of one chunk, use a token counter.
Up to 1,000,000 characters are handled and up to 10,000 chunks are made; above that the page asks for a larger size or less text. The table shows the first 1,000 chunks. In Markdown headings mode a line that starts with one to six # characters and a space is a heading, and lines inside a fenced code block are ignored. Setext headings, where a line is underlined with === or ---, are not recognised. The tool does not read a model's own limit, so check the limit of the embedding model you use. It does not count tokens exactly, remove duplicates or clean markup. Everything runs in this tab.
Choose Characters, Words or Approx. tokens, set the chunk size, for example 300, and an overlap smaller than it, for example 30. Each chunk then starts 30 units before the previous one ended. With size 10 and overlap 2 on the letters a to z you get 3 chunks.
The page uses 4 characters per token, which is OpenAI's rough guide for English. A size of 100 approx. tokens is therefore 400 characters. It is an estimate: code, numbers and other languages use more tokens per character, so check a sample with a token counter.
With an overlap equal to the size, each chunk would start where the last one started and never move forward. Set the overlap below the size, for example 50 for a size of 500.
Choose Markdown headings and a level. H2 starts a new chunk at every # and ## heading, so a file with a # title line directly followed by three ## sections gives 4 chunks. Each chunk keeps its heading line.
Yes. The JSON result has start and end for every chunk, counted from 0 in the text with line breaks as one character each, with end not included. text.slice(start, end) in JavaScript returns the chunk.
No. The text is processed in your browser and not uploaded. Only the option values you pick, such as the mode and size, are saved in this browser, never the text.