LLM Token Counter & Chunker API

Count tokens with the exact OpenAI encodings (o200k_base, cl100k_base), estimate other models, cut text to N tokens and split it into RAG chunks.

GETPOSThttps://apibox.io.vn/v1/llm-tokens

Text▶ Try it online

Counts tokens with the real OpenAI tokenizers: o200k_base (GPT-4o, GPT-4.1, GPT-5, o1 / o3 / o4) and cl100k_base (GPT-4, GPT-3.5, text-embedding-3), using the tiktoken rank tables through gpt-tokenizer. Pass model (for example gpt-4o) to pick the encoding automatically. The count also returns characters, words, tokens per word, an optional cost (price_per_million) and a limit check (max_tokens). For Claude, Gemini, Llama, Mistral, Qwen and DeepSeek, whose tokenizers are not public or not bundled, the response gives clearly labelled estimates with a range, never presented as exact.

action=truncate cuts the text to at most max_tokens tokens (from the start, or keep the end with keep=end) without breaking a character. action=chunk splits long text for retrieval (RAG): it prefers Markdown headings, then paragraphs, lines, sentences (Vietnamese sentence rules such as "TP." or "ThS." are handled), then words, packs pieces up to max_tokens and adds an overlap made of whole sentences. Each chunk has exact start / end character offsets in your text. action=encode / decode converts between text and token ids.

For Vietnamese text the response shows how much more it costs than English (vietnamese_vs_english, measured on the same paragraph in both languages). Special tokens such as <|endoftext|> are counted as plain text unless special=true. Limits: 1 million characters (500,000 for chunking). The text is processed in memory and not stored.

Parameters

Name	In	Type	Required	Default
`text`	body\|query	string	yes		The text (up to 1,000,000 characters). A raw text/plain body also works.
`action`	body\|query	string	no	`count`	`count`, `truncate`, `chunk`, `encode` or `decode`.
`encoding`	body\|query	string	no		`o200k_base`, `cl100k_base` or `all` (default: both for count, o200k_base otherwise).
`model`	body\|query	string	no		OpenAI model name (e.g. `gpt-4o`, `gpt-4`) to choose the encoding.
`max_tokens`	body\|query	number	no		Limit for truncate, chunk size for chunk (default 512), limit check for count.
`overlap`	body\|query	number	no	`50`	Chunk overlap in tokens (whole sentences, at most half of max_tokens).
`keep`	body\|query	string	no	`start`	Truncate: keep the `start` or the `end` of the text.
`price_per_million`	body\|query	number	no		USD per 1M input tokens, to get `estimated_cost_usd`.
`ids`	body\|query	array	no		Token ids for `action=decode` (array or comma-separated).
`special`	body\|query	boolean	no	`false`	Treat `<\|endoftext\|>` and similar as special tokens.
`estimates`	body\|query	boolean	no	`true`	Include the estimates for other model families.

Example

curl -X POST "https://apibox.io.vn/v1/llm-tokens" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text":"Xin chào! Hôm nay trời đẹp, chúng ta đi dạo nhé."}'

const res = await fetch('https://apibox.io.vn/v1/llm-tokens', {
  method: 'POST',
  headers: {
    Authorization: 'Bearer YOUR_API_KEY',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    "text": "Xin chào! Hôm nay trời đẹp, chúng ta đi dạo nhé."
  }),
});
const data = await res.json();
console.log(data);

import requests

res = requests.post(
    "https://apibox.io.vn/v1/llm-tokens",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"text": "Xin chào! Hôm nay trời đẹp, chúng ta đi dạo nhé."},
    timeout=30,
)
print(res.json())

<?php
$ch = curl_init('https://apibox.io.vn/v1/llm-tokens');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST => 'POST',
    CURLOPT_HTTPHEADER => ['Authorization: Bearer YOUR_API_KEY', 'Content-Type: application/json'],
    CURLOPT_POSTFIELDS => '{"text":"Xin chào! Hôm nay trời đẹp, chúng ta đi dạo nhé."}',
]);
$response = curl_exec($ch);
print_r(json_decode($response, true));

Response

{
  "encoding": "o200k_base",
  "tokens": 17,
  "counts": {
    "o200k_base": 17,
    "cl100k_base": 28
  },
  "characters": 48,
  "bytes_utf8": 60,
  "words": 11,
  "language": "vi",
  "tokens_per_word": 1.55,
  "chars_per_token": 2.82,
  "vietnamese_vs_english": {
    "reference": "the same paragraph in English and Vietnamese",
    "english_tokens": 39,
    "vietnamese_tokens": 45,
    "ratio": 1.15,
    "english_tokens_per_word": 1.08,
    "vietnamese_tokens_per_syllable": 1.22,
    "this_text_tokens_per_word": 1.55
  },
  "estimates": [
    {
      "family": "anthropic",
      "models": "Claude",
      "estimated_tokens": 35,
      "range": [
        28,
        42
      ],
      "estimate": true,
      "basis": "cl100k_base × 1.25",
      "note": "Claude tokenizer is not public; use the free Anthropic count_tokens API for exact numbers."
    },
    {
      "family": "google",
      "models": "Gemini, Gemma",
      "estimated_tokens": 16,
      "range": [
        14,
        18
      ],
      "estimate": true,
      "basis": "o200k_base × 0.95",
      "note": "SentencePiece 256k vocabulary; exact counts via the Gemini countTokens API."
    },
    "…"
  ],
  "estimates_note": "Estimates are heuristics (an exact OpenAI count × a factor per script), not real tokenizer output.",
  "source": "gpt-tokenizer 4.1.2 (MIT), OpenAI tiktoken ranks"
}

Use cases

Check that a prompt fits the context window before calling the API
Split documents into token-sized chunks with offsets for a RAG index
Estimate the cost of a batch job in tokens
Count tokens from Apps Script, n8n, Make or PHP where tiktoken is not available

FAQ

Are the counts for Claude and Gemini exact?

No. Their tokenizers are not public, so those numbers are estimates derived from an exact OpenAI count and a factor per script, with a range. Use the vendors’ own count-tokens endpoints when you need exact numbers.

Why does Vietnamese use more tokens than English?

Tokenizers are trained mostly on English, so Vietnamese syllables with diacritics are split into more pieces. With o200k_base the gap is small (about 1.15× on our reference paragraph); older cl100k_base needs far more tokens.

Do chunks cut sentences in half?

Only when a single sentence is longer than max_tokens. Otherwise chunks end at a heading, paragraph, line or sentence boundary, and offsets let you map each chunk back to the source.

LLM Token Counter & Chunker API

Parameters

Example

Response

Use cases

FAQ

Related APIs

Text Diff

Slugify

Text Statistics