https://blog-dev.simpleenough.net/tags/openai/Recent content in OpenAI on Simple Enough BlogHugoenWed, 26 Mar 2025 09:05:00 +0100https://blog-dev.simpleenough.net/blog/chunk/Wed, 26 Mar 2025 09:05:00 +0100https://blog-dev.simpleenough.net/blog/chunk/<h2 id="i-chunk-size-why-1000-tokens" class="heading">I. Chunk Size: Why ~1000 Tokens?<a href="#i-chunk-size-why-1000-tokens" aria-labelledby="i-chunk-size-why-1000-tokens"> <!-- <i class="fas fa-link anchor"></i> --> <svg class="svg-inline--fa fas fa-link anchor" fill="currentColor" aria-hidden="true" role="img" viewBox="0 0 640 512"><use href="#fas-link"></use></svg>&nbsp; </a> </h2> <p>The default value of <strong>1000 tokens per chunk</strong> is not arbitrary:</p> <ul> <li>A chunk of this size generally contains <strong>enough information to remain semantically coherent</strong> without being too large.</li> <li>It remains <strong>compatible with the context window</strong> of modern LLMs (4k, 8k, 32k, or even 1M tokens).</li> <li>It helps avoid <strong>diluting meaning</strong> or breaking semantic units.</li> </ul> <p>In some cases, other sizes may be more appropriate:</p>