https://blog-dev.simpleenough.net/tags/chatgpt/Recent content in ChatGPT on Simple Enough BlogHugoenTue, 01 Apr 2025 17:52:00 +0100https://blog-dev.simpleenough.net/blog/token/Tue, 01 Apr 2025 17:52:00 +0100https://blog-dev.simpleenough.net/blog/token/<h2 id="i-what-is-a-token" class="heading">I. What Is a Token?<a href="#i-what-is-a-token" aria-labelledby="i-what-is-a-token"> <!-- <i class="fas fa-link anchor"></i> --> <svg class="svg-inline--fa fas fa-link anchor" fill="currentColor" aria-hidden="true" role="img" viewBox="0 0 640 512"><use href="#fas-link"></use></svg>&nbsp; </a> </h2> <p>A <strong>token</strong> is a unit of text that the model processes. It could be a full word, part of a word, or even a special character.</p> <h3 id="concrete-examples" class="heading">Concrete Examples<a href="#concrete-examples" aria-labelledby="concrete-examples"> <!-- <i class="fas fa-link anchor"></i> --> <svg class="svg-inline--fa fas fa-link anchor" fill="currentColor" aria-hidden="true" role="img" viewBox="0 0 640 512"><use href="#fas-link"></use></svg>&nbsp; </a> </h3> <table class="table"> <thead> <tr> <th >Text</th> <th >Number of Tokens</th> </tr> </thead> <tbody> <tr> <td >Hello</td> <td >1</td> </tr> <tr> <td >I am a developer</td> <td >4</td> </tr> <tr> <td >Artificial intelligence is fascinating!</td> <td >5</td> </tr> <tr> <td >GPT is a powerful model.</td> <td >6</td> </tr> </tbody> </table> <h3 id="specifics-of-tokenization" class="heading">Specifics of Tokenization<a href="#specifics-of-tokenization" aria-labelledby="specifics-of-tokenization"> <!-- <i class="fas fa-link anchor"></i> --> <svg class="svg-inline--fa fas fa-link anchor" fill="currentColor" aria-hidden="true" role="img" viewBox="0 0 640 512"><use href="#fas-link"></use></svg>&nbsp; </a> </h3> <ul> <li>In English, short words are often 1 token (e.g., <code>&quot;Hello&quot;</code> = 1 token).</li> <li>In French and other languages, longer words can be split into multiple tokens (e.g., <code>&quot;développeur&quot;</code> or <code>&quot;intelligence&quot;</code> = 2 tokens).</li> <li>Punctuation also counts as tokens.</li> <li>Spaces are included with the following word.</li> <li>Acronyms are usually treated as 1 token.</li> </ul> <p>On average, 100 tokens correspond to roughly 75 words, though this can vary depending on the language and writing style.</p>