How Does ChatGPT Answer?From tokens to RAG: a picture tour inside AI

01The AI map: narrow AI, AGI, superintelligence

“AI” covers a lot of ground. A program that spots cats in photos, a program that plays Go and a chatbot that talks back are all AI. Three steps keep it straight.

Where does today’s star, the large language model (LLM), sit on these steps? It writes, summarizes, translates and codes, an astonishingly wide range. In September 2026 industry leaders even started talking about an “AGI era” (our four-year recap). That is a company framing, though, and there is still no agreement on what AGI means. What is well understood is the basic way today’s chatbots produce answers. Let’s look at how an answer gets made.

02LLMs are next-token champions

Type “Good” on your phone and the keyboard suggests “morning”. A large language model (LLM) works from the same root idea. It looks at the text so far and computes which token is most likely to come next. A token can be a whole word, part of a word or a punctuation mark (more in section 3). The difference is scale: it tuned those probabilities by reading more text than a person could finish in many lifetimes.

Before training, a model has not yet learned useful language patterns. Its billions of internal numbers (weights) sit at their initial values, so it cannot predict what follows “Good”. By guessing, being wrong and adjusting over and over across a huge amount of text, the numbers shift little by little until the model has found the pattern on its own: “morning” is the most natural next word.

Each click adds one word, and that word becomes input for the next guess. Writing one piece at a time while rereading what you just wrote is called autoregressive generation. Even a long answer is this one-piece guess repeated hundreds or thousands of times.

The hard part is reading the earlier text well. “I saw a bat in the cave” and “He swung the bat” use the same word for different things. The Transformer, introduced by Google researchers in the 2017 paper “Attention Is All You Need”, solved this with attention, which calculates which tokens in the context to weigh more. The writing side is masked so it cannot see tokens that have not been written yet.^[1] Most widely used LLMs today are built on this design.

03Tokens: the Lego bricks an AI eats

An AI cannot read letters as they are. It first cuts text into small pieces and gives each piece a number. Those pieces are tokens. Think Lego: the AI looks at brick numbers and picks the next brick.

How do you cut? Split at every space (word level) and you are stuck on new slang or typos that are not in the dictionary. Split every character (character level) and you get so many bricks that sentences become endlessly long. Modern models take the middle road: common chunks become one big brick, rare words become several small ones. This is subword tokenization.^[2] “Unbelievably”, for example, becomes “Un” + “bel” + “ievably”.

Even with the same meaning, token counts depend on the language and the tokenizer. Across the five sentence pairs we chose, Korean used about 1.54 times as many tokens as English with the o200k_base encoding, and 2.44 times with cl100k_base.^[3] That is the result for these examples, not a fixed ratio for every Korean sentence. With APIs that charge per token, the difference can show up in cost. For reference, OpenAI says one token is roughly four characters of common English text.^[4]

04Temperature: model-student mode vs poet mode

Ask the same question twice and you may get two different answers. Nothing is broken. The AI does not always take the top candidate token; it rolls weighted dice. Temperature is the knob that sets how boldly it rolls.

Turn it down and selection concentrates on high-probability candidates, so answers vary less. Turn it up and probability spreads out, so more surprising candidates get picked. OpenAI’s API reference sets temperature between 0 and 2 and says higher values like 0.8 make output more random, while lower values like 0.2 make it more focused and deterministic.^[5]

Here is a common misunderstanding: does a lower temperature make answers more accurate? No. Temperature only sets how adventurous the model is with the probabilities it has already assigned; it does not fix those probabilities. Suppose that after “The capital of Australia is”, the model gives “Sydney” the highest probability (the answer is Canberra). The lower the temperature, the more likely Sydney is to be picked. At the lowest setting it says Sydney almost every time. The wrong answer just becomes consistent.

So use it on purpose. Keep it low when you want answers in a steady format; nudge it up for brainstorming and storytelling. Push it too high and sentences can fall apart. Accuracy comes from sources and verification, not from the temperature knob.

05Hallucinations: why does AI answer even when it does not know?

The most famous LLM weakness is hallucination: stating something false as if it were fact. Models have invented paper titles and awards that never existed.

Why? OpenAI offered a striking explanation in September 2025: today’s training and evaluation reward guessing more than honestly saying “I don’t know”.^[6] Think of a multiple-choice exam. Leave a question blank and you get zero; guess and you might get lucky. If a model does not know someone’s birthday and says “September 10”, it has a 1-in-365 chance of being right. Over thousands of questions, the guesser scores higher.^[6]

So the researchers propose fixing the scoreboards: penalize confident errors more and give partial credit for appropriately saying “I don’t know”.^[6] Users have tools too: prompts, below, and RAG in section 9.

Prompts: the rulebook and the order

The instructions you give an AI are prompts. Here we focus on two kinds. The system prompt is the rulebook for the whole conversation: “You are a science teacher explaining to a 9-year-old. If you don’t know, say so.” The user prompt is the order you place inside that rulebook. Change the rulebook and the same order gets a different tone and level of caution.

06Embeddings: GPS coordinates for meaning

How does an AI know that “puppy” and “dog” are close in meaning? Embeddings. They turn a word or sentence into coordinates made of many numbers. Just as nearby cities have similar latitude and longitude, words with similar meanings get nearby coordinates.

You can even add and subtract coordinates. In 2013 Google researchers showed that taking “King”, subtracting “Man” and adding “Woman” lands closest to “Queen”.^[7] Meaning is stored as directions in the space. It does not always work out this neatly.

Real embeddings are not flat like the toy map. Each of the 10,000 words below has a coordinate made of 200 numbers. Squashed from 200 dimensions down to 3, “king” still sits near queen, prince, throne and emperor.

Coordinates are not just for text. CLIP, released by OpenAI in 2021, put photos and captions on the same map.^[9] It learned from a huge number of photo-caption pairs to pull the coordinates of a dog photo close to the coordinates of a caption describing a dog. Now you can search photos with words and pick the right caption for a photo. Putting several kinds of data on one map is called multimodal embedding.

07Cosine similarity: do the arrows point the same way?

How do you measure that two coordinates are “close”? For text embeddings, a widely used measure is cosine similarity. Draw arrows from the origin to the two coordinates and look only at the angle between them. Same direction gives 1, a right angle gives 0, opposite directions give -1.

Cosine similarity compares how similar two vectors’ directions are, rather than their size. That is why moving the length slider leaves the value unchanged. But do not read arrow length as the number of characters or the length of a document. Some embedding models normalize every vector to length 1; OpenAI’s embeddings do, and in that case cosine similarity and straight-line (Euclidean) distance produce identical rankings.^[10] A high cosine score also does not prove two texts mean the same thing. The number only tells you how similar the directions are.

08Vector databases: finding neighbors among millions

Once millions of company documents are coordinates, you need to find the ones closest to a question, fast. Measuring every single one is too slow. Vector databases are built for this job. Options range from Meta’s FAISS library to dedicated databases such as Milvus, Qdrant, Pinecone and Chroma, and search engines like Elasticsearch that added vector search.

The trick is approximate nearest neighbor (ANN) search: find neighbors that are almost certainly the closest. HNSW, one of the most widely used methods, borrows the small-world idea that anyone is a few handshakes from anyone else.^[11] The top floor has only a few points, so you leap across the map like a highway; each floor down is denser, until you walk the side streets to the exact neighbor.

Coordinates take a lot of memory, so vector databases also use quantization, compressing the numbers with slightly lower precision. Tags (metadata) such as date or department matter too: “documents from last year’s marketing team that resemble this question”.

09RAG: letting the AI take an open-book test

What a model learned in training is limited in time and scope. Without fresh material or a search tool, it struggles to answer accurately about a policy that changed yesterday or today’s stock price, yet it may answer as if it knew. That is the hallucination from section 5.

In 2020 Patrick Lewis and colleagues at Meta (then Facebook AI Research) proposed a fix: RAG, retrieval-augmented generation.^[12] It pairs the model’s built-in memory (its parameters) with an outside document store and has the model look up relevant documents before answering. The retrieved documents are not stapled onto a finished answer; they go in with the question as material to answer from. The closed-book test becomes an open-book test.

RAG brings three benefits. New information can be used by updating documents and the search index, without retraining the model; with vector search, changed documents also need fresh embeddings. Retrieved documents can be shown as evidence. And by connecting user permissions to search filters, you can design it so different people see different documents. Tools such as LangChain make it possible to assemble a RAG system without much code.

How small should the pieces be? Chunking

Long documents are often split into chunks, with input limits and search efficiency in mind. Cut too small and it is easy to find the passage that matches the question, but each piece carries too little context to build a good answer. Cut too big and context is rich, but unrelated material dilutes the match and the chunk slides down the rankings. So instead of chopping every N characters, many systems cut along paragraph and sentence boundaries.

Retrieving more chunks is not automatically better either. In 2023 researchers at Stanford and elsewhere reported “lost in the middle”: with long inputs, models make poor use of information placed in the middle, and do relatively better when it sits at the beginning or end.^[13] That is a reason to adjust where key information and documents go, and to measure the effect yourself.

10RAG vs fine-tuning: the chef sent abroad vs the kitchen tablet

There are two main ways to let an AI use your own material. Fine-tuning trains the model’s weights further. RAG leaves the model alone and hands it documents when needed.

Many teams now combine both: fine-tune for house style and report format, and pull exact numbers and current rules in with RAG.

11Smarter RAG: merge two searches, hold interviews, draw a map Deep dive

Hybrid search and RRF

Meaning-based embedding search will find the “lodging limit table” when you ask about “hotel costs”, but it sometimes misses exact strings such as a policy code like “TRV-23” or a rare name. Old-school keyword search (BM25 and friends), which checks for exact matches, is strong there. Running both and merging the results is called hybrid search.

Their scores use different scales, so adding them as-is can let one side dominate. A popular fix is RRF (reciprocal rank fusion), which ignores scores and uses only ranks.^[14] For each document, compute 1 ÷ (k + rank) in each search and add them up; the original paper used k = 60. Documents that rank well in both searches tend to beat ones that top only one.

Rerankers: after the resume screen comes the interview

Once hybrid search has picked 50 candidates for the travel-policy question, a reranker reorders them. The first-stage vector search is a resume screen: it turns the question and each document into coordinates separately and compares them. Fast, but it can miss subtle word-to-word relationships. A cross-encoder used as a reranker is the interviewer: it reads the question and the document together and scores how related they are. The score range depends on the model and its settings.^[15] It is usually more accurate but slower, so it interviews only the shortlist.

GraphRAG: connecting scattered dots

“Where did Steve Jobs’s biological sister get her undergraduate degree?” The answer often is not in any single document: one mentions his sister’s name, another mentions her school. Plain RAG, which pastes in similar chunks, struggles with questions that need two hops like this.

GraphRAG, published by Microsoft Research in 2024, builds its index in two stages. First it extracts entities (people, companies, places) and their relationships from the documents to form a knowledge graph. Then it pre-writes a summary for each group (community) of closely related entities.^[16] At question time it can follow relationships to connect scattered facts and combine the community summaries to reason over the whole dataset. What the paper targets most are questions that a few chunks cannot answer, such as “What are the main themes in this dataset?”^[16] Think “Which rules in the whole policy handbook changed most often in the past year?”

12Is it working? The RAGAS scorecard Deep dive

Once you build RAG you have to grade it, and a wrong answer only helps if you know whether retrieval fetched the wrong documents or generation misread the right ones. RAGAS, released in 2023, uses an LLM as the judge and scores retrieval and generation separately. The paper focused on metrics that work without human-written reference answers.^[20] Today’s library has metrics that use a reference answer and metrics that do not.^[21] Here are four that look at retrieval and answer quality.^[22]

Take the travel-policy question. If the answer says “lodging limit: $110 a night” but no retrieved document contains that number, faithfulness drops. When faithfulness is low, check whether the answer includes claims the retrieved documents do not support, and adjust both how documents are used and the instructions so the model does not assert what its sources do not say. If context precision is low, try a reranker; if recall is low, change the chunking or retrieve more pieces.

High faithfulness and relevancy still do not mean the answer is true in the real world. If the retrieved policy is an outdated version, an answer faithful to it is outdated too.

13So when does superintelligence arrive?

The idea is older than you might think. In 1965 the statistician I. J. Good imagined “a machine that can far surpass all the intellectual activities of any man however clever”. Such a machine could design even better machines, setting off an “intelligence explosion” that would leave human intelligence far behind. He called it the last invention humanity would need to make, with a condition: that the machine is docile enough to tell us how to keep it under control.^[23]

The philosopher Nick Bostrom describes superintelligence as “a general intelligence that vastly outperforms the best human brains in every significant cognitive domain”.^[24] A chess program that beats people in one field does not qualify.

Back to today’s parts. Today’s conversational AI services combine parts such as an LLM that picks the next token, embeddings that turn meaning into coordinates, and RAG that fetches outside documents. Systems built this way already do many tasks faster than people. It still makes things up, retrieves the wrong documents and needs a scorecard to keep it honest. Whether systems like this can reach superintelligence, and if so when, is a question on which experts sharply disagree. This article does not predict a date. Real cases of AI agents crossing the line are collected in our roundup of AI agent control failures.

You can now see where to start when an AI answer looks off. Was fresh material missing? Did it fetch the wrong document? Or did it add something the material never said?

Knowing how it works will not prevent every error. But it does make you less likely to trust an answer just because it sounds smooth. For answers that matter, open the sources and check dates and numbers. No one knows when superintelligence will arrive, but using today’s AI better can start now.

Five questions readers ask next

Does an AI understand a question the way a person does?

Not in the same way. An LLM looks at the text so far, computes the probability of the next token and writes one piece at a time. Along the way it learned patterns that handle meaning well, but it can still say plausible things that are wrong.

Does asking in Korean cost more?

It can with APIs that charge per token, because token counts depend on the language and the tokenizer even when the meaning is the same. In the five example pairs we chose, Korean used about 1.54 times the English count with o200k_base. That is the result for these examples, not a fixed ratio.

Does turning the temperature all the way down stop hallucinations?

No. Temperature only sets how adventurous the model is with the probabilities it has already assigned; it cannot change what is true. At the lowest setting the model picks its top candidate almost every time, and if the model has the fact wrong, that top candidate is the wrong answer, repeated every time. To reduce hallucinations you still need RAG to supply sources, and you still need to check them.

With RAG, is the AI never wrong?

No. It can retrieve the wrong documents, misread the right ones, or rely on documents that are outdated. That is why teams add rerankers and use scorecards such as RAGAS to check retrieval and generation separately.

Is superintelligence here, or about to be?

There is no settled answer. Superintelligence means an intellect that greatly exceeds the best humans in nearly every field. Companies talk about an AGI era, but there is no agreement on what counts as AGI, and experts sharply disagree on whether or when superintelligence could arrive.