Part two of ten. Most of the surprises people hit with AI at work trace back to this one chapter. A model reads in tokens, holds only so many at once, and charges for each one.
1. What is a token?
A token is the unit of text a model reads and writes. It can be a whole word, part of a word, a chunk of a number or a punctuation mark. For common English text, a rough rule is about four characters per token.
Why it matters. Tokens are how models are measured and billed. Context limits, prices and speed are all counted in tokens. Code and SQL use more tokens than you’d guess, because a name like SalesOrderDetailID splits into several pieces.
Watch out. The four-characters rule is a rough guide for English. Other languages, numbers and code can run much higher. When the number matters, use the vendor’s own token counter.
2. Why does AI struggle to count letters?
Because a token isn’t a word. A short, common word is usually one token, while a long or rare word splits into several. The model sees a few tokens, not ten letters, which is why it can miscount the letters in strawberry.
Why it matters. The same blind spot shows up in data work. Character counts, string lengths and exact spelling checks are weak spots. Let SQL or a script do the counting, and let the model explain the result.
The follow-up. “Why does it get simple arithmetic wrong sometimes?” Numbers are split into tokens too, and the model predicts digits rather than calculating them.
3. What is a context window?
The most text a model can consider at once, measured in tokens. It holds the instructions, the conversation so far, any files you attached and the answer being written. Anything outside the window doesn’t exist for the model.
What the interviewer wants. That on most models the answer counts against the same window. A long input can leave too little room for the reply.
Watch out. Window sizes change with every model release, so don’t memorise a number. Say how you’d find it: the model’s documentation lists it.
4. Why does AI lose track of a long file?
Fitting in the window doesn’t mean every detail gets used. Models can miss details in the middle of a long input, even when all of it fits. If the file doesn’t fit at all, the tool cuts, summarises or picks parts, and it doesn’t always say which.
Why it matters. This is how you get a fix that uses a variable the AI never saw declared. It’s also how a summary skips the clause on page 40.
Watch out. “Use a tool with a bigger window” is half an answer. A bigger window holds more, and it doesn’t decide what matters. Send the part that matters plus the definitions it depends on.
5. What do temperature and top-p do?
Both control how the model picks the next token. Temperature sets how adventurous the choice is. Top-p keeps only the likeliest options whose chances add up to a set share, such as 90 percent.
Why it matters. For SQL, summaries and data extraction, you want low temperature and repeatable answers. For brainstorming names or test data, a higher setting gives more variety.
What the interviewer wants. That you match the setting to the job, and that you don’t promise identical output. Even at zero temperature, many services don’t guarantee the same answer twice.
Five minutes, any AI assistant
Ask for the same query twice. First with nothing:
Write a SQL query for our top customers.
Then in a fresh chat, with the window filled properly:
Write a SQL query for SQL Server. Table: Sales.Orders (CustomerID, OrderDate, Amount). Return the 10 customers with the highest total Amount in 2024, with their totals.
Count the guesses in the first answer: table names, column names, the date range and what “top” even means. That gap is the whole lesson about context, and it took you a minute.
Why this is worth a book
Chapter 2 has ten questions. It also covers system prompts, what happens when a conversation outgrows the window, zero-shot against few-shot prompting, prompt caching and why longer prompts cost more.
The Senior question in that chapter is the one worth rehearsing: how would you give a model a large schema or codebase without blowing the window?
100 AI Interview Questions and Answers for Data Professionals has 184 key terms alongside the hundred questions. It’s in Kindle (also on Amazon.in), paperback and audiobook.
Part three is about the square on the grid where the money goes: wrong, and sounding sure.
Out of the window is not out of scope, it is out of mind.
Published by Pinal Dave on SQLAuthority. More of my work at pinaldave.com.
Discover more from SQL Authority with Pinal Dave
Subscribe to get the latest posts sent to your email.
