The SQL Was Never the Hard Part

Text to SQL works now. It works, on real schemas, with joins and date logic and window functions. I’ve watched three companies hand it to their business users. All three quietly stopped using it within a few months.

The technology was never the obstacle. We spent twenty years believing it was.

The Promise Arrived Before the Machine

Ask your data a question in plain English. That sentence has been sold to executives since the nineties. Every reporting product shipped a version of it, and every version was quietly retired.

Each time it failed, we blamed the ears. The machine couldn’t understand the question, which was fair, because it couldn’t.

So the promise sat on a shelf for two decades, waiting for hardware that could keep it. That excuse has now expired. Ask for the top ten customers by revenue last quarter, excluding internal accounts, and you get correct, well written SQL.

We finally ran the experiment we’d been unable to run. And we found out the question was never the hard part.

The Map and the Land

A database is a map of a business. It was drawn on a particular Tuesday, by people in a hurry. The business has been moving ever since.

Think of mapmakers who keep adding detail. Every year they add the new road, the renamed village, the bridge that washed out. They never remove anything, because somebody could still be using it.

Your database has a Customers table. It also has CustomersNew, Customer_Master, a view called vw_Customers_Current, and Customers_Backup_2019 that somebody swears one integration still reads.

You know which one is real. You were there, or somebody told you. Or you queried the wrong one once and spent a day explaining the numbers.

The machine has the whole map and has never walked the land. It picks the most convincing name. That is how it lands on Customer_Master, because whoever built the real replacement couldn’t use the good name. It was taken.

So the SQL is perfect and the answer is wrong. It’s wrong in the worst way available, which is plausibly.

The Word That Means Five Things

This is the one that ends these projects.

Ask five people in a company what revenue means and you’ll get five answers. Gross or net. Before or after refunds. Booked when the order is placed, when it ships, or when the money lands.

Finance has an answer. Sales has a different one. The board deck uses a third. All three are defensible, and all three are used daily by people who have never discovered that the others exist.

A business user types “show me revenue by region.” The machine has to choose. It chooses reasonably, and it doesn’t mention that it chose, because it doesn’t feel the ambiguity as ambiguity.

Your analyst, given the same question, asks one back. Revenue how, the finance way or the sales way? That question is the entire job. It has never appeared on anyone’s job description.

Ninety Five Percent Is Not a Number You Can Use

Here’s the pattern I watched three times.

Week one, everyone is delighted. Real questions, real answers, genuine excitement in the room.

Week three, someone gets a number that looks off. An analyst checks it. The machine used the wrong table.

Week four, everyone is back to asking the analyst. They now know they’ll have to ask the analyst to verify anyway. Asking twice is slower than asking once.

That’s the whole failure, and notice what it isn’t. It isn’t accuracy in aggregate. It’s the impossibility of knowing which single answer to trust.

A tool that’s right ninety five percent of the time isn’t ninety five percent useful. Not when you can’t tell which five percent. For anything that matters it’s close to nothing, because every answer needs the same checking the old way needed.

What the Ones That Worked Did

The companies getting value from this aren’t pointing it at the raw database.

They drew an honest map first. A curated set of views and defined metrics, where revenue has exactly one meaning, agreed with finance, written down. That’s the only revenue the tool can reach, so the ambiguity is settled before the question is asked.

That isn’t an AI project. It’s the data modeling work companies have postponed for a decade. AI made the postponement expensive enough to notice.

They aimed it at analysts, not executives. The tool is worth most to people who can read what it writes. An analyst gets a correct first draft and adjusts it. A business user gets a number they can’t check.

They made it show its work. Every answer displays the SQL and names the tables it touched. That sounds like it would scare people off. In practice it’s the only thing that catches the wrong table before the number reaches a slide.

The Four Fifths Nobody Wrote Down

We assumed the analyst’s job was writing SQL, because writing SQL is the part you can watch someone do.

It was never the SQL. The SQL was maybe a fifth of it.

The rest was knowing which table is real. Knowing which definition the asker meant. Knowing the numbers before the system migration aren’t comparable. Knowing that when the head of sales says customers, they mean accounts, not contacts.

None of that is written down anywhere. It lives in people. It transfers by sitting near someone for two years, which is an apprenticeship, and nobody ever called it that.

Here’s the part I find strange. We automated the visible fifth, and the automation was flawless. That made the invisible four fifths appear for the first time. They were always holding the building up, and now they stand in plain view, entirely undocumented.

That’s the opportunity here, and it has almost nothing to do with AI. Write the four fifths down, starting with what revenue means. Delete the tables that lie. Name the real one clearly enough that a stranger could find it.

Do that, and the machine becomes useful. Skip it, and you’ve bought a fast way to be confidently wrong.

I wrote about the part of this that lasts in Prompting Is Not the Skill. It’s an essay about why the tricks evaporate and the judgment compounds. It’s one of thirty essays in my book AI: Nobody’s in There. But we’re still in here. All of them are free to read online, and the paperback is on Amazon.

If your map needs redrawing before you point a machine at it, that’s work I do with teams. You can see how it goes on my consulting page.

Text to SQL is not a translation problem, it is a memory problem, and the memory was never written down.

Published by Pinal Dave on SQLAuthority. More of my work at pinaldave.com.