A Hard Memory Ceiling for PHP Scripts That Run for Hours

You inherit a legacy table full of auto-incrementing integers and a new architecture that wants UUIDs. You write a CLI worker, build an in-memory lookup array mapping old IDs to new ones, and let it run. Forty thousand rows in, the terminal stops mid-scroll and hands you the line every PHP developer eventually meet one day: Allowed memory size exhausted.

The instinctive fix is to raise memory_limit. The correct fix is to notice that you built an unbounded array and called it a lookup table. And once you’ve fixed that, a second problem is waiting right behind it: this migration runs for hours, across a table nobody trusts, on infrastructure nobody promised wouldn’t reboot. If the fix for memory doesn’t also make the script resumable, you haven’t solved anything: you’ve just delayed the crash until row two million, at which point restarting from zero stops being an inconvenience and starts being your weekend.

One structure fixes both problems, because they’re the same problem wearing two hats: an LRU cache backed by SQLite, doing double duty as a checkpoint log. Let’s see how one can do that.

Why WeakMap doesn’t apply here

PHP 8.0 gave us WeakMap, and it’s tempting to reach for it the moment “memory” and “cache” appear in the same sentence: the garbage collector drops an entry the moment nothing else references its key, no manual eviction required. It’s a real improvement over SplObjectStorage, which has been quietly holding strong references to its keys since PHP 5.3 and leaking memory in exactly the scenario you’d use it to avoid.

But WeakMap keys must be objects. In our case, the lookup table is keyed on database IDs, that is strings or integers, and WeakMap won’t accept them at all. It responds with WeakMap key must be an object, which is basically the exact contrary of the reaction of an array. Even granting yourself the trouble of wrapping every ID in a throwaway object, you’d still be trusting PHP’s generational garbage collector, incrementally improved since 7.4, but never deterministic, to decide when to reclaim memory. You want to keep control. You want a hard cap of exactly $maxItems entries, enforced on every write, not a random moment, sometime after PHP notices the cycle. WeakMap solves a reference-tracking problem. You have a capacity problem. Different tool.

Hot data in RAM, everything else on disk

The actual shape you want is a two-tier cache: a small PHP array holds the most recently touched entries, and SQLite holds all of them, unbounded, on disk. Reads check the array first; on a miss, they pull from SQLite and promote the entry into the array. Writes go to both, immediately.

PHP arrays are ordered by insertion, which makes LRU bookkeeping nearly free: unset() a key and reassign it, and it moves to the end. array_key_first() hands you the eviction candidate without a separate tracking structure. The only work SQLite does is being reliable, fast enough, and, crucially, still there after the process dies.

db = new PDO('sqlite:' . $dbPath);
        $this->db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
        $this->maxMemoryItems = $maxMemoryItems;

        $this->db->exec('
            CREATE TABLE IF NOT EXISTS data_store (
                key TEXT PRIMARY KEY,
                value TEXT,
                created_at INTEGER
            )
        ');

        $this->selectStmt = $this->db->prepare('SELECT value FROM data_store WHERE key = ?');
        $this->insertStmt = $this->db->prepare(
            'INSERT OR REPLACE INTO data_store (key, value, created_at) VALUES (?, ?, ?)'
        );
    }

    public function get(string $key): ?string
    {
        if (array_key_exists($key, $this->memoryCache)) {
            $value = $this->memoryCache[$key];
            unset($this->memoryCache[$key]);
            $this->memoryCache[$key] = $value; // promote to most-recently-used
            return $value;
        }

        $this->selectStmt->execute([$key]);
        $value = $this->selectStmt->fetchColumn();

        if ($value === false) {
            return null;
        }

        $this->remember($key, $value);
        return $value;
    }

    public function set(string $key, string $value): void
    {
        $this->insertStmt->execute([$key, $value, time()]);
        $this->remember($key, $value);
    }

    private function remember(string $key, string $value): void
    {
        unset($this->memoryCache[$key]);
        $this->memoryCache[$key] = $value;

        if (count($this->memoryCache) > $this->maxMemoryItems) {
            unset($this->memoryCache[array_key_first($this->memoryCache)]);
        }
    }
}
?>

It is worth pointing at the remember() extraction, because its absence is exactly where a first draft of this class would broke. set() on an existing key looks harmless, $this->memoryCache[$key] = $value, but PHP does not reorder an array on reassignment to an existing key. It only moves to the end on unset() + insert. If you sSkip that step, you have a structure calling itself an LRU cache that never promotes a re-set key. Eventually, it evicts values based on first-write order instead of last-touch order. Cheap bug, and one you’d only notice under load, exactly when you can least afford to.

The checkpoint you already own

set() writes to SQLite before it writes to the array. That means the database on disk is never more than one call behind the true migration state, which is a checkpoint mechanism, whether you asked for one or not. Use it deliberately: keep a progress table, read last_id on startup, resume from there.

setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

$db->exec('
    CREATE TABLE IF NOT EXISTS progress (
        id INTEGER PRIMARY KEY CHECK (id = 1),
        last_id INTEGER NOT NULL
    )
');
$db->exec('INSERT OR IGNORE INTO progress (id, last_id) VALUES (1, 0)');

$lastProcessedId = (int) $db->query('SELECT last_id FROM progress WHERE id = 1')->fetchColumn();
echo "Resuming from ID {$lastProcessedId}\n";

$fetchBatch = $oldDb->prepare('SELECT * FROM users WHERE id > ? ORDER BY id LIMIT 1000');
$saveCheckpoint = $db->prepare('UPDATE progress SET last_id = ? WHERE id = 1');
$cache = new MemoryCappedSQLiteCache('./migration_state.db');

do {
    $fetchBatch->execute([$lastProcessedId]);
    $rows = $fetchBatch->fetchAll(PDO::FETCH_ASSOC);

    foreach ($rows as $row) {
        $cache->set((string) $row['id'], generateUuid());
        $lastProcessedId = (int) $row['id'];
        $saveCheckpoint->execute([$lastProcessedId]);
    }
} while (count($rows) === 1000);
?>

That INSERT OR IGNORE seeding line is doing more work than its size suggests. An empty progress table with only an UPDATE statement pointed at it is a checkpoint that silently never checkpoints. UPDATE against zero matching rows succeeds, reports nothing wrong, and the resumable script will restart from ID 0. It’s the kind of bug that passes every manual test, because you only ever run it interrupted-and-resumed by accident once, in production, at 2am. I would recommend grepping your own scripts for that exact pattern: any UPDATE ... WHERE used as a substitute for an upsert, on a table, should be reviewed. Yes, this is me giving you homework.

The do...while around the fetch matters too. A single LIMIT 1000 query with no outer loop processes one batch and exits, which quietly turns a resumable migration into a migration you have to manually re-invoke a few thousand times. This is all fine if that’s what your process supervisor is for; but it is worth being honest with yourself about which one you want to built.

The trade off of this approach

Reading through SQLite is slower than reading through a PHP array, for sure. But, what it buys back isn’t just a memory ceiling; it’s the automated checkpoint, because durability and eviction turned out to be the same write. The cost worth naming explicitly: every set() here is its own implicit transaction, so PDO commits to disk on every single row. For a migration touching millions of rows, that’s millions of fsyncs. Wrapping each 1000-row batch in $db->beginTransaction() / $db->commit() buys real throughput. This is the price to pay for the risk of losing up to one batch’s progress on a hard crash mid-transaction, instead of losing at most one row. Decide which of those numbers you can live with before you’re the one explaining the choice in an incident channel.

The bigger picture

Once a structure is durable, resumable, and survives its own process dying mid-write, it is tempting to call it a cache. In fact, a cache is allowed to lose data. In this case, it is deliberately built so it doesn’t and can’t. Every property that makes it safe for a nine-hour migration is a property a database has, not a property a cache has. The eviction policy is the only thing left distinguishing data_store from a table you’d just call your database.

In the end, we have now set up a robust system can both limit the amount of memory it uses to process even more data, and can even restart. This is a feature that rest on the shoulders of the data organisation, rather than PHP. Yet, with generators, and tools that walk datasets rather than process then in one time, it is not difficult to prepare PHP to be reborn after dying. Again, and again, and again…