Concurrency bugs are some of the hardest problems to track down. I keep coming back to the Producer-Consumer problem because it captures exactly what goes wrong when threads operate at different speeds. Let me walk through what it is and how to solve it in Python.
The core issue: one thread produces data, another consumes it, and they share a fixed-size buffer. If the producer runs faster than the consumer, the buffer overflows. If the consumer runs faster, it reads from an empty buffer. Plus, both threads can step on each other if they access the buffer at the same time.
What is the Producer-Consumer Problem?
The problem describes three components working together:
- Bounded Buffer – a fixed-size array that holds items temporarily. The producer adds to it, the consumer removes from it.
- Producer Thread – generates items and places them into the buffer. Stops when the buffer is full.
- Consumer Thread – reads items from the buffer and processes them. Stops when the buffer is empty.
Here is what can go wrong when threads run at different speeds:
- Producer is fast – buffer fills up, producer tries to write past capacity
- Consumer is fast – buffer is empty, consumer tries to read nothing
- Both access buffer at once – race condition corrupts data
How Semaphores Solve It
A semaphore is a counter that threads can increment or decrement safely. Python’s threading.Semaphore gives us three semaphores to address each problem:
- empty – counts free slots. Starts at buffer capacity. Producer acquires it before adding data. Consumer releases it after removing data.
- full – counts occupied slots. Starts at 0. Consumer acquires it before reading. Producer releases it after writing.
- mutex – a lock that only one thread holds at a time. Both threads acquire it before touching the shared buffer.
The key insight is that each semaphore blocks the calling thread if the operation cannot proceed. The producer blocks when empty is 0 (buffer full). The consumer blocks when full is 0 (buffer empty). And mutex ensures only one thread modifies the buffer at any moment.
Python Implementation
Here is a working implementation with a producer generating 20 items and a consumer processing them. The producer is intentionally faster – it sleeps for 1 second per item while the consumer sleeps for 2.5 seconds.
import threading
import time
CAPACITY = 10
buffer = [-1] * CAPACITY
in_index = 0
out_index = 0
mutex = threading.Semaphore()
empty = threading.Semaphore(CAPACITY)
full = threading.Semaphore(0)
class Producer(threading.Thread):
def run(self):
global CAPACITY, buffer, in_index, out_index
global mutex, empty, full
items_produced = 0
counter = 0
while items_produced < 20:
empty.acquire()
mutex.acquire()
counter += 1
buffer[in_index] = counter
in_index = (in_index + 1) % CAPACITY
print("Producer produced:", counter)
mutex.release()
full.release()
time.sleep(1)
items_produced += 1
class Consumer(threading.Thread):
def run(self):
global CAPACITY, buffer, in_index, out_index
global mutex, empty, full
items_consumed = 0
while items_consumed < 20:
full.acquire()
mutex.acquire()
item = buffer[out_index]
out_index = (out_index + 1) % CAPACITY
print("Consumer consumed item:", item)
mutex.release()
empty.release()
time.sleep(2.5)
items_consumed += 1
producer = Producer()
consumer = Consumer()
consumer.start()
producer.start()
producer.join()
consumer.join()
The output shows the producer running ahead until the buffer fills up. Once the buffer hits capacity, the producer blocks on empty.acquire(). The consumer slowly drains the buffer, releasing empty slots, which unblocks the producer. This back-and-forth continues until all 20 items are produced and consumed.
Producer produced: 1
Consumer consumed item: 1
Producer produced: 2
Producer produced: 3
Consumer consumed item: 2
Producer produced: 4
Producer produced: 5
Consumer consumed item: 3
Producer produced: 6
Producer produced: 7
Producer produced: 8
Consumer consumed item: 4
...
Producer produced: 20
Consumer consumed item: 20
FAQ
Q: Why do we start the consumer before the producer?
Starting the consumer first ensures it is waiting on full.acquire() when the producer starts. If the producer starts first on a cold run, it might fill the buffer before the consumer is ready. Starting the consumer first is a common pattern, though not strictly required.
Q: Can this be solved without semaphores?
Yes. Python’s queue.Queue handles all of this internally. It manages a bounded buffer with built-in locking and blocking behavior. The equivalent using queue.Queue is much shorter – you call put() to add and get() to remove, both of which block automatically.
Q: What happens if the producer finishes before the consumer?
The producer calls full.release() after each item. The consumer calls full.acquire() and blocks only when no items are available. Once all 20 items are produced and consumed, the consumer exits its loop and the threads terminate cleanly via join().
The Producer-Consumer pattern shows up everywhere – printing documents from multiple apps, downloading files while processing them, a web server handling requests with a worker pool. Once you see the pattern, you notice it everywhere.
