Python Concurrency and Async for Interviews

Concurrency is dealing with many things at once (structure); parallelism is doing many at once (hardware). Classify the work first, bound it with a semaphore sized from arithmetic, and test on invariants recorded during the run.

The GIL

• GIL (global interpreter lock): one lock a thread must hold to run Python bytecode, so exactly one thread runs Python code at any instant, regardless of core count.
• A thread releases the GIL while waiting on I/O, so I/O waits overlap. It holds the GIL while computing, so CPU work does not overlap.
• I/O-bound = limited by waiting on network/disk/database. CPU-bound = limited by the processor doing arithmetic. First question of any workload.
• Threads on CPU-bound work: same total runtime as serial plus context-switch overhead, so measurably worse.
• C libraries (NumPy) can release the GIL themselves. The GIL is a CPython detail, not the language.

Choosing a model

• Threads are preemptive: interpreter can switch mid-bytecode, even inside counter += 1. Concurrency for free, races for free.
• Async is cooperative: a coroutine runs until it hits await, then may yield. Classic shared-state race mostly vanishes; new failure is blocking the loop.

asyncio mechanics

• Coroutine: what async def produces; calling it returns an object, does not run it.
• await: “suspend here, let something else run”; the only place a coroutine yields.
• Event loop: single-threaded scheduler over a queue of ready coroutines.
• Task: a coroutine handed to the loop to run concurrently via asyncio.create_task(...).
• await coro = run now and wait. create_task(coro) = start now, collect later.
• Concurrency = create all awaitables first, then await the collection.
• asyncio.gather(*coros) = run all concurrently, results in argument order. By default first exception propagates while others keep running; return_exceptions=True returns exceptions as list items in position. gather starts everything at once.

SEQUENTIAL  await each as created:  A|B|C  = 300 ms
CONCURRENT  create all, then await: A,B,C  = 100 ms
same work, one thread, only the waiting overlapped

Primitives, by the bug each prevents

• counter += 1 is read, add, write; a thread switch between read and write loses an update. Race is permitted, not guaranteed, and usually passes when you run it.
• Semaphore(1) is a lock. In async, in_flight[0] += 1 needs no lock (no await between read and write); in threads the same line is a race.
• Bounded queue = backpressure: await q.put() suspends the producer when full, so memory stays bounded. A semaphore limits what runs; a bounded queue limits what waits. Use one sentinel (None) per consumer to signal shutdown.

The four bugs every fetch-many ships

1. Coroutine never awaited: record(...) instead of await record(...); creates object, does nothing, only a RuntimeWarning.
2. await inside the loop: sequential code in async syntax; correct output, no test catches it, only a stopwatch. Fix: gather all at once.
3. Lost order or lost failures: as_completed yields in completion order, not input order; bare gather raises on first error and discards the rest. Fix: gather(..., return_exceptions=True), then split ok/failed.
4. Blocking the loop: one sync call (time.sleep, requests.get) freezes everything. Symptom: slower with more concurrency, one core pegged at 100%. Fix: await asyncio.sleep, async-native client, or loop.run_in_executor(None, fn, ...) (pass a ProcessPoolExecutor for CPU work).

Sizing with Little’s Law

L = λ · W
L = requests in flight (pool size you solve for)
λ = throughput, requests/sec (QPS)   W = latency, in SECONDS

• Round any fractional answer up. W in seconds: 200 ms → 0.2.
• Backwards: max throughput = L / W is the ceiling a pool imposes.
• Clamp to the provider’s rate limit: the smaller of your ceiling and their limit wins. Make the limit a config value, not a constant, because it moves when latency moves.

Testing and gotchas

• Assert on the invariant during the run, not the outcome. Track peak in-flight and assert peak <= limit; deleting the semaphore fails it deterministically while an output test still passes. An assertion that can’t fail (unsafe <= max) is not a test.
• Make the schedule deterministic: a fake service with latency and fail_first as inputs. Capture per-call state (n = self.calls) before the await, since other coroutines run during the await.
• Backoff sleeps OUTSIDE async with sem: holding a permit while sleeping shrinks the pool exactly when the service struggles. peak and calls are unchanged either way; only wall-clock catches it (~2.7× slower inside).
• Exponential backoff = base_delay × 2**attempt, spreads retries to avoid a thundering herd. Add jitter (random offset) in production. Transient (ConnectionError) → retry; permanent (ValueError 400) → return immediately.
• Contract of the capstone: one result per key in input order, failures returned not raised, never more than limit in flight, retries only on transient errors.

Agent harness mapping

• Parallel tool calls: one response can carry several tool_use blocks (I/O-bound → asyncio). Run with gather(..., return_exceptions=True), zip results back to calls, return all tool_result blocks in one user message in call order. A failed tool becomes is_error: true, not a raised exception.
• Bounded model fan-out: Semaphore sized to the provider’s rate limit, not a CPU count (tasks hold no CPU while waiting). Continuous briefs → use the Queue with long-lived consumers.