To Infinity And Beyond

Working With Infinite Iterators In Python

The itertools module in Python has built-in support for infinite sequences, and it's one of those things most people overlook until they need it. The standard approach is to combine itertools.count(), itertools.cycle(), and itertools.repeat() with slicing or filtering to create bounded views of unbounded data. This pattern shows up constantly when you're dealing with batch processing, simulation loops, or any situation where you want to generate values on demand rather than pre-computing everything into memory. I ran into a concrete problem last year that forced me to think about this more carefully. I was working on an ETL pipeline that needed to process records in groups of 500, cycling through a list of 12 worker handlers. Naively, I created a full expanded list by repeating the workers list thousands of times, which ballooned memory usage to around 800 MB for a job that should have stayed under 50 MB. The fix was simple but not obvious to someone coming from a different language: I used itertools.cycle(workers) paired with itertools.islice() to grab exactly 500 items at a time. Memory dropped to roughly 12 MB and the runtime went from about 4 minutes down to 47 seconds because the interpreter wasn't spending cycles managing a huge list object. Here's what the basic pattern looks like in practice:

from itertools import count, cycle, islice
workers = ['handler_a', 'handler_b', 'handler_c']
for batch_start in count(0, 500):
  batch = list(islice(cycle(workers), 500))
  process batch
  if some_condition: break The key insight is that cycle() doesn't actually store an infinite list in memory. It maintains a single iterator over your original sequence and yields values from it repeatedly. islice() consumes exactly the number of items you request before stopping. Together they give you lazy evaluation without any upfront allocation. There's a common trap with this approach that I wish I'd noticed earlier. If you call list(cycle(some_list)) without an islice bound, you will hang the process indefinitely and consume memory linearly. I've seen this happen in production because someone wrote a helper function that accidentally materialized the entire cycle. The workaround is to never pass a bare cycle() object to list() or any function that forces eager evaluation. Always wrap it with islice() first and make that a hard rule in your codebase.

Another thing people get wrong is assuming count() is the right tool for every infinite sequence. It only works for integer steps. If you need floating-point increments or a non-linear progression, you have to build a custom generator. I wrote one recently that produced Fibonacci-indexed batch sizes because the downstream system had variable capacity constraints that grew exponentially. Here's the shape of it: def fib_batches():
  a, b = 1, 1
  while True:
    yield a
    a, b = b, a + b This generated batch sizes of 1, 1, 2, 3, 5, 8, 13 and so on. Paired with the same islice pattern above, it let me start small and scale up dynamically. The downstream system could handle the initial tiny batches without choking, then grow into larger throughput as the queue stabilized.

The limitation nobody talks about is that these patterns don't compose cleanly when you need parallel execution. If you split the work across multiple processes, each process needs its own cycle() instance starting at a different offset, otherwise they'll all grab the same worker. I solved this by assigning each process an initial offset based on process_id % num_workers, then cycling from there. It's a small detail that causes subtle bugs if you miss it—multiple processes competing for the same resources creates contention that looks like a race condition but is actually just uncoordinated iterator state. For most use cases, the standard library is sufficient. You don't need third-party packages like boltons or toolz unless you're doing something particularly unusual. The built-in itertools is implemented in C and runs faster than any pure-Python equivalent you'd write. The documentation for itertools is sparse on the infinite-sequence patterns though, so you end up learning this stuff through trial and error or by reading other people's code on GitHub. I don't have a download link to share because this isn't a standalone tool or library. It's a pattern. If you want to see it in action, I'd recommend writing a small script that processes a CSV in infinite cycling batches and watching the memory usage with psutil. You'll see the difference immediately between the naive approach and the lazy one.