Fun with generators
I’ll admit to being a little scared of Python generators when I first became aware of their existence. I was fairly sure that I didn’t understand them properly, and I failed to see how useful they really are. A few things have opened my eyes in recent months. First, Wesley Chun’s Core Python Programming helped to clarify a couple of issues, and then there was a session at SPA2008 which generated one or two “Aha!” moments, despite ultimately making my head spin.
Most recently, a few folk have blogged about David Beazley’s PyCon ‘08 presentation, Generator Tricks for Systems Programmers. I concur with the person who said it was one of the best introductions he’d come across. I particularly like the approach David’s taken, of avoiding trivial Fibonacci sequence examples and jumping straight into real problems of the kind faced by sysadmins. The performance stats he’s gathered are striking; I hadn’t realised just how competitive writing generator-based scripts for log file analysis can be with non-generator approaches or the more traditional tools used for this purpose, such as awk.
All this has made me more determined to make much better use of generators in my day-to-day Python programming.
April 24th, 2008 at 4:11 pm
One great use for generators is to save memory. I had a program once that would grind to a halt on older machines. Couldn’t figure it out at first. Turned out I was using an iterator expression to build an ENORMOUS list, and then run through it.
Simply changed it to a generator expression, and suddenly it ran without eating all the machine’s RAM. There was no point in building the list first, I just needed to iterate over the elements.
April 24th, 2008 at 7:46 pm
Good point. Memory use is thing that’s always come top of my list when thinking of the potential benefits of generators. David Beazley’s post suggested other benefits that I hadn’t really grokked - like the fact that a ‘pipeline’ approach leads to cleaner, much more readable code.
April 25th, 2008 at 4:39 am
I had the same first impression of generators! I had a similar ‘aha’ moment with generators in the last month or so. I wanted to extend feedparser so that it would follow paging feeds. My first implementation was a bit cludgy. Then I took a couple hours to actually read about generators. I ended up with much cleaner code, fewer lines, much less memory consumption, and a much more ‘pythonic’ interface to my pagingfeedparser. something like
for page in pagingfeedparser.parse():