Posts/2011/12

New Year Python Meme - December 2011

I'm normally a curmudgeon about this kind of thing, but I enjoyed reading some of the other posts in this series Tarek kicked off, so I decided to make my own contribution.

1. What's the coolest Python application, framework or library you have discovered in 2011?
The move to Red Hat marked my entry into the world of web development (previously I'd merely been in interested observer of that world, rather than a participant). By far my favourite discovery since making that change is django-rest-framework - with that, I can use my web browser to browse early iterations of my server's REST API directly, without needing to write custom clients to process the JSON data from APIs that are still in a state of flux.

As a service, ReadTheDocs has been an absolute revelation - between that, code hosting & issue management sites like BitBucket and GitHub and of course PyPI itself, it's now possible for an open source project to have a quite respectable web presence without the developers needing to understand anything more than Sphinx, source control and the project they're working on.

2. What new programming technique did you learn in 2011?
REST would be the big one. I'd had some general exposure to the concept in the past, but there's no substitute for sitting down and building it into a product when it comes to understanding a programming or API design technique.

3. What's the name of the open source project you contributed the most in 2011? What did you do?
CPython, by far - kibbitzing on python-dev and python-ideas (and import-sig too these days), writing and reviewing several different PEPs, documentation updates, code reviews and patch applications, as well as working on my own things (including the still-in-progress integration work for the 'yield from' expression that's coming in 3.3).

I also recently started up 4 separate open source projects - 3 PyPI modules to hopefully address deficiencies I see in the current standard library offerings, plus the upstream open source project for my current development efforts at Red Hat:
  • contextlib2 (ContextStack has some potential as a new building block)
  • WalkDir (the idea here is to be the "itertools for os.walk()")
  • Shell Command (let Python handle control flow, the shell actual commands)
  • PulpDist (Bringing a semblance of order to small-scale rsync mirror networks)

4. What was the Python blog or website you read the most in 2011?
Planet Python.

5. What are the three top things you want to learn in 2012?
From a work point of view, getting my RHCSA (Red Hat Certified System Administrator) is at the top of the list. Coming up to speed on AMQP (Advanced Message Queuing Protocol) is a close second. Finally, I want to fill in more of the gaps in my very sketchy knowledge of web UI development (i.e. HTML/CSS/Javascript).

6. What are the top software, app or lib you wish someone would write in 2012?
I want to see the __preview__ namespace (in particular, the regex module) make it into Python 3.3. But that requires a volunteer to step up and write the PEP, write the code and generally champion the idea (if we have to wait for me to do it, there's no way it will happen before 3.4).

Want to do your own list? here's how:
  1. copy-paste the questions and answer to them in your blog
  2. tweet it with the #2012pythonmeme hashtag

Help improve the Python 3.3 Standard Library...

... and hopefully help yourself with current programming projects, too.

Some recent programming activities left me underwhelmed by a few of the standard library's included batteries. This has already led to a significant revamp of the subprocess module documentation to steer new users away from the Popen Swiss army knife (unless they really need it) and to explain the commonly needed parameters more clearly. It still needs work (the notes and warnings are far too repetitive), but it at least introduces things in the right order now (high level convenience API that most people want first, lower level Popen API that some people need second).

However, for 3.3 I'd like to improve things even more in at least three areas: invocation of the system shell for administration tasks, better tools for traversing filesystem directories and programmatic management of deterministic resource cleanup (i.e. not relying on the garbage collector).

Accordingly, I have 3 projects up on PyPI (with docs on ReadTheDocs and source control and issue tracking on BitBucket):

  • WalkDir: os.walk() style iterators with file and directory filtering (both inclusion and exclusion), depth limiting and symlink loop detection, as well as convenience iterators to flatten os.walk() style iterators into a series of paths (either all walked paths, just the directories or just the files). I currently plan to make (at least some of) these part of the shutil module, but exactly what gets added will be based on the feedback I receive on this module and its API design.
  • Shell Command: Convenience APIs that combine subprocess invocation with string interpolation. Interpolated strings are escaped with shlex.quote() by default, with a custom conversion specifier ("!u", for unquoted) used to invoke the standard interpolation process. It also features an experimental API where I'm tinkering with the use of select.select() on subprocess pipes (I'm not sure it achieves a lot over simple blocking IO in its current form, though). The current plan for this API is that it will be added directly to the subprocess module (well, the stable and sensible parts will be, anyway - I still have my doubts about the select.select() experiment)
  • contextlib2: This module basically exists to let me publish and gather feedback on ContextStack, a proposed addition to contextlib for 3.3 that should make it easier to manage deterministic resource cleanup programmatically (i.e. without coupling it as directly to code layout as simple with statements do).

Feedback on any and all of these is appreciated, either here or on the respective issue trackers. It isn't a foregone conclusion that any of these APIs will be added at all, so examples of real world use cases would definitely be helpful.