Correcting ignorance: learning a bit about Ruby blocks

Gary Bernhardt pointed out at PyCodeConf that I didn't know Ruby even half as well as I should if I wanted to really understand why Ruby programmers rave about blocks so much (I started this before his talk, but it touches on his key point about the centrality of blocks to Ruby's design, and Python's lack of a similarly endemic model for code interleaving). So I set about trying to fix that (at least, to the extent I can in 24 hours or so). Unsurprisingly (since I'm not interested in becoming a Ruby programmer at this point in time), I approached this task more in terms of what it could teach me about Python (and its limitations) rather than in figuring out the full ins and outs of idiomatic Ruby. So feel free to bring it up in the comments if you think I've fundamentally mischaracterised some aspect of Ruby here.

The first distinction: two kinds of function

It turns out the first distinction shows up at quite a fundamental level. Ruby has two kinds of function: named methods and anonymous procedures. The semantics of these are quite different, most notably that named methods create their own local namespace, while anonymous procedures just use the namespace of the method that created them (so they're almost like ordinary local code).

Python also has two kinds of function: ordinary functions and generator functions. The name binding semantics are identical, but the invocation style and semantics are very different. Lambda expressions and generator expressions provide syntax for defining these inside an expression, but under the hood the semantics are still the same as those of the statement versions.

The closest you can get to a Ruby style anonymous procedure in Python is to create a named inner function and declare every otherwise local variable explicitly 'nonlocal' (in Python 3 - nonlocal declarations aren't available in Python 2). Then all name binding operations in the inner scope would also affect those names in the outer scope.

Actually, make that three kinds of function

The named method vs anonymous procedure distinction actually doesn't fully capture Ruby's semantics. Blocks (which is what I was most interested in learning about), add a new set of semantics that don't apply to the full object versions: they not only use the namespace of the defining method for their local variables, but their parameters are pass-by-reference (so they can rebind names in the calling namespace) and their control flow can affect the calling method (i.e. a return from a block will cause the calling method to return, not just the block itself). While somewhat interesting, I don't think these are actually all that significant - the core semantic difference is the one between Ruby's anonymous closures and Python's generators, not the dynamic binding behaviour of blocks.

The implications: blocks versus coroutines

This initial difference in the object model for code execution has created a fundamental difference in the way the two languages approach the problem of interleaving distinct pieces of code. The Ruby way is to define a separate piece of the current function that can be passed to other code and invoked as if it was still inline in its original location, then resuming execution when the called operation is complete. The Python way is to suspend execution, hand control back to the invoking piece of code, and then resume execution of the current code block at a later time (as determined by the invoking code).

Hence, where Ruby has specific syntactic sugar for passing a block of code to another method (do-end), Python instead has syntactic sugar for various invocation styles for coroutines (iteration via for loops, transactional code via with statements).

It's also the case that coroutines are not (yet) as deeply bound into Python's semantics as blocks are into Ruby. Whereas Ruby had blocks from the beginning and defined key programming constructs in terms of them (such as iteration and transactional style code via blocks), Python instead is built around various task specific protocols that may *optionally* be implemented in terms of coroutines (e.g. for loops, the iterator protocol and generators, the with statement, the context manager protocol and the contextlib.contextmanager decorator applied to a generator).

Callback programming and hidden control flow

One interesting outgrowth of the Ruby approach is that callback programming actually becomes a fairly natural extension of the way the language works - since programming with blocks is callback style programming, the invoking code doesn't really care if the called method runs the passed in block immediately or at some later time. Whether you consider this a good thing or a bad thing is going to depend on how you feel about the merits and dangers of hidden control flow.

During the discussions that led to the introduction of the with statement in Python 2.5, Guido made a clear, conscious design decision: he wanted the possible flows of control through the function body to be visible locally inside a function, without being dependent on the definitions of other methods (raising exceptions, of course, being an exception - catching them, though, largely obeys this guideline). Most code is run immediately, code in if statements and exception handlers is run zero or one times, code in loops is run zero or more times, code in nested function definitions is executed at some later time when the function is called. The Ruby blocks design is the antithesis of this: your control flow is entirely dependent on the methods you call. The downside of wanting visible control flow, of course, is that iteration, transactional code and callback programming all end up looking different at the point of invocation. (If you read PEP 340, Guido's original proposal for what eventually became the with statement, and contrast it with PEP 343, the version that we finally implemented, you'll see that his original idea was a fair bit closer to Ruby's blocks in power and scope).

So Ruby's flexibility comes at a price: when you pass a block to a method, you need to know what that method does in order to know how it affects your local control flow. Naming conventions can help reduce that complexity (such as the .each convention for iteration), but it does move control flow into the domain of programming conventions rather than the language definition.

On the other hand, Python's choice of explicit control flow comes at a price in flexibility: callback programming looks starkly different to ordinary programming as you have to construct explicit closures in order to pass chunks of code around.

Two way data flow

With their functional API, blocks natively supported two-way data flow from the beginning: data was passed in by calling them, and then either returned as the result of the block or by manipulating the passed in name bindings.

By contrast, Python's generators were originally output only, reflecting their target use case of iteration. You could input some initial data via parameters, but couldn't readily supply data to a running calculation. This has started to change in recent years, as generators now provide send() and throw() methods to pass data back in, and yield became an expression in order to provide access to the 'send()' argument. However, these features do not, at this stage, have deep syntactic support - there's a fairly obvious mapping from continue to send() and break to throw() that would tie them into the for loop syntax, but this capability has not garnered significant support when it has been brought up (I believe because it doesn't really help with the last major code execution model that Python doesn't provide nice native support for: callback programming).

In Python 3.3, generators will gain the ability to return values, and better syntax for invoking them and getting that value, moving the language even further towards full coroutine support (see PEP 380 for details). However, that is merely the next step along the path rather than arrival at the destination.

Reinventing blocks

I think the folks who accuse us of (slowly) reinventing blocks have a valid point - Python really is on the path of devising ways to handle tasks neatly with coroutines (i.e. functions that can be suspended and transparently resumed later without losing any internal state) that Ruby handles via blocks (i.e. extracting arbitrary fragments of a function body and passing them to other code). The fact that generators were not built into Python from the outset but instead have been added later to make certain kinds of code easier to write does show through in a variety of ways - coroutine based code often doesn't play nicely with ordinary imperative code and vice-versa.

Ruby's way has a definite elegance to it (despite the hidden control flow). I think aspiring to that kind of elegance for callback programming in Python would be a good thing, even if the semantic model is completely different (i.e. coroutine based rather than block based). The addition of actual block functionality remains unlikely, however - if they were as powerful as Ruby blocks, then it would create two ways to do too many things (with no obvious criteria to choose between the current technique and the block based technique), but if they were strictly less powerful, then reusing Ruby's block terminology would likely be confusing rather than enlightening. For better or for worse, Python is now well down the path of coroutine based programming and we likely need to see how far we can take that model rather than trying to shoehorn in yet another approach.