A real Python “wat”

November 15, 2015 Django, Pedantics, Python

A few weeks ago I went through and explained the various items in a list of “Python wats” — behaviors of Python which seemingly made no sense. Calling them “wats” is a bit of a stretch in most cases, though, because most of them were simply consequences of fairly reasonable design decisions in how Python or its standard libraries work, but presented in ways which obscured what was actually going on.

Lest I be accused of defending Python too much there, I’d like to point out an absolutely genuine “wat” moment in Python that I helped someone with recently.

On reddit, someone was asking about Django’s length template filter, and specifically its behavior with an undefined variable. The bit of template in question was something like:

{% if some_var|length > 1 %}
    It's more than 1
{% else %}
    It's not more than 1
{% endif %}

When some_var was undefined in the template, the if condition was evaluating True. Huh?

Version requirements

If you want to puzzle this out yourself, then before reading on I’ll mention that I had difficulty reproducing this behavior. I was using Django 1.8 and Python 3.4, and consistently was getting the if evaluating to False as expected.

So in order to figure this out, it’s important to know you’ll only see this behavior on a Django version earlier than 1.8, and only when running on Python 2. Using either of Django 1.8 or Python 3 will remove the “wat”.

Aside on undefined variables

And to avoid spoilers from seeing the answer further down the page, here I’ll talk a bit about why it’s not just automatically an error, since after all some_var is undefined. Shouldn’t this raise a TypeError or some other kind of exception?

The answer to that is Django’s template language is a little bit forgiving of some types of errors. The philosophy here is that we shouldn’t necessarily take down your entire site with a hard HTTP 500 error any time you have a typo in a template variable name, or forgot to provide a variable you were expecting to be able to access, so Django’s template language will silence errors related to accessing/manipulating undefined variables. In older versions of Django, the setting TEMPLATE_STRING_IF_INVALID controlled what Django would output in that case, and it defaulted to the empty string.

In current (1.8) Django, the behavior is a little bit more complex:

Template engines support the configuration parameter string_if_invalid. It still defaults to the empty string.
string_if_invalid can contain the %s formatting marker, and if so it will tell Django to interpolate in the name of the invalid variable (useful for debugging).
When you attempt to apply a filter to an invalid variable, the filter will only be applied if string_if_invalid is the empty string. Otherwise, the filter is not applied.
Except in the case of the if, for and regroup template tags, which always apply filters to their variable argument. An unfiltered invalid variable is treated as None for purposes of those tags.

One other Django 1.8 change: previously, if the length filter was applied to an invalid variable (which, as noted above, only happens if it’s used in certain tags), it returned an empty string, consistent with the TEMPLATE_STRING_IF_INVALID behavior. Starting in Django 1.8, if length gets applied to an invalid variable, it returns 0.

This is verging on “wat” territory all by itself (though, again, tracing back to a reasonable-seeming design decision in terms of when and how templates should raise exceptions), but isn’t the direct cause of the weird behavior shown above. For that, we need to go look at Python.

Going deeper

Now that we understand a bit about how Django handles invalid variables, we can start to pick apart what’s happening. When the Django template engine gets this:

{% if some_var|length > 1 %}

then, in Django versions prior to 1.8, it will get translated (through the invalid-variable behavior) into the following Python:

if '' > 1:

And here the behavior is different depending on Python version. Here’s the Python 3 behavior:

$ python
Python 3.5.0 (default, Sep 26 2015, 18:41:42)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> '' > 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: str() > int()

That’s what we’d expect and hope should happen: strings and integers aren’t and should not be orderable with respect to each other. But in Python 2:

$ python
Python 2.7.10 (default, Jun  6 2015, 18:12:33)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.49)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> '' > 1
True

WAT.

Just in case someone still wants to figure this out on their own, here are some hints as to what’s going on:

>>> '' > ()
False
>>> () > []
True
>>> import sys
>>> () > sys.maxint
True

Ah, a tuple than which no greater integer can be conceived. That makes perfect sense, right?

The answer

All right, enough dancing around the problem. To see why Python is doing this, we have to read a note in the Python 2 comparison documentation:

CPython implementation detail: Objects of different types except numbers are ordered by their type names; objects of the same types that don’t support proper comparison are ordered by their address.

Comparing two numbers of different types (say, an int and a float) is exempt from this because Python knows how to order them (except for complex numbers, which raise a TypeError on ordered comparison). But otherwise Python is falling back to the names of the types. So why is '' > 1? Because "str" > "int". Similarly, all tuples are “greater” than all integers because "tuple" > "int".

To which the only appropriate response is…

Giant rubber ducky, captioned WAT

But not anymore

Thankfully, Python 3 fixed this — as we saw above, Python 3 simply raises TypeError and tells you the types of the operands are not orderable relative to each other. It also got rid of the ordering by address, apparently, as that entire note is gone in the Python 3 documentation. And of course we saw how Django’s behavior with invalid variables has changed, and especially how the length filter has changed so that if, by some chance, it gets applied to an invalid variable it’ll sensibly return 0, which fixes this behavior and probably also some other “wat” moments on Python 2.

But should you want an example of a Python “wat” that isn’t so easy to puzzle out and doesn’t come down to unexpected consequences of otherwise-reasonable design, well, this certainly is one. I like to pride myself on knowing the reasons behind weird things in Python, but I am utterly at a loss as to why CPython 2 had this ordering behavior, and honestly I’m not sure I want to know; such things may be best left decently in the past, where they can cause no madness from gazing upon their faces.