Web frameworks and web hosts
Today John Gruber linked to a post on the official Dreamhost blog lamenting the state of web-framework deployment on consumer-level shared hosting; while the post itself is largely concerned with Ruby on Rails, the current state of deploying popular (non-PHP-based) web frameworks on hosts like Dreamhost is, well, abysmal. A search for “dreamhost” in the archives of the django-users mailing list turns up nearly five hundred results, many of them people struggling to get Django working, and I imagine that similar searches of list archives for other frameworks would turn up similar numbers of problems.
So. What’s so hard about this, and what can be done to improve the situation?
The problem
Well. Actually, “problems”. As I see it, there are three major issues facing frameworks on shared hosting.
First off, the new wave of popular frameworks all make a clean break from a traditional CGI-style model where the application is simply loaded and executed fresh on each and every request; this is a necessary change (reloading a framework like Rails or Django on every request would result in horrific performance), but one that causes headaches for web hosts.
Second, the deployment options are so varied as to be bewildering; even when there’s some sort of standard gateway protocol — as with WSGI in Python — there’s still plenty of room for disagreement on how to actually deploy an application. To run with the WSGI example (and pick on Python a bit just to show I’m not engaging in any favoritism here): I can write an application that speaks WSGI fairly easily, and WSGI applications are for the most part interchangeable. But on the server side, it’s not so clear: should the server learn to speak WSGI directly (à la mod_wsgi)? Should there be an intermediary to bridge the gap, like flup? If so, what protocol should the server and the intermediary use (flup can speak FastCGI, SCGI and AJP, for example)?
Third, the new wave of frameworks are bringing languages like Ruby and Python onto shared hosting for the first time, and with that comes the problem of library support; there’s a whole new world of standard modules hosts need to be aware of and be ready to support if they want to get serious about these frameworks, because asking the typical shared-hosting customer to go out and compile database adapters or image-manipulation libraries (if your AUP even allows that) simply won’t work.
Go long
The fact that the deployment model for the new frameworks centers around long-running processes — rather than launching a clean copy of the application on each request — is one that’s hard on hosting providers. Unlike traditional CGI or PHP setups, a framework like Rails or Django or TurboGears simply can’t be run on a “launch a new copy every time” basis; the overhead of loading and initializing the framework makes it unusable in that situation. Having a persistent process which loads the code once and keeps it in memory is the only way this can work.
But hosts have traditionally been reluctant to allow users to launch long-running processes of the type needed to efficiently serve a framework-based application; more than once in my freelancing days, I ran into hosts with automated process monitors which simply killed any user-owned process that stayed resident for more than about thirty seconds. Others have been known to kill anything not on an approved whitelist (read: Rails), which has led to the amusing phenomenon of people renaming their non-Rails FastCGI handlers to dispatch.rb as a workaround.
This situation is slowly getting better, but a whole lot more progress is needed. Yes, a long-running process will chew up valuable memory even when it’s not actually serving a request, but that’s something that has to be accepted; hosts which don’t support this are going to be hosts that don’t get business (or word-of-mouth referrals) from framework users.
One obvious way to do it
One of the reasons behind PHP’s ubiquity is the fact that, for the longest time, the deployment pattern was simple and standard: install PHP and mod_php, tell Apache to use that for anything ending in .php and you’re pretty much done. These days there’s a bit of fragementation in the options — mod_php versus FastCGI versus straight CGI — but they’re all well-documented, well-understood and easy to set up (and, from the perspective of an application author, they all work the same way: just upload a .php file). What’s more, most Linux distributions can set up the whole thing for you with a single dip into the package manager.
Ruby and Python, right now, have nothing approaching this. Yes, Python has WSGI, but see above for why that doesn’t solve the problem. At most, there needs to be one standard stack per language, preferably of the form “install this Apache module”. For Python I think the most promising option is mod_wsgi which, though it doesn’t let you do everything in .htaccess, seems like it’d lend itself quite easily to a plugin for the popular hosting control panels. For Ruby I don’t know; I’m not familiar enough with the various options to be able to form an opinion.
This doesn’t mean that frameworks shouldn’t support multiple deployment options, but it does mean that, for whatever language you’re working in, there should be a standard option and the framework should support it.
import this
The other big reason behind PHP’s success is that it targets web development specifically, and comes with everything and the kitchen sink. The downside is that PHP’s standard function namespace is horrifically crowded, but at least there aren’t a whole lot of common cases where you have to go out and install a new library just to get something basic like database support.
Ruby and Python, on the other hand, are general-purpose modular languages; this means there are a number of modules which, though necessary for web application development, aren’t included in the standard library of either language. Database connectivity, for example, largely comes from third-party modules; Python only supports SQLite out of the box (and only in Python 2.5 and up), and as far as I know Ruby doesn’t ship modules for any of the “big three” open-source databases (SQLite, MySQL and PostgreSQL) in its standard library.
Some of this could be helped by changes to standard libraries or by more convenient packaging of some of the important third-party items, but if hosts want to get serious about frameworks they need to be ensuring the availability of any key modules that don’t come as part of the language; telling someone to go compile their own copy of, say, MySQLdb for Python is a non-starter.
Work with me here
You’ll note that, of the three things I’ve outlined above, two lay a lot of responsibility on the hosting providers (though if you think that solving the “standard deployment” issue is going to be easy on the framework side, I’ve got a bridge in Brooklyn I’d like to sell you). That’s pretty much inevitable: when a new method of building web applications becomes popular, web hosting needs to change to keep up.
But by and large, this process needs to be a dialogue between hosts who want to offer solid framework support, and developers and users of frameworks who want to make the hosting experience as painless as possible. Neither side can afford to sit back and hope that the other will deal with the problem, and neither side alone has enough expertise or resources to solve all of the issues.
And there are certainly hosts out there which are doing their part: Joyent, with whom this site is happily hosted, pride themselves on Rails support and have also done a lot of largely-unsung work to ensure that Python frameworks will be easy to use on their shared offerings (to the point of contributing patches to key third-party modules so they’ll work on Joyent’s Solaris environment). WebFaction has literally gone all-out, with integrated support for popular frameworks right in their control panel (see their Django deployment screencast for an example).
Going forward, I think that’s going to be the only way to resolve the issues I’ve listed above: the hosts who want to provide a great framework experience will get together with the frameworks which want to provide a great hosting experience, and sit down and solve these issues. And if they’re looking for good solutions to the problems they’re having with frameworks, I’d love to see Dreamhost get in on that.