Django, Accelerated
As of yesterday, this site is served from a Joyent Accelerator, running Django via Apache/mod_python backed by a PostgreSQL database. This setup probably isn’t for everybody, but if it’s right for you (and only you can make that call), here’s how I got everything running.
First things first
Make sure you’ve taken preliminary steps to secure your Accelerator before you do anything with it; this means things like setting up key-based SSH authentication and turning off password-based login, disabling SSH logins by root, etc.; the Joyent wiki has a “getting started with your Accelerator” guide, as well as a guide to turning off password logins and links to documentation which covers much of the rest of what you’ll want.
I also highly recommend disabling Webmin, the default Apache and Postfix; we’ll be setting up our own Apache to replace the default one, Webmin will still be accepting password authentication (and you don’t really need it to use Django) and it’s relatively easy to use an external mail server with Django (I use a Joyent Connector) so you won’t need Postfix:
sudo svcadm disable webmin:default sudo svcadm disable http:cswapache2 sudo svcadm disable postfix:default
Choose a package system
By default, an Accelerator comes configured to take advantage of Blastwave, a community-driven packaging system designed specifically for Solaris and OpenSolaris (Accelerators all run OpenSolaris). This is the sort of package manager you’re probably used to if you’re coming from a Linux or Linux-ish background: you type a command (in this case, pkg-get), followed by some options and a package name, and the package manager goes off and does its business. So, for example, to install mod_python for Apache 2, you’d type
sudo pkg-get -i ap2_modpython
And it would dutifully fetch and install the ap2_modpython package, checking dependencies along the way.
But there have been some complaints about Blastwave: it’s a fairly young system, and so the range of available packages is somewhat limited (and downright spotty in some areas) and there are quite a few Django-related packages which are fairly out-of-date; for example, Blastwave is still on Python 2.3.5 (which will work with Django, but is still two major revisions behind the Python language).
If you’re OK with that, don’t read any further; Jared Kuolt has written a step-by-step guide for setting up Django using Blastwave packages which should fulfill all your needs. His tutorial uses MySQL, but it’s fairly easy to adapt for other databases. It’s also shorter and quicker than what I’ve done, so if you’re looking for a quick payoff go with Blastwave.
I wanted a more recent Python and a better package selection, so I went a different route and used pkgsrc, the NetBSD packaging system. Don’t be fooled by the name; NetBSD focuses heavily on portability, and one of the fruits of that is that their package system works on a variety of different operating systems including Solaris. If you’re used to apt-get or yum or similar tools from the Linux world, pkgsrc is going to seem somewhat strange at first: instead of downloading binary packages, you’ll (most of the time) be downloading the source code for the packages and compiling them; pkgsrc just makes it easy by resolving dependencies for you and keeping track of what’s installed.
If you want to be adventurous and go the pkgsrc route, start by following this excellent getting-started guide on the Joyent wiki (courtesy of Filip). Once you’ve followed his instructions to get pkgsrc bootstrapped on your Accelerator, come back over here to hit the Django-specific bits. Pay special attention to the cron job you’ll be instructed to set up for downloading security notices, and make a note to regularly use the audit-packages utility to see whether there are any packages you need to remove or upgrade. The official pkgsrc guide has more information on this, and you’ll want to read it thoroughly before you get your feet wet.
Setting up required packages
Assuming you’ve got pkgsrc set up per Filip’s instructions, there’s one last bit of configuration you’ll need to do before you start installing things; open up the file /opt/local/etc/mk.conf and add the following line to the configuration:
ACCEPTABLE_LICENSES+= fee-based-commercial-use
Without this line, OpenSSL will not build.
Now, start installing things (remember that you’ll either need to become root or use sudo for all of these, and that you’ll have to wait in between commands for each package to download, compile and install):
cd /opt/pkgsrc/lang/python-24/ bmake && bmake install cd /opt/pkgsrc/databases/postgresql82/ bmake && bmake install cd /opt/pkgsrc/databases/py-psycopg2/ bmake && bmake install cd /opt/pkgsrc/www/apache22/ bmake && bmake install cd /opt/pkgsrc/www/ap2-python/ bmake && bmake install cd /opt/pkgsrc/textproc/py-docutils/ bmake && bmake install
Once that’s done you’ll have fairly recent versions of Apache, mod_python, Python (as the binary /opt/local/bin/python24), PostgreSQL, psycopg2 and docutils (for the Django admin documentation) installed. The Django 0.96 release is also available (as www/py-django), but I prefer to work from a Subversion checkout. If that’s living too dangerously for your taste, go ahead and install the package to stick with the release version of Django.
Setting up a place for Python modules
For sake of convenience, you’ll want to create a directory to hold all of your assorted Python code (SVN checkout of Django, third-party modules that aren’t in pkgsrc, all the projects/applications you’ll be using, etc.) which will be on your Python import path. Ideally this will be somewhere that’s outside your administrative user’s home directory (so that you don’t have to give Apache permission to read in there), but still writable by that user. I’m a fan of having that directory be /home/code:
cd /home
mkdir code
chown youruser:yourgroup code
Now edit your user’s .bashrc file, and the global .bashrc, to include these lines, which will set up paths properly and ensure you’re using the python24 binary:
export PATH=/opt/local/bin:/opt/local/sbin:/usr/xpg4/bin:/usr/bin:/usr/sbin:/usr/ccs/bin:/usr/sfw/bin:$PATH export PYTHONPATH=/home/code:$PYTHONPATH alias python='python2.4'
Then either log out and in again, or use source ~/.bash_profile && source ~/.bashrc to update your environment.
Get Django (and anything else you need)
Do a Subversion checkout of Django:
cd /home/code
svn co http://code.djangoproject.com/svn/django/trunk/django
You can test that it works by firing up a Python interpreter and doing from django import VERSION, then printing VERSION; it should say (0, 97, ‘pre’). While you’re in there, go ahead and do checkouts or downloads of any other Python modules or Django applications you want, and also upload your Django project. For templating purposes, I usually like to create a /home/html or /home/templates in much the same fashion as /home/code, so feel free to borrow that practice and put your templates in there (and remember to edit your TEMPLATE_DIRS setting if you do).
Set up PostgreSQL
At this point PostgreSQL is installed, but isn’t configured and isn’t running, and we need to fix that. First, we have to initialize it with a data directory, and this — like all PostgreSQL management tasks — must be done as the user postgres; PostgreSQL will not let you do these things as root:
su - bash cd /opt/local mkdir var cd var mkdir pgdata chown postgres:postgres pgdata su postgres initdb -D /opt/local/var/pgdata
This creates a PostgreSQL data directory at /opt/local/var/pgdata, sets its ownership to the postgres user, and initializes it properly. Inside that directory you’ll find the relevant PostgreSQL configuration files; the first thing you should do is make backup copies of them:
cp pg_hba.conf{,.bak} cp postgresql.conf{,.bak}
This will create pg_hba.conf.bak as a copy of pg_hba.conf, and postgresql.conf.bak as a copy of postgresql.conf. It’s a good idea to also download copies of those files and put them somewhere safe just in case you ever need them.
Next you’ll want to edit the configuration files; pg_hba.conf controls authentication, and postgresql.conf is the main configuration file. First off, open up pg_hba.conf and scroll down to the bottom, where you’ll see the following:
# TYPE DATABASE USER CIDR-ADDRESS METHOD # "local" is for Unix domain socket connections only local all all trust # IPv4 local connections: host all all 127.0.0.1/32 trust # IPv6 local connections: host all all ::1/128 trust
Comment out or remove the last four lines, so that it looks like this:
# TYPE DATABASE USER CIDR-ADDRESS METHOD # "local" is for Unix domain socket connections only local all all trust
This will “trust” connections on the local Unix socket PostgreSQL listens to, meaning that those connections will not need a password. Now, edit postgresql.conf and add this line:
listen_addresses = ''
This means PostgreSQL will not accept any connections over IP; it will only listen to the local socket (which we’ve already set it up to trust). If you do need to have it listen on an IP address, you’ll want to change your pg_hba.conf to require a password for those connections, but it’s easiest to just listen on the Unix socket. The definitive guide to PostgreSQL’s authentication system is the official documentation for pg_hba.conf from the PostgreSQL online manual; if you want to do something different, follow that guide.
Aside from telling PostgreSQL to only listen on the local socket, you probably don’t want to mess with the rest of the configuration; on a dedicated database server with a lot of RAM you’d have to change some things to get decent performance, but the default settings aren’t too bad for most of the Accelerator options (where you have less RAM available and have to take other things, like Apache, into account). So leave the rest of that file as-is for now, and only come back to it if you’re seeing performance problems; if you do have problems, Frank Wiles’ guide to PostgreSQL tuning is one of the best out there (disclaimer: Frank used to work for a different branch of the same company I work for; all that means in this case is that I know he knows his stuff).
Now that it’s configured, start the server manually (as the postgres user):
pg_ctl -D /opt/local/var/pgdata start
Now you’ll be able to connect to it and create a database user for Django (again, do this as the postgres user); PostgreSQL will ask if the new user should be created as a superuser, be able to create roles and databases, etc.; answer ‘no’ to these questions:
createuser django
And finally — still as the postgres user — create the database for Django to connect to:
createdb -O django your_database_name
Then shut down the PostgreSQL server (in a moment, we’ll register it with Solaris’ service-management facility, which will start it and keep it running automatically):
pg_ctl -D /opt/local/var/pgdata stop
If you’re used to Linux, you’re probably familiar with the system of init scripts which start and stop various services; for example, to start Apache on Linux you usually type something like /etc/init.d/httpd start, and have a symlink into an /etc/rc.d directory which tells the system to start and stop it automatically. Solaris uses a system called SMF, which is a bit more complicated but has several big advantages:
- It presents a single unified interface to all services, so you only have to learn one set of commands.
- It actively monitors all of the registered services and can re-start them automatically if they crash, or report problems if they can’t be started.
- It lets you specify dependencies between services so that, for example, you can tell it not to try to start Apache unless the network interfaces are running and the relevant filesystems have been mounted.
This is pretty handy, so it’s worth the extra trouble to set things up via SMF; if you’re interested in more detail, Joyent’s Ben Rockwood has an excellent explanation of key SMF concepts, and regularly blogs about other useful Solaris topics.
What we’re concerned with here is that there are two parts to registering a new service with SMF:
- A shell script which includes some SMF-specific files and knows how to start, stop and reload the service, and
- An XML “manifest” which tells SMF about the service.
Examples of both of these files are available from Sun’s documentation for PostgreSQL on Solaris, but we’ll need to edit both of them to make them work with the version of PostgreSQL we set up with pkgsrc.
First you’ll need to get the shell script in place; I’ve already prepared an edited version and — assuming you followed the instructions above and have everything installed from pkgsrc in /opt/local — you should be able to just use that without any trouble, so do the following (either as root, or using sudo):
cd /opt/local/lib mkdir svc cd svc mkdir method cd method wget http://media.b-list.org/files/postgresql.txt mv postgresql.txt postgresql chmod 555 postgresql
Now you need the XML manifest; I’ve also prepared an edited version of the file which should work, so you can download and register it like so:
wget http://media.b-list.org/files/postgresql.xml.txt mv postgresql.xml.txt postgresql.xml svccfg import postgresql.xml
Now run svcs -a to verify that the service svc:/application/database/postgresql:default is listed as imported successfully; if it shows up there, you can enable it by running svcadm enable postgresql:default. This will start the PostgreSQL server, and keep it running automatically. To verify that it’s working, try connecting with the user you set up:
psql -U django your_database_name
You should get an interactive PostgreSQL prompt; if you don’t, go back and check that you’ve set everything up properly.
Set up Apache
Configuring and starting Apache is a little bit less involved; you’re using mod_python, so the official Django/mod_python documentation should cover everything you need to do apart from the defaults; remember that /home/code needs to be added to the Python import path using the PythonPath directive, and remember that you’ll need to explicitly load mod_python by having this line in your httpd.conf:
LoadModule python_module /opt/local/lib/httpd/mod_python.so
You’ll also want to disable HTTP keep-alive, because that will just tie up a server process longer than needed:
KeepAlive Off
Before you edit your httpd.conf file (which will be in /opt/local/etc/httpd), remember to make a backup copy:
cd /opt/local/etc/httpd cp httpd.conf{,.bak}
I’ve also put together a minimal httpd.conf file you can look through if you’re interested in some of the relevant settings, but I recommend using that only as a guide to writing your own configuration file; you’ll probably want at least some of those settings to be different.
Once your configuration file is in place, test that Apache configuration works by manually starting the server:
/opt/local/sbin/apachectl start
If there are no error messages, hit your site’s IP address in a browser to make sure it’s serving up Django correctly. If it is, stop the server:
/opt/local/sbin/apachectl stop
Now you can register it with SMF; again, I’ve provided working samples of the shell script and XML manifest which you can use:
cd /opt/local/lib/svc/method
wget http://media.b-list.org/files/apache22.txt
mv apache22.txt apache22
chmod 555 apache22
wget http://media.b-list.org/files/apache22.xml.txt
mv apache22.xml.txt apache22.xml
svccfg import apache22.xml
Again, use svcs -a to make sure it imported properly, and then svcadm enable apache22:default will start Apache and keep it running.
Where to go from here
At this point you should have Django installed properly with a recent Python, PostgreSQL running with the psycopg2 adapter for Django to talk to it, and Apache with mod_python running to serve your site. From here on out, you can make whatever changes you like (when you change your code and need to restart Apache, use svcadm refresh apache22:default), but this setup is what I’m using to serve this site, so if you’re reading this it works.
The only major thing I’ve left out here is how to handle static media; as always, it’s recommended that you use a separate server to handle that, though the Django documentation will tell you how to configure Apache to serve static files alongside Django. For this site, I purchased a BingoDisk account; Bingo is a pretty simple service which simply sells storage space and lets you serve files directly out of it, and lets you point a DNS CNAME record at it so that it can appear to be your domain (or a subdomain; I have media.b-list.org pointing at mine), and so far I’ve been pretty happy with it (just be careful to keep everything public in your public directory, and not to do anything which would force a browser to go outside of that — if, for example, a browser tries to request /favicon.ico as part of a web page, your visitors will see an ugly authentication prompt). I’d recommend Bingo or a similar dedicated file-serving system just because it’s easier than running two servers or reconfiguring Apache, but how you handle it is up to you.
And, as always, remember to keep your server secure: set up the cron job to download pkgsrc security announcements, and regularly use audit-packages to look for problems. It’s also a good idea to have a cron job do a dump of your database and send it to a backup service (I recommend Strongspace).
Beyond that, the sky’s the limit; I’ve only been in production mode on the Accelerator for about 24 hours, but I’ve been tweaking and testing for a little while now and so far I’ve been very pleased with the performance and the flexibility. So if you find shared hosting a bit too limiting and don’t mind paying the premium (both in terms of pricing and of becoming your own sysadmin) for more power, or if you’re setting up a site which needs more flexibility or a more reliable platform, I’d highly recommend checking out Joyent’s various Accelerator plans.
Addendum: how to handle security updates
Updated September 10, 2007: I just saw from running audit-packages (you did remember to do that regularly, right?) that there was a vulnerability in Apache; it only affects mod_proxy, but I decided to go ahead and do the upgrade anyway. For the curious, what happens is that audit-packages tells you there’s a vulnerability in one or more packages, lists them and provides URLs for the relevant security notices; in this case the vulnerable package is www/apache22. Here’s what I did to upgrade:
cd /opt/pkgsrc && cvs -q up -d cd /opt/pkgsrc/www/apache22 bmake update
The first line updated my pkgsrc tree to include the relevant package files for a new version of Apache; then the bmake update in the Apache package’s directory built and installed a new copy of Apache and any dependencies which also needed to update. And that’s it; my site was offline very briefly as Apache was being switched out for the new version, but otherwise the whole thing was pretty painless.