Pylons deployment with Lighttpd and Flup

Pylons configuration

I've started off with the FastCGI protocol here for reasons discussed below. SCGI, for those who prefer, is a drop-in replacement (you can actually just change all instances of fcgi and fastcgi to scgi in the Pylons and Lighttpd configuration examples.) HTTP proxying is only slightly different.

You will need the flup package installed. Use easy_install flup or pip install flup, or visit the Flup home page.

Only a small change needs to be made to the INI file for your Pylons application (e.g.: development.ini) to use Flup instead of the default Paste HTTP server.

[server:main]
use = Flup#fcgi
host = localhost
port = 5000

You can then run the app with paster serve --reload development.ini, as usual.

Lighttpd configuration

Below is a minimal lighttpd.conf containing configuration that you can adapt to your needs. It's missing a lot of what you would actually want, such as logging and MIME types, so cut and paste liberally.

server.modules = (
    "mod_fastcgi"
)

server.port = 80
server.document-root = "/dev/null"

$HTTP["host"] =~ "^example.com|localhost$" {
    server.document-root = "/path/to/files/"
    $HTTP["url"] !~ "^/static/" {
        fastcgi.server = ("/" => ((
            "host" => "127.0.0.1",
            "port" => 5000,
            "check-local" => "disable",
            "disable-time" => 1,
            "fix-root-scriptname" => "enable"
        )))
    }
}

This configuration uses Lighttpd's regular-expression selectors to create a host that can be accessed either as example.com or as localhost on port 80. This is suitable if you are sharing one configuration file between your local development machine and the example.com server, but unnecessary if you are using separate configuration files, or remapping example.com to your local machine for development purposes.

The positioning of server.document-root and the second regular-expression selector means that URLs that start with /static/ will be served by Lighttpd from /path/to/files/static/, while everything else will be served by the application.

Disabling check-local means that for those URLs handled by the application, Lighttpd will not attempt to look at the filesystem at all. disable-time, on the other hand, is the amount of time Lighttpd will wait between tries if it fails to connect to the application server. Setting this to a small value makes restarts or redeployments of your app go faster, if you have not implemented a seamless restarting process for your app.

The fix-root-scriptname argument is necessary for any WSGI app, Pylons or otherwise, that is mounted on /. Otherwise SCRIPT_NAME and PATH_INFO will not be set properly.

Why FastCGI?

Some perceive FastCGI as outdated, but it works well, and given that Flup and Lighttpd handle the protocol for you there's no real need to worry about it. Still, there are options.

If you would like to use SCGI instead of FastCGI, simply load mod_scgi in server.modules and change fastcgi.server to scgi.server in the config above. Lighttpd 1.4.15 and previous have a segfaulting bug in mod_scgi, and it should not be used. Since SCGI and FastCGI are otherwise drop-in replacements and their performance seems equivalent, I simply used FastCGI.

You could also use Paste's HTTP server (the same one used for development) behind mod_proxy. This looks like the above, but load mod_proxy in server.modules, and put following in place of the fastcgi.server block:

proxy.server = ("" => ((
    "host" => "127.0.0.1",
    "port" => 5000
)))

Unfortunately, in my tests Paste's HTTP server (based on the standard library's BaseHTTPServer) did not hold up under load, whereas Flup did spectacularly. You could also try another Python HTTP server that speaks WSGI.

Load balancing

All three options (FastCGI, SCGI, and proxying) in Lighttpd come with load-balancing capability. See the "extra" set of parentheses around the host and port definition in the configuration examples above? Simply add multiple such blocks, like this:

proxy.server = ("" => (
    ("host" => "10.0.0.10",
    "port" => 5000),
    ("host" => "10.0.0.11",
    "port" => 5000)
))

If using mod_proxy, you can then alter the load-balancing algorithm using the proxy.balance parameter (see the mod_proxy docs).

What about mod_wsgi?

Apache (not Lighttpd) has mod_wsgi, which runs one or more embedded Python processes in each Apache process, passing requests directly to WSGI applications. This is a solid way to deploy Pylons apps.

See the Integration With Pylons document on the mod_wsgi site for information on how to use paste.deploy.loadapp to make Pylons compatible with mod_wsgi.

An advantage of mod_wsgi is that you don't have to keep track of as many processes. However, if you use Supervisor to manage your Pylons apps and Lighttpd, it's no trouble at all.

A disadvantage of mod_wsgi is that you don't easily have the ability to split your HTTP server and app servers across multiple machines, which is easy with Lighttpd and a separate Python process. You can do it, of course, by running Apache on each machine and using mod_proxy or a load balancer.

Also, if you are building your own Apache and Python from source, you need to be very careful about library versions. If Apache and Python are built against different versions of the Expat library, for instance, mod_wsgi will segfault. This is not a concern when Python is running as a separate process, rather than embedded.

(See Graham Dumpleton's comment, below, for some updates -- mod_wsgi can run daemon processes, although they're managed by mod_wsgi, and apparently the Expat library hasn't been a problem for a while.)

This isn't too farfetched -- if you want to use a more recent version of Python than is available in your package manager (some stable Linux distributions seem to habitually lag behind Python point releases), or if you want to use the exact same version on different platforms, you'll probably build your own.

One final note on mod_wsgi: writing to STDOUT (e.g. using the print statement) will cause serious errors, so make sure to excise all print statements and use logging instead.

Addendum

Lighttpd is my personal favorite way to deploy Python web applications. (Some people prefer nginx, also a solid option.)

Not only does separating the HTTP process from the Python process allow for easy and flexible deployment, but Lighttpd has an amazingly useful configuration system. The basic syntax is easy to work with, and for advanced uses you can generate configuration files dynamically (see: include_shell) or run Lua scripts in the server process to do advanced URL rewriting/redirecting, maintenance notices, application firewalls, etc. (see: mod_magnet).