Pylons apps on Google App Engine

This will show you how to get a Pylons web app running on Google App Engine (GAE) in about 15 minutes, assuming you already have your GAE account and a familiarity with Pylons and virtualenv.

(Otherwise, start out with the Google App Engine docs and the Pylons docs).

The approach described here is focused on using as "normal" a process as possible for creating a Pylons app, with minimal modifications for GAE. It uses a stock virtualenv, and a stock paster create template, instead of an entirely different setup just for GAE. Several manual edits are required (but fewer now!)

This means that you could use an existing app or an existing virtualenv, with a few modifications. This is up to you, as I don't know what your exact setup looks like, but most of the steps below should apply.

Getting started

To begin, run one of the following scripts in an appropriate directory, replacing $MYPROJ and $MYAPP with whatever values you like (they can be the same, though if they are the directory structure can end up slightly confusing. paster create will make two directories named $MYAPP as it is.)

I've provided instructions for both pip and easy_install, so you can continue to use your preferred package manager.

You could also browse app.yaml and app.py to get an idea of what's going on.

Getting started: pip version

console
virtualenv --python=python2.5 --no-site-packages --unzip-setuptools $MYPROJ
cd $MYPROJ
pip install -U -E . pylons
touch ./lib/python2.5/site-packages/paste/__init__.py
rm -f ./lib/python2.5/site-packages/simplejson*/_speedups*
./bin/paster create -t pylons $MYAPP template_engine=mako sqlalchemy=False
wget http://files.countergram.com/pylons-gae/app.yaml
wget http://files.countergram.com/pylons-gae/app.py

Getting started: easy_install version

console
virtualenv --python=python2.5 --no-site-packages --unzip-setuptools $MYPROJ
cd $MYPROJ
./bin/easy_install -UZa pylons
rm -f ./lib/python2.5/site-packages/simplejson*/_speedups*
./bin/paster create -t pylons $MYAPP template_engine=mako sqlalchemy=False
wget http://files.countergram.com/pylons-gae/app.yaml
wget http://files.countergram.com/pylons-gae/app.py

If you use easy_install, you must then look in the lib/python2.*/site-packages directory and see if multiple versions of any packages (especially WebOb) were installed due to differing dependencies. Delete all but the most recent; otherwise they will conflict. The -U argument to easy_install should prevent this but does not necessarily do so.

Python versions other than 2.5

GAE runs Python 2.5. As long as your code runs on 2.5, you can use whatever (2.x) version of Python you want locally. You'll need to change all instances of python2.5 in the script above, and also edit the value of libdir in app.py.

Other required edits

You'll need to manually edit a few files to get things working.

  1. Edit app.yaml and replace $REGNAME with the name of your application as registered with Google. For example, if your hosted URL is myapp.appspot.com, the registered name is myapp.
  2. Edit app.py and replace $APPNAME with the name of your application that you passed to paster create (i.e. the directory name).
  3. In your Pylons app's environment.py, comment out or delete the line beginning module_directory= in the instantiation of TemplateLookup. GAE does not have a writable filesystem, so Mako's file-based template caching will not work on it. If you are using a different template system, you'll have to figure out what is required to stop it from trying to write any files.

Running locally

You can run your app locally in a GAE environment using the Google App Engine SDK/Launcher. It's fairly simple, and documentation is available and beyond the scope of this article, but here are a few tips:

  • In the GUI version of the SDK/Launcher, be sure to use File->Add Existing Application rather than File->New Application. The path to use is the directory that contains app.yaml and app.py (i.e. $MYPROJ).
  • Using the command-line version, dev_appserver.py, don't run it from within the virtualenv; it won't be able to find the Google libraries.
  • The SDK/Launcher has a primitive (compared to paster serve) reloading capability. In order to force an "automatic" reload while your app is running, you'll need to edit (or touch) app.py.

Deploying

There are two ways to deploy your application: the Google App Engine SDK/Launcher application, or the appcfg.py command-line script.

The SDK/Launcher application gives you a nice push-button GUI, while appcfg.py is more flexible with various options such as verbose mode (-v), which prints a list of which files are being uploaded, and which are being skipped.

File count

When I deployed a barebones Pylons app in this way, with the included skip_list, the file count came out to 821 using pip and 814 using easy_install, both, well under the 3,000-file limit. Your "base" file count will probably vary slightly as new library versions come out.

If you are using easy_install (but apparently not if you are using pip), you can reduce the file count by excluding EGG-INFO directories from being uploaded. This will probably not break anything (but see below), unless part of your code depends on entry points of modules other than your app. You'll need to add a line to the middle of the skip_list regex in your app.yaml, as follows:

(.*/site-packages/.*\.egg/EGG-INFO/.*)|

For me, this reduced the "base" file count to 691, a savings of about 120 files. Unfortunately, excluding .egg-info directories when using pip breaks the setup.

If you do this, you must also add template_engine=None as a keyword argument to the config.init_app function call in your Pylons app's environment.py. Otherwise, some legacy code in Pylons (supporting the outdated Buffet template system) will break because it can't find some egg info.

Notes on workarounds, for the curious

  • GAE does not appear to use .pth files, which are files that are placed on the Python path and contain additions to the path. This is why sys.path must be set up in app.py. Also, easy_install uses its easy_install.pth file to make sure only the latest installed version of each package is on the path. This is why you have to manually delete duplicate packages using this GAE method.
  • The provided app.py has a rather odd line: os.mkdir = lambda *x:None. The latest version of setuptools (as of 0.6c11) attempts to import mkdir from os, and mkdir does not exist on GAE. This monkey-patch allows setuptools to be imported.
  • The provided app.yaml has a skip_list that prevents files that are part of the virtualenv but useless on GAE, such as binary shared objects, from being uploaded.
  • The "speedups" part of simplejson (_speedups.py and _speedups.so) uses a C extension, which is not supported by GAE. But the package is set up so that it will still work if these files are missing.
  • Paste uses a wonky system to allow separate packages (Paste, Paste Deploy, Paste Script) to inhabit the same paste namespace. When installed using pip, this system is dependent on several .pth files, which don't work on GAE. But sticking an __init__.py in makes it work normally.

Conclusion

At this point, you should be up and running and see the Pylons welcome page when you visit your local server run by the SDK. There are still more steps to a real GAE app, of course, such as familiarizing yourself with Google's datastore models.

I can't provide "support" on any of this, per se, but if the steps here don't work for you, please e-mail me and I'll try to correct or amend the article.

Prev: Managing multiple Pylons apps with Supervisor

Next: Pylons multiple-domain (virtual hosting) configuration

Share this content

Comments

Avatar picture for stuart Nice work. I think its worth pointing out that the dev appserver (dev_appserver.py) should not be run from within the virtual environment as it causes import errors, took me a while to figure that out.
Avatar picture for Jason Stitt Thanks. I've added that to the article.
Avatar picture for Christof Haemmerle crazy think i am doing something wrong. when i start the app in the local dir i get: ImportError: No module named config.middleware
Avatar picture for Jason Stitt Hi, Christof. I responded on the mailing list as well. I think this was due to the script version by another contributor, which mixed in some steps from the previous version of this article. Let me know if it still doesn't work after removing those or just using the steps above.
Avatar picture for Ken Kinder I'd be nice to know when this article was written. There's no date and the HTTP header says today.

Sometimes when I'm looking up documentation like this, it's handy to just check to see if it's recent. Just a thought.
Avatar picture for Jason Stitt Ken, good thought. I just did a redesign and what got lost in the shuffle was applicable software version numbers next to the byline, which I actually prefer to dates because you don't have to keep track of what dates new releases came out. I'm working on putting them back.
Avatar picture for Pablo Rosales Thanks!! I have a few days trying to get this to work, you made it so easy!!

App engine monkey does not work for me... too bad you are not first place on google for the search: "pylons app engine" that could have saved me time :)

Regards,
Pablo Rosales
Avatar picture for Sunil Arora Jason,

Thanks a ton for a writeup which is right on target for the job.

I followed your article line by line and had my pylons homepage up on just 25 minutes.


Avatar picture for Ben Rousch I'd like to try this with the new Pyramid web framework and current version of Google App Engine (1.4.2). Any chance you've given it a go recently?
Avatar picture for Evgeny What I do wrong?
I got message:
INFO:root:Starting admin server at: [Link to localhost:8000
ERROR]
2013-07-11 11:55:07,816 cgi.py:121] Traceback (most recent call last):
File "/home/evgeny/PycharmProjects/pylons/app.py", line 41, in <module>
run_wsgi_app(loadapp('config:development.ini', relative_to=appdir))
File "lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
return loadobj(APP, uri, name=name, **kw)
File "lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 271, in loadobj
global_conf=global_conf)
File "lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
global_conf=global_conf)
File "lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
return loader.get_context(object_type, name, global_conf)
File "lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 454, in get_context
section)
File "lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 476, in _context_from_use
object_type, name=use, global_conf=global_conf)
File "lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 406, in get_context
global_conf=global_conf)
File "lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
global_conf=global_conf)
File "lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg
return loader.get_context(object_type, name, global_conf)
File "lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 620, in get_context
object_type, name=name)
File "lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 646, in find_egg_entry_point
possible.append((entry.load(), protocol, entry.name))
File "lib/python2.7/site-packages/distribute-0.6.19-py2.7.egg/pkg_resources.py", line 1991, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File "project/project/config/middleware.py", line 9, in <module>
from routes.middleware import RoutesMiddleware
File "lib/python2.7/site-packages/routes/__init__.py", line 140, in <module>
from routes.mapper import Mapper
File "lib/python2.7/site-packages/routes/mapper.py", line 7, in <module>
from repoze.lru import LRUCache
ImportError: No module named repoze.lru

Post Comment

All comments are personally reviewed and must be:

  • On-topic
  • Courteous
  • Not self-linking or spam
(Optional. This is your one self-link.)