I have it on my todo list to convert my current CherryPy environment to Werkzeug as a proof of concept and see which one of the two I prefer and will ultimately use for my Japanese-Dutch dictionary project.

CherryPy uses a regular expression syntax for URL dispatching that people familiar with Django might recognise. It is a decent way to dispatch URLs, but once you’ve seen Routes’ maps using such regular expressions feels hackish. Well, at least to me, use whatever works for you.

There was only one issue, how on earth did I put all this in a FastCGI setup? I had previously, for my own script, used flup‘s fcgi WSGIServer class to kickstart my application. This means that my Lighttpd environment configures a Python file as a FastCGI script and creates a Unix domain socket to connect through. This worked quite well, so I set out to convert my old way to see how to use CherryPy and Routes. The first hurdle I encountered was that using Routes with CherryPy is not documented well (of course, it is/was not at the time of this writing). Nowhere in that page does it mention the magic incantation to switch dispatchers from the default to, say, RoutesDispatcher. Using some Google-magic as well as discussing this with Alec Thomas (of Trac fame) I arrived at the following Python code to switch the dispatcher around:

import cherrypy
from project.controllers import *
dispatcher = cherrypy.dispatch.RoutesDispatcher()
dispatcher.connect('home', '', controller=HomeController())
config = {'/': {'request.dispatch': dispatcher}}
app = cherrypy.tree.mount(None, config=config)

Next I started to pass app to my WSGIServer class to run it. I noticed that something strange was actually happening when I checked with sockstat on my FreeBSD machine if there were any Unix domain sockets left after stopping Lighttpd. And indeed, there was a stray socket left. Now that was funny, since normally after Lighttpd has sent a termination signal (SIGTERM) to the spawned processes, they shut down and the socket gets cleaned up. What was left was something like this:

USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
www      python     955   0  stream /tmp/labs.sock-0

So the file descriptor 0 socket is still left. This is, according to Unix tradition, standard input, so in effect it is still waiting to handle data. But wait a second, we told it to shut down, but it didn’t completely. Using the top command I looked for the process id (PID) and found a line like this:

PID USERNAME   THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
955 www          9  20    0 14784K 11556K kserel 0   1:25  0.00% python

Normally when everything is still running normally it displays as:

PID USERNAME   THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
955 www         10  20    0 14784K 11556K kserel 0   1:25  0.00% python

Notice the difference in the thread (THR) column (going from 10 to 9). So apparently Lighttpd sends a SIGTERM to the process and it succeeds in killing off one thread and subsequently lets the other nine stay and wait for new requests to serve. Now, this would not be potentially bad, were it not that every stop/start cycle spawns another process with ten threads and thus wasting valuable resources. So clearly this problem had to be solved.

The current code I had in place was (partially lifted from an older Trac FCGI start script):

app = cherrypy.tree.mount(None, config=config)
cherrypy.engine.start(blocking=False)
WSGIServer(app).run()

The traceback should get printed to the browser if the WSGIServer cannot be started for whatever reason or if it raises an exception.

I finally realised that apparently it had to be CherryPy that was not shutting down as it should, especially since this worked with flup’s WSGIServer and my own code before! Furthermore, in a thread I started on the CherryPy-users group over at Google, Robert Brewer pointed me (mistakenly as he later pointed out) to a page detailing the CherryPy HTTPServer API. Even though it was not correct in this case –CherryPy is not the controller in this case– it did point out one thing a thread_pool attribute set to a default value of 10! So this really confirmed my thought that CherryPy was not getting closed down as it should.

The solution to such a problem, as with most things in life I guess, was rather an anti-climax, the code above had to changed to be like this:

app = cherrypy.tree.mount(None, config=config)
cherrypy.engine.start(blocking=False)
try:
    WSGIServer(app).run()
finally:
    cherrypy.engine.stop()

And that’s it!

Update 2007-06-02 10:57: Stripped the exceptions, they actually do not add much in this case.