In Nomine – The Lotus Land

Expect more information about OpenGL and DirectX related subjects in the coming period as I will start to delve into the basics of these.

At work we use Jira for some projects, but the projects I work on use Trac. For many projects an environment like Trac is sufficient. Recently I started to use Eclipse with Pydev for the Python development. Eclipse features a plugin called Mylyn, which allows you to configure a task repository and directly access the tasks in that repository. This is extremely useful, since the repository can also be an instance of Jira or Trac. To connect Mylyn to these repositories you will need a so-called connector. For Jira there’s the Atlassian Connector for Eclipse and for Trac the standard Eclipse repository already has a Mylyn Connector: Trac available. Make sure this is installed.

Next, we need to install some plugins for Trac. On my setup we use AccountManager for authentication and managing of the password files. To make Mylyn talk with Trac we need to add the XML-RPC as well as the HTTPAuth plugin. The XML-RPC plugin is the one that provides XML-RPC methods for Mylyn to talk with Trac. The HTTPAuth plugin is needed if your Trac’s login has been replaced by AccountManager (as is the typical way to use it). HTTPAuth forces certain URL paths (/xmlrpc, /login/xmlrpc) to use standard authentication.

A typical trac.ini will look like this after enabling the appropriate plugins:

[account-manager]
password_file = /usr/local/projects/project1/svn-auth
password_store = HtPasswdStore
 
[components]
acct_mgr.admin.accountmanageradminpage = enabled
acct_mgr.api.accountmanager = enabled
acct_mgr.htfile.htpasswdstore = enabled
acct_mgr.web_ui.accountmodule = enabled
acct_mgr.web_ui.loginmodule = enabled
acct_mgr.web_ui.registrationmodule = disabled
httpauth.* = enabled
trac.web.auth.loginmodule = disabled
tracrpc.* = enabled
 
[httpauth]
paths = /xmlrpc, /login/xmlrpc

After restarting Apache (with mod_wsgi, of course), if you go to http://trac.example.com/project1/xmlrpc it should present you with a user/password prompt. But most likely this will keep repeating without authorising you. After a bit of head scratching and looking around I found out that that is because mod_wsgi requires to pass authorization through to the WSGI application by means of WSGIPassAuthorization On in the appropriate Apache configuration file.

After all this works, you can configure a new Mylyn repository within Eclipse (Windows > Show View > Other > Tasks > Task Repositories). If you then click the icon that stands for create new repository it will present you with a configuration window for adding new repositories. Select the Trac option, followed by Next. In the Server field you can just add http://trac.example.com/project1, followed by a, for you, descriptive Label for this repository. The User ID and Password are those you use to log into the Trac instance. Under Additional Settings is an option for selecting the Access Type, make sure you select XML-RPC here. Ensure that Task Editor Settings has TracWiki selected. Now, by selecting Validate Settings, you ought to see a message stating “Authentication credentials are valid” at the top of the window.

That’s it. You can now follow the Mylyn User Guide.

For some reason setup.py can fail with docutils complaining it cannot find the roman module. One thing that works is just removing docutils from your site-packages and reinstall it.

So I had implemented a simple JSON data server with Werkzeug for a classroom experiment. Unfortunately in my haste to get everything up and running I totally forgot about the fact that, since we cannot allow uploads to this server of various custom made webpages, using jQuery’s $.ajax() everything just fails since it will then be a cross-site scripting request.

So, normally you would do something like the following in order to return JSON data:

return json.dumps(data)

Which would be used with the $.ajax() call in a way like the following:

$.ajax({
  type: "POST",
  url: "http://example.com/json/something",
  data: "parameter=value",
  dataType: "json",
  error: function(XMLHttpRequest, textStatus, errorThrown){},
  success: function(data, msg){}
});

Which is perfectly fine for scripts getting and using the data on the same host/domain. But, as said before, this will fail with warnings similar to: "Access to restricted URI denied" code: "1012" nsresult: "0xdeadc0de (NS_ERROR_DOM_BAD_URI)".

One way out of this is using JSONP. jQuery has a $.getJSON() function, which loads JSON data using a HTTP GET request. Now, the simplistic way to convert your code would be to change it as such:

$.getJSON("http://example.com/json/something",
  function(data){}
);

But this causes another issue. Since $.getJSON() GETs the JSON data, but doesn’t use eval() on it, but instead pulls the result into script tags, it somehow causes,on Firefox at least, an invalid label error. In order to fix this you need to set up the JSON data server to properly support a callback argument, to use $.getJSON() how it is meant to be used:

$.getJSON("http://example.com/json/something?jsoncallback=?",
  function(data){}
);

In the code above the additional parameter jsoncallback will, thanks to jQuery, get the question mark replaced by an alphanumeric string (typically in the form of jsonp followed by a timestamp). This value should be used to wrap the resulting JSON data with. This means you would have to change the initial Python code to something like this:

return request.args.get('jsoncallback') + '(' + json.dumps(data) + ')'

Of course this causes problems when you want to reuse the code for both AJAX use on the same host/domain and use it from outside. So in order to make both work you can test on whether or not the callback parameter is available and return the appropriate data. I came up with this little snippet for that:

def jsonwrapper(self, request, data):
    callback = request.args.get('jsoncallback')
 
    if callback:
        return callback + '(' + json.dumps(data) + ')'
    else:
        return json.dumps(data)

As I said in http://www.in-nomine.org/2009/02/22/tinymce-in-wordpress-271-not-working-on-freebsd/ I had problems making TinyMCE work.

After some digging I discovered that the tiny_mce.js file was being oddly truncated and repeated when fetched. Subsequent tests showed that md5 hashes differed every single time with the original file.

During these fetch tests my kernel crashed twice in the network stack. Since my lighttpd configuration uses kqueue and the likes I suspected that perhaps something was not right in lighttpd 1.4.21, which was only released a couple of days ago. So I downgraded to 1.4.20. ‘Lo and behold, the problems disappeared. To be absolutely sure, I recompiled lighttpd 1.4.21 as well. And at first it seemed to work, but then the corruption kicked in again. After talking about it on the #lighttpd channel on FreeNode I found out it was a problem with lighttpd 1.4.21’s handling of sendfile() on FreeBSD.

The fix is now also in the lighttpd port of FreeBSD, so other people should not encounter this problem.

Discovered today that with both Firefox 3.0 and Opera 9.63 on FreeBSD, TinyMCE within Wordpress 2.7.1 is not allowing me to use the visual editing mode. I tried the example of TinyMCE and it works without problems. Based on this and the fact it works on Windows, there must be something weird in either Wordpress or its included version of TinyMCE with FreeBSD. I logged a post over at the Wordpress forums.

Marc Balmer (of the OpenBSD Project) investigated reports of weird filesystem behaviour and found a 25-year old bug in the BSD libc implementation of readdir().

The fix should be in the trunk of all BSDs now and scheduled for merges or backports soon (e.g. see http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libc/gen/readdir.c revision 1.15’s diff).

OpenSSH has a fantastic feature called ControlMaster. Basically this option allows you to create a socket that will share your already opened ssh session to the same host. To enable this option for all you put the following snippet in your $HOME/.ssh/config after creating something like $HOME/.ssh/sockets:

Host *
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h:%p

For every username@host:port it will create a socket in $HOME/.ssh/sockets. The only problem is that current Subversion (1.4.6 on my FreeBSD box) cannot work well with control sockets when using the svn+ssh:// URI identifier. In order to work around this problem you can add a specific host before the wildcard entry, for example:

Host svn.example.com
  ControlMaster no

Host *
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h:%p

Of course, doing it like this is a bit tedious for every Subversion repository you use in this manner. Thankfully there is another way to do this. In $HOME/.subversion/config there is a section called [tunnels]. If you add the following entry to that section it will disable the ControlMaster:

[tunnels]
ssh = ssh -o ControlMaster=no

So after yesterday’s post about some compiler results with Python 2.6 I wanted to show how some of GCC’s architecture-specific compiler flags affect the execution of pybench. As I explained in comments I think most people will never even touch the flags passed to Python’s build. Nonetheless, some people asked if I had tuned it in any way. Pádraig Brady had asked me if I had used the optimal GCC architecture flags. On my FreeBSD 7.0-STABLE machine at home (AMD Athlon(tm) 64 X2 Dual Core Processor 4600+ (2411.13-MHz K8-class CPU)) his script stated I had to pass along “-m32 -march=k8 -mfpmath=sse”. My machine is fully 64 bits so I left out the -m32 (since it will not link anyway) and used “-march=k8 -mfpmath=sse” (using -march=native instead of k8 resulted in a 0,1 seconds faster result and -mtune=native -march=native instead of k8 resulted in a 0,1 – 0,2 seconds faster result).

The default option flags are on my system: -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes.

Considering some other comments about how I did not use a 0-origin for my y-axis I have to point out two things: firstly, given the sometimes close results zooming out too much can eliminate detailed information (of course you have to be careful not to zoom in too much as well); secondly, I like to make sure the graph itself is appropriately centered so you do not get a whitespace skewing in the resulting image. I think, being a follower of the Edward Tufte school of graphic displaying, I did reasonably well. The graphs were made with a tool called Ploticus.

I was curious how the optimization level influenced the resulting program and as such I removed the -O3 option from the compiler flags. As is evident from the graph you are looking at a bit more than a doubling of execution time (an average of 14,2 seconds versus the previous 6,6 and 6,5 seconds).

So, given the huge performance hit by merely leaving out the -O3, I was interested how the other optimization levels worked out. Holger Hoffstätte asked to use -O2 -fomit-frame-pointer instead of -O3. Basically the results of -O3 (average of 6,5 seconds) and -O2 -fomit-frame-pointer (average of 6,5 seconds) were equal. The result of using -O1 (I could not really discern much of a speed difference by adding -fomit-frame-pointer, also for the -O2 case it was still an average of 6,5 seconds) was quite interesting. It already improves execution by ~86%. From -O1 to -O2/-O3 we are looking at another increase of ~16%. From the no optimization case to -O2/-O3 execution improves by ~118%

I tried a profile-guided optimization build, but I have some issues on my FreeBSD 7.0-STABLE with libgcov. Apparently only a libgconv.a is provided and linking gives me a relocation warning. Thankfully I also had a GCC 4.2.4 snapshot from March installed and did a PGO build, but I managed to only shave of about 0,2 seconds on the average time.

Due to recent concerns with memory use and execution speed I was curious how Python would behave with different compilers. I took Python 2.6a2 r62288 from the Subversion repository and compiled it with the flags: –with-threads –enable-unicode=ucs4 –enable-ipv6. The machine is a HP dc7700p with 1GB memory with an Intel Core2 6300 @ 1.86GHz running Ubuntu 7.10. I installed GCC 3.3.6, 3.4.6, 4.1.3, 4.2.1 from the Gutsy repository, and Intel 10.1.015. The MS Visual Studio 2008 Python was the MSI snapshot of 2008-04-10 from the main Python site. I ran this through Wine 0.9.46 after installing the VC2008 runtime.

First various GCC versions: 3.3.6, 3.4.6, 4.1.3, 4.2.1:

It is good to see that the 3.4 series is faster than the 3.3 series and the 4.2 series is faster than the 4.1 series. I am a bit worried about the 4.1 series drop in performance compared to the 3 series though.

Next we have Python compiled with GCC 3.4.6, 4.2.1, Intel CC 10.1.015, MSC from Visual Studio 2008:

It is nice to see how the Microsoft Visual Studio 2008 compiler produces a binary that, when run through Wine, still performs quite well compared to GCC. I am not quite sure if Wine incurs a performance penalty or not. What’s quite impressive is the performance of the Intel CC compiled Python. If we take the fastest GCC, which is 4.2.1 at the moment, take the average of the 10 rounds of execution, which is 6,574 seconds, and compare that to the average of ICC, which is 5,412 seconds, we see that ICC is about 21% faster. If we take the slowest, GCC 4.1.3 with an average of 7,002 seconds, we even get a result that ICC is about 29% faster.

So it seems for people who want to get the full performance out of Python compiling with ICC might be quite beneficial. I want to check out how ICC progressed from version 8 to version 10 performance-wise.

The raw data can be found at http://www.in-nomine.org/~asmodai/python-pybench.txt.