Category Archives: Python

Mercurial, python keyring 3, and mercurial keyring

In an earlier post I documented how to set up an encrypted file store for your keyring. With recent versions of Python keyring (at least 3 and up) the CryptedFileKeyring backend got removed and replaced by EncryptedKeyring. So in your $HOME/.local/share/python_keyring/keyringrc.cfg you need to now have the following:

[backend]
default-keyring=keyring.backends.file.EncryptedKeyring

PyCharm, Mercurial and keyrings

If PyCharm complains that it Can't start Mercurial: /usr/bin/hg Probably the path to hg executable is not valid, then check if running hg from the command line triggers a problem running a certain extension. In my case I had a version of keyring and mercurial_keyring that did not play nice with each other. After upgrading these to 3.0.5 and 0.6.0 respectively, the problem went away. I guess PyCharm tests the run of the hg binary and if the shell return code (echo $?) is something other than 0 will show this warning.

Selenium, Chromedriver2, and SSL pages

If you are doing Selenium testing using Chromedriver2 0.8 and are having problems with self-signed SSL certificates, this is a known problem and will be fixed in a subsequent release. In the meantime I found that using the Chromedriver 26.0.1383.0 still worked without problems for Chrome 27 and also did not have this SSL certificate regression in it.

Mercurial and safely storing passwords

Mercurial allows for tying in keyring configuration for those of us who do not want to store passwords in plain-text in our .hgrc files or constantly using SSH.

First install the Python keyring library by running pip install keyring. After that is installed, checkout https://bitbucket.org/Mekk/mercurial_keyring/ and add to $HOME/.hgrc the following:

[extensions]
mercurial_keyring = ~/path/to/mercurial_keyring/mercurial_keyring.py

Next up, configure your repositories, e.g. in the case of Bitbucket I use:

[auth]
bitbucket.prefix = bitbucket.org/asmodai
bitbucket.username = asmodai
bitbucket.schemes = https

Mercurial keyring will automatically decide on the best keyring to use. On a FreeBSD system with no Gnome or other systems providing a keyring, if you do not specify a specific keyring, the system will use the file ~/.local/share/python_keyring/keyring_pass.cfg. This keyring file stores the passwords encoded in Base64 in plain-text. This is not quite what you would want from a security point of view. You can configure which backend store to use by editing $HOME/.local/share/python-keyring/keyringrc.cfg. To get a plain-text file with encrypted keys use the following configuration:

[backend]
default-keyring=keyring.backend.CryptedFileKeyring

This will create the file ~/.local/share/python-keyring/crypted_pass.cfg after initializing the backend store with a password. Look at the documentation for keyring on what other configuration options are available.

Note: make sure the PyCrypto dependency is installed with the _fastmath module. This in turn depends on the gmp library.

PyCharm and external lint tools

PyCharm already has a number of features present in various tools to lint/check your source code with, but offers a way to hook up external tools. Under File > Settings is a section called IDE Settings. One of the headings here is called External Tools. Select this heading and then press the Add... button on the right hand pane to configure a new external tool.

In the Edit Tool window that now appeared fill in a name, e.g. PEP8 and a group name Lint and add a description. Next point the Program to the location of the pep8.exe executable, e.g. C:\Python27\Scripts\pep8.exe. For Parameters you need to use $FilePath and Working directory should be filled in by default. Once done, you can close it by pressing the OK button.

Now, pyflakes has no .exe or .bat file to accompany it. You will need to add a pyflakes.bat in your Scripts directory inside Python with the following contents:

@echo off
rem Use python to execute the python script having the same name as this batch
rem file, but without any extension, located in the same directory as this
rem batch file
python "%~dpn0" %*

Within PyCharm you follow largely the same settings as for pep8, however make sure to point to the batch file of pyflakes under Program. Close the external tools configuration windows by clicking OK twice. Under the menu heading Tools you should see an submenu heading Lint which, in turn, should contain two menu items: PEP8 and Pyflakes.

Now open a Python file, go to Tools > Lint > PEP8 and you should get output like the following in your Run (4) window:

D:\Python26\Scripts\pep8.exe D:\pprojects\babel\babel\tests\__init__.py
D:\pprojects\babel\babel\tests\__init__.py:16:1: E302 expected 2 blank lines, found 1

Process finished with exit code 1

JSONP with Werkzeug

So I had implemented a simple JSON data server with Werkzeug for a classroom experiment. Unfortunately in my haste to get everything up and running I totally forgot about the fact that, since we cannot allow uploads to this server of various custom made webpages, using jQuery’s $.ajax() everything just fails since it will then be a cross-site scripting request.

So, normally you would do something like the following in order to return JSON data:

return json.dumps(data)

Which would be used with the $.ajax() call in a way like the following:

$.ajax({
  type: "POST",
  url: "http://example.com/json/something",
  data: "parameter=value",
  dataType: "json",
  error: function(XMLHttpRequest, textStatus, errorThrown){},
  success: function(data, msg){}
});

Which is perfectly fine for scripts getting and using the data on the same host/domain. But, as said before, this will fail with warnings similar to: "Access to restricted URI denied" code: "1012" nsresult: "0xdeadc0de (NS_ERROR_DOM_BAD_URI)".

One way out of this is using JSONP. jQuery has a $.getJSON() function, which loads JSON data using a HTTP GET request. Now, the simplistic way to convert your code would be to change it as such:

$.getJSON("http://example.com/json/something",
  function(data){}
);

But this causes another issue. Since $.getJSON() GETs the JSON data, but doesn’t use eval() on it, but instead pulls the result into script tags, it somehow causes,on Firefox at least, an invalid label error. In order to fix this you need to set up the JSON data server to properly support a callback argument, to use $.getJSON() how it is meant to be used:

$.getJSON("http://example.com/json/something?jsoncallback=?",
  function(data){}
);

In the code above the additional parameter jsoncallback will, thanks to jQuery, get the question mark replaced by an alphanumeric string (typically in the form of jsonp followed by a timestamp). This value should be used to wrap the resulting JSON data with. This means you would have to change the initial Python code to something like this:

return request.args.get('jsoncallback') + '(' + json.dumps(data) + ')'

Of course this causes problems when you want to reuse the code for both AJAX use on the same host/domain and use it from outside. So in order to make both work you can test on whether or not the callback parameter is available and return the appropriate data. I came up with this little snippet for that:

def jsonwrapper(self, request, data):
    callback = request.args.get('jsoncallback')
 
    if callback:
        return callback + '(' + json.dumps(data) + ')'
    else:
        return json.dumps(data)

Python 2.6 compiler options results

So after yesterday’s post about some compiler results with Python 2.6 I wanted to show how some of GCC’s architecture-specific compiler flags affect the execution of pybench. As I explained in comments I think most people will never even touch the flags passed to Python’s build. Nonetheless, some people asked if I had tuned it in any way. Pádraig Brady had asked me if I had used the optimal GCC architecture flags. On my FreeBSD 7.0-STABLE machine at home (AMD Athlon(tm) 64 X2 Dual Core Processor 4600+ (2411.13-MHz K8-class CPU)) his script stated I had to pass along “-m32 -march=k8 -mfpmath=sse”. My machine is fully 64 bits so I left out the -m32 (since it will not link anyway) and used “-march=k8 -mfpmath=sse” (using -march=native instead of k8 resulted in a 0,1 seconds faster result and -mtune=native -march=native instead of k8 resulted in a 0,1 – 0,2 seconds faster result).

The default option flags are on my system: -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes.

Considering some other comments about how I did not use a 0-origin for my y-axis I have to point out two things: firstly, given the sometimes close results zooming out too much can eliminate detailed information (of course you have to be careful not to zoom in too much as well); secondly, I like to make sure the graph itself is appropriately centered so you do not get a whitespace skewing in the resulting image. I think, being a follower of the Edward Tufte school of graphic displaying, I did reasonably well. The graphs were made with a tool called Ploticus.

GCC 4.2.1 default and architecture options

I was curious how the optimization level influenced the resulting program and as such I removed the -O3 option from the compiler flags. As is evident from the graph you are looking at a bit more than a doubling of execution time (an average of 14,2 seconds versus the previous 6,6 and 6,5 seconds).

GCC 4.2.1 with no O3

So, given the huge performance hit by merely leaving out the -O3, I was interested how the other optimization levels worked out. Holger Hoffstätte asked to use -O2 -fomit-frame-pointer instead of -O3. Basically the results of -O3 (average of 6,5 seconds) and -O2 -fomit-frame-pointer (average of 6,5 seconds) were equal. The result of using -O1 (I could not really discern much of a speed difference by adding -fomit-frame-pointer, also for the -O2 case it was still an average of 6,5 seconds) was quite interesting. It already improves execution by ~86%. From -O1 to -O2/-O3 we are looking at another increase of ~16%. From the no optimization case to -O2/-O3 execution improves by ~118%

GCC 4.2.1 with various options

I tried a profile-guided optimization build, but I have some issues on my FreeBSD 7.0-STABLE with libgcov. Apparently only a libgconv.a is provided and linking gives me a relocation warning. Thankfully I also had a GCC 4.2.4 snapshot from March installed and did a PGO build, but I managed to only shave of about 0,2 seconds on the average time.