Hoffstadter's Law and Jython 2.5

Hoffstadter's Law states: "It always takes longer than you think, even when you consider Hoffstadter's Law."

I thought by now I'd have a Jython 2.5 release, but alas I do not. I keep wanting to get just one more feature in, but I think the time has come to freeze things and get to release candidates. Most of 2.5 is quite solid, and it's time to get it out. I plan to put together what I think will be the last beta for Jython 2.5 this Friday.

Some things are not going to make it in, for example I don't think ctypes will make it for 2.5.0. I looked over what we have so far and what I think it would take me to implement it. I don't feel confident that I can really get it together as quickly as I would like and still have it be solid. My C skills are just a little too rusty to whip it together quickly. I may make ctypes my personal sprint target for PyCon 2009 since there will be lots of ctypes knowledge floating around, particularly from the PyPy folks. Jim Baker has started a ctypes implementation which uses the PyPy work.

I'm going to try to get modjy integrated into trunk, but if I don't make it by this Friday it will just have to stay separate for this release. Jythonc work has moved to the "clamp" project that Charlie Groves started on github. Hopefully we can get that together at the same time or shortly after 2.5.0 releases.

If any committers have something they would like to get in at the last moment, now would be the time!

Progress with Mercurial on Jython: part 2

So yesterday's foray into Mercurial's test engine started with much excitement, but the excitement was short-lived. It turns out that the Mercurial tests are mostly shell scripts, and some blundering on my part permitted some CPython to bleed through and run some of the tests (so they appeared to succeed, but did not).

Since I really want to at least assess the scope of getting Mercurial to run on Jython, and it really is a bit closer, I decided to start reading the Mercurial book. At the same time, I am gearing up to play around with the Da Vinci Machine project, which is hosted on a Mercurial repo. So I'm doubly motivated to get a good understanding of hg. So, going along with the book, I first tried "hg version" (I have a local hg that points to Jython -- and I am quite sure it points to Jython this time). Woohoo! -- version works (you have to start somewhere...):

[frank@pacman ~]$ hg version
Mercurial Distributed SCM (version b8d750daadde+20090218)

Copyright (C) 2005-2008 Matt Mackall  and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Next I find that "hg help" works:

[frank@pacman ~]$ hg help init
hg init [-e CMD] [--remotecmd CMD] [DEST]

create a new repository in the given directory

  Initialize a new repository in the given directory.  If the given
  directory does not exist, it is created.

  If no directory is given, the current directory is used.

  It is possible to specify an ssh:// URL as the destination.
  Look at the help text for the pull command for important details
  about ssh:// URLs.

options:

-e --ssh        specify ssh command to use
  --remotecmd  specify hg command to run on the remote side

use "hg -v help init" to show global options

The next bit in the book uses "hg clone". Unfortunately "hg clone" uses buffer(), which Jython lacks. Jython has never supported the buffer() protocol from CPython, and almost certainly never will. For one thing, buffer() is very C-oriented, and buffer() did not make the cut into Python 3.0, so implementing buffer makes little sense for us. We may very well look into the buffer() replacements defined for 3.0, but that is something for another time. So I used regular hg to execute the clone.

Next "hg log" revealed a bug in Jython that was pretty easy to fix: we where
not handling writes past the end of the buffer in our cStringIO. That's a nice benefit of trying out larger apps on Jython -- they are good exercise :)

And so now "hg log" runs:

changeset:   4:2278160e78d4
tag:         tip
user:        Bryan O'Sullivan 
date:        Sat Aug 16 22:16:53 2008 +0200
summary:     Trim comments.

changeset:   3:0272e0d5a517
user:        Bryan O'Sullivan 
date:        Sat Aug 16 22:08:02 2008 +0200
summary:     Get make to generate the final binary from a .o file.

changeset:   2:fef857204a0c
user:        Bryan O'Sullivan 
date:        Sat Aug 16 22:05:04 2008 +0200
summary:     Introduce a typo into hello.c.

changeset:   1:82e55d328c8c
user:        mpm@selenic.com
date:        Fri Aug 26 01:21:28 2005 -0700
summary:     Create a makefile

changeset:   0:0a04b987be5a
user:        mpm@selenic.com
date:        Fri Aug 26 01:20:50 2005 -0700
summary:     Create a standard "hello, world" program

So Jython's mercurial story isn't quite as good as I thought it was yesterday, but it is certainly much better than it was in 2007. If you are interested in Mercurial, I am finding the book to be extremely well written and easy to read, again you can find it here.

Progress with Mercurial on Jython

Back in 2007, Charles Nutter posted a suggestion: get Mercurial (also called hg) running on Jython. At the time I downloaded hg to give it a try. This was before Jython 2.2 was released, and it did not go well. Mercurial would not even compile, and if you know much about compiling Jython or Python, you might know that getting something to compile is about 1/4 of the battle.

Much has happened in Jython since 2007, 2.2 is out, and we are a getting very close to a release of 2.5. Recently, Dirkjan Ochtman has been making changes to Mercurial to better support other implementations of Python . For example, the parts of Mercurial that where implemented in C have pure Python fallbacks now [Update: Dirkjan tells me that the specific work of re-exposing pure Python fallbacks from hg past was done by Martin Geisler]. To avoid performance issues in the default case, these pure python fallbacks need to be compiled in.

To test this out in Jython (you need trunk, or at least beta1), get a copy of the latest mercurial

hg clone http://selenic.com/repo/hg

Go to the hg directory and compile hg with the "pure" option (thanks to mohbana on #jython irc for the recipe used):

jython setup.py --pure build_py -c -d . build_ext -i build_mo

[Update: the patch below was applied by Dirkjan Ochtman last night with the comment "use same popen strategy for jython as for nt"]
then cd to the "tests" directory and (for now) apply this change to run-tests.py and run

jython run-tests.py --pure

This should start the hg test harness. There are *lots* of failures, but also plenty of heart warming success dots.
[Update: at least some of these success dots are leaks to CPython... I'll do another weblog post when I know more]

mohbana (again from #jython irc) says that he has had success with "hg init", "hg tip", and "hg add".

We are getting closer!