Now that I have a reasonable parser in Antlr, I'm back to thinking about the compiler again. Currently Jython has a hand crafted bytecode emitter. It was built in a generic fashion and is actually a very nice implementation. For an earlier Jython this was a necessity, because there where no good open source frameworks out there that could be used for bytecode emission. These days, there are a number of good open source frameworks for this task. ASM stands out as a particularly good one if you are looking for a small, fast, and well documented framework with a vibrant community of users and developers. The advantage of using a framework like ASM over a hand rolled implementation is that the developers of Jython can let the ASM folks worry about new JVM features like annotations, backwards compatibility issues, etc, leaving Jython developers more time to make Jython a better implementation of Python.
A few months ago I did a one for one translation of the current compiler into ASM and found that this was a straightforward (if a bit tedious) task. This translated compiler lives in the svn branch asm_compiler. The asm_compiler branch's build.xml uses jarjar to bundle the asm libraries together into a jython-complete.jar. I called it jython-complete mirroring the jruby project's jruby-complete.jar vs a jruby.jar that requires dependencies. I think Jython will eventually follow suit with one jar that bundles everything together and another that does not. asm_compiler currently passes all but four of the regression tests that trunk passes on my machine.
Since I did not try to go beyond the current functionality of 2.3, this is just a proof of concept step. The next steps will be to start re-writing the "newcompiler" work in Java. The newcompiler work which sits in the svn trunk and sandbox prototypes an improved compiler that implements 2.5. newcompiler is the impressive outcome of a Google Summer of Code project by Tobias Ivarsson mentored by Jim Baker. Having said that I'd still like to see if I can make asm_compiler the compiler of Jython's trunk -- it may work better to incrementally move to newcompiler compared to trying to replace the current strategy all at once.
-
The majority of my Jython time for the last couple of months has been spent working on a new parser for Jython. This parser is being written in Antlr3 and lives in the Jython sandbox.
CPython 2.5 introduced a new library called _ast that allows direct access to its internal parser. Jim Baker wrote some python that pretty prints the _ast results, and I extended it to do a diff with the output of the new parser. The current results of this comparison with all of the .py files in CPython 2.5's Lib directory is a perfect match for just over 75% of these files (there are 1230 of them). The majority of the remaining problems are mismatches in the representation of floats and unicode, although there are plenty of real bugs remaining, especially in the lexer pass. The sandbox project has a README that lists the bugs that I know about.
There are two grammars, a combined lexer/parser in grammar/Python.g that emits a simple parse tree which is a reasonable representation of the code in an AST form. The real work occurs in the tree grammar grammar/PythonWalker.g. This one walks the output of Python.g and emits a tree of Java classes that will later be consumed by a future Jython compiler. PythonWalker does a fair amount of processing on the parse tree to shape the AST so that it matches _ast.
The code for the nodes that PythonWalker.g emits is also generated. This is done using asdl_antlr.py. asdl_antlr.py uses the same core code that CPython uses to create their parser, which is copied into the Jython sandbox (asdl.py and spark.py).
Plenty of work remains. First I will need to get the ASTs to match better -- I'm hoping to at least get a 100% match for the Lib of CPython 2.5. Once that is done Python.g and PythonWalker.g are going to need a cleanup pass -- there is plenty of kludginess in there to get it up and running. And of course then there is the compiler, but that is a topic for another day.1View comments
-
Work on Jython since 2.2 has been proceeding at a blistering pace. Commits are coming in so fast that I have been having trouble keeping up at times. Recently some Jython folks got together with some Django folks and together figured out what was missing. I participated in the early work (and I still have a mysql backend that I need to contribute to the effort) but then I got too caught up in rewriting Jython's parser in antlr and so lost track. More on that another time :).
Jim Baker has a nice write up of Django on Jython and the news was picked up by The Server Side.
This is just the beginning -- we are also looking at Pylons, TurboGears, and much much more. Exciting times.4View comments
View comments