Posts Tagged ‘ast’
This is a summary
The soft deadline has passed and the hard deadline is not far away. Soon, what is likely to be my last GSoC will be over. And it was great! These three years I’ve had the summer job of my dreams. I worked on projects I was passionate about, using tools I liked and with people I liked.
I’ve learned a lot about many aspects of software development. Before GSoC, I had never worked with any version control system. I had never made a software release. I had never written any reasonably large program. I had never had to work with existing code, and make compromises for backwards compatibility or stability. I had never contributed patches, or otherwise coordinated with others.
I made many friends.
Missing a beat
For this project, I ended up having less time than I’d thought I would initially. This was partly my fault, and partly the fault of “the elements”, like GSoC midway being late, UK uni ending too late and me not having any other source of income. The end result was that for the past few weeks, I had another job for 2 days/week, with the rest left for GSoC.
I chose to use an existing Python parser (the ‘ast’ module). That was a great time-saver. I managed to write a very basic compiler in a day.
I chose Winxed for the bits I couldn’t write in Python. Winxed is by far the best low-level Parrot language. Even outside the context of Parrot, it’s a decent language.
I chose to focus on the object system, partly because it’s what Pynie’s missing, partly because it was more challenging and interesting. I wrote a guest object system in Winxed based on Parrot’s Object/Class that behaves just like CPython’s. There are a few differences (mainly that my object system is more flexible), but they wouldn’t affect Python’s semantics. It is also, obviously, very incomplete. For example, while I have full support for classes and metaclasses, you can’t substract numbers (you can only add them).
I never had the delusion that I would be able to implement a significant part of Python. All throughout the project, I worked on a prototype, a subset, an incomplete implementation. Having realistic expectations helped.
Parrot works beautifully on ARM(el). I’ve been developing on an Efika MX Netbook for the past few weeks, and I’ve yet to encounter any ARM-specific issues.
I only used a ready-made parser. I should have forked an entire compiler even if not quite pure-python. PyPy’s compiler would’ve been a good choice, since RPython is a strict subset of Python.
I should have targeted Python 2. There would’ve been more compilers to choose from (in particular PyPy’s). Switching to Python 3 afterwards wouldn’t have been hard, and I wouldn’t bother implementing old-style classes anyway.
Perhaps I should have used 6model. From what I’ve seen of it so far, it’d be much more suitable. It’s much, much more extensible than Object/Class. The reason I didn’t use it was the lack of documentation, so perhaps I should have pestered Jonathan more about that.
Perhaps I didn’t focus enough on the compiler. I still don’t think it’s particularly important to write it now (since I consider the object system much more important), but it would most certainly have been a more impressive demo than a bunch of unit tests.
Parrot is a little slow. I know I’m the lasts person to worry about speed, but developing on an 800Mhz ARM machine makes one wish for more. A JIT would be welcome, but almost any optimisations done before that are entirely irrelevant.
Some of this I’ve said before. Parrot’s Object/Class is pretty bad, and the duality with PMCs isn’t helpful. IMCC is terrible. It’s the main reason I gave up on adding features to the compiler. I still don’t know how I could fully control the namespacing mechanism without explicit namespaces/hashes, or the exception system without putting everything in one huge try.
This I’ve also said before: I didn’t work on the project as much as I would’ve wanted. In particular, the project timeline was backloaded: the second half implied much more work than the first half. I hadn’t realised this until quite late.
Parrot has no native, pervasive bool type. This is very annoying and can only promote ugly hacks, like what Winxed and Rosella.Test do, or like I did in my get_integer override. This particular issue also introduced at least one ugly, hard to find, bug in my code.
In general, I’m disillusioned with Parrot. I though it’d be better, but in my opinion the deprecation policy has really held it back.
I didn’t blog enough. I really didn’t, and it was entirely my fault. Sorry.
I hope to keep working on puffin in my spare time. It was fun and interesting and challenging. I would like to get a better compiler and complete the object system. I think it would pay to switch to 6model, especially if NQP becomes optional at some point.
I would like to work on Parrot in the future. M0 and 6model seem of good design. I’d like to try writing an M0 interpreter in Python with PyPy, so I could get a JIT for free.
Thank you whiteknight, NotFound, allison, benabik, jnthn, dukeleto, cotto and anyone else I’m forgetting. You were a real help and I’m happy to have met you.
Did you really think I’d forget to post the quota of puffin pics? Here’s a lovely bunch of puffins!
Finally done with exams, yesterday was my last. Ever, hopefully. So I had time to do some work. You can find it at either http://bitbucket.org/lucian1900/puffin or http://github.com/lucian1900/puffin. I pull from bitbucket, but push to both.
At the moment I am focused entirely on correctness and completeness. I care little about interop with other parrot languages and not at all about performance. I don’t want to waste time on issues that don’t have established solutions on parrot anyway. Instead, I want a correct python implementation on parrot. Interop and performance can be fixed later.
I started writing a compiler using python3’s ast module, generating PIR. I focused on a subset of python that supports int literals, int addition, assignment and printing. I wrote some tests, to check both the PIR output and its execution. This was straightforward, the ast module is very good.
Python’s semantics are almost entirely defined by its object system, so I decided to start implementing it. I looked at whiteknight and NotFound’s experiments with prototype objects (found in rosella/unstable/prototype). I found it unsuitable to directly support Python’s object system, but that it could be a useful source of inspiration.
I don’t want to write assembly and I have almost no experience with Perl, so I decided to use Winxed to implement an object system, on top of Object/Class. Objects are backed by a Hash, for __dict__, which contains all attributes of the object. Both types (classes) and objects (instances) are instances of ‘instance’, a parrot Class. ‘type’ inherits from ‘instance’, since types are also objects in python.
Python objects can be interracted with from Winxed/PIR similarly to how CPython does: “foo.bar” becomes “foo.__class__.__dict__[‘__getattribute__’](foo, ‘bar’)”.
The object system boostraps itself to the point where there is a significant subset of builtins, with almost entirely correct pythonic behaviour: type, object, int, tuple, function, BaseException etc. From there on, in theory, everything can be implemented in pure python.
The compiler doesn’t yet generate code for this object system, since there are a few vital bits missing, such as correctly working methods and metaclasses. Also, I have few tests for the object system, I’d like concentrate on improving that situation.
I had initially also considered 6model, but its unfortunate lack of documentation prevented me from properly evaluating it. After some chats with jnthn and his recent (very useful)docs, I have a much better understanding of 6model, it is in fact quite similar to my object system (but more general). Since Python’s interaction with its objects follows a very clearly defined interface, I believe it’ll be very easy to rewrite the object system from under the compiler, using whatever ends up being the recommended method. Since I’ve already built much of what 6model would offer me, I’ll stick with this (I know, it’s a bit NIH) until 6model gets integrated into parrot better.
Another issue is building&packaging. Packaging for Python(3) is well established, and Parrot has its own distutils. However, since I have code in both Python and a parrot language, the interraction is a bit tricky. The same goes for testing, in fact. I see two possible solutions: 1) write build scripts in python/distutils for building winxed&pir, likely based on allison’s work in pynie/setup.py or 2) write setup.py for python code and setup.winxed for parrot code, and add a command to setup.py that calls setup.winxed. I’m inclined to prefer the second option.
After I’ve tested the builtins in my object system, I’ll start targeting the compiler to use them. Afterwards, I’ll look at supporting more ‘exotic’ features, such as I/O or module importing.
Sadly, exams have proven to take up more time than expected. Today’s exam was ok, easier than I expected. Also easier than previous exams from this lecturer. At least my last exam is on Wednesday, so after that I’m free.
I have however managed to do some investigative work.
I looked at pynie, to determine whether there’s anything worth reusing. As it happens, pynie hasn’t been working for quite a while now. I tried to bring it out of bitrot, but I didn’t have enough knowledge of NQP/PCT and didn’t bother to do more than make it build. Furthermore, PCT isn’t particularly friendly towards python developers.
Allison suggested that I might reuse the tests, but the rest is of little use. The bootstrap tests can’t easily use a python testing tool, since it during boostrap Python code can’t run yet. Py.test might help with that, but I’m not sure I should bother using it (as opposed to unittest).
I had planned to decide this week between using 6model or building my own object model over Parrot’s existing objects. While 6model appears to be able to fully support Python, I’m reluctant to jump in and use it while it’s still an external dependency. I don’t particularly want to pioneer in the usage of 6model on parrot, I’d much rather someone with more 6model experience did that instead.
I’ve started playing with the ‘ast’ module in Python3. I’ve figured out how to walk the AST, now I have to decide what to do with it. One option might be trying to transform it to PAST and letting parrot generate the code, but afaik PAST doesn’t have a textual form that I could target. The other option is linearising the AST and generating PIR, which is likely the one I’ll take unless things change or someone tells me it’s a stupid idea.