Pugs

KindaPerl6 now runs on sbcl, clisp and ecl

I've been meaning to do something about testing for the kp6-lisp backend. Previously we compiled all tests to stand-alone executables using sbcl, the reason for this being that I didn't know about --noinform --noprint at the time and dumping the lisp image was the hacky solution to making sbcl not print out all that superfluous output.

This meant we had to wait for sbcl to compile each test before we could run any, and each test produces an approximately 30MB executable which meant the test directory grew to several hundred megabytes.

Everything is much better now, you can select the backend you want when writing the makefile:

KP6_TARGET=KP6-LISP-sbcl perl Makefile.PL

or

KP6_TARGET=KP6-LISP-clisp perl Makefile.PL

or

KP6_TARGET=KP6-LISP-ecl perl Makefile.PL

And then run the tests on the selected backend:

make test

All the backends fail the same 56 tests out of a total of 85 so it looks like the backend code is pretty portable. There's a significant speed difference between the implementations, since most of their time is taken up by compilation this makes a lousy general benchmark but testing uses clisp is the best of the three:

ecl: real 2m3.895s
clisp: real 1m7.318s
sbcl: 4m16.725s

Things could be made a lot faster for all of them by only compiling the backend runtime once instead of doing it for every test, or running lisp as it's supposed to be run, in one persistent compiler.

That and adding options to kp6(1) for running code directly with a given runtime is a task for another day.

KindaPerl6 Common Lisp status update

A lot of work has been done on the KindaPerl6 Common Lisp backend for KindaPerl6 since I last 89wrote about it. Variables, subs, lexicals, closures, basic datatypes (int, str, num, hash, array, ..), control structures (if, for, while, ..) are working. We're missing user-defined classes & methods, some internal methods (sort, map, grep, ...), junctions, pairs, gather and probably some other things. Aankhen++ has been doing most of the work with the occasional contribution from myself.

Things sped up a bit recently after fax++ wrote a meta-object model (MOP) for us based on CLOS. The one in the Perl 5 backend does not use the Perl 5 object system (implements its own @ISA and so on) but so far it looks like we can build the Perl 6 MOP on top of CLOS which is more flexible.

A method in the MOP uses the same calling convention regardless of whether it's internal to the compiler or compiled from a user defined class. An example of an internal method using the new calling convention is the dispatch method that implements the .elems method for hashes:

(defmethod kp6-dispatch ((invocant kp6-Hash) interpreter (method (eql :elems)) &rest parameters)
  "Returns the number of elements in the hash"
  (declare (ignore parameters interpreter))
  (make-instance 'kp6-Int
     :value (hash-table-count (slot-value invocant 'value))))

In the case of %hash.elems this would be called as:

(kp6-dispatch (lookup-lexical-variable (kp6-generate-variable "%" "hash")) |Main| :elems)

The object system is used for pretty much everything. For example the .Str method is used to get the string representation of an object, .Int for its integer form, .true to check whether it wants to be true (used in all boolean contexts) etc. Not everything has been moved to the new system internally. Helping with that would be an easy beginner task.

Speaking of beginner tasks there's a lot more of them. And since we have MOP now implementing things is often just a matter of seeing how the Perl 5 backend does things and filling in the gaps in the Lisp backend.

We have some bugs like my @a; @a[123] = 456; which causes an error currently since the underlying CL array isn't extended properly. As well as a lot of things like substr() which are easy to implement but simply aren't at the moment.

So if you're interested in hacking a Perl 6 compiler in Perl 6 and Common Lisp head over to #perl6 and poke us for a commit bit.

KindaPerl6 released on CPAN

One of the things I've been working on in KindaPerl6 is making it CPAN
ready, first by rewriting its custom Makefile as a Makefile.PL and
then by making various small fixes. Yesterday I wrote a unified
frontend to it (kp6(1)) and made `make install' work and released in
on CPAN as version 0.001.

Since KindaPerl6 is still in active and unstable development `make
test' does not run successfully at the moment. So the CPAN install has
to be forced:

    cpan -f -i KindaPerl6

That should install KindaPerl6 along with its kp6 frontend, the hello
world example is then:

    $ echo 'say "hello" ~ "world"' | kp6 | PERL5LIB=$(kp6 -lib) perl
    helloworld

The PERL5LIB part is required so that the Perl 5 backend can find its
runtime libraries.

If you want to check out the AST of the program just do:

    $ echo 'say "hello" ~ "world"' | kp6 -ast | perltidy

And to try another backend such as the Common Lisp one try one of the
emitter options:

    $ echo 'say "hello" ~ "world"' | kp6 -lisp

There's no user friendly way to run the generated lisp code yet but
hopefully future releases will allow for passing options to specific
emitters. I'm hoping to make a user-friendly way to generate
stand-alone executables with sbcl from the command line for instance.

Playing with the new "Not Quite Perl"

Today I tried a new product of the Perl 6 project: Not Quite Perl.  "NPQ" is a subset of the Perl 6 syntax. It runs on Parrot.

To put this in context, Pugs has been the most feature-complete implementation of Perl 6, but has suffered from poor performance and lack of Haskell-enabled contributors.

KindaPerl6 currently supports a subset of Perl6, and is executed with Perl5, but also has slow performance now.

"Not Quite Perl" is a similar effort, using Parrot to execute the code. Parrot is expected to be the fastest performing implementation of Perl 6.  Right now "KindaPerl6" and "Not Quite Perl" are developing separately, but there have been discussions to coordinate, according to fglock, a KindaPerl6 contributor.

Here's what I found tried when I tried both technologies today.

YAPC-SA-2007 Hackathon

Here is the report on the 4 days of the YAPC-SA-2007 Hackathon, from April 12 to 15, 2007, in Porto Alegre, Brazil (during fisl8.0).

v6/docs
- added the KindaPerl6 diagram
- added a KindaPerl6 memory layout diagram - with help from #perl6
- added the MiniPerl6 talk (pdf)

MiniPerl6
- started the Perl6-in-Parrot target: desugars MiniPerl6 into p6parrot supported features
- grammar fix: semicolon is optional after a block
- perl5 backend: implemented warn()
- new: mp6-to-AST script
- nferraz started porting Test.pm to mp6

KindaPerl6
- memory management: BEGIN-block side effects are logged
- updated plan, discussed a possible release schedule
- fixed the container proxy algorithm (the implementation needs to be fixed; pugs has the same problem)

v6.pm
- fixed the "fish-operator" =<>
- fixed the quoted-word-operator <...>

Pugs::Compiler::Rule
- ratchet emitter fix, array interpolation now works

misc/pX
- nferraz and fglock started an adventure game implementation (with v6.pm), to be presented by nferraz at the Nordic Perl Workshop.

The network proxy at the conference did not allow commits to pass through, and there will be some delay until all the changes get to the repository.

Thanks to cmarcelo, david_fetter, eden_c, lorn, merlyn, and nferraz for 4 great days!

- Flavio S. Glock (fglock)

Val migration

The primary difficulty in moving over to the new internals are that it's hard to make a huge all-encompassing change in a project. We're not just changing the internal representation of strings from one form to another. Encapsulation and type inference would make that task certainly manageable. The present change affects virtually all Perl types, obsoletes the widely-used VRef, and changes the way AST expressions are strung together.

Another problem is that the new AST is much more detailed than the old one, and putting all its definitions in one file makes it hard to maintain/understand all at once. Also, although this AST definition should be shared by all Perl 6 implementations, the native value representations are backend-specific.

Armed with techniques from this paper, we can address both issues at the same time. Instead of a closed datatype for Val (starting from values instead of expressions makes migration easier), we make use of typeclasses where appropriate. A new (:>:) class lets us stipulate subset relationships cleanly, so a single cast method convers (old) Strings to (new) ByteStrings, but also a value to its Id when in Id context.

The work is going along nicely. We've added a VV node type in the old AST, and already we can construct newstyle values from pugs by casting them with the interim vv prim. This helps to keep us honest, as our code comes under constant scrutiny of the type checker and actually gets to be executed.

pugs> vv "Moose"
VPure (MkStr "Moose")      # look, not VStr!
pugs> vv 42
VPure (IFinite 42)
pugs> vv 3.1415
VPure (NRational (6283%2000))
pugs> vv "22"/7
VPure (NDouble 3.142857142857143)

Hs -> P6

When I was giving a talk at YAPC, someone asked me if a Haskell data type is like a blessed object in Perl. I said that they were profoundly different. Well, that was probably too offputting: a Haskell datatype can be represented as a Perl 6 Role, and its variants as different Classes.

To facilitate the Perl 6-on-Perl 6 efforts, Audrey asked me to write a general automatic DrIFT-based Hs->P6 converter. The first use is to put the new AST scheme in a language where data can be more readily manipulated. For example, here is a function parameter in both Haskell and Perl 6:

-- Haskell
data Param = MkParam
    { p_variable    :: Ident
    , p_types       :: [Class]
    , p_constraints :: [Code]
    , p_unpacking   :: Maybe Sig
    , p_default     :: Maybe Exp
    , p_label       :: Ident
    , p_slots       :: Table
    , p_hasAccess   :: ParamAccess
    , p_isRef       :: Bool
    , p_isLazy      :: Bool
    }

# Perl 6
role Param is TaggedUnion;

class MkParam does Param {
    has Ident $.p_variable;
    has Class @.p_types;
    has Code @.p_constraints;
    has Sig $.p_unpacking;
    has Exp $.p_default;
    has Ident $.p_label;
    has Table $.p_slots;
    has ParamAccess $.p_hasAccess;
    has Bool $.p_isRef;
    has Bool $.p_isLazy;
};

Here is the latest AST translation (and where that came from). This is a step towards assuring that Pugs' Perl 6 AST can be shared by other implementations, and (for example) macros that manipulate parsed bits of Perl can work on more than one compiler.

So far this was much easier to do than our YAML class (that bridges why++'s libsyck with Haskell for any data). But the next step is figuring out how to dump actual arbitrary Perl 6 values...

use v6-pugs;

now invokes the new Perl5-based Perl6 compiler:

use v6-pugs;
"hello, world".say;

$ perl -Ilib hello_world.pl

It can even be called from the command line:

$ perl -Ilib lib/v6.pm -e ' "hello, world".say; '

It still fails many of the sanity tests, as well as nearly all of the other tests under t/.

DrIFTing YAML.

Three more backends joined Pugs today:

• "./pugs -CPIL1-YAML" dumps the PIL1 tree using gaal's Syck emitter binding.
• "./pugs -CPIR-YAML" serializes the PIR syntax tree generated from PIL1.
• "./pugs -CParse-YAML" does the same for the parse tree itself, upon theorbtwo's request.

They are made possible thanks to the DrIFT technology. It lets us specify how to convert any Haskell structure into YAML nodes, with a simple rule file and some base instances.  From there on, say if we'd like to add various serialization for PIL^N native types, all it takes is a single pseudo-comment in the .hs-drift file:

{-! global : YAML, Perl5, JSON, HaskellXml, ... !-}

Debugging PIL1, PIR and the parse tree is made much simpler with this effective visualization tool. I've also released a new YAML::Syck that can load Haskell structures into blessed Perl5 objects, although Ingy's YAML.pm 0.50 also does this in a more elegant fashion, without involving Class::Rebless.

Loading from YAML to arbitrary Haskell structures should also be straightforward; now we just need a Syck emitter binding at Parrot's side, and we can shuffle Parrot objects (e.g. PGE Match objects or TGE trees) with Pugs's compiler backend across separate compilation units. That should be fun...

PIR compilation lives again!

After a 18-hours hacking run, I fixed all three blocking issues today.

Thanks to generous help from Leo and #parrot folks, Pugs now generates correct PIR; the parrotBrokenXXX global flag we used to dodge around register alligators is finally gone. Hurray for a new dawn of non-scheduled Parrot!

The lexical pad spec proved to be quite sane and easy to target, except for the somewhat irritating demand of having to manually initialize lambda lifted closures, even for package-scoped subroutines. For example, this Perl 6 code:

my $x;
sub f { $x++ }
f();

becomes this PIR code (you can try it out with "./pugs -CPIR test.p6"):

.sub "main" :anon
    .local pmc s__x
    .lex "$x", s__x
    $P8 = find_name "&f"
    $P8 = newclosure $P8
    store_global "main", "&f", $P8
    # ...
.end
.sub "&f" :outer("main")
    # ...
.end

Running newclosure for each subroutine at runtime is quite silly, because its outer subroutine "main" cannot be entered again. Moreover, it prevents code in BEGIN and other packages from calling &f using fully qualified syntax.

It would be nice if the newclosure can be performed at compile time, e.g.:

.sub "main" :anon
    .local pmc s__x
    .lex "$x", s__x
.end
.sub "&f" :outer("main") :closure
    # ...&f is initialized  automatically to see "$x"
.end

 

Leo suggested me to write this request up and post it to p6i, so here it is. :-)