Pugs

A graphical tracer for Perl 6 regexes based on PCR

Tracing parse failures by hand while developing a compiler can be really a daunting task, because the real problem can appear everywhere -- the grammar definition, the input string being matched, or even the regex engine itself.

So I've implemented a graphical tracer for Perl 6 regexes atop PCR (Pugs::Compiler::Rule). You can find some small online demos here:

• http://agentzh.org/misc/tracer/digits/index.html (Firefox only)

• http://agentzh.org/misc/tracer/adder/index.html (ditto)

• http://agentzh.org/misc/tracer/langs2/index.html (ditto)

• http://agentzh.org/misc/tracer/grammar/index.html (Tracing PCR's own grammar parser)

To generate the HTML pages (say, the first demo) yourself, simply check out the Pugs repository , ``cd'' into perl5/Pugs-Compiler-Rule, and enter the following commands:

  $ perl util/compile_p6grammar.pl -D \
      examples/digits.grammar > Digits.pm
  $ echo '7c3d54' > digits.input
  $ perl -Ilib -MDigits -e \
     'undef $/; print Digits->count(<>)->(), "\n"' \
     digits.input > trace.out
  $ perl util/gen-tracer-view.pl --outdir tmp/digits \
      examples/digits.grammar digits.input < trace.out
  $ firefox tmp/digits/index.html

As of this writing, the tracer interface still needs love and I'm adding cooler features like ``random jump'', ``stepping in a specified pace'', and ``stepping back''. If you like to help, just join #perl6 and get a Pugs commit bit ;)

I think this tool should be very useful for both regex engine developers and compiler writers, especially when parsing fails in an unexpected way. And it can also be beneficial to Perl 6 beginners who want to learn the shiny new regex syntax and complicated matching behaviors by just ``stepping through'' the actual parsing process. moritz++ said on #perl6 that he would build a CGI script to make my demos above ``alive'' when he had the tuits; let's just wait and see ;)

It should be warned that the regex syntax supported by the current PCR implementation is already a little out-of-date regarding the lastest Synopsis 5 . (Thanks TimToady++ for tweaking the regex syntax during the meantime ;)) I'll try to port KindaPerl6 's perl5rx backend over to PCR later.

Hey, it'll be nicer to have KindaPerl6, PGE , Parse::RecDescent, or even the Perl 5 regex engine to work with my tracer as well! :)

Stay tuned!

-agentzh

KindaPerl6 now runs on sbcl, clisp and ecl

I've been meaning to do something about testing for the kp6-lisp backend. Previously we compiled all tests to stand-alone executables using sbcl, the reason for this being that I didn't know about --noinform --noprint at the time and dumping the lisp image was the hacky solution to making sbcl not print out all that superfluous output.

This meant we had to wait for sbcl to compile each test before we could run any, and each test produces an approximately 30MB executable which meant the test directory grew to several hundred megabytes.

Everything is much better now, you can select the backend you want when writing the makefile:

KP6_TARGET=KP6-LISP-sbcl perl Makefile.PL

or

KP6_TARGET=KP6-LISP-clisp perl Makefile.PL

or

KP6_TARGET=KP6-LISP-ecl perl Makefile.PL

And then run the tests on the selected backend:

make test

All the backends fail the same 56 tests out of a total of 85 so it looks like the backend code is pretty portable. There's a significant speed difference between the implementations, since most of their time is taken up by compilation this makes a lousy general benchmark but testing uses clisp is the best of the three:

ecl: real 2m3.895s
clisp: real 1m7.318s
sbcl: 4m16.725s

Things could be made a lot faster for all of them by only compiling the backend runtime once instead of doing it for every test, or running lisp as it's supposed to be run, in one persistent compiler.

That and adding options to kp6(1) for running code directly with a given runtime is a task for another day.

KindaPerl6 Common Lisp status update

A lot of work has been done on the KindaPerl6 Common Lisp backend for KindaPerl6 since I last 89wrote about it. Variables, subs, lexicals, closures, basic datatypes (int, str, num, hash, array, ..), control structures (if, for, while, ..) are working. We're missing user-defined classes & methods, some internal methods (sort, map, grep, ...), junctions, pairs, gather and probably some other things. Aankhen++ has been doing most of the work with the occasional contribution from myself.

Things sped up a bit recently after fax++ wrote a meta-object model (MOP) for us based on CLOS. The one in the Perl 5 backend does not use the Perl 5 object system (implements its own @ISA and so on) but so far it looks like we can build the Perl 6 MOP on top of CLOS which is more flexible.

A method in the MOP uses the same calling convention regardless of whether it's internal to the compiler or compiled from a user defined class. An example of an internal method using the new calling convention is the dispatch method that implements the .elems method for hashes:

(defmethod kp6-dispatch ((invocant kp6-Hash) interpreter (method (eql :elems)) &rest parameters)
  "Returns the number of elements in the hash"
  (declare (ignore parameters interpreter))
  (make-instance 'kp6-Int
     :value (hash-table-count (slot-value invocant 'value))))

In the case of %hash.elems this would be called as:

(kp6-dispatch (lookup-lexical-variable (kp6-generate-variable "%" "hash")) |Main| :elems)

The object system is used for pretty much everything. For example the .Str method is used to get the string representation of an object, .Int for its integer form, .true to check whether it wants to be true (used in all boolean contexts) etc. Not everything has been moved to the new system internally. Helping with that would be an easy beginner task.

Speaking of beginner tasks there's a lot more of them. And since we have MOP now implementing things is often just a matter of seeing how the Perl 5 backend does things and filling in the gaps in the Lisp backend.

We have some bugs like my @a; @a[123] = 456; which causes an error currently since the underlying CL array isn't extended properly. As well as a lot of things like substr() which are easy to implement but simply aren't at the moment.

So if you're interested in hacking a Perl 6 compiler in Perl 6 and Common Lisp head over to #perl6 and poke us for a commit bit.

KindaPerl6 released on CPAN

One of the things I've been working on in KindaPerl6 is making it CPAN
ready, first by rewriting its custom Makefile as a Makefile.PL and
then by making various small fixes. Yesterday I wrote a unified
frontend to it (kp6(1)) and made `make install' work and released in
on CPAN as version 0.001.

Since KindaPerl6 is still in active and unstable development `make
test' does not run successfully at the moment. So the CPAN install has
to be forced:

    cpan -f -i KindaPerl6

That should install KindaPerl6 along with its kp6 frontend, the hello
world example is then:

    $ echo 'say "hello" ~ "world"' | kp6 | PERL5LIB=$(kp6 -lib) perl
    helloworld

The PERL5LIB part is required so that the Perl 5 backend can find its
runtime libraries.

If you want to check out the AST of the program just do:

    $ echo 'say "hello" ~ "world"' | kp6 -ast | perltidy

And to try another backend such as the Common Lisp one try one of the
emitter options:

    $ echo 'say "hello" ~ "world"' | kp6 -lisp

There's no user friendly way to run the generated lisp code yet but
hopefully future releases will allow for passing options to specific
emitters. I'm hoping to make a user-friendly way to generate
stand-alone executables with sbcl from the command line for instance.