Pugs

Short notes on migrating a SVK mirror to Git.

While mugwump++'s notes are concise enough, see below for an even more concise version suitable to save as a shell script:

mkdir pugs
cd pugs
git svn init --use-svm-props \
    file://$HOME/.svk/local/mirror/pugs
git svn fetch
rm -rf .git/svn .git/config
git svn init http://svn.pugscode.org/pugs
git svn fetch
git repack -a -d -f --window 100

Also here are a couple of useful aliases:

alias pull='git svn rebase'
alias push='git svn dcommit'

Note that I still think SVK is a great revision control system -- to me, its design strikes a fine balance between practicality and elegance.

However, with a 300MiB+ repository (51MiB under Git) and 20000+ revisions, the N in O(N) vs O(log N) is large enough for common operations to matter... :-)

Pugs now builds again from SVN.

Thanks to testing efforts from kolibrie++, masak++, moritz++ and xinming++, Pugs has migrated back to the public repository from my local hard disk.

Now all you need is GHC 6.8 on your system, and "perl Makefile.PL ; make ; make install" should take care of everything.

Note that the refactored-out parts like MetaObject and HsPerl5 are not committed back yet, so you'd need a working network connection during the build.

However, I'm glad to report that dcoutts++ is working on adding local repository support to cabal-install, and once it's ready, we'd simply commit a MiniHackage containing Pugs dependencies, and bundle them into something installable offline (and uploadable to CPAN).

Pugs.hs is back.

(The excellent news from fglock++ that v6.pm is back prompted me to write something about Pugs, too, so here it is.)

During the past month, I've been quietly releasing Pugs and its dependencies on Hackage.

Installation for Pugs is now much simpler than before; see http://svn.pugscode.org/pugs/INSTALL for details.

There has been no action in terms of features, but the internal has been refactored to reduce compilation time, and the individual components has been released as separate packages to reduce maintenance overhead.

Startup time is also greatly improved.  As a consequence, running the full smoke test suite is no longer a multi-hour-long ordeal; it now takes less than 15 minutes on modern machines.

As of the 6.2.13.11 release, the smoke numbers say it passes 17215 tests out of 19260, which is not significantly different from the 6.2.13 release.

Going forward, I think GHC 6.10's Quasiquoting and View patterns are going to be vital for the sanity of Pugs internals, so please expect no Pugs 6.28.* releases before GHC 6.10.1.

And finally, a shameless plug: I'm still looking for ways to make ends meet, so any offers of telecommuting, project-based/part-time jobs would be very much appreciated. :-)

A graphical tracer for Perl 6 regexes based on PCR

Tracing parse failures by hand while developing a compiler can be really a daunting task, because the real problem can appear everywhere -- the grammar definition, the input string being matched, or even the regex engine itself.

So I've implemented a graphical tracer for Perl 6 regexes atop PCR (Pugs::Compiler::Rule). You can find some small online demos here:

• http://agentzh.org/misc/tracer/digits/index.html (Firefox only)

• http://agentzh.org/misc/tracer/adder/index.html (ditto)

• http://agentzh.org/misc/tracer/langs2/index.html (ditto)

• http://agentzh.org/misc/tracer/grammar/index.html (Tracing PCR's own grammar parser)

To generate the HTML pages (say, the first demo) yourself, simply check out the Pugs repository , ``cd'' into perl5/Pugs-Compiler-Rule, and enter the following commands:

  $ perl util/compile_p6grammar.pl -D \
      examples/digits.grammar > Digits.pm
  $ echo '7c3d54' > digits.input
  $ perl -Ilib -MDigits -e \
     'undef $/; print Digits->count(<>)->(), "\n"' \
     digits.input > trace.out
  $ perl util/gen-tracer-view.pl --outdir tmp/digits \
      examples/digits.grammar digits.input < trace.out
  $ firefox tmp/digits/index.html

As of this writing, the tracer interface still needs love and I'm adding cooler features like ``random jump'', ``stepping in a specified pace'', and ``stepping back''. If you like to help, just join #perl6 and get a Pugs commit bit ;)

I think this tool should be very useful for both regex engine developers and compiler writers, especially when parsing fails in an unexpected way. And it can also be beneficial to Perl 6 beginners who want to learn the shiny new regex syntax and complicated matching behaviors by just ``stepping through'' the actual parsing process. moritz++ said on #perl6 that he would build a CGI script to make my demos above ``alive'' when he had the tuits; let's just wait and see ;)

It should be warned that the regex syntax supported by the current PCR implementation is already a little out-of-date regarding the lastest Synopsis 5 . (Thanks TimToady++ for tweaking the regex syntax during the meantime ;)) I'll try to port KindaPerl6 's perl5rx backend over to PCR later.

Hey, it'll be nicer to have KindaPerl6, PGE , Parse::RecDescent, or even the Perl 5 regex engine to work with my tracer as well! :)

Stay tuned!

-agentzh

KindaPerl6 now runs on sbcl, clisp and ecl

I've been meaning to do something about testing for the kp6-lisp backend. Previously we compiled all tests to stand-alone executables using sbcl, the reason for this being that I didn't know about --noinform --noprint at the time and dumping the lisp image was the hacky solution to making sbcl not print out all that superfluous output.

This meant we had to wait for sbcl to compile each test before we could run any, and each test produces an approximately 30MB executable which meant the test directory grew to several hundred megabytes.

Everything is much better now, you can select the backend you want when writing the makefile:

KP6_TARGET=KP6-LISP-sbcl perl Makefile.PL

or

KP6_TARGET=KP6-LISP-clisp perl Makefile.PL

or

KP6_TARGET=KP6-LISP-ecl perl Makefile.PL

And then run the tests on the selected backend:

make test

All the backends fail the same 56 tests out of a total of 85 so it looks like the backend code is pretty portable. There's a significant speed difference between the implementations, since most of their time is taken up by compilation this makes a lousy general benchmark but testing uses clisp is the best of the three:

ecl: real 2m3.895s
clisp: real 1m7.318s
sbcl: 4m16.725s

Things could be made a lot faster for all of them by only compiling the backend runtime once instead of doing it for every test, or running lisp as it's supposed to be run, in one persistent compiler.

That and adding options to kp6(1) for running code directly with a given runtime is a task for another day.

KindaPerl6 Common Lisp status update

A lot of work has been done on the KindaPerl6 Common Lisp backend for KindaPerl6 since I last 89wrote about it. Variables, subs, lexicals, closures, basic datatypes (int, str, num, hash, array, ..), control structures (if, for, while, ..) are working. We're missing user-defined classes & methods, some internal methods (sort, map, grep, ...), junctions, pairs, gather and probably some other things. Aankhen++ has been doing most of the work with the occasional contribution from myself.

Things sped up a bit recently after fax++ wrote a meta-object model (MOP) for us based on CLOS. The one in the Perl 5 backend does not use the Perl 5 object system (implements its own @ISA and so on) but so far it looks like we can build the Perl 6 MOP on top of CLOS which is more flexible.

A method in the MOP uses the same calling convention regardless of whether it's internal to the compiler or compiled from a user defined class. An example of an internal method using the new calling convention is the dispatch method that implements the .elems method for hashes:

(defmethod kp6-dispatch ((invocant kp6-Hash) interpreter (method (eql :elems)) &rest parameters)
  "Returns the number of elements in the hash"
  (declare (ignore parameters interpreter))
  (make-instance 'kp6-Int
     :value (hash-table-count (slot-value invocant 'value))))

In the case of %hash.elems this would be called as:

(kp6-dispatch (lookup-lexical-variable (kp6-generate-variable "%" "hash")) |Main| :elems)

The object system is used for pretty much everything. For example the .Str method is used to get the string representation of an object, .Int for its integer form, .true to check whether it wants to be true (used in all boolean contexts) etc. Not everything has been moved to the new system internally. Helping with that would be an easy beginner task.

Speaking of beginner tasks there's a lot more of them. And since we have MOP now implementing things is often just a matter of seeing how the Perl 5 backend does things and filling in the gaps in the Lisp backend.

We have some bugs like my @a; @a[123] = 456; which causes an error currently since the underlying CL array isn't extended properly. As well as a lot of things like substr() which are easy to implement but simply aren't at the moment.

So if you're interested in hacking a Perl 6 compiler in Perl 6 and Common Lisp head over to #perl6 and poke us for a commit bit.

KindaPerl6 released on CPAN

One of the things I've been working on in KindaPerl6 is making it CPAN
ready, first by rewriting its custom Makefile as a Makefile.PL and
then by making various small fixes. Yesterday I wrote a unified
frontend to it (kp6(1)) and made `make install' work and released in
on CPAN as version 0.001.

Since KindaPerl6 is still in active and unstable development `make
test' does not run successfully at the moment. So the CPAN install has
to be forced:

    cpan -f -i KindaPerl6

That should install KindaPerl6 along with its kp6 frontend, the hello
world example is then:

    $ echo 'say "hello" ~ "world"' | kp6 | PERL5LIB=$(kp6 -lib) perl
    helloworld

The PERL5LIB part is required so that the Perl 5 backend can find its
runtime libraries.

If you want to check out the AST of the program just do:

    $ echo 'say "hello" ~ "world"' | kp6 -ast | perltidy

And to try another backend such as the Common Lisp one try one of the
emitter options:

    $ echo 'say "hello" ~ "world"' | kp6 -lisp

There's no user friendly way to run the generated lisp code yet but
hopefully future releases will allow for passing options to specific
emitters. I'm hoping to make a user-friendly way to generate
stand-alone executables with sbcl from the command line for instance.

Playing with the new "Not Quite Perl"

Today I tried a new product of the Perl 6 project: Not Quite Perl.  "NPQ" is a subset of the Perl 6 syntax. It runs on Parrot.

To put this in context, Pugs has been the most feature-complete implementation of Perl 6, but has suffered from poor performance and lack of Haskell-enabled contributors.

KindaPerl6 currently supports a subset of Perl6, and is executed with Perl5, but also has slow performance now.

"Not Quite Perl" is a similar effort, using Parrot to execute the code. Parrot is expected to be the fastest performing implementation of Perl 6.  Right now "KindaPerl6" and "Not Quite Perl" are developing separately, but there have been discussions to coordinate, according to fglock, a KindaPerl6 contributor.

Here's what I found tried when I tried both technologies today.

Perl6-regex on Perl5-regex

Following with KindaPerl6 development, and inspired by Yves Orton (demerphq++) Perl 5.10 Advanced Regular Expressions, I started working on an implementation of Perl6 regex on Perl5 regex.

After some experimentation, I ended up with a prototype program that works in Perl 5.8.

The algorithm is now being implemented as an alternate KindaPerl6 backend. It can compile some complex patterns already. The command line for the perl5rx backend looks like:

$ perl kp6-perl5.pl --perl5rx < examples/token-test.pl | perl -I lib5regex -I lib

The program output is a dump of the resulting Match object.

- Flavio S. Glock (fglock)

"KindaPerl6" project roadmap

I finished the first version of the KindaPerl6 project roadmap, after a whole month of planning, prioritizing, and procrastination.

This version is a filtered out "main development track", extracted from the otherwise anarchistic development structure (somebody called it "genetic programming").

The goal of phases and milestones here is really just to break the project down into bite-sized pieces that regular hackers can get their heads around.

Specifically, the plan doesn't tell people what they should not to do. Small projects are much welcome. You might even get a Perl 6 Microgrant . I'm told that the TPF really wants to give out a couple of kp6/mp6/6STD grants!

The roadmap as html - http://moritz.faui2k3.org/pugs_pod/kp6-roadmap

The roadmap as pod - http://svn.pugscode.org/pugs/v6/docs/kp6-roadmap.pod

- Flavio S. Glock (fglock)