Pugs

Pugs 6.2.11-pre0 calls for test.

After a long day of release engineering, I'm glad to report that Pugs 6.2.11 prerelease 0 is upon us.  Please give it a try and report any problems on irc.freenode.net #perl6 and/or [email protected] -- I'll write a more elaborate changelog and release notes tomorrow.

The day before (the day before (6.2.11)).

Since the main subtest failures were taken care of yesterday, today we focused on fixing parsefails, cleaning up unit tests of bundled modules, as well as build system improvements.  Thanks to bsmith, both "make ghci" and "make haddock" targets are repaired, and Makefile.PL now contains a simple build system overview section added by putter.

As of this writing, "make test" passes 100% here, and only four ext/ modules (Set-Infinite, Span, Perl6-Container-Array and Perl6-Value-List) remains to be triaged.

Tomorrow we'll push out preflight builds; if all goes well, another round of changelogging should be the only thing left before the anniversary release of Pugs 6.2.11.  Wish us luck. :-)

Main test-suite triaging done.

Today dduncan rewrote File::Spec to work with the new lexical import semantics, so many module tests that uses File::Spec now passes.  He and putter also triaged readline brokenness on OSX 10.4 introduced yesterday, which was due to an old NetBSD version of readline without signal handling support.

Meanwhile, I TODOized all the the non-regression failures; most of them are OO-related, and a switch to the 6.28.0 runcore in the next release should make them pass. A comprehensive regression test suite is lovely -- I accidentally broke whitespace handling around "$foo .= chomp" parsing, and it's the test for "pugs -n -e ..." that caught this problem, because "-n" desugars into a while loop that does an in-place chomping.

In the background, I spent a few hours with Bestian to grok Haskell is not not ML, the newest paper from the relentless paper-producing machine known as SPJ (the next-newest paper, lock free data structure with STM, is also worth a look). The not-not-ML paper reconciles the gap between lazily-evaluated and eagerly-evaluated languages, a gap previously considered irreconcilable (i.e. a language has to pick one side and emulate the other side at a space/speed penalty).

The paper introduces an intermediate language (IL), that allows a surface language (say Perl 6) to annotate individual expressions as lazy or strict, translate them into the IL, and compile them efficiently to an abstract register-based virtual machine, using various clever thunking tricks.

Thanks to the Curry-Howard isomorphism (types are formula and programs are proofs), Bestian was able to grok the para-consistent logic system side of the paper, and merge them with my operational understanding. Now that I have a much clearer understanding of how to mix lazy and strict evaluations in the same program, a lot of useful optimizations for the Parrot backend just follows from it.  Stay tuned... :-)

 

Building the next Haskell.

One week ago, the Haskell' (Haskell prime) process was announced to the public. Our goal is designing a successor to the Haskell 98 standard.  The current timeline is to finish up the standard by October 2006, backed with conforming implementations shortly after that.

The process is worth writing about, as it's markedly different from the Perl 6 process, with different community dynamics and design constraints.  From now on I'll record my musings here in the newly created "Haskell" category.

Perhaps the most obvious difference is that of "only perl runs Perl" versus multiple implementations.  The HaskellExtensions page lists the beyond-Haskell-98 features supported by the four leading compilers (GHC, Hugs, NHC, JHC), and most of the proposals are simply ratifying those experimental features that worked.  Some proposals are expressed as textual diffs from the Haskell 98 standard, which allows for concise discussion on both denotational and operational grounds.

The good thing about this model is that it's sane, mostly predictable, and very efficient. The drawback is, of course, that it only makes existing Haskell users (much) happier -- the scope of Haskell stays essentially unchanged.

On the other hand, the Perl 6 process is exploratory. It has a evolving spec, an evolving runtime, and an evolving compiler that targets multiple evolving backends, relying on the sheer number of library writers and users to keep the use cases real and the unit tests accurate.

This is good for expanding Perl from its (already very diverse) niches to even more places, by  keeping the language as multi-paradigmed as possible. The drawback is that without a solid semantic model, most arguments become effective only on ergonomic fronts, and does little to improve the language's soundness.  Many times we've found that two features that looks sane independently simply doesn't mix together (magical pairs and bindings comes to mind), and it's discovered only through trying out real programs in Pugs.

So... I think there's something to learn from each other, and that will be my primary focus in the Haskell' comittee in the months to come.  As a concrete example, Haskell' can really use Perl 6's Unicode model -- and I still owe them a proposal on this.  But sleep+release Pugs 6.2.11 comes first...

Happy new year!

Today I'm back to Tamsui, the location for the first Pugs Hackathon with my brother Bestian, exactly one lunar year ago.  However, it seems silly to write two anniversary journal entries, so that will wait until the common-era anniversary in Feb 1st.

Meanwhile, putter, gaal, bsmith and I were working in around-the-timezones shifts on 6.2.11 bug triaging, with a new TASKS file introduced by putter and bsmith as a lightweight issue tracking system.

We are finally close to the finishing line -- less than 200 subtests failures in the main test suite. In particular, interaction between named arguments and demagicalized pairs now conforms to the much saner subroutine spec.

However, I keep getting distracted of implementing useful feature suggestions from #perl6. For example, jvoorhis (our new committer who hacks ruby as part of day job) tried constructing a new class in the interactive Pugs shell, and found it very broken:

pugs> class Foo {
Internal error while running expression:
***
    unexpected end of input
    expecting end of input, ";", statements or "}"
    at <interactive> line 1, column 12

So he asked on #perl6 if Perl 6 allows runtime class creation. Of course we do; the bug is in the use of line-based input.  Compare the behaviours of entering "1+\n2" in Python and irb (Ruby shell):

$ python
>>> 1+
  File "<stdin>", line 1
    1+
     ^
SyntaxError: invalid syntax
>>> (1+
... 2)
3

$ irb
irb(main):001:0> 1+
irb(main):002:0* 2
=> 3

Clearly, irb's behaviour is far more intuitive, as it allows people pasting whole programs into the interactive shell -- although the Python behaviour is also quite consistent, as it allows "1\n+2" but not "1+\n2" in a file-based program either.

It turned out to be a simple 10-line patch to Pugs, so these cases now work:

pugs> class Foo {
....> has $.bar;
....> }
undef

pugs> 1+
....> 2
3

Moreover, I've fixed the annoying bug of readline segfaulting under threaded Pugs, by disabling signal and sigwinch handlers.  Expect more triaging tomorrow...

(Intermission.)

$Job is in the final 48-hours deadline push before the 10 days of Lunar new year holiday, so I had to drop my online presence to get this SQLite/Jifty/Lighttpd/LiveGrid thing beaten into shape.

Apologies for the inconveniences... I'll be back. :-)

36-hours yak shaving with Ingy.

The original focus was about interoperability between YAML::Syck, YAML.pm, and Pugs's Syck support.

We thought that we'd need a shared, specification-based test suite for them. So we started porting Test::Base to Perl 6, starting with this Perl 6 rules grammar of the Test::Base specification format. (If you havn't been exposed to Test::Base before, Ingy's YAPC::NA 2005 talk is a good start. (To navigate to the next page, press the Enter key.))

Then, because Ingy is learning Haskell and trying out parser technologies, we looked into how to combine top-down rules parsing with bottom-up tokens parsing, which is (as of this writing) an underspecified part of the Subroutine spec (look for "tokens").

As we are discussing this push/pull style of parsing, we realized there's an ideal use case: the Kwid format, with its no-backtracking, no-syntax-errors-possible, cross-level-end-tags (think HTML's <b><i><u>...</b>) characteristics, should be the first test case.

Plus, there's this whole Perldoc project from Pugs's inception -- Ingy lured the lambdacamels into writing our changelog and much of the module documentations in Kwid, not to mention the Kwid spec, which was also written in Kwid.

So we set out to work.  Ingy has picked up Haskell from the excellent Haskell tutorial for C programmers, so we paired up a prototype in Haskell as Text.Parser.Kwid in the Pugs tree pretty quickly.  The prototype uncovered many corner cases in the algorithm, so we kept simplifying the rules until they agreed with each other.

However, having it available in Haskell is not going to help Test::Base (and other Perl 5 modules) to switch from POD to Kwid; what we need is a seamless way of turning Kwid into Pod/HTML/Manpage, plus the neccessary Module::Install hooks to make this transformation an automagic part of make dist.

Fortunately, during the Pugs Hackathon in Toronto, Ingy had already designed how this system should work -- see his Perldoc talk in YAPC::NA 2005 for details. Therefore we need to port this Kwid parser to Perl 5, and have it generate Perldoc-compatible events to hook into HTML/Pod/Manpage emitters.

Fortunately, Perl 5 proved to be a fine functional programming language: the port took me two hours, carrying over mutual recursive structures, lazy instantiation, thunks, closures, et cetera. (Readers of Higher Order Perl may find them familiar.)  The only thing missing is type safety, but it's not really a concern when it's been typechecked in another language already.

After Ingy woke up, he took my decidedly non-OO code, put an OO shim around it, and released the new Perldoc.pm framework.  The bundled perl-doc command line utility can convert Kwid to HTML now -- something I've longed for for many months. (Prior to this I had to use the very fragile, not-released-to-CPAN, unmaintained Spork::Formatter::Kwid and extract the HTML from Spork slides. Ow.)

With Kwid and Perldoc back in an usable state, Ingy is looking into filling in PPI's longstanding gap of a subparser for =directive ... =cut segments. Again, read the presentation for the wonderful things that can follow from this path.

Now... Where were we anyway? :-) It's been a lot of fun, surely, but this yak is much bigger than I anticipated...

More random recaps.

The weekend approaches. Tomorrow I'll continue release engineering for Pugs 6.2.11; much kudos to putter++ and other lambdacamels on #perl6 for continuing bug triaging in the past few days.

I'll also hack with Ingy in realspace; we plan to port his underrated-but-really-good Test::Base to Pugs, so the t/ tree can become more specification-based and easier to maintain.

It's lovely to see Luke getting into Scala's vObj/component systems.  I've been studying the vObj as one of the base calculi for Pugs 6.283 for a couple months; although it lacks Perl6's requirement of staged computation (i.e. having the full language in compile time to control macro/inlining/constraints), its treatment of classes and roles is very well rounded.

Furthermore, many of new features in Scala 2  (GADTs, implicit parameters, etc) and the class hierarchy can be imported verbatim to Perl 6.

On the $job front, I ported a simple version of MySQL's extremely handy GROUP_CONCAT() aggregate function to SQLite. The Rico LiveGrid based rendering is now 20x faster, thanks to Jifty::DBI's support for group_by and aggregate functions. (Come to think about it, those were hacked in by yours truly many years ago.)  We'll see next week whether it's good enough to run on AIX servers with 250Mhz CPUs...

By the way, Perl 5.8.8 RC1 is out. We can't have a Ponie yet, but the various core improvements produced by Nicholas along the way is already improving camelfolk's lives. For a moment I thought I was looking at perl59delta -- it's an amazing list of enhancements for a point release, especially when compared to perl586delta and perl587delta.

Ever since I ported PGE to Haskell, I've been pondering producing a librules, the equivalent of libpcre for Perl 6 rules, so we wouldn't have to reinvent the engine for each underlying runtime. Similar to pcre (and syck), it needs to interact closely with the host environment, to provide dynamic callbacks and interpolations during a match.

I think it'd make sense to write it in DigitalMars D, one of those faster than C++ languages that happens to maintain native-C linkability, while supporting high-level abstractions like closures, interfaces, mixins and type inferencing. Maybe there'd be time around the lunar new year to download GDC and play with it a bit...

Random recaps.

Back to bullet list format for $job days...

• Lypanov, a newcomer to #perl6 and an implementor for ruby-on-ruby (akin to PyPy and eventually Pugs), asked about our plan to make Perl 6 run fast. Some discussions followed, intermingled with a discussion with Rob Kinyon on PIL^N representation types.
• Stevan submitted a TPF grant proposal to pay for his airfare to the YAPC::Asia hackathon, so we can work with Larry and Damian on formalizing the object model and encoding it to PIL^N and Perl 5.
• Release engineering continues. I fixed a bug in HTML::Entities that caused infinite loops; putter made {("a"=>3),} a Hash (instead of Code) again.
• "make smoke" can now skip the ext/ module tests in the default runcore as well,  by setting the PUGS_SMOKE_EXCLUDE_EXT environment variable to a true value.

As seen on #perl6...

<putter> audreyt: say you sat down with a copy of yourself, similar CS capabilities (perhaps not as yet identical reading lists), but who had never hear of perl or p6. how would you describe p6? what about it is not just "another been there and done that" language? what is its "related work" connection to the rest of the field?
<putter> I'd love to read a transcript of that conversation.
<audreyt> I think p6 is mostly about reconciling irreconcillable gaps.
<audreyt> prototype vs class-based; multi dispatch vs constrained types; inheritance vs interface; typesafety vs ducktyping; early vs late binding; etc
<audreyt> some of them are socially irreconcillable, some of them provably irreconcillable. but p6 reconciles them anyway.

Hmm. A more detailed/comprehensive list of such gaps may be interesting.