Pugs

Day 245: Haskell Workshop.

Today's workshop has been a blast. David Roundy's darcs talk covered many challenges and advantages in using Haskell to develop a real-world application; he even mentioned SVK once, in a positive light. Among other cool tricks, he employed interleave IO to separate program logic with I/O parts. I have used that trick in OpenAFP, but I'm beginning to see that has wider applications as a Pugs/Perl6 idiom -- AOP with Lazy Lists, if you will.

The Cabal/Hackage system (the equivalent of Module::Build/CPAN) generated much interest. Discussions centered around how to incite spotaneous mutual goodwill using social software frameworks, which is imho the key part of the CPAN cultural mechanism.

I'm glad to hear that the huge GHC distribution will be broken apart into various packages, to promote the sense of community library ownership. In the future, the GHC core will be minimal, just enough to run Cabal to install packages -- Perl5 porters have bounced this idea for quite some time now, and it's nice to see it happen here too. In addition, the Haskell people are also reducing the committer-bit deadlock, by adopting decentralized darcs repositories as the norm, so a change of maintainership is simply a URL redirection apart.

Another focal point is hIDE, a Haskell development environment coded in Haskell, using the GHC bindings and build structures developed in the Visual Haskell effort. The choice of Yi as the editor component is particularly interesting -- it can dynamically load and rebind plugins, including the editor core itself, preserving the editor state (undo history/workspaces) without requiring a restart like Ecplise does. I suggested to dons that we should integrate the bootloader with GHC/Cabal, so all Haskell applications compiled this way can be code-swapped on the fly, even by eval'ing code from the user. I like dynamic languages with static typing...

I helped the library effort by polishing dons's Data.FastPackedString module, so it can subsume the old Data.PackedString implementation. The new string module is very fast and supports FFI, gzip and mmap natively. It is already used in the new PIL runtime; once it's merged into GHC core, it should give string manipulation-heavy programs a nice speed boost.

Hacking of GHC core continued. Pairing with wolfgang, the new ICFP 2005 contest champion, is very exciting and fruitful. We reasoned out how to support record updators for existentially-quantified GADT types, which is starting to dangerously resemble ML-style modules.

Also, from discussion with oleg and ccshan, I learned that GHC is actually well-equipped to natively support delimited continuations in the runtime. If implemented, Pugs's Haskell runtime can do away with the space-intensive ContT monad transformer thunks, and get a massive speed boost without changing a line of code. simonmar showed me the GHC runtime scheduler, and indeed there seem to be native support in the form of thunk-capturing asynchronous exceptions (which will incidentally give native support for the Perl6 $!.resume). Using a similar mechanism, we can also get a stack trace -- easily the #1 requested debug feature -- from fatal errors, at least when compiled for profiling.

Whew. That's a lot of hacking. Tomorrow I'll go back coding in MM2 and full Cabal support, and investigate full continuations on top of Perl5 VM. Stay tuned!

Day 244: Hacking GHC.

Continuing yesterday's work, today I got seriously addicted in GHC core hackery, skipping the entire Curry workshop. It was a real pleasure pairing with spj on this. He reminded me of lwall -- both have an obsession on providing useful feedback to programmer, strong intuition on how language features intersect, and above all a deep enthusiasm to their respective languages.

Thanks to spj's continuous guidance, I hacked in support for existential record types on both vanilla and GADT types, which allows us to write:

data Counter public = forall this. New
    { self   :: this
    , inc    :: this -> this
    , output :: this -> IO ()
    , tag    :: public
    }

The tag function is well-typed as Counter public -> public; the other three fields hides the real implementation from the Counter data type, so one can have a counter using Int, another using String, and have both be of the same type.

This representation is quite close to how encapsulation works in OO langauges, and the feedback from #haskell has been very positive. Additionally, the soundness of the GADT/Labels treatment passed scrunity by Oleg today, which made me feel much safer.

Of course, any attempt to use the three private fields as normal functions raises a compile-time error that advises the use of pattern-matching syntax instead. Also, in the GADT form, the forall this may be omitted to improve readability.

While I was madly hacking away, brendan passed by, and graciously accepted a Pugs T-Shirt from me. We talked about many things -- I'm very excited to hear that they are considering to adopt Rhino's continuation engine for SpiderMonkey (which will make Perl6-on-JavaScript insanely faster), and that the XUL scripting is proceeding well. They would like to use user's system python/perl/ruby/etc if possible, but that complicates the versioning problem.

Inevitably, the topic of shared runtime came up, and it seems that Mono's RAND patent position is still not yet very satisfatory. Parrot has matured a bit since two years ago when brendan checked it out, so I filled him on some recent development. With 0.3.0 out in a couple days, we can finally start targetting Parrot again -- hopefully the long wait will be proven worthwhile.

Oh, do check out geoffb's writeup of his musings on #perl6; it's a nice survey of some of the good ideas from other language that are finding their way into Perl6.

After yesterday's discovery of ILX codegen, spj pointed out that there is a more robust Java codegen on the same directory. Targetting JVM suddenly looks more attractive too...

All in all, I think my time on hacking GHC core is well-spent; I learned a lot of Haskell idioms that should make Pugs code cleaner (such as the case x of DataCon {} -> ... syntax that works even on non-record types), and about complexity management of a large real-world language implementation -- i.e., what makes it to evolve extremely fast, despite a relatively small core team and contributor base, and an utterly broken issue-tracking system. RT-powered bugs.haskell.org seems like a good idea...

Tomorrow is the Haskell workshop, the final day and highlight of this conference. See you!

Day 233: Reports from ICFP!

The main ICFP conference session concluded today, so I'm finally getting a tiny slice of time to write down what I've been doing.

I enjoyed handing out Perl6/Timeline T-shirts to various lambdafolks, including bpierce (author of TaPL), spj (author of GHC), as well as several cool hackers from #haskell. At any time in the conference hall, there was always someone with a Pugs shirt in sight, leading to many interesting discussions and opportunities to join forces.

I spent much time hacking with the GHC core team (spj and simonmar, also known as "Simon and Simon"), to get my current #1 feature request -- GADT with record types -- into GHC. Among Haskell people, GADTs are all the rage this year; see luqui's notes for a short introduction from Perl6's perspective.

During the YAPC::NA 2005 Toronto hackathon, I presented the then-brand-new PIL1 data structure to @Larry, using GHC's then-brand-new GADT syntax:

data PIL a where
    PCode       :: SubType -> [TParam] -> PIL [Stmt] -> PIL Expression
    PBind       :: [PIL LValue] -> PIL Expression -> PIL LValue
    PAssign     :: [PIL LValue] -> PIL Expression -> PIL LValue
    -- ...etc...

luqui looked at it and asked quite innocently: "Why are the arguments not labelled, such as 'lhs' for 'PAssign'?" -- To which lwall said, half-jokingly: "Because autrijus is a lazy bastard."

However, the real reason was because GHC's extended Haskell syntax only allowed positional argument types for GADT. To name them with the record syntax, we'd need to go back to the vanilla data types, and forgo the ability to encode type information using the constructor's return type (as demonstrated in PIL Expression and PIL LValue above).

A month later, I diligently re-encoded the structure using vanilla record types, so we can export them into hash-based Perl (and JSON) data structures, instead of array-based ones. This significantly improved the readability of iblech's PIL2JS code, but the Haskell data declaration became 3 times longer, because each GADT variant needs to be re-encoded into vanilla data types.

Ever since then, I was hoping that GHC may one day remedy this by allowing GADTs and record syntax to play together. When I met spj in ICFP, he was gladly surprised that I'm eager to hack GHC to make it happen. After bouncing several design ideas, we settled on this syntax:

data PIL a where
    PCode   { typ :: SubType, params :: [TParam], body :: PIL [Stmt] }
        :: PIL Expression
    PBind   { lhs :: [PIL LValue], rhs :: PIL Expression }
        :: PIL LValue
    PAssign { lhs :: [PIL LValue], rhs :: PIL Expression }
        :: PIL LValue
    -- ...etc...

The neat thing is that PBind and PAssign can share the lhs and rhs labels, as long as their return value PIL LValue stays the same.

During the reception, spj was kind enough to guide me through the GHC core to unify the two data type definitions together back. The next day, simonmar walked me through the Happy parser to get the new syntax recognized, and today spj showed me how the GHC typechecker works. All there's left is installing the labels as record selectors, and the upcoming GHC 6.6 will have a nice feature to fit Pugs's needs. Hacking GHC is evidently not as difficult as I feared! :-)

Talking about GADTs, I was also involved with the crazy project of encoding the Darcs patch algebra into the GADT type system, letting the compiler catch entire classes of common logic errors in darcs, bringing us dagerously close to the ideal of proof-carrying code.

Two days ago, when droundy told me this idea, I was very excited, started prototyping it, then promptly ran into errors on existential types. Heffalump took my laptop and shuffled it with Igloo, arriving at something that seemed to compile, which evolved into an embedded DSL for patches. After today's conference sessions, a dozen darcs hackers gathered together, learned about the recent developments of the patch theory, then worked on the GADT encoding. The hacking session was quite successful -- we didn't run into any showstoppers, much to everybody's surprise -- and I visiolized the whiteboard into this diagram.

On the Pugs backend front, I learned from simonmar the existence of the ILX emitter library, hiding in a forgotten corner of GHC source tree. This enables us to emit pre-CLR-2.0 MSIL code that could run on Mono, under a 3-clause BSD license. Targetting CLR suddenly looks more attractive...

I also learned how to generate native machine code straight from C--, using the GHC API distributed as part of Lemmih's ghc-api Cabal package.

Speaking of Cabal, I learned from SyntaxNinja about the design of the Cabal/Hackage system, which is the Module::Build/CPAN equivalent for Haskell. We shared our experiences, and he helped me setting up the basic Cabal build rules for Pugs. With Cabal support, Pugs's Makefile.PL would only need to care about the Perl parts in the source tree, and we can finally get a libPugs to link with pugscc and other programs -- Inline::GHC is definitely on the radar, and the idea of using Perl6 to script the Haskell IDE (hIDE) is also gaining some traction.

Compiling with continuations, another pet interest of mine, also got a boost from ICFP. The Generalized Stack Inspection talk outlined how to use vanilla try/catch mechanism to implement full continuations, which is much more efficient than the regular trampolining approach as used in Pugs's JavaScript backend, since only people who actually use full continuations need to pay the cost for catch. Their technique operates on the ANF form, which seems to be a good low-level intermediate form for Pugs.

Interestingly, the paper went as far as saying that Parrot might have made a suboptimal choice in paying the full cost of going full-CPS, since we could emulate full continuations using exception handling efficiently. They gave concrete implementations in CLR and JVM -- we'll see if this result can carry over to Perl5 and JavaScript runtimes.

The invited talk of the first day tackled impredicative type inferencing, a long-standing problem in inferring types for a rich system such as Perl6. I had much trouble grokking the Wobbly Type idea of spj's, so I was much delighted to see that fpottier gave a very intuitive treatment of splitting the user-annotation-driven phase and the complete inferencing phase apart.

Oh, the ICFP contest. Haskell won again, which came at no surprise, but the Dylan Hacker team made another strong showing. From the brief investigation, I think Dylan is even closer to the "Perl 6 without advertisement" mark than Ruby, largely due to its incremental soft typing system, where one can start rapid prototyping with full dynamism, then generate more efficient and robust C code by adding type annotations.

Tomorrow is the Curry workshop, then the Haskell workshop on the day after. After that I'll get two weeks of Pugs hacking time. I can barely wait to apply those new techniques, tools and thoughts into Pugs!

Day 230: My CUFP Talk...

...was very well received. The other talks in CUFP were very interesting too.

Talked with David Roundy about the svk-darcs isomorphism, and learned how to use darcs tags to simulate svk merge points and improve on two-way mirroring.

Daan's Morrow language is quite inspiring, especially due to its use of UUAG. I learned attributive grammars from the TMR article, and was happily surprised that @Larry seemed to be in favor of adopting AG as part of Perl 6's self-host compilation.

I'm totally sold to the Takahashi method for presentation. It takes 3x time to prepare but is worth it. The slides are available online as odp, pdf, swf and ppt. Enjoy!

Day 228: Blondie wants you.

So I'm in Zurich airport and (finally) working toward the slides for my talk, cheduled to go on air in 24 hours.

I'm adopting the Takahashi method, which basically means doing six lightning talks in a row, and somehow string them all back together in the end. It's fun and very challenging...

The State of the Onion 9 talk transcript is quite entertaining. I hope to run into other fun amplifiers in ICFP, so the two weeks after that can be optimised to get Pugs 6.2.10 (or even 6.28.0) out in a useful shape in time for Euro OSCON.

Perl6-MetaModel News

Role work is now (mostly) complete. We still need more tests, and there is still more Class and Role integration planned but not yet implemented.

To exercise the meta-model I have ported chromatic's Perl 6 version of Test::Builder (complete with tests) to the meta-model. The translation was fairly seamless, and even used the meta-model based package system instead of Perl 5 packages.

You can see it here and you will find the Perl6 code in the POD documentation for reference.

Blondie wants You (pointing finger) to give it more backends

Blondie is a side project whose purpose is to prototype some approaches towards compiling the dynamic/static conglomerate that is Perl 6 into something that runs efficiently and portably, in as little effort as possible.

Blondie needs as many backends as possible as early as possible to stress it's design.

Blondie is already interpreted in perl, and partially compiled to C (supported by a primitive type inferrencer), but we want more!

The backends we're still interested in are:

Parrot and PIR

This is what Perl 6 will run on by default, so we better support it

Other virtual machines, e.g. LLVM, Ruby's YARV, the JVM through Jasmin, C--

For two reasons: Running Perl 6 on foreign runtimes designed for high level languages Learning what VMs look like when targeted uniformly

GCC's GENERIC

Although we're already harnessing GCC with the C backend, making a GCC frontend for Blondie has two interesting aspects: it's higher level than a VM, so this illustrates AST conversion, and it requires completely manual memory management

Other interpreted backends, e.g. JavaScript

Interpreting approaches are by far the quickest way to get Blondie to run on strange platforms. This will stress the way the shared prelude is handled.

Mock backends

Harsh environments for a dynamic language, like no IO, hard real time (all builtins which cannot be realtime are masked from the prelude), etc.

Please see the todo file for more tasks under perl5/Blondie/TODO in the pugs repository.

Day 227: I survived!

Wow, I'm still alive after this tumulous week, which featured not one but three met-at-the-final-minute deadlines, a rescheduled and forced-upgraded plane ticket, my passport stuck in limbo for three days due to DHL system errors, and of course even more hardware failures.

At long last I'll be able to catch some sleep now, wake up and board the plane, step off it and sleep for a few hours, then immediately step up to speak in CUFP, with nary a page of slides in sight (or in mind). Wish me luck, as I'd need it very much...

Perl6-MetaModel News

Some re-arranging has been done, ::Object now shares pneuma.pl with ::Package and ::Module, while ::Role has been moved to the new psyche.pl file. This means that genesis.pl is concerned simply with bootstrapping things, and nothing more.

The Role work is progressing slowly becuase of $work. However, we have added pictures (p6_role_model.jpg) and there is some general thoughts/comments in psyche.pl as to how the bootstrapping will actually work.

We recently added a test which implements the inside-out style of objects within the meta-model. This basically shows that the meta model is capable of itself hosting other models.

Basic support (read: very experimental) for parameterized classes has been added recently. This is implemented entirely on the "macro" level, and not actually part of the meta-model itself. Here is an example of how it is used:

my $Scalar = class 'Scalar' => [ '::T' ] => sub {
    my %params = @_;
    my $T = $params{'::T'};

    $::CLASS->add_method('STORE' => ::make_method(sub {
        my ($self, $value) = @_;
        $value->isa($T->name) or die "Incorrect Type";
        # ... store the value here
    }));
};

At this point $Scalar is not actually a Class instance, but more a Class instance factory (actually it's a CODE ref). Here is how it is used:

my $Int = class 'Int' => {};
my $ScalarInt = $Scalar->('::T' => $Int)

Now $ScalarInt is actually an instance of Class parameterized to only accept Int objects. For a working version of this example see 32_parameterized_classes.t.

Day 221 (r7023): More updates from lambdacamels.

PIL2JS

Reminded by gaal's t/pugsbugs/value_alias_readonly.t, iblech fixed .values and .kv on hashes and arrays to return aliases.

This means that %hash.kv[1]++ or @array.kv[1]++ modifies %hash or @array respectively. Additionally, the tests of .kv and .pairs were appropriately improved.

Perl6-MetaModel

Some basic work on Roles was done, and several new tests added for this. Mostly just getting the basic Role object working as expected. Next step is to do Role compositing and Role consumption by Classes.

cperl6-mode (Perl code editing commands for (X)Emacs)

Many new keywords/builtins, much better indentation, better working of parameterized rx// modifiers, highlighting of sub prototypes, non-parenthesized conditions for most statements, more Perl6'ish sigil interpretation, dedicated "Pugs" style for pugs::hack code conventions.

Blondie

nothingmuch checked in the experimental Blondie language.

Blondie is a dumb intermediate language and a compiler collection for the purpose of demonstrating an approach to compilation. It has no syntax, only an abstract syntax tree, and some very simple operations.

Day 220 (r7000): Updates from lambdacamels.

I'm still scrambling, rather unsuccessfully, to redo all the lost works in this week, before my plane to Estonia takes off next Tuesday.

Fortunately, lambdacamels from #perl6 decided to collaboratively write this journal in my wahnsinnig absence, so here is the first issue. Enjoy!

PIL-Run

With newly added my variables in the weekend, the Perl 5 backend now supports complex definitions of lazy lists:

sub fat ($x) {
  if ($x == Inf) { $x }
  else { if ( $x == 1 ) { $x } else { fat( $x-1) * $x }  }
}
@a = (1..Inf).map:{ fat($_) }

(1, 2, 6 ... Inf)

Perl6-MetaModel 2.0

We decided today, after some discussion on #perl6 that class attributes need not be handled by the metamodel, and could be just treated as package level our variables. This of course simplifies the meta-model, which is always a good thing.

The latest version of the object model diagram was posted to perl6-langauge for review. This represents the current state of the meta-model prototype, our next step is to add Roles.

Debian package updates

pugs 6.2.9 was recently uploaded to Debian unstable and will go into the archive with the next dinstall run on 14.Sept 05.

The Debian package now Build-Depends on libghc6-plugins-dev. So now we have a pugs with perl5 and haskell eval support. Now the only thing missing for a full featured pugs in Debian is parrot, which is being worked on currently. General Pugs updates

Pugs can now be called with a specific runtime engine. The Perl 5 backend is invoked with pugs -B Perl5, and the JavaScript backend can be invoked with pugs -B JS.

The $?PUGS_BACKEND variable tells which is the current backend.

The wrapper script around pugs_bin, pugs, was modified so pugs -BJS respectively pugs -BPerl5 will launch the interactive shells of PIL2JS/PIL-Run. This also means that make test-js works now.

Day 216: OT - autrijus.org gone.

Mail is back to service, but only barely. Thanks to the helpful interface of a Sil RAID1 controller, I managed to low-level format all of autrijus.org's two disks, as well as the extra backup, which was selected as the target -- somehow the controller cheerfully formatted everything else too.

Combine this with the recent laptop HD failure, it means all I have left are publicly aggregated data and whatever archive.org has in it.

Among the definitely-lost category are mail archives, several active websites, and lots of unpublished works -- not the end of the world, but far from pleasant, either.

Day 215: OT - Another hardware failure.

The power supply of my home server (autrijus.org) burned up unexpectedly, possibly taking the motherboard with it. For the next 18 hours or so, mails should be able to reroute to autrijus at gmail.com automatically.

Also today I realised that Estonia does not take Schengen visa after all, which leaves me with ten days to get two visas done -- which normally would take more than one month. I'm trying to speed up the process so I can get a remote chance of presenting Pugs at CUFP.

So today's all spent on errands... Sorry for the lack of update again. :-/

Oh, btw, Edd Dumbill's interview with me was posted to perl.com today. Edd does know how to ask intelligent questions, but sadly I only had tuits to answer very briefly.