Pugs

Day 59: 1 commit out of 33

Of all commits today, only 1 was mine (a very small step on getting on-the-fly compilation to work), and 32 are from other committers, doing great work at an amazing rate as usual.

My subpar productivity was due partly to carsick and mild food poisoning (again), partly from this silly CPAN drinking game which I won, and also from this even sillier but surprisingly addictive Liquid War game, which clkao largely dominates over me, obra and acme.

However, I did find time to read over the extremely concise, well-written and robust YAML spec, and have started some work on a Parsec-based YAML parser that can deal with infinite streams, which according to ingy will make the world a happier place.

Now, let's see what other people have done to Pugs today...

• corion has made great progress on command line switches, making -I foo and -l work, added @*ARGS access to pugs -e one-liners, and written tests for -C and friends.
• asavige the mad golfer checked in the first Pugs Obfuscation. It looks really nice -- but I'm not going to tell you what it looks like. Try running it with pugs!
• kungfuftr worked on pugs-smoke graph, so we can now all see in plain daylight how the trunk is burning. There's been lots of work into it, making pretty tooltips on the test failure messages, and hyperlinks into the actual tests, even down to the failing line itself. I'm as amazed as the first time I've seen the Test::Cover graph. Wonderful work!
• mugwump keeps churning out large chunks of Perldoc code. DOMs can now send events to each other, which means we are closer and closer to the nirvana of interoperable Perldoc dialects.
• lightstep took the plunge to restore sanity to option parsing, by introdusing an abstract data type for options. corion and shapr followed up with some discussions and debug actions.
• stevan took a clue from #perl6 and changed the name of CGI::Pugs to be just CGI. He then proceeded to port lots of functionalities from Perl 5's CGI.pm, as well as quite a few tests.
• stevan also added various tests for string and math processing builtins.
• theorbtwo rewrote our test cataloger that links tests with design documents.
• kcwu added tests for pointy blocks, and fixed the test for rand().
• hcchien added tests for multidimensional arrays.
• rgs added some more keywords into perl6.vim.
• darrens continues to p6ify SQL::Routine. Strangely, they look even more perlish to my eyes. Maybe that's because Perl 6 no longer forces us to pay the 30%+ "language tax" of code overhead anymore...

Oh, and I've talked with Leo on IRC today, and there is a distinct possibility that I may stay in his place for a couple weeks in the months to come -- I'll keep you posted. But, just imagine the unstoppable combined productivity! :-)

Day 58: IMC compiler in an hour

Today, on clkao's van from the hotspring to the dinner place, I implemented a minimal Parrot IMC compiler, parser, evaluator and pretty printer in an hour, using GADTs and TH2. The reason for this exercise was so that Pugs can target IMC AST without being bound by the actual state of IMC, i.e. Pugs can still run without a parrot installation. It also makes it possible to express "feature suggestions" to Parrot as code, instead of words. :)

After that, I started doing a YAML parser/compiler with ingy, but my battery went out of power. Oh well. Still I think a fast, native YAML library for Pugs will be a good idea.

Today Stevan unleashed a working CGI.pm implementation in Pugs. His commit log notes: " Why? I have no idea, but hey, why not?"

Darren continued working on Locake::KeyedText and SQL::Routine. In particular, their Makefile.PL are now written in Perl 6, making them the first 100% Perl 6 code in modules/ directory. Woot!

gaal worked on perl6.vim some more; mugwump and ingy continues figuring out the Perldoc model (I think %=POD should be renamed to %=DOC now, really); there's more developments, but I'm a bit drunk from this 3-hour dinner thing, so I'll stop right here for tonight. :)

Day 57: GADTs and Hualian

(Network outage prevented this journal from being posted on time; sorry)

Today was a short day for me; the first 5 hours are spent on travelling to Hualian, then the post-conference fatigue kicked in right after dinner, so I took a nap, woke up to write this journal, and about to sleep again.

However, I did spend some time playing with GADTs, for the upcoming IMC compiler and Eval.hs refactoring. It is a very powerful feature introduced in GHC 6.4, along with Software Transactional Memory (lock-free, composable, atomic threaded code) and Template Haskell v2 (which allows for type-safe BEGIN blocks in Haskell code that can lexically rebind the builtins based on console input -- extremely Perl6ish).

What are GADTs good for? Well, In most ASTs, it is possible to construct a "nonsense" AST that would not be detected until the runtime. For example, here is a valid Pugs AST:

    # Representing the nonsense program: ++[42=]
    Syn "[]" [ App "&infix:++" [] [] ] [ Syn "=" [ Val (VInt 42) ]

Furthermore, the compiler and evaluator spent lots of dead code on unpacking this Exp structure and repacking them into the Val structure. GADTs solve all this by give each node in AST a well-typed representation, so it becomes impossible to construct invalid ASTs, which greatly simplifies the compiler and evaluator. More details are available in the Wobbly Types paper, and the upcoming Pugs Apocryphon 2: Architecture.

Hualian is evidently a great place for hacking. clkao and obra worked tirelessly on getting svk 1.0rc1 out, with a huge list of changes. svk was the reason why Pugs can adopt a wiki-style development model, and I owe a lot of my productivity to svk. Hopefully one day Pugs can pay back by freeing clkao from the Swig/XS mess, and put an end of this written in Perl, so gets no respect situation.

In other Hualian news, mugwump and ingy worked on Perldoc, their new name for the unified framework that includes POD, Kwid and other inline DOM structures. Magically they managed to agree with each other, producing a detailed process flow document, as well as a comprehensive set of modules for the Perldoc toolchain. Excellent work!

Oh, and mugwump continued to make sense out of the Perl 6 OO model. I love his comments in Class.hs, e.g.:

∃ MetaCategory C₁, C₂ : C₁.catPair = C₂ ∧ C₁.catIsComposite
→ ¬(C₂.catIsComposite)

Other developments from across the world:

• stevan discovered a bug with IO in loops, when working on his excellent Pod::Streamer module. He also wrote a README explaining his plan on hacking our way out of the Stone Age.
• vladtz submitted TODO tests for unpacking in subroutine signatures.
• corion added tests for pugs -I, and noticed that ~~ was broken when the right hand side is a variable; I fixed it promptly.
• fayland committed POD.t to test for the %POD directive.
• kcwu added tests for **, Inf and NaN.
• darren synced SQL::Routine and Locale::KeyedText with their Perl 5 counterparts on CPAN. Side-to-side comparison of p5 and p6 code is a really nice way to learn and appreciate Perl 6!
• scw, gaal and rgs hacked on perl6.vim, bringing it up to speed with today's Perl 6.
• bsmith and corion worked on the tentatively named %?CONFIG hash, and hacked in support for pugs -V:foo.
• gaal began a Perl 6 port of Cache::Memcached.
• nothingmuch worked some more on tests, graphs, and the beautiful testgraph.pl.

That's it for today. See you tomorrow!

Day 56: Leopold Toetsch on IRC!

Two days ago I invited Leo to coordinate Pugs/Parrot issues on IRC; tonight he showed up on #perl6 and #parrot. Much discussions ensued -- this is definitely my wisest course of action during the past 56 days!

Alas, unlike #perl6, #parrot is not yet logged on the web, so here is an abridged transcript of the hour before the (brain)storm...

* acme summons leo
<@Psyche> now that's a spell gone wrong, eh?
<@acme> have faith, young Psyche
<@acme> drat, i really thought he was going to pop up
-!- leo_ has joined #parrot
<@Coke> holy smokes.
<@Coke> now, if autrijus were awake...
<@autrijus> hi leo!
-!- Coke changed the topic of #parrot to: OMG, is that *LEO*?

A two-hour brain tennis session followed, focusing on how to extend the calling convention and IMC types, so that the Parrot AST interface can work correctly. We reached many fruitful courses of action:

Pugs: Multi-pass compilation architecture.

• The evaluator and three independent compiler backends will now work in concert.
• Perl 6 source code will still be parsed to Pugs AST.
• The current evaluator in Eval.hs becomes a compiler, generating a Parrot AST.
• Parrot AST will then be compiled into Haskell, or pretty-printed into IMC code.
• Existing code in the Eval monad need not to be rewritten, thanks to the monadic architecture and Template Haskell,

Pugs: Combinator library to emit Parrot AST.

• Unlike existing IMC macros, it can be freely used without worrying about producing illegal IMC.
• Pugs will ship with its own IMC parser, based closely on Parrot's one; a Lexer has already been transcrbed.
• Pugs also needs to compile the Parrot AST into Haskell, probably using an IMC monad, with its own implementation of Perl* PMCs,
• That means Pugs can still run Perl 6 by itself, but enhancement on it can then be shared with Parrot and other languages targetting IMC.

Parrot: IMCC enhancements.

• IMC may need to have a .lexical pragma to simplify pad creation and management.
• A Lazy PMC is needed to support the VThunk postponed-evaluation value type in Pugs.
• The calling convention needs to be extended to allow the caller to propagate the calling context (want in Perl 6) to the callee, via tags like "PerlContext" = "Int" stored in a Hash PMC register.
• A new @ARGS may be introduced to IMC to abstract away parameter processing code in the calling convention, and let us change the underlying convention mroe freely.

Here is some closing notes:

<@autrijus> leo: I need to sleep soon too. I'll be online ~20hr later
<@leo> and as IRC isn't that bad I'll try it again soon
<@leo> any final questions?
<@autrijus> leo: thank you a lot for doing this immensely sane system :)
<@autrijus> leo: also, if you have a server somewhere
<@autrijus> you can use GNU screen
<@autrijus> to stay on irc
<@leo> yep - thanks for the hint WRT sreen
<@leo> see you all

Other committers did not relax either, churning out enormous amounts of code in the past couple days:

• rgs, our new committer and Perl 5.10 pumpking, implemented q// literals, fixed numerous problems in the build system, and hacked on luqui's perl6.vim.
• ingy fought with MakeMaker again and emerged victorious: Pugs can now build modules with inline Haskell, link against the original .o files, and install them correctly. Now I need to get some time to make it work with Cabal to build the pugs package.
• scw produced debian packaging files for Pugs, tweaked pugscc so it can generate IMC code anywhere, then hacked madly on getting :key(123), :key<abc>, :key[1,2,3] and :key working. He succeeded.
• Stevan helped me with TODOing and unTODOing tests during 6.0.13 release engineering, then resumed his work on Pod::Stream::Parser, a POD parser that already works in Pugs.
• Corion took the herculean task of implementing Perl5-compatible command line argument parsing, including proper support for many one-liner idioms.
• mugwump, our resident OO ninja, single-handedly modeled out much of the Perl 6 OO system, with metaclasses, modules, traits and packages. He also made a funny Powered by PhD image, after hearing me saying that GHC is powered by dissertations. :)
• darren continued working on SQL::Routine to make it into idiomatic Perl 6.
• nothingmuch and theorbtwo continued to improve the excellent test result visualization tool, with the aim of refactoring it out into a general-purpose module that other projects can use.
• vladtz added and refactored some tests. However, his commit log is empty, which makes summarizing his works a bit difficult...
• gaal made the parser recognize $.x and $:y as variables, in anticipation of the upcoming OO core.
• kcwu fixed a GHC warning, and made the t/pugsrun/ tests run pugs in a saner way.
• luqui fixed a syntax error in Context.hs and normalized test files to begin with #!/usr/bin/pugs, instead of #!/usr/bin/perl6.

Finally, Jeff Horwitz released mod_parrot 0.2, which greatly helped my understanding of Parrot's embedding API. He also added a new high-level-language layer to support plug-in style development for languages targetting Parrot. Specifically, he plans to learn Haskell and implement a GHC layer for Pugs. This is likely to be very useful, and may pave a way for us to unify Haskell FFI with Parrot NCI in the future. Way to go, Jeff!

Day 55: 6.0.13; Perl 6 The Ultimate

YAPC::Taipei day 2 was a raging success, and my two-hour talk was very well received. Again, see Leon's journal. Thanks to Damian and Allison's previous slides, which formed a considerable part of my talk's materials. Also, thanks to Leo who kindly fixed mandel.imc before I give the talk, so it actually renders the identical picture as Perl 5's version.

I've had one bottle too many beer and eight hour too few sleep, so I'm sorry to say I need to punt on today's journal summaries, except to say that Pugs 6.0.13 has been released, and I'm going to focus on fixing the t/* failures, on-the-fly compilation, and a standalone implementation of IMCC Parser, AST, PrettyPrinter and Compiler this week. We'll see how much I actually manage to get done...

Day 54: YAPC::Taipei Day 1.

YAPC::Taipei day 1 was a blast, even though all of us were (and still are) severely sleep-deprived. See acme's journal for details. Here's my photo taken by obra.

So I'm totally fried. But, as usual, an awful lots of Pugs development happened today. Here's some of them:

• I met scw in person; he just walked to me and offered a solution to the END { say 2 }; die 1; not printing 2 as it should, so I slapped a resetT to evaluateMain and it Just Worked. hurray for continuations!
• Stevan helpfully prepared a ChangeLog for tomorrow's 6.0.13 release, and I kwidified it.
• Stevan also officially finished File::Spec porting. Well, except the VMS bit, but I don't think Pugs runs on VMS anyway...
• Mad Porter iblech ported File::Basename, then was convinced by Stevan to become the Mad Tester, adding an absolutely huge number of OO tests, which exercises almost all of A12/S12.
• theorbtwo added UTF8 support for fixauthors tool, and worked a lot of features into the test harnesser/reporter/cataloger.
• asavige committed the first Pugs JAPH. I want more JAPHs! :)
• darren tidied up some Perl 6 syntax in Locale::KeyedText, then promptly committed a first sketch at his SQL::Routine port.
• kcwu, corion, mugwump et all did various commits to clean up the source some more.

And yes, I still had not written all the slides for my talk tomorrow. Need sleep first and work on it on the MRT tomorrow morning. ;)

Day 53: Day before the conference.

Don't really have time to write detailed report, as I need to wake up in ~4 hours for YAPC::Taipei, so just some quick links:

• Chip releases Perl6::Subs that lets you use Perl 6 subroutine parameters in Perl 5.
• iblech ported another 5 or so modules.
• A very nice Perl 6 Modules Porting HOWTO was also written by iblech.
• Stevan added a modules/README at my request.
• gaal did tons of work on the test harness and reporter
• I implemented s:perl5:g/// with $1 etc on the substitution side.
• asavige reported that $*EXECUTABLE_NAME is broken; I fixed it along with $*PROGRAM_NAME.
• metaperl hacked in lots of mathematic functions.
• corion started doing tests for multiple -e and other pugs bugs.
• iblech added tests for proxy objects and attributes. which brings us to...
• mugwump and I designed and checked in the first sketch of class meta-model, marking the beginning of OO and introspection support in Pugs.

See you tomorrow!

Day 52: pugscc --runparrot mandel.p6

Pugs can now compile the mandelbrot generator, example/mandel.p6 , into mandel.imc .

Running the line below generates 100 copies of this pretty picture. Time for some benchmarking!

% time pugscc --runparrot examples/mandel.p6 > /dev/null
real    0m5.548s
user    0m5.325s
sys     0m0.078s

Compare this with the perl5 version:

% time perl examples/mandel.p5 > /dev/null
real    0m5.774s
user    0m5.629s
sys     0m0.016s

It is also worth noting that this is on FreeBSD 5-STABLE, where perl5 was built with -O by default. The --runparrot above uses an unoptimized parrot, with vanilla runcore, and with no type inference (so all variables are PMCs). That means even though Pugs/Parrot is heavily handicapped, it still performs better than Perl 5.

Our Mad Porter iblech exhibited superhuman productivity, and ported five CPAN modules to Perl6 in one day:

• Commands-Guarded
• Email-Simple
• Geo-Distance
• Mail-Address
• URI

All I can say is "Wow". In other news, after ingy's 2-days-fight with MakeMaker, Inline Haskell/C modules now finally works. It requires GHC 6.4 for the Template Haskell powered wrapper generation code. SHA1.pm is our first working example.

Mad Golfer asavige committed our first golf-based test script, which combines the best parts of golf-driven development with test-driven development. :-) I added in $*EXECUTABLE_NAME and $*PROGRAM_NAME support for him, too.

Stevan asked for s:perl5:g/// to complete his File::Spec port, which I promptly implemented. He also started picking up Haskell, and hacked in -d and -f all by himself. Yay!

Some other developments of today:

• nothingmuch hacked in pugs -Ilib support.
• hcchien and viirya both committed tests to make sure that grep {...} @list still works.
• gaal implemented a nice YAML harness as a replacement for the vanilla Test::Harness.
• theorbtwo wrote a Test Graphing tool that generates HTML coverage reports from test results produced by the yaml harness.
• And, in Kwid-related news, ingy and gugod just ported Kwid as a Spork/Kwiki formatter, so YAPC::Taipei speakers can write their slides in Kwid, and have them converted to S5

It's getting late, so I'll sleep now. YAPC::Taipei starts in 30 hour or so. Stay tuned!

Day 51: Inline Haskell, s:perl5///

I woke up to ingy's arrival on today afternoon. The first thing he said to me was "let's do an ext/YAML module together with inline Haskell!" We end up choosing the SHA1 extension, because it is easier to implement, while still having a significant practical value.

After three hours of intensive hacking, we have managed to work out the design of inline() directive in Makefile.PL, as well as the semantics of an inlined module. I have also managed to get require_haskell("") working, thanks to the DynamicLoader module. Unfortunately, ingy did not check in his Makefile.PL magic before he went to sleep, so we have to wait until tomorrow morning for a working implementation for inlining Haskell and C code -- I predict an explosion of extension authors once that is in place, and even more so when we figure out how to generate .pmc from such extension modules.

nothingmuch spent another full day hacking on currying, and posted a nice proposal on p6l for currying positionals. Simple named arguments like sub_call(name => expression) is also made to work today.

Stevan has almost completed his port of File::Spec to Pugs, lacking only s/// support for Perl5 regular expressions. At his prodding, I promptly hacked substitution support in, but that raised much confusions for the actual types and semantics for ~~ s/// calls. Stevan helpfully summarized the questions we have met; hopefully it will meet with suitable answers soon.

I think the audience of YAPC::Taipei are in for a real treat this weekend. :-)

Day 50: pugscc --runparrot -e "'Hello, Parrot'.say"

The title is worth repeating:

% pugscc --runparrot -e "'Hello, Parrot'.say"

Yes, it works now, as of revision 1024. And yes, it does what you think it does. Pugs takes that Perl 6 source code, produce an AST, triggers the Compiler backend, generates some PIR code, saves it into a.pir, and runs it with the parrot executable in path. The compilation and execution speed are both very fast, and will get faster with embedded Parrot and/or Ponie support in the future.

This also works:

% pugscc --runhaskell -e "'Hello, Haskell'.say"

Instead of PIR, it generates raw Haskell code using Template Haskell, compiles it with GHC, and runs the resulting executable. Template Haskell is really good for prototyping and verifying a compiler; the code it produces is also quite fast, within 20% of Perl 5 speed.

The old, undecorated form of pugscc also works. It can also be written thus:

% pugscc --runpugs -e "'Hello, Pugs'.say"

The --pugs backend is the only one that guarantees to run identically to the pugs interpreter. Unsurprisingly, it merely bundles the evaluator with the Pugs AST, and makes an executable out from that.

On a somewhat more interesting note, if you can build Pugs with the PUGS_EMBED environment variable set to perl5 (that works on FreeBSD for me, but apparently not on many other platforms), then you can do this:

% pugscc --runpugs -e "eval_perl5('print q[Hello, Perl5!]')"

The eval_perl5($str) form is but a temporary kluge; it will likely become eval($str, :language<perl5>), or whatever form preferred by the perl6 design folks. Suggestions welcome!

Aside from perl5, I plan to add embedding support for Haskell and C via hs-plugins, and Parrot support via the embedding API. Of course, once Ponie gets up to speed, I can replace Perl5 and Parrot embeds with it, which will save a lot of value casting works.

Tomorrow I'll hack some more to get mandel.p6 working correctly on the two new compiler backends. Thanks to Dan Sugalski's help on #parrot, I have figured out the PIR-level supports to get this done. Parrot is really much, much saner than I had hoped; it eeriely supports lots of things needed by a Perl 6 implementation. I wonder why that is the case. :-)

There are many developmenets today. nothingmuch checked in a working .assuming() implementation; GHC 6.2.2 now works again Win32; scw fixed the behaviour of $! after eval; gaal started working on a YAML-based harness; and more things that I can't quite summarize. I'll try to catch up again tomorrow...