Pugs

Day 147: Sleep, at last.

During my YAPC talk, I commented that the past five months feels like five really long days to me -- I kept the Reality window firmly minimised; Pugs virtually occupied all my waking moments, if not totally, at least in part. Even my dreams are populated with syntax trees, code generators, and runtime dispatch algorithms.

This month especially feels like a really long day, constantly interacting with some of the most interesting people I know, with no much room left for rest -- I'm glad I have the window of one-day sleep through tomorrow, before I fly back to Taipei and have to face $jet_lag and $work again. :-)

This afternoon I had some really good talk with cwest about his JavaScript Archive Network ideas, and his ongoing quest to expand the Perl community's norms -- the culture that let us fill in practical utilities in all possible gaps, with brutal efficiency -- to areas beyond the current CPAN. We also talked at length about compiling Javascript to PIL and PIL to Javascript, and the benefits of using one single well-defined vocabulary for database, server-side logic and client-side scripting. The idea is similar to Wadler's Links project, but with a vast body of CPAN-rooted norms (eg. DBI, DateTime, Test::*, LWP/Mechanize, etc.), so maybe we will have a better chance of adoption when the technology becomes available.

Btw, my set of Test::Base demos for ingy's talk is online. I think the Test::Base methodology (a natural extension to Cunningham's FIT) is sound, and I look forward to bring it to Perl6 as well as adding a specification-based generator interface (ala Test::LectroTest, QuickCheck) to it.

...I see that chromatic, stevan and nothingmuch are still hacking madly when I'm writing this, but now I have to sink back into blissful sleep again...

Day 146: Post-YAPC hacking

At the town hall meeting, the organizer rdice made fun of the lambdacamel table, where lwall, stevan, nothingmuch and luqui are still hacking madly, largely oblivious what happens on stage. :-)

After the town hall meeting -- I'm told that Andy mentioned Pugs and it got an ovation, but I sadly missed it -- we moved to my room to continue hacking. Stevan finished a less hacky model of Perl 6 object space in Perl 5, so I can finally start porting it to Haskell; we figured out ways to encode Pad in PIL, and nothingmuch/luqui switched to Code::Perl instead of PPI for the codegen.

We also got some really amazing people trying out Pugs: drolsky is trying to rethink his DateTime API in Pugs, and coke (of ParTcl fame) committed the long-sought message in Makefile.PL that warns OSX 10.4 users that they need a gcc_select 3.3 before running make.

chromatic and I worked for an hour to get down to the weird bug that reverts the instance attribute in in submethod BUILD (?$.x=3) { $.x=4 } to its binding value. I joked that I need to get to the bottom of this, or I'll commit seppuku -- and promptly traced it down to see that a single return will fix it, but falling through the block won't -- however I was stuck there, so I'm forced to commit seppuku.hs (to my local svk branch). Oh well, at least we hacked in useful stuff like $obj.perl. I'll get to the object alligator when I get enough sleep...

...but chromatic promptly fixed it with a two-lines patch. It turns out his original implementation runs the parameter-defaulting code after the body, so of course it broke -- but it's all good now. :-)

Showstoppers for 6.2.8:

• Prelude.pm needs to be precompiled -- it's too slow to load it. 10x slowdown for a new release cancels all our optimization work in the past few releases. :-/
• Fix qualified name lookup that broken Darren's Locale::KeyedText
• Of course, solve all the regressions... which is going quite well already.

Day 145: YAPC - We made it!

If the previous journal sounds desperate, that's because it was the case... I woke up at 6am and completed only the first half of the slides; by the time I started my talk, I've still got a few missing images and broken links, not to mention lots of typos, and a bootstrap graph produced 5 minutes before the talk. But somehow, it all worked out!

Thanks to earthy, masak, iblech and other people who committed fixes and additions to the slides source. The rendered HTML is now online.

I spent much time working on #30 and #70, but I think I said what I had to say, and I'm very glad that people seem to take it in a constructive fashion. Thank to all lambdacamels for being there with me. :-)

But now I need to crash (and sadly miss the cruise). If I manage to wake up on midnight I'll update this space again...

...which I did. Ingy summoned my coding-on-the-fly-fu to write a series of tests, refactored from the most mundane Test::More style that we all take for granted, into something so absolutely amazingly concise that it's just one use line with expected input and outputs. It has to be seen to be believed (and I want that in Pugs now), so if you hate writing tests as much as I do, be sure to attend Ingy's talk on Wednesday 10:45am about Test::Base. :-)

Day 144: Night before the talk

...no journal tonight, for obvious reasons...

I didn't get to finish my talk that will go on air in 8 hours; help very much welcome on cleaning up the typo/thinko/grammo, finding images from image.google.com, and filling in the missing parts. Otherwise... I'd need to finish the second half during the 15 minute intermission, or speak to a blank screen. :)

6am Update: I'm up now and resuming working on slides. Thanks Stevan for checking in the MetaModel code so I can fill in the XXXs better, and thanks to darren/spinclad for sanity-checking the existing slides.

Day 143: Night before YAPC::NA

Last night luqui entered the zone, and cranked out a PIL to Perl 5 codegen interface. Stevan's Perl 6 object environment in Perl 5 is progressing nicely; today nothingmuch extended the codegen with PPI, so we can work with quasiquoting Perl 5 syntax trees instead of concatenating strings.

The upshot of all this is, in addition to GHC and Parrot, Perl 5 is going to be the third runtime that we can run Perl 6 on. Because the codegen and PIL representation are themselves written in Perl 6, we may recompile this:

    use perl5:PPI;
    use Perl::Compiler::CodeGen::Perl5_PPI;
    # ...codegen from PIL to Perl5...

Into this Perl 5 code:

    use PPI;
    use Perl6::Loader 'Perl::Compiler::CodeGen::Perl5_PPI';
    # ...codegen from PIL to Perl5...

and run it with the Perl 5 runtime. Perl 6's base types and the roles/class system are evaluated in Stevan's Perl6::MetaModel module to be released later this week; Coro and Perl6::Junction etc may be needed to simulate the more complex parts of the control flow.

At the Cabal meeting today, Allison mentioned that it would be nice to have the same set of PIR/PASM abstraction tree I'm using available as Parrot objects, so people targetting Parrot can generate their code in a more robust form, and said it would be nice to have it at OSCON time. Well, by tonight's dinner time I've hacked the Parrot objects emitter from any Haskell data structures to PIR objects. To make use of them in the stand-alone PIR, we'd just need to port the 100ish line of instance Emit defintions. We can also use the same Haskell-declarations-to-objects process to keep PIL's Haskell form and Perl 6 form in sync, which would ensure reusable macro and compiler code, which allows gradual porting of Haskell to Perl 6.

There's plenty of other news, for example the first templating system being developed on Pugs, but I really need to crash now after less than 3 hours of sleep last night. And I also got a 3hr talk on Tuesday, which I have zero slides prepared... ;-)

Day 142: Hackathon day #3

Subversion repository is finally back, and I'm committing things we've been hacking on in the past two days. luqui and nothingmuch hacked on reifying the PIL AST as Perl 6 objects (so we get true macro support for free), and a codegen written in Perl 6 that takes PIL and emits Perl 5. Stevan worked on a Perl 5 metamodel that runs Perl 6 roles and classes from Perl 5 space, so we can use Perl 5 as our host VM (in addition to Parrot and Haskell); nothingmuch further joked about a PIL codegen that targets Forth (but turns out he was entirely not serious); Patrick implemented builtin rules for PGE, and put qualified and inheritable+lexical grammar/subrules support in place. Ingy worked on his new Perldoc framework that provides unified, comment-driven, haddock/javadoc-like metadata and macro-friendly way to generate Kwid and POD and translate between them. lwall continued to hack through mad segfaults on PDD/MAD_SKILLS, refactoring Perl 5's toke.c to be more informative.

Allison arrived today and we debated for nearly an hour on the implementation strategy. This Pugs Hackathon work was focused on having the Perl 6 compiler written in Perl 6, that translates the parse tree into PIL, then from PIL to either a Parrot syntax tree that emits PIR, or to Perl 5, Mono or even Javascript. However, that means to compile a newer version of Perl 6 compiler, we need to have an older version of Perl 6 around, first via the non-Parrot side of Pugs, and then in the form of a Perl6.pbc runtime. This is just like how the Mono C# compiler is written in C#, or GCC itself written in C, or how PyPy is written in Python.

Allison prefers instead to see the production Perl 6 compiler and related tools to be written entirely in PIR or other non-Perl6 Parrot languages, so that we can compile any version of Perl 6 without having access to previous versions of Perl 6, and she suspects that a parser/compiler/emitter written in PIR would be easier to write and maintain than the same toolchain written in Perl 6. The parser and emitter tools will then be reusable by other languages (eg. PHP) that want to target Parrot, because PHP folks would prefer to use the compiler suite written in PIR than one written in either Perl 6 or PHP. That ties Perl 6 to PIR, but one can persumably link in libparrot in Mono or JVM to run the PIR code via their foreign call interface. This is similar to how the Perl 5 runtime is written in XS/C, and just like a (hypothetical) Mono-targetting compiler written in the .NET/IL high-level assembly language.

Thus, we have two possible implementation strategies that will evolve separately. People will, well, use whichever one that actually works. :-)

Now... revelation time!

• To use a perl 5 module, put a perl5: in the front of the module name:

    use perl5:DBI;

  Extending this metaphor, to use a python module:

    use python:Zope;

• When calling a function, the unary splat * takes an aggregate, or reference to an aggregate, and flatten them out on the invocation list. Unary splat on hash arguments flattens it out as pairs for named bindings; splat on scalars deref it to find an array/hash reference; for code and non-reference-to-aggreate scalars it's a no-op.
• &prefix:<int> now always mean the same thing as &int. In the symbol table it's all stored in the ``prefix'' category; &int is just a short name way for looking it up -- it's just sugar, so you can't rebind it differently.
• Here is a more clarified role hierarchy:

    Any | Object  | Item | ...pretty much everything else goes here...
                  | Pair
                  | Junction
        | int
        | str
        | num

• Constrained types in MMD position, as well as value-based MMDs, are not resolved in the type-distance phase, but compile into a huge given/when loop that accepts the first alternative. So this:

    multi sub foo (3) { ... }
    multi sub foo (2..10) { ... }

  really means:

    multi sub foo ($x where { $_ ~~ 3 }) { ... }
    multi sub foo ($x where { $_ ~~ 2..10 }) { ... }

  which compiles two different long names:

    # use introspection to get the constraints
    &foo<ANONTYPE_1>
    &foo<ANONTYPE_2>

  which really means this, which occurs after the type-based MMD tiebreaking phase:

    given $x {
        when 3 { &foo<ANONTYPE_1>.goto }
        when 2..10 { &foo<ANONTYPE_2>.goto }
    }

  in the type-based phase, any duplicates in MMD is rejected as ambiguous; but in the value-based phase, the first conforming one wins.

• Closure composers like anonymous sub, class and module always trumps hash dereferences:

      sub{...}
      module{...}
      class{...}

• The do form is now taking a single statement (that may still be a block); what it does is turning the statement into an expression form, immediately evaluating it when the left hand side demands a value.

    my $val = do use CGI;   # same as
    my $val = BEGIN { use CGI };
  
    # This assigns 4 to $foo
    my $foo = do given 3 {
        when 3 { 4 }
    };

• A closure form of but is desugared into a do given block that eliminates the need of returning $_ explicitly. So those two forms are equivalent:

    my $foo = Cls.new but {
        .attr = 1;
    };
  
    my $foo = do given Cls.new {
        .attr = 1;
        $_;
    };

• The anonymous class is allowed to both is and does any class/roles; it will be composed in compile time into an anonymous (but unique) class, same way as an anonymous closures remembers its original place of definition:

    role Foo {...}
    class Boo is Baz {...}
  
    (class{ does Foo; is Boo }).new(1);

• The is Foo and does Bar declarations inside class body is always lifted up as class traits and executed at class composition time.
• There's another pseudopackage, OUR:: and symbol table form %OUR:: that contains the symbols in your current package namespace.
• trust is lexical: It controls accessor generation for all my, our and has forms in its scope. Inside a class body, the my $.x and our $.x always generates public accessors on the spot:

    class Foo {
        trusts Bar; # sees the accessor methods in the scope
        { # some inner scope
            my $.x; # This creates accessors by inserting these two lines
            # die "Duplicate accessor" if %OUR::<&x>;
            # our &x := method () is rw { $.x };
  
            my $:y; # This create accessors by  inserting these two lines
            # my &:y := method () is rw { $:y };
            # $?SELF.trust_access.push(&:y); # adds to Bar -- $?SELF is class object
        }
        trusts Baz; # this is always an no-op because there's nothing below
    }
  
    class Pie is Foo { }
  
    Pie.x = 5;      # lvalue method - writes back to lexical $.x
    say Bar.x;      # 5 - shared by Bar and Pie

  The twigil . and : controls the generated accessor's scope (our and my respectively). The scope of the variable itself is orthogonal to the accessor.

• In order to pass by read-only reference, we need a way to bind a container into a new name, but say that it is read-only while the original is read-write. This is a contradiction, since binding by definition binds a container to two names. Still, we have to do it.

  In parameter lists, the is constant default trait on parameter variables is not really acting on their containers; it creates a transparent container on top of an existing container. It is ``transparent'' because it autoderef for everything except for write-type STORE/PUSH/SPLICE methods; all read methods like FETCH are passed to the underlying container. Even .ref and .does calls are passed through to the underlying container, but the .tied call does get you the wrapped implementation object.

  The upshot is that these are now errors:

    sub foo ($x) is rw { $x }
    my $a;
    foo($a) = 4;    # runtime error - assign to constant
  
    sub constref ($x) { \$x }
    my $a;
    my $r = constref($a);
    $$r = 4;        # runtime error - assign to constant

  To get a normal reference, use the is rw trait on the parameters.

Day 141: Hackathon day #2 (continued)

This single Perl 5 expression:

print "1..2\n";

has been translated by lwall's PERL5_MADSKILLS (also known as PPD) into a XML dump of the OP tree that looks like this (abridged):

<op_leave seq="1 -> DONE" targ="1" flags="VOID,KIDS,PARENS" private="REFCOUNTED" <op_leave seq="1 -> DONE" targ="1" flags="VOID,KIDS,PARENS" private="REFCOUNTED" refcnt="1">
<op_enter seq="2 -> 3" />
<op_nextstate seq="3 -> 4" flags="VOID" line="5" package="main" />
<op_print seq="6 -> 7" flags="VOID,KIDS">
<madprops>
<mad_pv key="o" val="print"/> </madprops>
<op_pushmark seq="4 -> 5" flags="SCALAR" />
<op_const seq="5 -> 6" flags="SCALAR" PV="1..2&#10;">
<madprops> <mad_op key="F">
<op_stringify seq="39 -> DONE" targ="1" flags="SCALAR,KIDS"> <op_null seq="0 -> (40)" was="pushmark" flags="SCALAR" /> <op_const seq="40 -> 39" flags="SCALAR" PV="1..2&#10;">
<madprops><mad_pv key="X" val="1..2\n"/></madprops>
</op_const>
</op_stringify>
</mad_op>
<mad_pv key="o" val="&#34;1..2\n&#34;"/> <mad_pv key="_" val=" "/>
<mad_pv key="(" val=""/>
<mad_pv key=")" val=""/> </madprops>
</op_const> </op_print>
</op_leave>

This, in turn, goes through a xml2pl program in Perl 5 that turns it into a P5AST object. After that, the object is dumped into Haskell expression:

P5AST [ OpLeave [ OpPrint [Items [ ] , Items [ Bare " " , OpConst [Bare "\"" , OpStringify [Bare "" , OpConst [Bare "1..2\\n" ] , Bare "" ] , Bare "\"" ] ] ] ] ] ...

Pugs's new Pugs.Frontend.P5AST module takes that expression, then compile to a Pugs parser tree of the Exp type:

Stmts (App (Var "&print") Nothing [Val (VStr "1..2\n")]) Noop

After that, the Pugs.Compile module takes over and turn it into a PIL intermediate code expression:

PStmts PNoop (PStmts (PExp (PApp TTailCall TCxtVoid (PExp (PVar "&print")) [PLit (PVal VStr "1..2\n"])) PNil))

Which can emit this very much unoptimised PIR code:

.sub "main" @ANON
    .local pmc P1263_lex
    P1263_lex = new .PerlUndef
    P1263_lex = find_name "&print"
    .local pmc P1264_lit
    P1264_lit = new .PerlUndef
    P1264_lit = assign "1..2\n"
    .local pmc P1265_app
    P1265_app = new .PerlUndef
    set_args '(16)', P1264_lit
    get_results "(0)", P1265_app
    invokecc P1263_lex
.end

...and Parrot happily runs it and prints the string. Pretty cool, no? Especially cool is that we are getting the P5AST in its desugared (post-source filter) form, so it's indeed using perl to parse Perl.

Day 141: Hackathon day #2: More revelations

(to be continued...)
(and no, Perl 6 was too much fun so that we didn't tackle the ICFP contest in the end. sadly.)

• Roles are also classes! They can be instantiated just fine if they are concrete enough. Basically they mean composable classes or mixin-able classes. Hence, RoleName.new() instantiates an object that will probably fail on all stubs.
• Class-defined methods with the same short name as their roles must conform to the same signature as the role methods; otherwise a fatal exception occurs.
• Abstract pure methods in Roles really just return an Exception object, exactly the same way as if the ... is evaluated normally.

  method foo () { ... }

• Perl 6 is defined with separate compilation in mind; each compilation unit is going to pretend they are compiled in a different process. They are allowed to bind names (exports) in other namespaces, but they cannot read anything from others.
• &chomp and &wrap are now nondestructive; chomp returns the chomped part, which can be defined by the filehandle that obtains the default string at the first place. To get destructive behaviour, use the .= form.
• Filehandles opens chomped by default; you need to add the :newlines flag to turn off chumping.

  my $fh = open 'file';                   # autochomp
  my $fh = open 'file', :newlines;        # non-autochomp
  $fh.newline = "\r";                     # now split on \r
  $str = $fh.readline;
  $str.newline;                           # \r
  
  for chomp =$fh { ... }

  If .newline is set to a rule, then its captured variables are made available to the calling block as if it has done a matching.

• The chained associativity level is associative with the precendence level it is in; hence, in the same level all chainfix operators glue together the same way the chained comparisons glue together. The listfix associativity always accept one trailing operator, so 1 Y 2 Y 3 Y is legal.
• Rules have their grammatical category, &rule:<foo>, that can reside in either grammars, non-grammar packages, or the true global namespace where they get names like &*rule:<ident>. Inside grammars they are called &Foo::rule:<blah>.
• Grammar is a specialized form of Role (which is a special form of Class, which is a special form of Package). When a grammar does another grammar, it mixes in the grammar's rules as package globals.
• To invoke an unqualified subrule, look at the lexical scope of the invocation site, then package globals, then finally global builtins. This is exactly the same order as a normal subroutine lookup. One can think rules as submethods that gets mixed in with a does call. You call them with the current invocant (i.e. the Grammar):

  # calling <ident>
  &rule:<ident>.($?SELF: $state);

• To invoke a qualified subrule, one look the package-level named rule up, then invoke it with two parameters: The invocant is the package object that subrule belongs to; the argument is the state object the rule engine is currently treading upon:

  # calling <Grammar::ident>
  &Grammar::rule:<ident>($Grammar: $state);

Day 139/140: Hackathon day #0/1

The Hackathon is a huge continuous flow of ideas, chatters, pizzas, keystrokes, etc...

All Pug's uncatchable value cast errors are eliminated by theorbtwo's sweeping AST.Internals change, which uncovered various bad assumptions in the evaluator core -- I spent most of time today chasing them down. So, the planned release has (yet again) delayed. :-)

During my flight to Toronto, I jotted down many questions to ask Larry; I'd glad that Larry resolved most of the show-stopping ambiguities, with help from luqui and other hackathoners.

Yesterday, Test.pm finally works correctly with PIR compilation, so I ran make pirsmoke for the first time, which passed 2.5% of all subtests. As I'm writing this, luqui is close to get eval working on Pugs-generated PIR code -- this should improve our percentage by a significant amount.

stevan brought a new paper that describes a concise, dynamically extensible, polymorphic typed metamodel based on kind/class/objects. He is implementing it in Perl 5 right now; once that works we can port it to Haskell and use it in both the evaluator and the codegen.

lwall is hacking on PPD, the Perl5 optree dumper that defeats optimisations. The Perl 5 to Perl 6 translator may then work by compiling PPD to PIL, and then pretty-print PIL back to Perl 6.

Aside from cooking delicious and inflammable pizza, nothingmuch managed to get a Perl 5 implementation of Forth going, as a prototype of the Harrorth work. We didn't get around to generate PIR from that, though.

ingy showcased his Test::Base superior technology, where you can get specification-based testing done automatically for your projects just by writing two lines of harness. He is porting it to Perl 6 as I write this, which would be very cool indeed.

theorbtwo is tracing out the remaining failing tests, so we can have a reasonable chance of cutting a Pugs release tomorrow. Patrick arrives tomorrow; I hope to get PGE repaired against the trunk parrot, and get closures in Rules working.

There's much, much, much more activity going on than I can describe here -- I covered maybe about 3% of it. Below is a glimpse of a few Perl 6 design glitches we've been ironing out so far -- there more to come...

• deref is now 0-level; $x = 3; $y = \$x; $y++. # now an exception
• coercion is now done in mmd level in coerce:<as> calls. sqrt("element") now yields to 0. my Int $x = "foo" as Str casts.
• is lazy parameters in MMD now causes the argument to be delayed even if other MMD candates mandate them as eager; it enters MMD not with the type of its evaluated result, but with its inferred type at compile time.
• &prefix:<=> is just .shift in item context; in slurpy context it just turns the iterator into a generator. All arrays are concatenations of generators (which may or may not be preflattened)
• &slurp() is the eager variant of list context &prefix:<=>; you can, alternately, use the steamroller &prefix:<**>.
• my() declarations scopes lexically to the rest of the block; using $MY::x or $::("x") in the block before the actual declaration is erroneous.
• Only Roles take type parameters. Hence, Array and Hash and Scalar are roles; PerlArray/PerlHash/PerlScalar are classes that conforms to them. A variable is bound to containers of the Perl* types by default, depending on its sigil.
• Any is now really Any (includes Junction); the role hierarchy looks like this:

•      Any
       /  \
  Item    Junction

• Non-source-filter-ish macros work on the PIL(AST) level, not on parse tree level. The AST should preserve enough information to derive the original parse tree and source code back, for the compiler to use.
• return inside blocks always throw out exception specific to its containing Routine; if the caller chain does not include that routine, it's just a fatal exception.
• The while loop assigns the evaluated result into the block, but only if it takes an one mandatory parmaeter:

• while foo() -> $_ {...}   # binds $_
  while foo() -> ?$_ {...}  # does not bind $_

• The Hash, Int, Str etc are just roles; implementation classes are known as PerlHash, PerlInt etc; they are not generally used by the user.

• role Hash[?::returns=Any, ?::shape=Str] { ... }

• For pair/named binding to work, the inferencer needs to know it is a pair before putting it in the named binding position.

• sub foo ($x) { ... }
  sub bar returns Pair () { (x => 3) }
  my $y = (x => 3);
  
  foo({ x => 3 }.()); # This assigns (x=>3) to $x
  foo(bar());         # This assigns 3 to $x
  foo($y);            # This assigns 3 to $x, too
  
  sub foo (?$x, ?$Inf) {}
  my $y = (x => 3); foo($y); # binds $x
  my $z = (+Inf => 3); foo($z); # binds $Inf

• Parametric type is now gone. Bare is merged with Block -- it's all of the Block type now.

Day 138: Safely arrived to Toronto.

...is all I could write. Need to crash now. Thanks nothingmuch and his friends for hosting me for tonight. :-)