Pugs

Day 206: Updates on JavaScript and Perl5 runtimes.

Starting last week, there's an effort to combining common infrastructure and design for the JavaScript runtime and the Perl5 runtime, headed by iblech and fglock respectively, with plenty of help from putter at both sides.

To that end, putter added a new magical $?PUGS_BACKEND variable to track the codegen used. He also added backend-specific make test targets: `make test-js, make test-perl5, make test-pir, and make test-all` to run for all backends. Same applies to make smoke, too.

If you want to track the progress on all our runtimes, check out iblech's Pugs Smoke Reports page It's now linked from Pugs's homepage; smoke testers are encouraged to submit smoke.html there. Below is a few of selected news on JS and Perl5 backends -- I still had not backlogged through all relevant changes, let alone other changes in the Haskell runtime...

JavaScript

• Passing 3954/5117 subtests, 70 unexpected successes, 77.27% OK.
• Of the ext/ modules, Algorithm::TokenBucket now runs correctly in addition to Test.pm.
• Much more extensive README with implementation details.
• Perl5-compatible regular expression support via JavaScript-side PCRE, including captures.
• &?CALLER_CONTINUATION works fully, thanks to the codegen's CPS transform, marking the first runtime that support reentrant call/cc correctly.
• Support for coroutines via coro and yield.
• Flow controls, including next, last, redo as well as postfix while and until.
• Container equivalence test =:= distinguished from object identity test eqv.
• Correct semantics for reference to containers and auto-enreferencing arrays/hashes to refs.
• Restored junctive autothreading support.
• Slurpy hashes and arrays, which according to the latest ruling can't be passed in by name now.
• A new Perl5 module, Perl6::Run::JS, that wraps JavaScript::SpiderMonkey to work as an alternative runtime to bin/js and HTML browsers. This allows for mixed Perl6/Perl5 code to work even with the JavaScript codegen.
• (1,2,3) no longer allocates new containers, the way [1,2,3] does.
• UTF8 support for both bin/js and browsers.
• The CALLER:: pseudo package, as well as symbolic deference based on that.
• A new command: :e exp , for evaluating an expression in the interactive jspugs.pl shell.
• Fully qualified code variables such as &foo::bar() no longer autovivify to no-op, as it still is in the Haskell runtime.
• Emulated &sleep support by burning CPU, for SpiderMonkey which has no setTimeOut builtin.
• Simple list binding, eg. ($a, $b) := ($b, $a). Full subroutine-param syntax is not yet there.
• LValue subs (sub is rw) emitted as PIL and compiled directly to JS.
• Support for temp and let declarations.
• Macros for JS written in Perl5 via PIL::P5Macro; part of Prelude::JS is now written that way.

Perl5

• Test.pm does not yet run correctly, so sanity tests are the only tests that pass.
• All builtin classes migrated to use MetaModel 1.0, allowing full reflection from user space.
• Primitives can now be written as Perl5 macros in PIL::Run::PrimP5, as well as in Perl6 with PrimP6.pm.
• Support for read-only variables, which can only be assigned once: eg. my $x is readonly; $x = 3 (this replaces the old is constant trait.)
• Scalar containers handles autoderef/autoenref, binding and tieableness correctly.
• Hash containers is now based on Perl6::Container::Hash::Object, allowing any object to potentially serve as hash keys.
• Array containers are now fully sparse, allowing for efficient storage like [ 1, undef x 1_000_000, 2 ].
• Support for lazy Array slices; all destructive updates to Array preserve as much laziness and sparseness as possible. For example, @a = 1..9999999; @a[1..1000000] = @a[1000001..2000000] would take constant time.
• Ref class, with autodereferncing when pointing to Array or Hash container objects.
• Pair class, with the => constructor and stringification.
• Code class, with Sub, MultiSub and Block exposed as Perl6 classes.
• Invoking Perl6's Code objects in Perl5 now carries over slurpy (i.e. wantarray) context information.
• Various primitives, including basic variable manipulations, anonymous array/hash, list constuction, coercion and arithmetics.
• Inline::Pugs hooked to Perl5 runtime. This now works without spawning off a separate process:

    use Inline Pugs => q{
        sub intone { say "Hello, $_!" }
    };
    intone('World');

Whew. That was a lot. To be continued tomorrow...

Day 205: ...and back again.

I was packing and moving to a new place, and $job got crazy for quite a while. Now I'm finally back, backlogging the 300 commits happened in the past ten days. There's been so much progress, that I would need to work hard to get the now-monthly 6.2.10 release in place.

Fortunately, there's a large typhoon coming up, and I don't expect to go anywhere in the next couple days. That should give me plenty of time to catch up.

Oh, it turns out I will speak about Pugs in Euro OSCON after all, instead of my originally scheduled Perl and Internationalization talk. There's a writeup about this in gnat's blog.

Day 194..204: (blank)

Took a whole day to get all data successfully salvaged to the new 80GB hard disk. Sorry, no journal today -- I'll try to make it up tomorrow.

Update: It is tomorrow now and I'm still neck deep in this $work deadline thing. Sorry.

Update: Another day passed; thanks to both Parsec and Catalyst I should be able to survive this. But I'm missing all the interesting p6-on-p5 and perl5 typecheck fun. :-(

Update: It's been a week; $work and moving has been throughly consuming. Maybe I'll catch up the rapid movements for p6-on-p5 this weekend, maybe not. Sigh.

Day 193 (r6344): JSAN and CPAN, here we come!

Good news: Thanks to iblech, Pugs-on-Javascript can now load and use JSAN modules:

use jsan:Number.Roman <to_roman>;
say to_roman(42); # XLII

Another good news: Thanks to putter, stevan and fglock, Pugs-on-Perl5 compiler now emits code that directly uses the metamodel and intrinsic classes. It now compiles and runs t/01-sanity/01-tap.t in Perl5 correctly; the logical next step is to fill in the primitives and interoperate with use perl5:DBI.

The bad news is my hard disk totally and suddenly died a couple minutes ago, took the unfinished journal with it, along with my private .svk work depot. Darn... I'll see if I can salvage some of it tomorrow. :-(

Day 191/192 (r6310): Shop, Kontent, JS/CPS, Underscores.

In the last two days, four people orderd T-shirts from the Pugs Store, a fact that made me unreasonably happy. :-)

mugwump's cute "Powered by Ph.D." design joined the store, along with a metamodel raglan shirt. We are bouncing a few new ideas in #perl6, most of which related to a hypothetical WebService::CafePress module (eg. per-release shirts that are available for a very limited time). This leads to conversations like:

<kolibrie> autrijus: update the AUTHORS shirt every release, it can lag some
<kolibrie> but WebService::CafePress would be good
* autrijus ponders a t-shirt that reflects the sky every minute
* kolibrie knows autrijus is crazy now
<autrijus> that would not be a news to many people here :)

Okay, enough shop talk. Today I succeeded in getting brentdax's Kontent running -- the default page looks elegant indeed. Thanks to kolibrie for reporting the Kontent failure with Perl 5 binding; it turns out that the perl5 glue was casting POPi as bool, causing a method call to only succeed if its CODE pointer address is an odd number! This weak typing thing is really annoying... Fortunately it's all fixed now.

Our new committer, Yiyi "xinming" Hu, reported that method $obj didn't call $obj.method as specced in the Synopsis. I fixed it promptly.

As an added bonus, I fixed the long-standing parse glitch that makes class Foo { syntax'error } report the error at the outer block level, instead of where the error happened. This should simplify debugging OO code a lot.

iblech finished converting PIL2JS to use the much faster -CPerl5, cutting down compile time by 50%. However, the CPS transform made the runtime 2x slower, so the end result for make jssmoke is about equal before and after the conversion. putter suggested some trampoline techniques to speed things up, so there may still be room for improvement.

iblech also made DrIFT.Perl5 emit Math::BigInt objects for NaN and Inf, so Perl6's numeric semantic can be correctly duplicated in Perl 5. As a result to the CPS conversion, various new features are working in JavaScript, including a true lexical $_; $CALLER::_ follows naturally.

putter added a link to mugwump's Wellinton.pm Perl6 talk in docs/talks/README.

Limbic_Region reported that 1_000_000.00_00 was not parsed, and I implemented it right away. Curiously, that broke some code, because dot is a method invocation now, and 1.e10 can take two meanings. An ongoing p6l thread is triggered to resolve this issue.

fglock cleared up much of the boxed/unboxed/reference semantics in the Perl 5 runtime implementation, and reimplemented the Scalar container to autoderef calls such as $x[2]. It's really exciting to see Pugs inch closer to the day of Perl6-on-perl5...

Day 190 (r6272): Pugs Store opens!

Based on numerous feedback to yesterday's images, I simplified the design somewhat and uploaded them to the newly established Pugs Store.

putter suggests we rotate selected items every three months. However, I decided to subject the entire site to that treatment. Hence the current Fall 2005 edition will expire on the end of October. New design ideas are most welcome! :-)

mjl69 mentioned that he has been looking for a T-shirt featuring Pythagoras's Theorem. So I produced one and filled it into another non-pugs-related store.

iblech got DrIFT working on his machine, tweaked DrIFT.Perl5 to escape things properly, then started porting PIL2JS from the old $node->[0][1][0] style to the new, much saner $node->{pFun}{pBody}{pStmt} based on the new -CPerl5 backend.

fglock hacked through the Perl 5 runtime, implementing Hash containers and various boxed value classes -- with tests, too!

putter conjured up a Rul class that serves as a pure-Perl6 representation of a Perl 6 rule. He took over :w and :i handling from PGE, and implemented a new :overlap adverb.

Another work day ahead, so this is again a brief entry. See you tomorrow!

Day 189 (r6253): Pictures, DrIFT(JSON, Binary, XML), MetaMod

Two new pictures joined the Pugs image gallery today:

• Perl 6 - Imaginary Timeline
• Pugs - Lambda Camels

Some #perl6 folks talked about turning them into T-shirts. I only know about Cafepress and Zazzle -- any better suggestions out there?

After the Visio session, I continued refactoring and adding DrIFT instances. The -C Binary backend enables Pugs to efficiently store and retrieve PIL trees, paving the way to separate compilation for Perl6 modules.

Aankhen suggested me not bothering doing XML backend, so naturally I did it right away. Thanks to Haskell2Xml, this only took a couple lines -- see hello.xml for a sample output. However, I disabled it for now, as there are no active consumer for this, so I don't need to worry about the HaXml dependency right now.

Next in my goal is YAML. I started with the Syck emitting interface, but soon it dawned on me that JSON is a subset of YAML. I can reuse Syck as a JSON parser, and the emitter is downright trivial. Here is the output of pugs -CJSON -e "say 'Hello, World!'":

{"PIL_Environment": {"pilGlob": [{"PSub": {"pSubName": "&*END","pSubType": {"SubPrim": null},"pSubParams": [],"pSubBody": {"PNil": null}}}],"pilMain": {"PStmts": {"pStmt": {"PNoop": null},"pStmts": {"PStmts": {"pStmt": {"PPos": {"pPos": {"MkPos": {"posName": "-e","posBeginLine": 1,"posBeginColumn": 1,"posEndLine": 1,"posEndColumn": 20}},"pExp": null,"pNode": {"PStmt": {"pExpr": {"PExp": {"pLV": {"PApp": {"pCxt": {"TTailCall": [{"TCxtVoid": null}]},"pFun": {"PExp": {"pLV": {"PVar": {"pVarName": "&say"}}}},"pInv": null,"pArgs": [{"PLit": {"pLit": {"PVal": {"pVal": {"VStr": ["Hello, World!"]}}}}}]}}}}}}}},"pStmts": {"PNil": null}}}}}}}

SpiderMonkey seems to like it just fine, so I can start idly wondering about meta-programming Perl 6 with JavaScript... Mmm bad ideas.

On the MetaModel front, nothingmuch hacked the minimetamodel to support various plugins, including C3, multiple inheritance, and autolaoding.

Taking a more refactorial path, Stevan checked in MetaModel 2.0, a much cleaner implementation of the Perl5 metamodel. Following the gnostic naming convention in the metamodel picture, the files are named as genesis.pl, pneuma.pl, etc. He also added a chaos.pl that contains primitive utilities that exists a priori.

putter finished the first milestone his long journey in writing a Rules engine in Perl 6. The Pugs parser can now calls back to Prelude.pm's rx_ macros, which can then call into a Rules compiler written in pure Perl 6. He recounted some post-mortem thoughts on #perl6. It seems to me that the main takeaway is centralized version-control hinders development; switch to svk today! :-).

fglock produced Perl6::Value implementation of Num, Int, Str and Bit classes, for the Perl 5 runtime.

There's more to chronicle -- eg. my planned .il hackathon in February, but it's $work day tomorrow, so I better sleep now. See you!

Day 188 (r6230): Perl 6 compiled to Perl 5 data structures.

This may not be exciting as compiling Perl 6 into Perl 5 programs, but it's exciting news nevertheless. With help from the amazingly flexible DrIFT, pugs -CPerl5 can now serialize the compiled PIL1 tree into a Perl 5 expression:

% ./pugs -CPerl5 -e "say 42"
bless({
    pilMain => bless({
        pStmt => 'PNoop',
        pStmts => bless({
            pStmt => bless({
                pPos => bless({
                    posName => "-e",
                    posBeginLine => 1,
                    posBeginColumn => 1,
                    posEndLine => 1,
                    posEndColumn => 7
                } => 'MkPos'),
                pExp => (undef),
                pNode => bless({
                    pExpr => bless({
                        pLV => bless({
                            pCxt => bless(['TCxtVoid'] => 'TTailCall'),
                            pFun => bless({
                                pLV => bless({
                                    pVarName => "&say"
                                } => 'PVar')
                            } => 'PExp'),
                            pInv => (undef),
                            pArgs => [
                                bless({
                                    pLit => bless({
                                        pVal => bless([42] => 'VInt')
                                    } => 'PVal')
                                } => 'PLit')
                            ]
                        } => 'PApp')
                    } => 'PExp')
                } => 'PStmt')
            } => 'PPos'),
            pStmts => 'PNil'
        } => 'PStmts')
    } => 'PStmts'),
    pilGlob => [
        bless({
            pSubName => "&*END",
            pSubType => "SubPrim",
            pSubParams => [],
            pSubBody => 'PNil'
        } => 'PSub')
    ]
} => 'PIL::Environment')

iblech expects PIL2JS compilation to be several times faster, once he switches it to use this representation. Furthermore, as the derive Perl5 works for any Haskell data type, we can load in PIL2 and their support structure in Perl 5 with no extra effort, which will greatly simplify the true Perl6-on-perl5 effort.

Tomorrow I'd like to work on -CYAML, -CXML and/or -CBinary, just to make sure that everybody can play with Pugs's internal structures, so we can have macros and refactoring browsers written in non-Haskell languages.

Day 187 (r6222): Visiolizing the MetaModel.

As I promised yesterday, today I worked on drawing out the metamodel, covering various factors involved in the bootstrapping -- i.e. exposing the internal implementation of classes into user space as ::Class. It's available in two formats:

• Static PNG
• GIF animation of the bootstrapping process, made by Stevan

Much thanks to #perl6 and #parrot folks for encouragement and feedback. Please refer to this previous journal for the origin of those gnostic names. Stevan's new Mini MetaModel is a much more readable Perl 5 implementation -- bootstrapped and tested in less then 200 lines!

Other documents under the same directory are helpful, too, including Stevan's 10k ft view and kolibrie's 20k ft view.

On the PIL2JS front, iblech sped up compilation by 300%+, and the compiler now takes much more reasonable amount of memory. He also hacked in junctive autothreading inhibition for Bool parameters, returning out of multiple blocks, and various other nifty things.

fglock polished the Array container and List value implementation for the Perl5 runtime, and plans to expand his work on other container and value types.

That's it for today. See you tomorrow! :-)

Day 185/186 (r6197): Inferencing and named returns.

Work got busy in the past two days, so I didn't do much on the Pugs front, except fixing a couple bugs (eg. lower-cased type names are now legal -- reported by jql). I also switched the new PIL runcore to use a faster Str storage. So we can't have infinite strings now, but fortunately that wasn't in the spec anyway.

The long discussion on p6l about type inferencing continued. I'm glad that Larry seems to like my named return idea:

sub init_dbh (Str $dsn --> DBI $dbh) {
    $dbh .= connect($dsn);
    $dbh.do('some init code');
    say 'I return $dbh anyway!';
}

This is suspiciously like Fortran's out variables, but I think it improves readability nonetheless.

fglock is churning out an amazing amount of Perl 6 code, covering Arrays, Lists and some of the newly-promoted Sets. He also translated modules and tests to Perl 5, reportedly with help from a five-line regex. Check out the TODO sections for more things to come...

After recovering the JavaScript runtime from the continuation-passing rearchitecting, iblech hacked the current PIL generator to handle multisubs. He also skipped some IO tests when $*OS eq 'browser'. I guess $*OS may become emacs some day...

geoffb, in his a helpful documentation gnome hat, updated the STATUS document and the source tree map.

Aankhen changed require to use in the HTTP::* family of modules, and changed &coerce:<as> from method to sub.

vkon revamped quickref/var to cover more magical variables in a more organized fashion.

Stevan eliminated the incestuous relationship between the OO metamodel in Perl 5 and Perl 5's own package system, and added ASCII arts to the 10k ft view. I look forward to Visiolize them during the weekend.

dudley carried on Stevan's preliminary work on >org.perl6.metamodel, the Java port of meatmodel -- I mean metamodel.

putter fixed a post-6.2.9 bug; Pugs was sometimes accidentally handing perl6 rules to PCRE, causing strange errors.

I think that's it for today... Fortunately the weekend begins tomorrow. :)