Pugs

YAPC-SA-2007 Hackathon

Here is the report on the 4 days of the YAPC-SA-2007 Hackathon, from April 12 to 15, 2007, in Porto Alegre, Brazil (during fisl8.0).

v6/docs
- added the KindaPerl6 diagram
- added a KindaPerl6 memory layout diagram - with help from #perl6
- added the MiniPerl6 talk (pdf)

MiniPerl6
- started the Perl6-in-Parrot target: desugars MiniPerl6 into p6parrot supported features
- grammar fix: semicolon is optional after a block
- perl5 backend: implemented warn()
- new: mp6-to-AST script
- nferraz started porting Test.pm to mp6

KindaPerl6
- memory management: BEGIN-block side effects are logged
- updated plan, discussed a possible release schedule
- fixed the container proxy algorithm (the implementation needs to be fixed; pugs has the same problem)

v6.pm
- fixed the "fish-operator" =<>
- fixed the quoted-word-operator <...>

Pugs::Compiler::Rule
- ratchet emitter fix, array interpolation now works

misc/pX
- nferraz and fglock started an adventure game implementation (with v6.pm), to be presented by nferraz at the Nordic Perl Workshop.

The network proxy at the conference did not allow commits to pass through, and there will be some delay until all the changes get to the repository.

Thanks to cmarcelo, david_fetter, eden_c, lorn, merlyn, and nferraz for 4 great days!

- Flavio S. Glock (fglock)

A beautiful calling convention.

The Beauty of Perl 6 Parameter Passing, a new article on perl.com by philcrow++, is an excellent intro into some of the wonders of the Subroutines spec (S06). We absolutely need more articles like this. :-)

Perl 5 Embedding Docs available

A revamped Guide to Perl 5 Embedding is now available. It provides details on how to pass the most common data structures to Perl 5, including arrays and arrayrefs, hashes and hashrefs, strings, and even code references.

Examples of using Perl 5 OO from Perl 6 are also provided. Some related bugs have also been fixed recently in Pugs.

Perl 5 embedding allows you create a project in Perl 6, while still using Perl 5 for parts that have not yet been ported to Perl 6.  Updated Perl 5 Embedding is just one of the many updates that will be available in the next release of pugs, expected soon.

Check smoke results while reading Synopses

Our new smoke server now has some extra links with the name of SYN in the right margin of the page, where one SYN link corresponds to one smoke report:

http://m19s28.vlinux.de/cgi-bin/pugs-smokeserv.pl

Click on one of these links and you will be redirected to a list of Synopses. Enter one synopsis, say, ``02 Syntax'', and you will find out the magic.

As you can see, these synopses contain test passing/failing marks in the code snippets. (This feature was originally suggested by Christopher++ and Gaal++.) For example:

√	is($foo, "blah", "lone block actually executes it's content");
	my $foo2;
	{$foo2 = "blah"};
√	is($foo2, "blah", "lone block w/out a semicolon actually executes it's content");
	my $foo3;
	({$foo3 = "blah"});
×	ok(!defined($foo3), "block enclosed by parentheses should not auto-execute (1)", :todo<bug>);

Note that TODO tests are marked as passing only if they're actually passing (i.e., unexpected passing from the perspective of implementers) and they're marked as failing only if they're actually failing (i.e. expected failing). I don't introduce special marks for these tests because added complexity will lead to unnecessary confusion.

Skipped tests have no marks at all and the intention is straightforward. Just don't mistake blank tests for smartlinks.pl's bugs or something like that. ;)

There's also brief stats info in the link name so that you can see how many sub-tests are failing and how many are not without actually expanding the code snippet. (Thanks Juerd++ for suggesting this one.) For instance:

   - Show the snippet from t/var/var.t (line 13 ~ line 132  —  30 √, 0 ×) -

On top of the web page, there's a header like this:

  This page was generated at 2006-09-17 05:58:59 GMT.
  (syn r12136, pugs r13398)

where the revision number for pugs is extracted from the corresponding smoker's report. Christopher has ensured the version of the Pugs test suite used by the smoke server matches exactly the smoker's local Pugs. The timestamp was proposed by Juerd while the revision numbers for both syn and pugs were proposed by [particle]++.

All the Synopses on the smoke server are updated periodically by cron, even including old smoke reports that had been processed. We're trying to offer you the latest Perl 6 Spec everywhere. :)

For more details on the original motivation and Christopher's design decisions, please read the p6c threads below:

Integrating the Pugs test suite into the Synopses

synopses on smoke server

Thanks Gaal and Larry's tests.yml while I was implementing smoke result rendering feature in smartlinks.pl many weeks ago. At last but not least, I really appreciate Christopher's work on the smoke server/client mechanism to make this feature truly useful. (We don't need auto-smoking on feather any more, whee!) And I also thank iblech++ (yes, iblech's alive!) for helping Christopher through the whole process.

The Synopses on feather will still go without smoke results, since smoke results are always specific to a certain Perl 6 implementation or a Pugs backend. We hope you will still like it. :)

Audrey once said on #perl6 that she was biased to fix tests with smartlinks. And [particle] recently said that he had started a port of smartlinks.pl to PIR so as to cross-reference PGE's tests with the Synopses and parrot PDDs. Furthermore, I've seen more and more people are adding both tests and smartlinks to the Pugs test suite, which is really really wonderful. :D

Major Perl 6 Wiki update

Conrad Schneiker recently gave the Perl 6 community a gift by loading this new Perl 6 wiki full of content. The wiki itself runs SocialText software written in Perl 5 and hits a nice sweet spot between being featureful and easy to use. I recommend trying it out, and plan to use it frequently myself.

This wiki is still young, and as Andy Lester notes in the comment at the bottom, it would be useful to break up the front page content into several pages. It's a wiki, so feel free to jump in and help with that!

Update: I went through and overhauled the front page and site structure myself, so the front page is simple new, with some embedded Perl 6 news. Of course, it's a wiki, so you can further refine it as you like.

       

Improved Perl6 Documentation available

Perl6 now has a notably improved entry point to find documentation. See the Official Perl 6 Documentation page. The center piece are  the Synopsis documents, which form the specification for the language.  A number of  additional useful documents are now linked from here as well.

I'm personally been working on Perl6::Perl5::Differences, documenting things that seem like key differences as I learn the new language. Both pages are available in the pugs repository for anyone to improve.

I also updated the front page of feather, the Perl 6 Community Development Server to make it a better general Perl 6 resource as well.

In the process of this documentation work, I discovered there is already quit a lot of Perl6 information out there, but it's a bit fractured-- there's more to be done to create an excellent, sustainable central resource. I'm interested to experiment more with this new Perl6 wiki powered by SocialText. There's little information there now, but it looks especially friendly to use and work with.

Integrating the Pugs test suite into the Synopses

When you see the title, you may wonder what it means. Here is the answer:

http://perlcabal.org/syn/S04.html

Search for links with names like "Show the snippet from ..." and click on them to find out the magical effect. :)

Well, in short, we have divided the .t files in the Pugs test suite into pieces and inserted every resulting snippet after the corresponding paragraph of the Synopses.

The job was done by the Perl 5 script util/smartlinks.pl living in the Pugs source tree. And all the *.html under http://perlcabal.org/syn/ are updated every hour with the latest Synopses and Pugs test suite. Thanks to the cron program on feather. ;-)

At this point, there is a hitch. How does smartlinks.pl manage to determine which snippet of the tests corresponds to which paragraph of a certain synopsis?

Hmm, I'm sure you may have already found the answer from the web pages. There're hundreds of "smartlinks" spreading over the whole test suite in the form of comments. Here're two examples:

  # L<S04/The do-once loop/"can't" put while modifier>

  # L<S04/"Switch statements" /"is exactly equivalent to">

You see, the "smartlink" has its deep root in our POD, but contains an extra "keyphrase" list after the second slash. This list of key phrases specifies which paragraph in the given section is relevant to the tests right below the comment.

Multiple adjacent smartlinks share the same group of tests:

  # L<S02/Context/boolean "?">
  # L<S03/Changes to Perl 5 operators/"?" boolean context>
  { ... }

Please see t/README in Pugs repository for more detailed information on smartlinks:

http://svn.openfoundry.org/pugs/t/README

Embedding test snippets into the Synopses bears several benefits:

Providing more sample code to the Synopses

Synopses are known for their rich samples. Unfortunately, the sample code in our Synopses is inadequate. Many sections go without sample code.

On the other hand, the Pugs test suite contains tens of thousands of tests, which can certainly serve as very good examples as long as they are correct and up-to-date.

Intuitive estimation for the quality of the tests

When we browse the web pages with test snippet embedded, it's quite easy to figure out the validity and also the integrity of the test suite.

For instance, if a whole section has no snippet links at all, then that probably means we need to add more tests as well as smartlinks. The HTML pages are good visual coverage reports per se.

Furthermore, the correctness of a group of tests is easiest to determine if it appears right in the context of the synopses.

Currently we also have TODO and SKIP marks in the test suite. Hence a good estimation of the proportion of the Synopses implemented by Pugs at a quick glance.

Easier to detect mismatch between tests and docs

Everyone can see from p6l that the Synopses are changing pretty fast. In contrast, many tests in the Pugs test suite are terribly out of date even though Larry has the habit of updating some tests after updating the documents.

But if we have enough smartlinks distributed in the test suite, the mismatch between tests and docs can be detected in the form of "broken" links. To get a list of broken links, we only need a single command:

  $ util/smartlinks.pl --check t/*/*.t t/*/*/*.t

Unmatched smartlinks (if any) will be reported in detail:

  ERROR: t/operators/quoting.t: line 173: pattern
         ``/Split result on words (no quote
         protection)/'' failed to match any
         paragraph in L<S02/Literals>.

  ERROR: t/data_types/arglist.t: line 6: section
         ``Operator renaming'' not found in S03.

As I have told Juerd on #perl6, the original goal of developing smartlinks.pl is simply to make myself more motivated to contribute tests to Pugs.

Thanks to other Pugs contributors who have been working strenuously on the smartlinks long before I joined the Pugs team. Without their work, smartlinks.pl won't produce anything meaningful today. :)

Update:

• We now use the term "keyphrase" instead of "keywords" since the order of the keys is significant. Thank TimToady++ for proposing this cool name. :)
• There is also a shorter URL for our Synopses:

  http://perlcabal.org/syn/

  which is essentially a symbolic link to

  http://feather.perl6.nl/~agentzh/syn

  on feather.

Perl 6 is coming (to the SysAdmin magazine).

Yay, merlyn++'s wonderful article is now available online!  I attended his and bdfoy++'s "Learning Perl 6"
course at YAPC::NA, and I'm really amazed of their ability to explain concepts so effectively.

Also, the instructor point-of-view is helpful to language design as well: For example, merlyn++ noted that the ==> and <== operators, known as the pipe operators, is likely to cause confusion with Unix pipes. Because they are shipping concrete objects instead of stream-of-characters, now they are renamed feed operators.

Another example: The implicitly-passed-with-each-calls variables were known as environmental variables, and declared with "env $x".  However, merlyn++ noted that it can easily be confused with environment variables in %*ENV.  So they are now contextual variables, and declared with "my $x is context".

...and now I stop procrastinating and go back to my slides...

Design cleanup.

The past three days weren't all about license; live design sessions on #perl6 happens every day for hours, resolving the remaining ambiguities that slows down Perl 6 parsing (due to ambiguities that requires infinite lookahead), or the parts that were easy in Parsec but was difficult to declare as Rules.

A concerte example for infinite lookaheads:

$hash<1;2;3;long;string;here>;  # one statemeent?
$hash<1;2;3;long;string;here;    # 6 statements?

Because of this, Perl 6 always require a whitespace before < to denote an infix operator; non-whitespaces always means the postcircumfix <> macro, which then desugars to a postcircumfix {} lookup.

Pugs currently uses very expensive backtracking to disambiguate these cases, so getting rid of them should dramatically reduce parsing time.

While switching to Packrat Parsing (gaal++ for the pointer) would allow infinite lookahead and backtracking with linear time, it doesn't make it easier for humans to disambiguate between the two cases above -- so we are not doing that. Instead, I'm using Christopher Kuklewicz's excellent Text.Regex.Lazy package, a compiler that turns extended regexes with customized backtracking strategies (read: Perl 6 rules) into Parsec functions.

This should allow for piecemeal migration from Pugs's Parsec parser to a grammar fully defined in Perl 6 rules syntax, and allow maximum sharing with pmichaud and fglock's parsers.

So, while I only checked in about 15 changes to Pugs this week, there were more than 20 commits to the Synopses, from TimToady, pmichaud and yours truly. Among the changes where:

• my @a = 1,2,3; # this now works, as list-context assignment is listop
• my (Int $x, Str $y) = 1, 'foo'; # this works too; "my" now takes full Signature syntax
• Backtracking patterns are now constructed by its old name, regex, instead of rules, which never backtracks.  The variant of rules that does not translate /a  b/ to /a<ws>b/ but to /ab/ is called tokens. Damian++.
• The *$foo and **$foo syntax now always means "inject content of $foo into the current argument list", and has nothing to do with list contexts anymore.
• A new form :!foo to mean the counterpart of :foo, namely a pair with foo as key and false as value.
• There is no native str type anymore; it's now always buf, so the confusions of Perl 5's buffer->string autopromotion will not appear. See my encoding::warnings for an explanation of the misdesign -- Dan Kogan and I both thinks that's the #1 Bad Idea of the Perl 5 string model, and I'm glad it's gone from Perl 6 now. (That also means Str are also immutable and Buf are always mutable -- think of the latter as Byte Arrays.)
• if foo { 4 } { 5 } now always means if (foo) { 4 }; { 5 }; instead of if foo({4}) { 5 }, regardless of  foo's signature.
• Per Migo (Mikhael Goikhman)'s suggestion during OSDC.il, a null first alternative in rules is now ignored, so we can write this BNF-ish parser:

rule answer {
    | Yes  { return 1 }
    | No   { return 0 }
    | N/A { return undef }
}

There are much more interesting changes that makes Perl 6 parsing more predictive, and Rules more expressive to capture common patterns in parsing Perl 6.  Ah, the joy of self-hosting... :-)

Dispatching out...

Thanks to all the feedback to yesterday's Capture.pod post, today I updated Capture.pod with more examples, including a more elaborate description on single/multiple vs subroutine/method dispatch (as requested by Humberrto), which I reproduce below:

---

What about multi-dispatch?

Multi-dispatch governs cases where multiple subroutines or methods share the same name, relying on the arity and types of tie-breaking parameters in their parameter list (Signature):

    multi f ($x: $z)             { say 'A' }
    multi f (Int $x, Int $y: $z) { say 'B' }
    multi f (Str $x, Str $y: $z) { say 'C' }

    f(1, 2);            # goes to A
    f(1, 2, 3);         # goes to B
    f('x', 'y', 'z');   # goes to C

During multi-dispatch, a tie-breaking parameter may bind to the invocant argument (e.g. for multi-methods), or one of the positional arguments.

However, regardless of single- or multi- dispatch, the argument list (Capture) can never have more than one invocants.  Typically, the presence of an invocant indicates a method-call (which may fall back to a subroutine-call); the lack of invocant means a subroutine-call.

    1.foo(2);  # Int.foo with 1 as "self"; if not found,
               # then fall back to foo(1, 2)
    foo(1: 2); # same as above

    bar(1, 2); # never looks at Int.bar;
               # calls &bar in lexical/package scope

Method/subroutine calls are determined by the presence of an invocant at the calling site.  Single/multi dispatch are determined by the presence of multi in the declaration site.  The two concepts are entirely orthogonal.

---

The last line was key -- it took me several months to realize that in the Synopses, the so-called "invocant parameters" in multisubs really have nothing to do to with "invocant arguments" in method calls.  Hence I've started referring to them as "tiebreaking parameters" to reduce general confusion.

Also, instead of mixing multi and method with the MMD acronym, I've taken to say sub/method calls and single/multi dispatch, to further clarify that they are distinct (and quite unrelated) concepts.  Hope that helps...

I'll have less cycles to stay on #perl6 the next four days, as all three projects at $job are drawing into their closure, and I'd like to deliver them properly before going back to Pugs. However, we are just 35 commits away from r10000, and it might make a nice birthday gift to myself... So we'll see if I manage to send in some commits here and there. :-)