Pugs

Day 4 of Text.Parser.Rule.

Today I've rewritten PArrow's primitive combinators to support proper backtracking and non-greedy matching, as well as subrule support.  This now works correctly:

miniLang :: Grammar
miniLang = grammar
    [ "literal" ~:~ "nil | true | false"
    , "call"    ~:~ "\\. <method>"
    , "method"  ~:~ "\\w+"
    , "expr"    ~:~ "<literal> <call>*"
    ]

ghci> "nil.foo.bar" ~~ miniLang .<> "expr"
Right (MatchObj
    { matchString = "nil.foo.bar"
    , matchSubPos = fromList []
    , matchSubNam =
        { "call" := MatchSeq (fromList
            [ MatchStr "."
            , MatchNam "method" (MatchStr "bar")
            ])
        , "literal" := MatchStr "nil"
        }
    })


Both named and positional captures are supported, as well as negated and noncapturing variants (<!name> and <?name> respectively).  This seems sufficient for porting the surface language parser of PIL^N and PIL² from Parsec; tewk already volunteered to give it a try.

If it turns out that we need more features (e.g. lookaround, integration with OpTable, callbacks and backreferences), I'll definitely hack them in. :-)

Scalar container lands.

Scalar container support has landed to PIL^N runtime.  To wit:

$ make pil
$ ./pil -e '::Scalar`create(3).STORE(9).FETCH()'
### Evaluated ###
9

Moreover, ::Int now represents a true BigInt (internally using the Haskell type Integer), which differs from its unboxed representation with limited precision.  This is correct according to the spec, and unlike the old runtime which uses BigInts everywhere.  Unicode-aware ::Str types will get the same treatment soon.

The next step is to implement Array and Hash container types, and revise PIL² so they can be desugared to PIL^N. Which should all happen tomorrow, unless I get distracted into more Rules hacking...
 

As seen on #perl6...

04:06 < gantrixx> and I wish there all the effort that was going into Python develompent was going into Perl6
04:07 < audreyt> oh, but they are
04:07 < audreyt> we stole^Wlifted a _lot_ of ideas from PyPy and IronPython
04:07 < gantrixx> very good
04:07 < gantrixx> but more than likely, the guy is going to move on to a new girlfriend, and just because his ex lost weight doesn't mean he will re-unite
04:07 < stevan> gantrixx: but if that ex was his true love,.. he might :)
04:07 < audreyt> ..and this is more like the daughter of his ex ;)
04:08 < gantrixx> yes, but waiting for her to be of legal age is a bit frustrating

Multipositional error reporting.

It's past midnight here, so I'll let the commit log speak for itself:

$ ghc --make -main-is Text.Parser.Rule.main \
      -isrc -Isrc/cbits src/cbits/*o \
 src/Text/Parser/Rule.hs
$ ./a.out '.abc|def' 'xyz'
Expecting: 'def' at line 1, column 1
       or: 'abc' at line 1, column 2


I have rewritten half of PArrow for this.  The neat thing is that if it matches at any given point, none of the expensive error collection would take place, so there should be no severe performance drawbacks.

Also, the returned match object is now compatible with PGE's structure, so hooking it back to Pugs is trivial.

Parsing is fun, especially when there's only 1.5hr of timeslice for hacking -- I'll go right back to container types tomorrow when the timeslice gets longer.  There may even be a Perl5 prototype from Stevan waiting for me when I wake up...

Second day of Text.Parser.Rule.

Today and tomorrow are both $job days, so I didn't get much time to play with Pugs.

Still I managed to transcribe most of PGE::P6Rule to Text.Parser.Rule, so about 80% of rule syntax is now parsed correctly into a Rule tree. Instead of using GADTs, I went for a more conventional cascading data type approach, with five levels of rule nodes:

(Alternation > Conjunction > Concat > Quantified > Term)

Moreover, list-associative infix support landed in OpTable parser, so /(a|b|c)/ is now parsed as alternation of a three-element list, instead of (a|(b|c)) in the current PGE implementation.

Most of those nodes have straightforward mappings to PArrows, so I implemented them trivially. However, capturing groups, subrules and binding will have to wait until tomorrow.  After this week's $job is done, I'll get more continuous timeslices to go back to finish container type support in PIL^N. Stay tuned!

Lexical import comes to PIL2JS.

Today's short hacking session with clkao was quite successful.  Apparently his $work involves using JavaScript for server-side scripting -- there's definitely a trend here -- and as the original author for Perl5 embedding support in Pugs, he thought it might be neat to add Perl 5 embedding to the JavaScript backend as well, so you can use both JSAN libraries and CPAN libraries in your Perl 6 code.

With a few days of mad hacking, this now works today:

./pugs -B JS-Perl5 -e \
'use jsperl5:DBI; use jsan:Number::Roman; ...'

(The "jsperl5:" hack is expect to go away and replaced by "perl5:" once our Parser can recognize that compilation to JS (instead of PIL) is currently in effect.)

After gaal's recent work on lexical imports (which I'll journal about later), clkao noticed that all use statements stopped exporting, so he asked me to fix this today.  In return, I nudged him into finishing up SVK::View support, which implements  Perforce Views (lightweight client-side branching), the one feature I missed the most when I migrated from Perforce to SVK.

Two hours later, we both finished our tasks: mine involved deleting a dozen lines of code, and his involving adding 1000+ lines. Yay for productive mini-hackathons! :-)

Perl 6 Rules implementations in Haskell and Perl 6.

A 20-minutes hack had paid off handsomely: In 33 lines, we have a fast, pure Haskell implementation of basic Perl 6 Rules support. Adding support for each new construct should take only 2 lines -- one at the OpTable precedence parser below to recognize it, and one to compile it to the underlying PArrows library.

(As an aside, I'm glad to have finally grokked what an Arrow is. As usual with lambdafolk technologies such as MonadsActions, it is very simple yet tricky to convey, and could really use a better name like Flow.)

Of course, the real work is done somewhere else. Aside from using PArrows for the heavy lifting in top-down parsing, we have the bottom-up OpTable.hs precedence parser (a straight port of Parrot's PGE::OPTable). This is my first successful use of GHC's implicit parameters, which are exactly like "env" variables as recently specced in the lexical syntax spec (S02), allowing a much faster and concise implementation than Reader monads.

Currently the OpTable port covers all features found in the PIR base code, and I plan to add list/chain-associative infix operators to it, so we can  perform interesting optimizations like turning /cat|car|cow/ into /c[a[t|r]|ow]/.

Much kudos to Patrick Michaud for doing all the design work for Rules and OpTable, so this coding monkey can just plunge ahead without any head-banging against the wall.

Luqui is also doing well in implementing both libraries in Perl 6, using a role-based composition architecture to handle multiple input types and evaluation strategies.  More details is available in its architecture document.

With luck, we can ship 6.28.0 with this Haskell-native Rule support, without having PGE (and thus Parrot) as a dependency. We'd still use PGE when compiling to Parrot, and rely on luqui's Parse::Rule when targetting JavaScript and Perl 5. PArrows can compile to JavaScript natively, which may speed up things a bit...  We'll see. :-)

 

Roles support in PIL.Native runtime.

Almost two months ago when I was in .nl, Stevan and I worked out the ObjectSpace bootstrapping plan, decomposing Perl 6's object model into method calls of a few native types, so we can build all boxed types, classes, roles and containers on top of them.

This approach is needed because our target runtimes (Perl5, Parrot, JavaScript) have wildly different method dispatch semantics. In the PIL¹ era, each backend needs to bootstrap the object model differently, resulting in massive code duplication and subtly different semantics on eg. private methods and role composition.

Today I'm glad to report that Stevan's Bootstrap code is capable of representing roles, private methods, multiple inheritance with C3 dispatch order, built on full introspection and object space enumeration capabilities offered by the PIL.Native (PIL^N) runtime.  PIL^N is a small language with only literals, variable lookups and primitive (potentially statically bound) method calls.  Typing make pil in the Pugs source tree will build its interpreter; tests are in the t/pil/ directory.

The next step is to represent container types as instances of ::Class, and group their methods into  a Roles hierarchy. Fortunately, Scala, Ruby and C# has done much of the hard work for us -- with Code, SigList, ArgList and other PIL² base classes implemented, we can then hook it to a Perl 6 parser and complete the 6.28.0 milestone.

-Ofun with YAML::Syck.

Tonight's hackathon session with Ingy was very enjoyable. I volunteered to be the guinea pig for his soon-to-be-released much-refactored pure-perl YAML 0.50, which has been sitting on the YAML-Dev subversion repository  for months, but Ingy was afraid of it breaking existing users.

The fear appears well-founded, as both SVK and Module::Install had incompatibilities (something to do with autouse.pm -- I'll spare you the details), but we had them squashed, so expect YAML 0.49_xx test releases appearing on CPAN today.

In addition to that, I mentioned that many other languages had decent Syck binding for ages, yet Perl 5 only had a randomly-segfaulting YAML::Parser::Syck and no dumper bindings.  Thanks to SWIG and Ingy's guidance, I coded up a working implementation of YAML::Syck in a few hours.  I think a fast lightweight YAML library will really benefit interoperability, and with some more polish (i.e. lifting more scary bits from Storable.pm) it may even be core-worthy in the future.

As Pugs had YAML parsing support for quite a while now with the eval($yaml_string, :lang<yaml>) builtin, I look forward to carry the same sort of proper YAML loading/dumping support to PIL.Native runcore sometime this week.

We also discussed plans on merging Spiffy.pm with Perl6::Subs (codenamed Spiffy6), merging Wikiwyg with Jifty into a WikiUI presentation layer, merging Takahashi resizer with next-generation Spork ported to Perl 6 and compiled to JavaScript, as well as other assorted optimized-for-fun ideas. :-)

Upcoming conferences.

My talks at OSDC::Australia and Sydney were quite well-received.  I received much feedback on how to improve the delivery. I'll meet the the also-recently-renamed Ingy döt Net tomorrow for a mini-hackathon, and new Takahashi-friendly slide making tools is definitely very high on our agenda, along with YAML/Syck, PerlDoc/Kwid and other crazy scheme of his.

I'll attend OSDC::Israel on February 26-28, arriving two weeks before that for an extended Pugs hackathon with lwall, gaal, nothingmuch et al.  Right after that, German Perl Workshop will take place on March 1-3, probably with another hackathon afterwards.

On March 29-30 I'll be in Tokyo for YAPC::Asia, and work with Perl and Ruby hackers there. Finally, on April 8-9 I'll be back to .tw for OSDC::Taiwan (web page to be announced later).