Integrating the Pugs test suite into the Synopses

When you see the title, you may wonder what it means. Here is the answer:

http://perlcabal.org/syn/S04.html

Search for links with names like "Show the snippet from ..." and click on them to find out the magical effect. :)

Well, in short, we have divided the .t files in the Pugs test suite into pieces and inserted every resulting snippet after the corresponding paragraph of the Synopses.

The job was done by the Perl 5 script util/smartlinks.pl living in the Pugs source tree. And all the *.html under http://perlcabal.org/syn/ are updated every hour with the latest Synopses and Pugs test suite. Thanks to the cron program on feather. ;-)

At this point, there is a hitch. How does smartlinks.pl manage to determine which snippet of the tests corresponds to which paragraph of a certain synopsis?

Hmm, I'm sure you may have already found the answer from the web pages. There're hundreds of "smartlinks" spreading over the whole test suite in the form of comments. Here're two examples:

  # L<S04/The do-once loop/"can't" put while modifier>

  # L<S04/"Switch statements" /"is exactly equivalent to">

You see, the "smartlink" has its deep root in our POD, but contains an extra "keyphrase" list after the second slash. This list of key phrases specifies which paragraph in the given section is relevant to the tests right below the comment.

Multiple adjacent smartlinks share the same group of tests:

  # L<S02/Context/boolean "?">
  # L<S03/Changes to Perl 5 operators/"?" boolean context>
  { ... }

Please see t/README in Pugs repository for more detailed information on smartlinks:

http://svn.openfoundry.org/pugs/t/README

Embedding test snippets into the Synopses bears several benefits:

Providing more sample code to the Synopses

Synopses are known for their rich samples. Unfortunately, the sample code in our Synopses is inadequate. Many sections go without sample code.

On the other hand, the Pugs test suite contains tens of thousands of tests, which can certainly serve as very good examples as long as they are correct and up-to-date.

Intuitive estimation for the quality of the tests

When we browse the web pages with test snippet embedded, it's quite easy to figure out the validity and also the integrity of the test suite.

For instance, if a whole section has no snippet links at all, then that probably means we need to add more tests as well as smartlinks. The HTML pages are good visual coverage reports per se.

Furthermore, the correctness of a group of tests is easiest to determine if it appears right in the context of the synopses.

Currently we also have TODO and SKIP marks in the test suite. Hence a good estimation of the proportion of the Synopses implemented by Pugs at a quick glance.

Easier to detect mismatch between tests and docs

Everyone can see from p6l that the Synopses are changing pretty fast. In contrast, many tests in the Pugs test suite are terribly out of date even though Larry has the habit of updating some tests after updating the documents.

But if we have enough smartlinks distributed in the test suite, the mismatch between tests and docs can be detected in the form of "broken" links. To get a list of broken links, we only need a single command:

  $ util/smartlinks.pl --check t/*/*.t t/*/*/*.t

Unmatched smartlinks (if any) will be reported in detail:

  ERROR: t/operators/quoting.t: line 173: pattern
         ``/Split result on words (no quote
         protection)/'' failed to match any
         paragraph in L<S02/Literals>.

  ERROR: t/data_types/arglist.t: line 6: section
         ``Operator renaming'' not found in S03.

As I have told Juerd on #perl6, the original goal of developing smartlinks.pl is simply to make myself more motivated to contribute tests to Pugs.

Thanks to other Pugs contributors who have been working strenuously on the smartlinks long before I joined the Pugs team. Without their work, smartlinks.pl won't produce anything meaningful today. :)

Update:

• We now use the term "keyphrase" instead of "keywords" since the order of the keys is significant. Thank TimToady++ for proposing this cool name. :)
• There is also a shorter URL for our Synopses:

  http://perlcabal.org/syn/

  which is essentially a symbolic link to

  http://feather.perl6.nl/~agentzh/syn

  on feather.

Comments

[Mark Stosberg]

Great work! I find this useful. If there were just an index page here:

http://feather.perl6.nl/~agentzh/syn/

I would find this more useful than the Synopsis posted on dev.perl.org.

[Mark Stosberg]

One more comment: the part of the link that says "/home/audreyt/pugs" could be removed, since it's implied the tests are part of pugs.

[Thomas Wittek]

Great idea, really!

[Audrey T]

Markstos: I fixed the link in your comment. Also I agree -- util/smartline.pl line 314 needs an extra:

$file =~ s{.*/t/}{t/};

to strip the generated output of the leading path. I'll commit it tomorrow when svn is back, if noone had beat me to it, and it'd be reflected automatically in 1hr...

[crzwdjk]

Wow, this is a wonderful idea and long overdue. Now it's possible to see at a glance what part of the synopses are covered, what tests need to be updated when synopses change. And also makes it easier for us lazy potential contributors to see what tests need to be written, just by looking through the synopses.

[Agent Zhang]

Markstos: Thank you for your suggestions. And there will be an index page on feather very soon. :)