TPF Devel::Cover grant report November 2012

[Paul Johnson]

In accordance with the terms of my grant from TPF this is the monthly
report for my work on improving Devel::Cover covering November 2012.

This month I released Devel::Cover versions 0.97 and 0.98.

As I mentioned in my monthly report for October, I've been rather busy since
returning from the Google Mentor Summit. I have, however, found some time for
working on Devel::Cover.

This month I tested Devel::Cover against the newly released Perl versions
5.12.5, 5.14.3, 5.16.2, 5.17.5, and 5.17.6. That's new releases on three
stable versions of Perl and two development releases. I think that's quite
impressive. Not by me, but by everyone who has worked on and released these
versions. Thank you!

Having got through a lot of bug fixes in the last few months, including some
which have taken quite a few hours to track down and fix, and which I would
never have been able to fix without this grant, I wanted to spend a little
time this month tackling a couple of more research orientated areas. In
particular I looked at two topics.

First, I took a look into adding a new coverage metric: path coverage.
From the Devel::Cover documentation:

There are classes of errors that branch coverage cannot detect, such as:

$h = undef;
if ($x)
{
$h = { a => 1 };
}
if ($y)
{
print $h->{a};
}

100% branch coverage can be achieved by setting ($x, $y) to (1, 1) and then
to (0, 0). But if we have (0, 1) then things go bang.

The purpose of path coverage is to ensure that all paths through the
program are taken. In any reasonably sized program there will be an
enormous number of paths through the program and so in practice the
paths can be limited to a single subroutine, if the subroutine is not
too big, or simply to two consecutive branches.

In the above example there are four paths which correspond to the truth
table for $x and $y. To achieve 100% path coverage they must all be
taken. Note that missing elses count as paths.

In some cases it may be impossible to achieve 100% path coverage:

a if $x;
b;
c if $x;

50% path coverage is the best you can get here.

Loops also contribute to paths, and pose their own problems which I'll
ignore for now.

100% path coverage implies 100% branch coverage.

Path coverage and some of its close cousins, are also known as
predicate, basis path and LCSAJ (Linear Code Sequence and Jump)
coverage.

There are a number of problems to be solved to collect this data. Not the
least of which is determining which paths to count. I suspect that a
reasonable compromise is to only cover paths from two consecutive branches in
the same subroutine. The currently executing subroutine is needed, and this
can be determined with the find_runcv() call.

I didn't manage to get to implementing a solution but at least the way forward
is much clearer now.

The second area I investigated was the way in which coverage data is matched
up with source code. This currently happens by (ab)using B::Deparse and
replacing some of its innards. This isn't a particularly nice solution on any
front, really, except that it works.

The biggest downside to getting rid of using B::Deparse is that without it the
reporting of condition coverage generally displays the condition incorrectly.
This is because when using B::Deparse to deparse just the condition, the
deparse context going into the condition is unknown, and that context is used
to determine the type of the condition.

So I've left that branch as it is for now. The fundamental concept is sound,
but something needs to be done about condition reporting.

I spent some time investigating RT 80340 (Block before "continue" is ignored)
which seems to be related to specific versions of Perl, but I didn't reach any
firm conclusions.

I had a request to merge databases from disparate modules, but Devel::Cover
isn't set up to work that way because it really wants to be able to locate
source files, and relative paths from different directories makes that
problematic. But I did improve and reduce the number of error messages
resulting from such problems.

The work I have completed in the time covered by this report is:

Closed Github tickets:

26 Insecure dependency in ``
30 Merge databases from multiple modules

Merged pull requests:

32 filter paths contained in CWD out of @Inc
33 Running Devel::Cover in taint mode

You can see the commits at https://github.com/pjcj/Devel--Cover/commits/master

Hours worked: 18:45

Total hours worked on grant: 181:55