TPF Devel::Cover grant report Week 18

[Paul Johnson]

In accordance with the terms of my grant from TPF this is the report for
week 18 of my work on improving Devel::Cover.

This report covers 15.09 to 21.09.

This week I released Devel::Cover version 0.94.

I was finally able to get back to doing some grant related work this week, and
the first thing I did was catch up on some of the Devel::Cover related mail I
have received whilst installing perl-5.16.1 and perl-5.17.3.

I had already installed 5.16.1 as my default perl via perlbrew, but I have a
suite of perl versions that I use to test every Devel::Cover release
candidate, and those perl versions aren't installed via perlbrew. I have
tried, but I couldn't get perlbrew to automatically apply appropriate patches
to old perls so they could be built today, although perlbrew's
install-patchperl seems to suggest that that should be possible.

If anyone knows how to do that, I'd love a hint. Or better yet, a pull
request against build() in
https://github.com/pjcj/Devel--Cover/blob/master/utils/all_versions

And if anyone is interested in my current regression testing solution, it's
basically that file above plus https://github.com/pjcj/buildperls to build the
perl versions. It's not overly pretty, but it does mean that I can build new
perls simply by dropping a tarball into a src directory and running a command.
And not having to think about stuff like that is always a win.

Having built those perl versions I ran them on Devel::Cover and, as expected,
saw that Devel::Cover fails its tests on 5.17.3. This is something I knew
about from discussions with Father Chrysostomos on p5p and also from failing
cpantesters reports. (Thanks again cpantesters!)

The failure was purely cosmetic and brought about by Father Chrysostomos
fixing up loopexes. The details are in
https://rt.perl.org/rt3//Public/Bug/Display.html?id=113684 and the explanation
of why the cosmetic change exists are in
https://rt.perl.org/rt3//Public/Bug/Display.html?id=106892#txn-1043534

That seemed sufficient for a new release, so I let 0.94 roam the CPAN mirrors.

Whilst all that was going on, I also set cpancover running against an official
5.16.1 release (The previous run was against an RC). The results are
available at http://cpancover.com, as usual.

I investigated github ticket 15: issue with Moo attribute which is Moo object.

This turned out to be another of those longstanding bugs that I knew a bit
about but had never been able to find the time to put in sufficient effort to
crack. Thanks to the grant I was able to crack it.

The problem was with condition coverage, and in particular when the first
operand of a LOGOP called (directly or indirectly) a subroutine in a file for
which coverage was not being collected. This meant that at the time
Devel::Cover decided whether or not to collect coverage data for the LOGOP the
last COP seen was indicating that coverage shouldn't be collected. The
solution was to check PL_curcop at appropriate times.

In conjunction with this I was able to add numerous test cases from RT and
github and close out various tickets.


Closed RT tickets:

63090 conditionals where one element is data from a Moose attrib are not evaluated
63698 Math::BigInt ':constant' confuses branch coverage
72027 Devel::Cover does not detect branch coverage with DBM::Deep
38258 inclusion/exclusion options to cover not reflected in summary row
34327 Condition coverage for constants

Closed Github tickets:

15 issue with Moo attribute which is Moo object

Merged pull requests:

20 failing test for branching coverage of Moo attributes

Fixed cpantesters reports:

http://www.cpantesters.org/cpan/report/ceee4f36-de73-11e1-a876-575485400d13
http://www.cpantesters.org/cpan/report/f296fa46-de73-11e1-bc68-855885400d13
http://www.cpantesters.org/cpan/report/abe8c2d2-de73-11e1-bad3-055385400d13

You can see the commits at https://github.com/pjcj/Devel--Cover/commits/master

Hours worked:

17.09 4:45
19.09 9:00
20.09 3:25

Total 17:10

[Pedro Melo]

Hi,

Post by Paul Johnson
I investigated github ticket 15: issue with Moo attribute which is Moo object.
This turned out to be another of those longstanding bugs that I knew a bit
about but had never been able to find the time to put in sufficient effort to
crack. Thanks to the grant I was able to crack it.
The problem was with condition coverage, and in particular when the first
operand of a LOGOP called (directly or indirectly) a subroutine in a file for
which coverage was not being collected. This meant that at the time
Devel::Cover decided whether or not to collect coverage data for the LOGOP the
last COP seen was indicating that coverage shouldn't be collected. The
solution was to check PL_curcop at appropriate times.

This alone made my week. :)

Thanks!

[Steffen Schwigon]

I have tried, but I couldn't get perlbrew to automatically apply
appropriate patches to old perls so they could be built today

Not sure how old you mean. Older than 5.8?
If 5.8+ is enough then for my app-bootstrap-perl I only needed

https://github.com/renormalist/app-bootstrap-perl/blob/master/bin/bootstrap-perl#L458

Kind regards,
Steffen

[Paul Johnson]

Post by Steffen Schwigon

I have tried, but I couldn't get perlbrew to automatically apply
appropriate patches to old perls so they could be built today

Not sure how old you mean. Older than 5.8?

Yes. Devel::Cover works (for some definition of) back to 5.6.1. As
long as it's not too much effort I'd like to keep that support. At the
moment the most effort is probably in keeping old working perls around
for testing purposes.

Post by Steffen Schwigon
If 5.8+ is enough then for my app-bootstrap-perl I only needed
https://github.com/renormalist/app-bootstrap-perl/blob/master/bin/bootstrap-perl#L458

That's a nice script. I use
https://github.com/pjcj/buildperls/blob/master/buildperl.pl and it works
well enough, but I'd prefer to work with perlbrew and let someone else
worry about that maintenance effort.

http://www.perlbrew.pl/Perlbrew-and-Friends.html implies that perlbrew
should use patchperl from https://metacpan.org/module/Devel::PatchPerl
(which is essentially the same software that I'm using) and it seems to
be doing that, but the build fails.

I don't really want to spend time debugging it, or my setup, or whatever
is wrong, although I know that ideally I should. But if someone were
able to say "just do X", where X is something really simple, I'd love to
switch to just using perlbrew. If nothing else, it would make it much
easier for anyone else to set up the full development/testing environment.

[Ovid]

Hi Paul,

I'm very happy to see the great work you've been doing for Devel::Cover.

I've been meaning to ask: have you considered adding basis path coverage information? Devel::Cover appears to be tracking all of the relevant data. It seems that now it would merely be a matter of writing the algorithm to generate the basis coverage. Many other languages have automated tools for generating the requisite cyclomatic complexity and basic path information. It would be awesome if Perl had this too :)
 
For others: yes, I know that PPI offers cyclomatic complexity information. I would like to see all code coverage information provided in "one-stop shopping" for Perl rather than relying on a separate tool (and I don't know that PPI exposes the control-flow graph used to calculate the cyclomatic complexity. Does it?).

Cheers,
Ovid
--
Twitter - http://twitter.com/OvidPerl/
Buy my book - http://bit.ly/beginning_perl
Buy my other book - http://www.oreilly.com/catalog/perlhks/
Live and work overseas - http://www.overseas-exile.com/

________________________________
Sent: Saturday, 22 September 2012, 3:14
Subject: TPF Devel::Cover grant report Week 18
In accordance with the terms of my grant from TPF this is the report for
week 18 of my work on improving Devel::Cover.
This report covers 15.09 to 21.09.
This week I released Devel::Cover version 0.94.
I was finally able to get back to doing some grant related work this week, and
the first thing I did was catch up on some of the Devel::Cover related mail I
have received whilst installing perl-5.16.1 and perl-5.17.3.
I had already installed 5.16.1 as my default perl via perlbrew, but I have a
suite of perl versions that I use to test every Devel::Cover release
candidate, and those perl versions aren't installed via perlbrew.  I have
tried, but I couldn't get perlbrew to automatically apply appropriate patches
to old perls so they could be built today, although perlbrew's
install-patchperl seems to suggest that that should be possible.
If anyone knows how to do that, I'd love a hint.  Or better yet, a pull
request against build() in
https://github.com/pjcj/Devel--Cover/blob/master/utils/all_versions
And if anyone is interested in my current regression testing solution, it's
basically that file above plus https://github.com/pjcj/buildperls to build the
perl versions.  It's not overly pretty, but it does mean that I can build new
perls simply by dropping a tarball into a src directory and running a command.
And not having to think about stuff like that is always a win.
Having built those perl versions I ran them on Devel::Cover and, as expected,
saw that Devel::Cover fails its tests on 5.17.3.  This is something I knew
about from discussions with Father Chrysostomos on p5p and also from failing
cpantesters reports.  (Thanks again cpantesters!)
The failure was purely cosmetic and brought about by Father Chrysostomos
fixing up loopexes.  The details are in
https://rt.perl.org/rt3//Public/Bug/Display.html?id=113684 and the explanation
of why the cosmetic change exists are in
https://rt.perl.org/rt3//Public/Bug/Display.html?id=106892#txn-1043534
That seemed sufficient for a new release, so I let 0.94 roam the CPAN mirrors.
Whilst all that was going on, I also set cpancover running against an official
5.16.1 release (The previous run was against an RC).  The results are
available at http://cpancover.com, as usual.
I investigated github ticket 15: issue with Moo attribute which is Moo object.
This turned out to be another of those longstanding bugs that I knew a bit
about but had never been able to find the time to put in sufficient effort to
crack.  Thanks to the grant I was able to crack it.
The problem was with condition coverage, and in particular when the first
operand of a LOGOP called (directly or indirectly) a subroutine in a file for
which coverage was not being collected.  This meant that at the time
Devel::Cover decided whether or not to collect coverage data for the LOGOP the
last COP seen was indicating that coverage shouldn't be collected.  The
solution was to check PL_curcop at appropriate times.
In conjunction with this I was able to add numerous test cases from RT and
github and close out various tickets.
  63090 conditionals where one element is data from a Moose attrib are not evaluated
  63698 Math::BigInt ':constant' confuses branch coverage
  72027 Devel::Cover does not detect branch coverage with DBM::Deep
  38258 inclusion/exclusion options to cover not reflected in summary row
  34327 Condition coverage for constants
  15 issue with Moo attribute which is Moo object
  20 failing test for branching coverage of Moo attributes
  http://www.cpantesters.org/cpan/report/ceee4f36-de73-11e1-a876-575485400d13
  http://www.cpantesters.org/cpan/report/f296fa46-de73-11e1-bc68-855885400d13
  http://www.cpantesters.org/cpan/report/abe8c2d2-de73-11e1-bad3-055385400d13
You can see the commits at https://github.com/pjcj/Devel--Cover/commits/master
  17.09  4:45
  19.09  9:00
  20.09  3:25
  Total  17:10
--
http://www.pjcj.net

[Jeffrey Thalhammer]

Post by Ovid
For others: yes, I know that PPI offers cyclomatic complexity information. I would like to see all code coverage information provided in "one-stop shopping" for Perl rather than relying on a separate tool (and I don't know that PPI exposes the control-flow graph used to calculate the cyclomatic complexity. Does it?).

To my knowledge, PPI doesn't compute cyclomatic complexity. But Perl::Critic approximates it just by counting conditional operators and keywords.

-Jeff

[Ovid]

Sent: Monday, 1 October 2012, 22:48
Subject: Re: TPF Devel::Cover grant report Week 18

Post by Ovid
For others: yes, I know that PPI offers cyclomatic complexity information. I would like to see all code coverage information provided in "one-stop shopping" for Perl rather than relying on a separate tool (and I don't know that PPI exposes the control-flow graph used to calculate the cyclomatic complexity. Does it?).

To my knowledge, PPI doesn't compute cyclomatic complexity.  But Perl::Critic approximates it just by counting conditional operators and keywords.
-Jeff

D'oh! You're right.

I didn't realize that P:C only approximates it.

For those who haven't seen it, it's the "McCabe score" when you use --statistics:

    $ perlcritic lib/ --statistics
    <snip>
      5 files.
     21 subroutines/methods.
    177 statements.

    450 lines, consisting of:
         59 blank lines.
         10 comment lines.
          4 data lines.
        207 lines of Perl code.
        170 lines of POD.

    Average McCabe score of subroutines was 1.52.
Cheers,
Ovid
--
Twitter - http://twitter.com/OvidPerl/
Buy my book - http://bit.ly/beginning_perl
Buy my other book - http://www.oreilly.com/catalog/perlhks/
Live and work overseas - http://www.overseas-exile.com/

[Paul Johnson]

Post by Ovid
Hi Paul,
I'm very happy to see the great work you've been doing for Devel::Cover.

Thanks!

Post by Ovid
I've been meaning to ask: have you considered adding basis path coverage information? Devel::Cover appears to be tracking all of the relevant data. It seems that now it would merely be a matter of writing the algorithm to generate the basis coverage. Many other languages have automated tools for generating the requisite cyclomatic complexity and basic path information. It would be awesome if Perl had this too :)
 
For others: yes, I know that PPI offers cyclomatic complexity information. I would like to see all code coverage information provided in "one-stop shopping" for Perl rather than relying on a separate tool (and I don't know that PPI exposes the control-flow graph used to calculate the cyclomatic complexity. Does it?).

Caclulating path coverage has been in the plan since day one. The low
level routines are there and, you are right, I think that I'm collecting
all the required information. It would be nice if I were able to
complete that within the scope of this grant.

As far as cyclomatic complexity is concerned, I don't think that that is
something which I would like to build into Devel::Cover. Static
analysis is a different problem. However, I would be very happy to have
an annotation which added cyclomatic complexity information to a
coverage report.

I could be persuaded to reconsider if static analysis solutions are
insufficient here.

Annotations are a feature of Devel::Cover which allow you to add
non-coverage information to coverage reports. At the moment, they're
really only used to add git blame information to coverage reports.

I would be happy to add such an annotation to Devel::Cover, or for an
author to upload the annotation as a separate distribution. I don't
think that all Devel::Cover reports and annotations need to be a part of
Devel::Cover itself. There are APIs for both reports and annotations.

Path coverage was already in the TODO, but I have added cyclomatic
complexity annotations. Thanks for mentioning it.