TAP::Harness and -w

[Ricardo Signes]

I just submitted:

https://github.com/Perl-Toolchain-Gang/Test-Harness/pull/16

...to document two undocumented changes. First, the removal of "turn on -w by
default with no easy way to turn it off" and secondly the reversion of that.

These commits are, respectively:

https://github.com/Perl-Toolchain-Gang/Test-Harness/commit/18794a85

https://github.com/Perl-Toolchain-Gang/Test-Harness/commit/99159fc2

The first has a bit of discussion attached. I will only VERY briefly summarize
it as:

schwern : We need to discuss this, and please see
https://rt.cpan.org/Ticket/Display.html?id=58229
ether : We should keep the original change.
mst : We should keep the original change.
miyagawa : We should keep the original change.
leont : We should keep the original change.
ewilhelm : We should keep the original change, but update EUMM to restore -w
ribasushi: I don't see a reason to change the old behavior.

I will also add that I was in favor of the change, and presumably was Andy A.,
who made the commit in the first place.

I am all for getting this discussed publicly and brought to a consensus-based
conclusion.

[Leon Timmermans]

Post by Ricardo Signes
https://github.com/Perl-Toolchain-Gang/Test-Harness/pull/16
...to document two undocumented changes. First, the removal of "turn on -w by
default with no easy way to turn it off" and secondly the reversion of that.
https://github.com/Perl-Toolchain-Gang/Test-Harness/commit/18794a85
https://github.com/Perl-Toolchain-Gang/Test-Harness/commit/99159fc2

The change would have probably been better discussed in a less
ephemeral medium than IRC, but people reverting it without a proper
discussion is really only making that worse.

We are doing a *terrible* job of communicating with each other here,
and this thread getting warnocked is not making that better.

Post by Ricardo Signes
ewilhelm : We should keep the original change, but update EUMM to restore -w

I'm not sure that's what he meant.

Post by Ricardo Signes
I am all for getting this discussed publicly and brought to a consensus-based
conclusion.

By what process? Define consensus? Given Andy is the official
maintainer and Ovid is the effective maintainer, I don't think they
need our consensus a priori.

Leon

[Ricardo Signes]

Post by Leon Timmermans
By what process? Define consensus? Given Andy is the official
maintainer and Ovid is the effective maintainer, I don't think they
need our consensus a priori.

06perms.txt says:

Test::Harness,ANDYA,m
Test::Harness,MSCHWERN,c
Test::Harness,OVID,c

So presumably at least getting the three of them to agree on some kind of
resolution process is a start.

[Ovid]

----- Original Message -----

Post by Leon Timmermans
By what process? Define consensus? Given Andy is the official
maintainer and Ovid is the effective maintainer, I don't think they
need our consensus a priori.

  Test::Harness,ANDYA,m
  Test::Harness,MSCHWERN,c
  Test::Harness,OVID,c
So presumably at least getting the three of them to agree on some kind of
resolution process is a start.

I'm winding up with astonishingly little bandwidth due to launching our company, so I was hoping to see a strong consensus from the users. I would also love to see examples of where the change or lack thereof is causing an issue. I am SWAMPED with so much email that receiving many opinions piecemeal makes it hard for me to follow along.

Were I not so bandwidth-constrained, this would be less of an issue, but I'd like to see a good Wiki page or something with the pro/con arguments laid out. If this is too much, I should turn over maintainership to someone with more bandwidth to ensure I'm not a blocker.
 
Cheers,
Ovid
--
IT consulting, training, international recruiting
       http://www.allaroundtheworld.fr/.
Buy my book! - http://bit.ly/beginning_perl
Live and work overseas - http://www.overseas-exile.com/

[Karen Etheridge]

Post by Ovid
Were I not so bandwidth-constrained, this would be less of an issue, but I'd like to see a good Wiki page or something with the pro/con arguments laid out. If this is too much, I should turn over maintainership to someone with more bandwidth to ensure I'm not a blocker.

wiki page created:
https://github.com/Perl-Toolchain-Gang/Test-Harness/wiki/TAP::Harness-and--the-w-flag

[Ovid]

----- Original Message -----

Post by Ovid

Post by Ovid
Were I not so bandwidth-constrained, this would be less of an issue, but

I'd like to see a good Wiki page or something with the pro/con arguments
laid out. If this is too much, I should turn over maintainership to someone with
more bandwidth to ensure I'm not a blocker.
https://github.com/Perl-Toolchain-Gang/Test-Harness/wiki/TAP::Harness-and--the-w-flag

Karen,

Thank you for putting that together. It made it much easier for me to follow this!

The Problem:

In Test::Harness, many people want to see -w no longer being enabled by default. Others object to this change because this alters the behavior and it we should tread carefully, particularly since this change effects everyone who installs modules. Backwards-incompatible changes  to toolchain modules should generally not be done lightly. What's worse, different people have different desired behaviors here (mainly ribasushi, as far as I can tell, but I don't know what the silent masses think). 

That being said, I tend to agree that -w should not be forced on those who do not want it.

Solutions:

1. Make the change.
2. Don't make the change.
3/4. Make this very easy to configure but keep/change the old behavior.

We have a change that people want, but as Eric pointed out, it introduces an incompatibility in the Test::Harness backwards-compatibility layer. If I'm to be conservative on this, I have to say that the one place we *don't* want incompatible changes is in a backwards-compatibility later!

I originally wrote TAP::Harness to be very configurable (we can argue later if I succeeded). Since Test::Harness is merely a compatibility layer on top of TAP::Harness, I like Eric Wilhelm's suggestion of software switching to TAP::Harness instead of Test::Harness. The basic change is simple. Instead of:

    use Test::Harness;
    runtests(@test_files); 


You do this:

    use TAP::Harness;
    my $harness = TAP::Harness->new( \%args );
    $harness->runtests(@tests);

Of course, the devil is in the details and I imagine that many tools will be more seriously impacted.

This allows us to maintain backwards-compatibility and gives uses a better interface, to boot!

The Real Question:

What toolchain software is being impacted by this and how hard would it be to make the switch? People have long known that TAP::Harness is a better alternative to Test::Harness, but the compatibility layer meant that people wouldn't have to make that switch right away. I suppose that some time in the past 5 1/2 years I should have been urging people to make the change. Perhaps now there is a reason to make that switch?

Admittedly, the above sounds like a remarkably self-serving way of passing the buck (very handy right now), but the question is very real: what are the obstacles to those wanting a different behavior in making a switch?

Cheers,
Ovid

--
IT consulting, training, international recruiting
       http://www.allaroundtheworld.fr/.
Buy my book! - http://bit.ly/beginning_perl
Live and work overseas - http://www.overseas-exile.com/

[Leon Timmermans]

Post by Ovid
What toolchain software is being impacted by this and how hard would it be to make the switch?

AFAIK only ExtUtils::MakeMaker and Module::Build use it. The latter
can already use TAP::Harness, but there are some details we'd might
want to fix first (such as environmental variables).
Module::Build::Tiny is already using TAP::Harness exclusively.

Post by Ovid
People have long known that TAP::Harness is a better alternative to Test::Harness, but the compatibility layer meant that people wouldn't have to make that switch right away.

Given toolchain still has to support perl < 5.10.1, using TAP::Harness
means an extra non-core dependency. I suspect that's the real reason.

Possibly we could use TAP::Harness but fall back on Test::Harness if
the former is not present. Disadvantage may be different test results
depending on this because of the -w issue.

Leon

[Leon Timmermans]

Post by Ovid
I'm winding up with astonishingly little bandwidth due to launching our
company, so I was hoping to see a strong consensus from the users. I would
also love to see examples of where the change or lack thereof is causing an
issue. I am SWAMPED with so much email that receiving many opinions
piecemeal makes it hard for me to follow along.
Were I not so bandwidth-constrained, this would be less of an issue, but
I'd like to see a good Wiki page or something with the pro/con arguments
laid out. If this is too much, I should turn over maintainership to someone
with more bandwidth to ensure I'm not a blocker.

Just as I expected, "make it a wiki" means it gets warnocked again.

Can we please make a decision, or if we must first come to an agreement on
how to make it?

Leon

[Mark Stosberg]

Post by Ovid
I'm winding up with astonishingly little bandwidth due to launching
our company, so I was hoping to see a strong consensus from the
users. I would also love to see examples of where the change or lack
thereof is causing an issue. I am SWAMPED with so much email that
receiving many opinions piecemeal makes it hard for me to follow along.
Were I not so bandwidth-constrained, this would be less of an issue,
but I'd like to see a good Wiki page or something with the pro/con
arguments laid out. If this is too much, I should turn over
maintainership to someone with more bandwidth to ensure I'm not a
blocker.
Just as I expected, "make it a wiki" means it gets warnocked again.
Can we please make a decision, or if we must first come to an agreement
on how to make it?

I think a pro/con list is a reasonable request. I've read all the
messages myself, am a regular user of Perl's testing tools, and the
benefits and drawbacks are not clear to me either.

An attempting-to-be-neutral summary would be very helpful.

Mark

[chromatic]

Post by Mark Stosberg
An attempting-to-be-neutral summary would be very helpful.

Pro of adding -w to test runs:
- that's how Test::Harness has always worked, and people might be relying
on that behavior

Cons of adding -w to test runs:
- you get warnings from dependencies (and their dependencies) because -w
enables global action at a distance
- using fatal warnings may cause your test suite to fail because of
warnings in dependencies you don't directly control

-- c

[Mark Stosberg]

Post by chromatic

Post by Mark Stosberg
An attempting-to-be-neutral summary would be very helpful.

- that's how Test::Harness has always worked, and people might be relying
on that behavior
- you get warnings from dependencies (and their dependencies) because -w
enables global action at a distance
- using fatal warnings may cause your test suite to fail because of
warnings in dependencies you don't directly control

Thank you.

I would rather TAP::Harness not enable warnings by default.

I would prefer to opt-in, rather than opt out.

Mark

[Buddy Burden]

Post by chromatic
- you get warnings from dependencies (and their dependencies) because -w
enables global action at a distance
- using fatal warnings may cause your test suite to fail because of
warnings in dependencies you don't directly control

- you could get different output from running tests via `prove` than you
do via running the test file directly, which could cause you to scrath your head
and go "WTF??"

I see Ovid mentioned in a later email that prove uses TAP::Harness instead of Test::Harness, implying that it doesn't have this problem, but I know I spent several hours a few months back trying to figure out just what the hell was going on, and it finally came down to _somebody_ flipping on -w unexpectedly.


-- Buddy

[Leon Timmermans]

Post by Mark Stosberg
I think a pro/con list is a reasonable request. I've read all the
messages myself, am a regular user of Perl's testing tools, and the
benefits and drawbacks are not clear to me either.
An attempting-to-be-neutral summary would be very helpful.

I agree that's a reasonable request. However, the odds are no one except
me, ether and Ovid have looked at that wikipage between the previous set of
emails and my email 2 hours ago. IMO, it's the wrong medium.

Leon

[Eric Wilhelm]

odds are no one ... looked at that wikipage ...
IMO, it's the wrong medium.

I agree. Also, Test::Harness is the wrong module for this discussion.

- that's how Test::Harness has always worked, ...

This really isn't a question of pros/cons of -w. That should be an
option for the user with sensible defaults. It already is an option
with sensible defaults in TAP::Harness.

I believe Test::Harness is a compatibility layer for the sake of
compatibility (and the first sentence of its docs seems to confirm it.)
If this is true, changing it in incompatible ways is counter to its
reason to exist. So, it seems better to deprecate and then delete it
rather than discuss changes.

Thanks,
Eric

[Ovid]

As I said in my previous email on July 7th: backwards-incompatible changes to the backwards-compatibility layer (Test::Harness) are not a good idea. The proper response is to have people impacted by this issue switch to TAP::Harness, as was suggested several years ago when Test::Harness 3.0 was released. For example, the 'prove' utility calls App::Prove which calls TAP::Harness. If others are using Test::Harness directly, perhaps Eric is right and it should be deprecated? However, it's a core module and I don't know the implications of that.
 
Cheers,
Ovid
--
IT consulting, training, international recruiting
       http://www.allaroundtheworld.fr/.
Buy my book! - http://bit.ly/beginning_perl
Live and work overseas - http://www.overseas-exile.com/

________________________________
Sent: Tuesday, 17 September 2013, 17:26
Subject: Re: TAP::Harness and -w
I'm winding up with astonishingly little bandwidth due to launching our company, so I was hoping to see a strong consensus from the users. I would also love to see examples of where the change or lack thereof is causing an issue. I am SWAMPED with so much email that receiving many opinions piecemeal makes it hard for me to follow along.

Post by Ovid
Were I not so bandwidth-constrained, this would be less of an issue, but I'd like to see a good Wiki page or something with the pro/con arguments laid out. If this is too much, I should turn over maintainership to someone with more bandwidth to ensure I'm not a blocker.

Just as I expected, "make it a wiki" means it gets warnocked again.
Can we please make a decision, or if we must first come to an agreement on how to make it?
Leon

[Ricardo Signes]

Post by Ovid
As I said in my previous email on July 7th: backwards-incompatible changes to
the backwards-compatibility layer (Test::Harness) are not a good idea. The
proper response is to have people impacted by this issue switch to
TAP::Harness, as was suggested several years ago when Test::Harness 3.0 was
released.

It looks like Module::Build already has use_tap_harness. So what's suggested
is a patch to ExtUtils::MM_Any to switch from Test::Harness to TAP::Harness...?

[Leon Timmermans]

Post by Ricardo Signes
It looks like Module::Build already has use_tap_harness. So what's suggested
is a patch to ExtUtils::MM_Any to switch from Test::Harness to
TAP::Harness...?

Or both just setting $Test::Harness::switches = ''


Leon

[Ricardo Signes]

Post by Leon Timmermans
Or both just setting $Test::Harness::switches = ''


Yes. I was cleaning out ~/tmp on the flight over here and I found this dzil
plugin:

package inc::STFUEUMM;
use Moose;
extends 'Dist::Zilla::Plugin::MakeMaker';
use namespace::autoclean;

after setup_installer => sub {
my ($self) = @_;
my ($file) = grep { $_->name eq 'Makefile.PL' } @{ $self->zilla->files };
my $content = $file->content;

$content .= <<'END_AWFUL_HACK';
sub MY::test_via_harness {
my($self, $perl, $tests) = @_;
return qq{\t$perl "-MExtUtils::Command::MM" }.
qq{"-e" "require Test::Harness; undef *Test::Harness::Switches; test_harness(\$(TEST_VERBOSE), '\$(INST_LIB)', '\$(INST_ARCHLIB)')" $tests\n};
}
END_AWFUL_HACK

$file->content($content);
};


I hadn't published that before because AndyA applied my patch to Test::Harness.
Hmm...

--
rjbs

[Ricardo Signes]

Post by Leon Timmermans
Just as I expected, "make it a wiki" means it gets warnocked again.
Can we please make a decision, or if we must first come to an agreement on
how to make it?

Allow me to offer this:

https://github.com/Perl-Toolchain-Gang/ExtUtils-MakeMaker/pull/72

Rather than change Test::Harness, which would break the compatibility of the
compatibility layer, we just fix the thing that *actually* bothers people: the
behavior of `make test`.

[Leon Timmermans]

Post by Ricardo Signes

Post by Leon Timmermans
Just as I expected, "make it a wiki" means it gets warnocked again.
Can we please make a decision, or if we must first come to an agreement

on

Post by Leon Timmermans
how to make it?

https://github.com/Perl-Toolchain-Gang/ExtUtils-MakeMaker/pull/72
Rather than change Test::Harness, which would break the compatibility of the
compatibility layer, we just fix the thing that *actually* bothers people: the
behavior of `make test`.

Given I've recently added a «use warnings;» to Test::Harness (not released
yet), '' may be a better value for $Switches than undef.

Leon

[Eric Wilhelm]

Post by Leon Timmermans

Post by Ricardo Signes
ewilhelm : We should keep the original change, but update EUMM to restore -w

I'm not sure that's what he meant.

Yeah no. I said get rid of Test::Harness.

TAP::Harness provides a cleaner interface and this default only exist in
the old crufty compatibility interface.

If you're going to keep the compatibility layer, don't change it. But
if we're going to argue about changing it, I say we should burn it.

" changing it in the Test::Harness reverse compatibility
layer seems incompatible with reverse compatibility."
"require TAP::Harness and use that
interface (where it would be off by default anyway, right?)

IIRC, Module::Build has --use-tap-harness or something, which should
probably just be the default these days."

--Eric

[Ricardo Signes]

Post by Eric Wilhelm

Post by Leon Timmermans

Post by Ricardo Signes
ewilhelm : We should keep the original change, but update EUMM to restore -w

I'm not sure that's what he meant.

Yeah no. I said get rid of Test::Harness.
TAP::Harness provides a cleaner interface and this default only exist in
the old crufty compatibility interface.

Thanks for clarifying!