Best practices for TODO test management?

[Mark Stosberg]

We have a large test suite with 20,000+ tests in it, and a number of
passing TODO tests in it.

Our problem is that once a test is marked as TODO, it falls off our
radar, and we are unlikely to notice it again.

What are some suggestions to make sure that passing TODO tests get
regular attention?

I was hoping that 'prove' might have a flag to treat passing-TODOs as
failures, but Id didn't see one.

I would prefer not to write a custom harness... this seems like a
feature that would be of general interest.

In case this context matters-- our tests are regularly through Jenkins,
so our TAP is getting converted to JUnit. We also run tests from the
command line with prove regularly as well.

Thanks!

Mark

[Pedro Melo]

Hi,

Post by Mark Stosberg
We have a large test suite with 20,000+ tests in it, and a number of
passing TODO tests in it.
Our problem is that once a test is marked as TODO, it falls off our
radar, and we are unlikely to notice it again.
What are some suggestions to make sure that passing TODO tests get
regular attention?

Make a TODO that passes all subtests a failure? Not sure how to go about
it, but it sure would get your attention :)


Bye,

[Ovid]

Hi Mark,

I know this isn't exactly what you want, but a long time ago I added a --directives switch to 'prove' to help with this. If you do this:

    prove -lr --directives t/

'prove' will *not* run in verbose mode, except for any tests with directives (TODO and SKIP).
 
It's not perfect, but at least it will make the TODO tests stand out (they're also colored differently if you're using colors).

Cheers,
Ovid
--
Twitter - http://twitter.com/OvidPerl/
Buy my book - http://bit.ly/beginning_perl
Buy my other book - http://www.oreilly.com/catalog/perlhks/
Live and work overseas - http://www.overseas-exile.com/

________________________________
Sent: Wednesday, 27 February 2013, 15:06
Subject: Best practices for TODO test management?
We have a large test suite with 20,000+ tests in it, and a number of
passing TODO tests in it.
Our problem is that once a test is marked as TODO, it falls off our
radar, and we are unlikely to notice it again.
What are some suggestions to make sure that passing TODO tests get
regular attention?
I was hoping that 'prove' might have a flag to treat passing-TODOs as
failures, but Id didn't see one.
I would prefer not to write a custom harness... this seems like a
feature that would be of general interest.
In case this context matters-- our tests are regularly through Jenkins,
so our TAP is getting converted to JUnit. We also run tests from the
command line with prove regularly as well.
Thanks!
  Mark

[Mark Stosberg]

Post by Ovid
Hi Mark,
I know this isn't exactly what you want, but a long time ago I added a
prove -lr --directives t/
'prove' will *not* run in verbose mode, except for any tests with
directives (TODO and SKIP).
It's not perfect, but at least it will make the TODO tests stand out
(they're also colored differently if you're using colors).

Thanks for that tip, Ovid!

I had overlooked this feature. I did find it helpful, but found it it
doesn't work in combination with the "-j" feature to run tests in
parallel. I'm not sure if that's a bug, or an inherent limitation.

Still, it's nice to know about this option, even if I have to have a
slower test run now-and-then to use it.

Mark

[Graham TerMarsch]

Post by Mark Stosberg
What are some suggestions to make sure that passing TODO tests get
regular attention?

[.....snip.....]

Post by Mark Stosberg
In case this context matters-- our tests are regularly through Jenkins,
so our TAP is getting converted to JUnit. We also run tests from the
command line with prove regularly as well.

Mark, what are you using to convert your TAP to JUnit?

I ask as TAP::Formatter::JUnit automatically treats "passing TODOs" as
"failures" and reports them as such in the JUnit output.

(disclaimer: I'm the author of TAP::Formatter::JUnit)

[Mark Stosberg]

Post by Graham TerMarsch

Post by Mark Stosberg
What are some suggestions to make sure that passing TODO tests get
regular attention?

[.....snip.....]

Post by Mark Stosberg
In case this context matters-- our tests are regularly through Jenkins,
so our TAP is getting converted to JUnit. We also run tests from the
command line with prove regularly as well.

Mark, what are you using to convert your TAP to JUnit?
I ask as TAP::Formatter::JUnit automatically treats "passing TODOs" as
"failures" and reports them as such in the JUnit output.

I use "TAP::Harness::JUnit", only because I got it working, and it works
well enough.

Thanks for highlighting this feature of TAP::Formatter::JUnit. I'll have
to give it another look.

When I looked at one one of my passing TODO tests, I found that it
marked as a TODO, because sometimes the test data it was working with
would be in the desired state, and sometimes it wouldn't be. So, it
wasn't a simply matter of of un-TODO'ing the test in that case, as it
would still fail sometimes.

In a case like that, I have to re-work the test so that it has
sufficient control over the test data to give consistent results.
Perhaps that test would be better written to "SKIP" if the data isn't
working with isn't in a state for the test to be meaningful.

Mark

[Steffen Schwigon]

Post by Mark Stosberg
We have a large test suite with 20,000+ tests in it, and a number of
passing TODO tests in it.
Our problem is that once a test is marked as TODO, it falls off our
radar, and we are unlikely to notice it again.

Maybe this helps:

http://www.nntp.perl.org/group/perl.qa/2009/07/msg12376.html

It's applying my personal hammer "DPath" to my personal nail "TAP::DOM".

Kind regards,
Steffen