preforking prove

[Jonathan Swartz]

We have a large slow test suite at work (Test::Class, 225 classes, about 45 minutes run time). Many of the tests start by loading a bunch of the same modules. Obviously we could speed things up if we could share that loading cost.

I'm aware of Test::Aggregate and Test::Aggregate::Nested, but a number of our tests run into the caveats (e.g. they change singletons -- yes, this is not ideal, but also not easily changeable).

I thought of an alternative to Test::Aggregate called "preforking prove", or pfprove, which does the following:

* Accepts the same arguments as prove
* Preloads the module(s) in -M
* Launches a Starman server in the background
* For each test file, makes a request to the Starman server. The Starman child runs the test and exits.

The idea is that you preload all the common stuff in the Starman parent, so that forking each child and running each test is fast (at least on Linux).

Potential advantages over Test::Aggregate:
* runs one test per process, so avoids some caveats
* keeps the TAP in traditional form (one TAP stream per file)
* works well with parallel prove

Potential disadvantages:
* lots of extra complexity (requires Starman or similar, need to make sure it shuts down, need to handle errors, etc.)

Curious what you think. Is there something like this out there already? Potential problems?

If I do this I might call it Test::Aggregate::Preforking, just to keep it in the same category.

Thanks!
Jon

[Eric Wilhelm]

Hi Jonathan,

I like the idea. I rambled along similar lines a few years ago but
don't think I got past some preliminary code and $work has not had any
pressing need for such a thing in the meantime.

Post by Jonathan Swartz
* works well with parallel prove

How do you get from the single prefork into parallel test runs?
Specifically, how do you manage getting the streams into the parser?

Post by Jonathan Swartz
If I do this I might call it Test::Aggregate::Preforking, just to keep
it in the same category.

I wouldn't think of this as being under Aggregate.

--Eric

[Jonathan Swartz]

Post by Eric Wilhelm
Hi Jonathan,
I like the idea. I rambled along similar lines a few years ago but
don't think I got past some preliminary code and $work has not had any
pressing need for such a thing in the meantime.

Post by Jonathan Swartz
* works well with parallel prove

How do you get from the single prefork into parallel test runs?
Specifically, how do you manage getting the streams into the parser?

For each test run, instead of loading a .t file, you're making a request against the Starman server. So you can obviously hit it with multiple simultaneous requests.

[Karen Etheridge]

Post by Jonathan Swartz
For each test run, instead of loading a .t file, you're making a request against the Starman server. So you can obviously hit it with multiple simultaneous requests.

For something so simple, you could also use Parallel::ForkManager, with
each child passing back a serialized Test::Builder object which contained
the results of all the tests that were run. The trickiest problem is
consolidating all the test results back together and then emitting the
proper TAP output.


Karen Etheridge
***@cpan.org

[Mark Stosberg]

Post by Karen Etheridge

Post by Jonathan Swartz
For each test run, instead of loading a .t file, you're making a request against the Starman server. So you can obviously hit it with multiple simultaneous requests.

For something so simple, you could also use Parallel::ForkManager, with
each child passing back a serialized Test::Builder object which contained
the results of all the tests that were run. The trickiest problem is
consolidating all the test results back together and then emitting the
proper TAP output.

Not long ago I worked on a project where I needed to move 8 million
images to S3, where each image had a file and some associated database rows.

We wrote a solution using Parallel::ForkManager, but it benchmarked to
take 9 days to complete.

I rewrote a solution much like the one that Jonathan describes, where a
small control script submitted jobs to a pre-forking Apache/mod_perl
serve to process.

It benchmarked to take 2 days, and ultimately bottlenecked at bandwidth,
rather than CPU power as the first solution had.

Based on that, I think Starman-prove could perform very well. I also
have a large test suite that I'm always trying to make run faster, so I
like the idea a lot.

Using a number of other techniques, I've already been get the run time
down from about 25 minutes to 4.5 minutes. We now run the full suite for
every "push", rather than a few times per day.

A lot of our run time reduction was getting the tests to be
parallel-friendly, which involved some different tricks to allow them to
share the same database without problems.

I also use Test::Class, and still run all of those tests one at a time.
One of our future optimizations is to use something like
Test::Class::Load, but I suspect we will run into some problems there,
that Starman-prove would solve.

Mark

[Ovid]

Hi Jonathan,

I have just one question. You wrote that you're using Test::Class and "many of the tests start by loading a bunch of the same modules". That confuses me as Test::Class was originally designed to speed up test suites and did so by loading everything *once* in the same process.

Are you using a separate .t script per test class? That would cause the reloading. Otherwise, a single .t script loading all of your test classes (Test::Class::Load helps) should help you load all classes at once. Assuming they have a sane setup and all call their own test control methods (startup/setup/teardown/shutdown), you could drop something like this in your base class:

    # see also http://www.slideshare.net/Ovid/a-whirlwind-tour-of-testclass
    sub setup : Tests(setup) {
        my $test = shift;
        $test->reset_singletons;
    }

Yes, it's a hack to work around the fact that you have a bunch of singletons with mutable state, but it seems like this would be much easier (though perhaps less fun :) than your preforking solution.
 
Cheers,
Ovid
--
Twitter - http://twitter.com/OvidPerl/
Buy my book - http://bit.ly/beginning_perl
Buy my other book - http://www.oreilly.com/catalog/perlhks/
Live and work overseas - http://www.overseas-exile.com/

________________________________
Sent: Tuesday, 6 November 2012, 2:03
Subject: preforking prove
We have a large slow test suite at work (Test::Class, 225 classes, about 45 minutes run time). Many of the tests start by loading a bunch of the same modules. Obviously we could speed things up if we could share that loading cost.
I'm aware of Test::Aggregate and Test::Aggregate::Nested, but a number of our tests run into the caveats (e.g. they change singletons -- yes, this is not ideal, but also not easily changeable).
* Accepts the same arguments as prove
* Preloads the module(s) in -M
* Launches a Starman server in the background
* For each test file, makes a request to the Starman server. The Starman child runs the  test and exits.
The idea is that you preload all the common stuff in the Starman parent, so that forking each child and running each test is fast (at least on Linux).
* runs one test per process, so avoids some caveats
* keeps the TAP in traditional form (one TAP stream per file)
* works well with parallel prove
* lots of extra complexity (requires Starman or similar, need to make sure it shuts down, need to handle errors, etc.)
Curious what you think. Is there something like this out there already? Potential problems?
If I do this I might call it Test::Aggregate::Preforking, just to keep it in the same category.
Thanks!
Jon

[Jonathan Swartz]

Hi Ovid - yes, we use a separate .t script per test class. This creates a more traditional TAP output (one stream per class) which works better with Smolder etc, and avoids the caveats with running potentially conflicting tests in the same process. The chances of one test interfering with another in a mysterious way explode if we run all 200+ in the same process. We don't have a magic reset_singletons. :)

Post by Eric Wilhelm
Hi Jonathan,
I have just one question. You wrote that you're using Test::Class and "many of the tests start by loading a bunch of the same modules". That confuses me as Test::Class was originally designed to speed up test suites and did so by loading everything *once* in the same process.
# see also http://www.slideshare.net/Ovid/a-whirlwind-tour-of-testclass
sub setup : Tests(setup) {
my $test = shift;
$test->reset_singletons;
}
Yes, it's a hack to work around the fact that you have a bunch of singletons with mutable state, but it seems like this would be much easier (though perhaps less fun :) than your preforking solution.
Cheers,
Ovid
--
Twitter - http://twitter.com/OvidPerl/
Buy my book - http://bit.ly/beginning_perl
Buy my other book - http://www.oreilly.com/catalog/perlhks/
Live and work overseas - http://www.overseas-exile.com/
Sent: Tuesday, 6 November 2012, 2:03
Subject: preforking prove
We have a large slow test suite at work (Test::Class, 225 classes, about 45 minutes run time). Many of the tests start by loading a bunch of the same modules. Obviously we could speed things up if we could share that loading cost.
I'm aware of Test::Aggregate and Test::Aggregate::Nested, but a number of our tests run into the caveats (e.g. they change singletons -- yes, this is not ideal, but also not easily changeable).
* Accepts the same arguments as prove
* Preloads the module(s) in -M
* Launches a Starman server in the background
* For each test file, makes a request to the Starman server. The Starman child runs the test and exits.
The idea is that you preload all the common stuff in the Starman parent, so that forking each child and running each test is fast (at least on Linux).
* runs one test per process, so avoids some caveats
* keeps the TAP in traditional form (one TAP stream per file)
* works well with parallel prove
* lots of extra complexity (requires Starman or similar, need to make sure it shuts down, need to handle errors, etc.)
Curious what you think. Is there something like this out there already? Potential problems?
If I do this I might call it Test::Aggregate::Preforking, just to keep it in the same category.
Thanks!
Jon

[Jonathan Swartz]

miyagawa sent me this proof of concept - using fork, not starman. personally I like the opportunity to combine with parallel testing. :)
-----
proof of concept - https://gist.github.com/4024197

* using Shotgun loader (plackup -L Shotgun) would make a clean environment for perl to run test for each request, while still preloading modules with -M. Combined with Starman, you'll get a benefit of running multiple workers for parallel testing.

* But if you just need to preload modules to run tests, using fork() like above is much easier and involves no HTTP server etc :)

* My POC above seems to work fine (I will probably make a github repo later), but it fails in some tests (with Moose, only a couple of tests fail) and it runs almost twice as fast. It is important to remember NOT to preload Test::More (and Test::Builder) in the parent process because doing so will mess up things.

Post by Jonathan Swartz
We have a large slow test suite at work (Test::Class, 225 classes, about 45 minutes run time). Many of the tests start by loading a bunch of the same modules. Obviously we could speed things up if we could share that loading cost.
I'm aware of Test::Aggregate and Test::Aggregate::Nested, but a number of our tests run into the caveats (e.g. they change singletons -- yes, this is not ideal, but also not easily changeable).
* Accepts the same arguments as prove
* Preloads the module(s) in -M
* Launches a Starman server in the background
* For each test file, makes a request to the Starman server. The Starman child runs the test and exits.
The idea is that you preload all the common stuff in the Starman parent, so that forking each child and running each test is fast (at least on Linux).
* runs one test per process, so avoids some caveats
* keeps the TAP in traditional form (one TAP stream per file)
* works well with parallel prove
* lots of extra complexity (requires Starman or similar, need to make sure it shuts down, need to handle errors, etc.)
Curious what you think. Is there something like this out there already? Potential problems?
If I do this I might call it Test::Aggregate::Preforking, just to keep it in the same category.
Thanks!
Jon

[Jonathan Swartz]

Now on cpan. A much simpler solution than what I suggested :) and apparently still works with parallel testing. Thanks miyagawa!

https://metacpan.org/module/forkprove

Post by Jonathan Swartz
miyagawa sent me this proof of concept - using fork, not starman. personally I like the opportunity to combine with parallel testing. :)
-----
proof of concept - https://gist.github.com/4024197
* using Shotgun loader (plackup -L Shotgun) would make a clean environment for perl to run test for each request, while still preloading modules with -M. Combined with Starman, you'll get a benefit of running multiple workers for parallel testing.
* But if you just need to preload modules to run tests, using fork() like above is much easier and involves no HTTP server etc :)
* My POC above seems to work fine (I will probably make a github repo later), but it fails in some tests (with Moose, only a couple of tests fail) and it runs almost twice as fast. It is important to remember NOT to preload Test::More (and Test::Builder) in the parent process because doing so will mess up things.

Post by Jonathan Swartz
We have a large slow test suite at work (Test::Class, 225 classes, about 45 minutes run time). Many of the tests start by loading a bunch of the same modules. Obviously we could speed things up if we could share that loading cost.
I'm aware of Test::Aggregate and Test::Aggregate::Nested, but a number of our tests run into the caveats (e.g. they change singletons -- yes, this is not ideal, but also not easily changeable).
* Accepts the same arguments as prove
* Preloads the module(s) in -M
* Launches a Starman server in the background
* For each test file, makes a request to the Starman server. The Starman child runs the test and exits.
The idea is that you preload all the common stuff in the Starman parent, so that forking each child and running each test is fast (at least on Linux).
* runs one test per process, so avoids some caveats
* keeps the TAP in traditional form (one TAP stream per file)
* works well with parallel prove
* lots of extra complexity (requires Starman or similar, need to make sure it shuts down, need to handle errors, etc.)
Curious what you think. Is there something like this out there already? Potential problems?
If I do this I might call it Test::Aggregate::Preforking, just to keep it in the same category.
Thanks!
Jon

[Mark Stosberg]

Post by Jonathan Swartz
Now on cpan. A much simpler solution than what I suggested :) and apparently still works with parallel testing. Thanks miyagawa!
https://metacpan.org/module/forkprove

I did some benchmarking last night and found no real benefit over prove
-j, but Miyagawa reports that "heavy" modules like Catalyst and Moose,
he's seen 40% to 50% speed-ups.

See the details of our discussion on Github:

https://github.com/miyagawa/forkprove/commit/ca2b0c2f55a250468c4f61f7cbd1b008a0eb91b4#commitcomment-2115186

Mark

[Jonathan Swartz]

Post by Mark Stosberg

Post by Jonathan Swartz
Now on cpan. A much simpler solution than what I suggested :) and apparently still works with parallel testing. Thanks miyagawa!
https://metacpan.org/module/forkprove

I did some benchmarking last night and found no real benefit over prove
-j, but Miyagawa reports that "heavy" modules like Catalyst and Moose,
he's seen 40% to 50% speed-ups.
https://github.com/miyagawa/forkprove/commit/ca2b0c2f55a250468c4f61f7cbd1b008a0eb91b4#commitcomment-2115186

I wasn't able to get forkprove to work with Test::Class, because of Test::Class's insistence that tests be declared at compile time.

swartz> cat t/Sanity.t
#!/usr/bin/perl
use CHI::t::Sanity;
CHI::t::Sanity->runtests;

swartz> forkprove t/Sanity.t
t/Sanity.t .. Test::Class was loaded too late (after the CHECK block was run). See 'A NOTE ON LOADING TEST CLASSES' in perldoc Test::Class for more details
t/Sanity.t .. No subtests run

Mark, you mentioned before that you use Test::Class before - did you use it in conjunction with forkprove?

If this truly can't work, it may be the straw that finally gets me away from Test::Class, or at least to use Test::Class::add_testinfo instead of the subroutine attributes.

[Mark Stosberg]

Post by Jonathan Swartz
I wasn't able to get forkprove to work with Test::Class, because of

Test::Class's insistence that tests be declared at compile time.

Post by Jonathan Swartz
swartz> cat t/Sanity.t
#!/usr/bin/perl
use CHI::t::Sanity;
CHI::t::Sanity->runtests;
swartz> forkprove t/Sanity.t
t/Sanity.t .. Test::Class was loaded too late (after the CHECK block was run). See 'A NOTE ON LOADING TEST CLASSES' in perldoc Test::Class for more details
t/Sanity.t .. No subtests run
Mark, you mentioned before that you use Test::Class before - did you use it in conjunction with forkprove?

Jonathan,

It "just worked" for me, using the documented forkprove syntax of
loading modules with "-M".

I ran it on a directory that primarily contained test class files. Each
one followed this general design:

###

package Project::Test::Foo;
use parent 'Test::Class';

# my tests here...

Test::Class->runtests;

###

Mark

[Jonathan Swartz]

Post by Jonathan Swartz

Post by Jonathan Swartz
I wasn't able to get forkprove to work with Test::Class, because of

Test::Class's insistence that tests be declared at compile time.

Post by Jonathan Swartz
swartz> cat t/Sanity.t
#!/usr/bin/perl
use CHI::t::Sanity;
CHI::t::Sanity->runtests;
swartz> forkprove t/Sanity.t
t/Sanity.t .. Test::Class was loaded too late (after the CHECK block was run). See 'A NOTE ON LOADING TEST CLASSES' in perldoc Test::Class for more details
t/Sanity.t .. No subtests run
Mark, you mentioned before that you use Test::Class before - did you use it in conjunction with forkprove?

Jonathan,
It "just worked" for me, using the documented forkprove syntax of
loading modules with "-M".
I ran it on a directory that primarily contained test class files. Each
###
package Project::Test::Foo;
use parent 'Test::Class';
# my tests here...
Test::Class->runtests;
###
Mark

Ok - and what did you pass to -M?