new Stream Test::Simple question, XS requirement

Discussion:

bulk88

2015-04-25 21:57:19 UTC

Since Test::Stream::Context's core design uses weaken() everywhere, that
means new Test::Simple always requires XS building (for Scalar::Util),
which means it will never run on miniperl, and hence, new Test::Simple
will never be in Perl core, is my understanding correct?

I am asking this since if new Test::Simple is not PP clean and never
will be, there are couple places I would like to throw XS at (the
accessors in Test::Stream::HashBase::Meta in particular).

Chad Granum

2015-04-25 22:20:38 UTC

Permalink

I have no objections to having OPTIONAL XS available to speed things up for
people who want it. However I will not accept making XS a requirement. I
thought that since Scalar::Util::weaken was in core that there would be no
issues using it in a core module. If using weaken makes Test-Simple
non-corable, then I will use a slightly alternate design. But I will wait
for rjbs to weigh in on if Scalar::Util::weaken will block Test-Simple from
core.

-Chad

Post by bulk88
Since Test::Stream::Context's core design uses weaken() everywhere, that
means new Test::Simple always requires XS building (for Scalar::Util),
which means it will never run on miniperl, and hence, new Test::Simple will
never be in Perl core, is my understanding correct?
I am asking this since if new Test::Simple is not PP clean and never will
be, there are couple places I would like to throw XS at (the accessors in
Test::Stream::HashBase::Meta in particular).

bulk88

2015-04-26 03:56:23 UTC

Permalink

Post by Chad Granum
I have no objections to having OPTIONAL XS available to speed things up
for people who want it. However I will not accept making XS a
requirement. I thought that since Scalar::Util::weaken was in core that
there would be no issues using it in a core module.

Core includes XS modules. You relied on an XS module, see below.

Post by Chad Granum
If using weaken
makes Test-Simple non-corable, then I will use a slightly alternate
design.

Current new Test::Simple is slightly slower than legacy, will your
non-weaken solution make new Test::Simple twice as slow as legacy?
Because of the rjbs post that supposedly (someone else should confirm
this) minitest does not use Test::*, new Test::Simple can use all the XS
it wants. Maybe you want to change the design rule and say XS is
mandatory to run new Test::More, but also remember, legacy Test::Simple
doesn't require XS. There are 2 levels of "requires XS", 1 is to say
you must have a working C compiler to install new Test::Simple, the
other level is to say, at some point in the past you or your predecessor
must have had access to a C compiler to build the core XS modules. If
your platform's perl doesn't have DynaLoader, or if your OS vendor/IT
dept decided to only distribute XS-free core modules, then sorry, new
Test::Simple is not supported on your system (must have working
Scalar::Util).

To prove new Test::Simple requires XS, I ran.

---------------------------------------------------------
C:\perl521\srcnewb4opt>miniperl -IC:\sources\testmorenew\lib -Ilib
-MTest::More
-e"0"
Can't load module List::Util, dynamic loading not available in this perl.
(You may need to build a new perl executable which either supports
dynamic loading or has the List::Util module statically linked into it.)
at lib/Scalar/Util.pm line 11.
Compilation failed in require at lib/Scalar/Util.pm line 11.
Compilation failed in require at
C:\sources\testmorenew\lib/Test/Stream/Util.pm line 5.
BEGIN failed--compilation aborted at
C:\sources\testmorenew\lib/Test/Stream/Util.pm line 5.
Compilation failed in require at
C:\sources\testmorenew\lib/Test/Stream.pm line 9.
BEGIN failed--compilation aborted at
C:\sources\testmorenew\lib/Test/Stream.pm line 9.
Compilation failed in require at C:\sources\testmorenew\lib/Test/More.pm
line 10.
BEGIN failed--compilation aborted at
C:\sources\testmorenew\lib/Test/More.pm line 10.
Compilation failed in require.
BEGIN failed--compilation aborted.

C:\perl521\srcnewb4opt>
---------------------------------------------------------

Legacy doesn't requre XS

---------------------------------------------------------
C:\perl521\srcnewb4opt>miniperl -Ilib -MTest::More -e"0"

C:\perl521\srcnewb4opt>
---------------------------------------------------------

So what is your (exodist) design rule, atleast for now, on XS and new
Test::Simple?

Chad Granum

2015-04-26 04:15:18 UTC

Permalink

I am not sure it is entirely my decision. If it is entirely my decision
than this is it:
Must have a working scalar::util::weaken to use the new Test::Simple. The
Test-Simple dist itself will not require a c compiler to install, as in no
xs is included in the dist. A Test-Simple-XS dist may be created if desired
that will be used by Test-Simple if it is installed, but it will not be a
requirement. Up to rjbs and p5p as to of the xs components will be part of
core.

However I want to know how the rest of perl-qa feels about this first. If
people are not bothered by using scalar::util then we have no problem. If
they are bothered by scalar::util being used then we need to look at
options.

Does anyone have a strong opinion about Test-Simple needing scalar::util?

Post by bulk88

Post by Chad Granum
I have no objections to having OPTIONAL XS available to speed things up
for people who want it. However I will not accept making XS a requirement.
I thought that since Scalar::Util::weaken was in core that there would be
no issues using it in a core module.

Core includes XS modules. You relied on an XS module, see below.
If using weaken makes Test-Simple non-corable, then I will use a slightly

Post by Chad Granum
alternate design.

Current new Test::Simple is slightly slower than legacy, will your
non-weaken solution make new Test::Simple twice as slow as legacy? Because
of the rjbs post that supposedly (someone else should confirm this)
minitest does not use Test::*, new Test::Simple can use all the XS it
wants. Maybe you want to change the design rule and say XS is mandatory to
run new Test::More, but also remember, legacy Test::Simple doesn't require
XS. There are 2 levels of "requires XS", 1 is to say you must have a
working C compiler to install new Test::Simple, the other level is to say,
at some point in the past you or your predecessor must have had access to a
C compiler to build the core XS modules. If your platform's perl doesn't
have DynaLoader, or if your OS vendor/IT dept decided to only distribute
XS-free core modules, then sorry, new Test::Simple is not supported on your
system (must have working Scalar::Util).
To prove new Test::Simple requires XS, I ran.
---------------------------------------------------------
C:\perl521\srcnewb4opt>miniperl -IC:\sources\testmorenew\lib -Ilib
-MTest::More
-e"0"
Can't load module List::Util, dynamic loading not available in this perl.
(You may need to build a new perl executable which either supports
dynamic loading or has the List::Util module statically linked into it.)
at lib/Scalar/Util.pm line 11.
Compilation failed in require at lib/Scalar/Util.pm line 11.
Compilation failed in require at
C:\sources\testmorenew\lib/Test/Stream/Util.pm line 5.
BEGIN failed--compilation aborted at
C:\sources\testmorenew\lib/Test/Stream/Util.pm line 5.
Compilation failed in require at C:\sources\testmorenew\lib/Test/Stream.pm
line 9.
BEGIN failed--compilation aborted at
C:\sources\testmorenew\lib/Test/Stream.pm line 9.
Compilation failed in require at C:\sources\testmorenew\lib/Test/More.pm
line 10.
BEGIN failed--compilation aborted at
C:\sources\testmorenew\lib/Test/More.pm line 10.
Compilation failed in require.
BEGIN failed--compilation aborted.
C:\perl521\srcnewb4opt>
---------------------------------------------------------
Legacy doesn't requre XS
---------------------------------------------------------
C:\perl521\srcnewb4opt>miniperl -Ilib -MTest::More -e"0"
C:\perl521\srcnewb4opt>
---------------------------------------------------------
So what is your (exodist) design rule, atleast for now, on XS and new
Test::Simple?

Chad Granum

2015-04-26 04:39:29 UTC

Permalink

Just to also be clear. My alternate implementation would be a fallback. If
scalar::util works it uses it, if not it falls back to a slightly different
system. The switch would happen when Context.pm is loaded. So even if the
fallback is slightly slower, few if any systems would ever experience it.
The fallback I have in mind also maintains the exact same interface, so it
would be seamless to users, and does not deviate from the current context
usage in any way.

Post by Chad Granum
I am not sure it is entirely my decision. If it is entirely my decision
Must have a working scalar::util::weaken to use the new Test::Simple. The
Test-Simple dist itself will not require a c compiler to install, as in no
xs is included in the dist. A Test-Simple-XS dist may be created if desired
that will be used by Test-Simple if it is installed, but it will not be a
requirement. Up to rjbs and p5p as to of the xs components will be part of
core.
However I want to know how the rest of perl-qa feels about this first. If
people are not bothered by using scalar::util then we have no problem. If
they are bothered by scalar::util being used then we need to look at
options.
Does anyone have a strong opinion about Test-Simple needing scalar::util?

Post by bulk88

Post by Chad Granum
I have no objections to having OPTIONAL XS available to speed things up
for people who want it. However I will not accept making XS a requirement.
I thought that since Scalar::Util::weaken was in core that there would be
no issues using it in a core module.

Core includes XS modules. You relied on an XS module, see below.
If using weaken makes Test-Simple non-corable, then I will use a

Post by Chad Granum
slightly alternate design.

Current new Test::Simple is slightly slower than legacy, will your
non-weaken solution make new Test::Simple twice as slow as legacy? Because
of the rjbs post that supposedly (someone else should confirm this)
minitest does not use Test::*, new Test::Simple can use all the XS it
wants. Maybe you want to change the design rule and say XS is mandatory to
run new Test::More, but also remember, legacy Test::Simple doesn't require
XS. There are 2 levels of "requires XS", 1 is to say you must have a
working C compiler to install new Test::Simple, the other level is to say,
at some point in the past you or your predecessor must have had access to a
C compiler to build the core XS modules. If your platform's perl doesn't
have DynaLoader, or if your OS vendor/IT dept decided to only distribute
XS-free core modules, then sorry, new Test::Simple is not supported on your
system (must have working Scalar::Util).
To prove new Test::Simple requires XS, I ran.
---------------------------------------------------------
C:\perl521\srcnewb4opt>miniperl -IC:\sources\testmorenew\lib -Ilib
-MTest::More
-e"0"
Can't load module List::Util, dynamic loading not available in this perl.
(You may need to build a new perl executable which either supports
dynamic loading or has the List::Util module statically linked into it.)
at lib/Scalar/Util.pm line 11.
Compilation failed in require at lib/Scalar/Util.pm line 11.
Compilation failed in require at
C:\sources\testmorenew\lib/Test/Stream/Util.pm line 5.
BEGIN failed--compilation aborted at
C:\sources\testmorenew\lib/Test/Stream/Util.pm line 5.
Compilation failed in require at
C:\sources\testmorenew\lib/Test/Stream.pm line 9.
BEGIN failed--compilation aborted at
C:\sources\testmorenew\lib/Test/Stream.pm line 9.
Compilation failed in require at C:\sources\testmorenew\lib/Test/More.pm
line 10.
BEGIN failed--compilation aborted at
C:\sources\testmorenew\lib/Test/More.pm line 10.
Compilation failed in require.
BEGIN failed--compilation aborted.
C:\perl521\srcnewb4opt>
---------------------------------------------------------
Legacy doesn't requre XS
---------------------------------------------------------
C:\perl521\srcnewb4opt>miniperl -Ilib -MTest::More -e"0"
C:\perl521\srcnewb4opt>
---------------------------------------------------------
So what is your (exodist) design rule, atleast for now, on XS and new
Test::Simple?

James E Keenan

2015-04-26 12:24:47 UTC

Permalink

As I suggested in an earlier post in this thread, I think this is mainly
a backwards-compatibility issue.

The new Test-Simple, like the old one, will be distributed primarily in
two ways:

* As a stand-alone distribution on CPAN, in which case it should work
with older versions of Perl 5 as far back as possible.

* As part of the Perl 5 core distribution, in which case it only has to
work with the version with which it's distributed.

We've got the second case covered, so we only have to worry about the
former. If the new Test-Simple is being installed against a Perl that
already has Scalar::Util::weaken -- which is guaranteed by Perl 5.8.5 or
later -- there is no problem. The version of Test-Simple distributed
with Perl 5.20 was 1.001002 and had a minimum Perl version of 5.006. So
for all practical purposes, the only production versions of Perl we have
to worry about are those from 5.6.0 to 5.8.4.

IIRC, we had to face problems like this several times in the era before
Scalar-List-Utils became part of the core distro and ... we solved them!
I wasn't part of p5p at the time, so I don't recall the details, but
I'll bet it's a SMOP and hence no obstacle to Test-Simple's ongoing
development.

Thank you very much.
Jim Keenan

Post by Chad Granum

Post by bulk88

Post by Chad Granum
I have no objections to having OPTIONAL XS available to speed things up
for people who want it. However I will not accept making XS a requirement.
I thought that since Scalar::Util::weaken was in core that there would be
no issues using it in a core module.

Core includes XS modules. You relied on an XS module, see below.
If using weaken makes Test-Simple non-corable, then I will use a slightly

Post by Chad Granum
alternate design.

Current new Test::Simple is slightly slower than legacy, will your
non-weaken solution make new Test::Simple twice as slow as legacy? Because
of the rjbs post that supposedly (someone else should confirm this)
minitest does not use Test::*, new Test::Simple can use all the XS it
wants. Maybe you want to change the design rule and say XS is mandatory to
run new Test::More, but also remember, legacy Test::Simple doesn't require
XS. There are 2 levels of "requires XS", 1 is to say you must have a
working C compiler to install new Test::Simple, the other level is to say,
at some point in the past you or your predecessor must have had access to a
C compiler to build the core XS modules. If your platform's perl doesn't
have DynaLoader, or if your OS vendor/IT dept decided to only distribute
XS-free core modules, then sorry, new Test::Simple is not supported on your
system (must have working Scalar::Util).
To prove new Test::Simple requires XS, I ran.
---------------------------------------------------------
C:\perl521\srcnewb4opt>miniperl -IC:\sources\testmorenew\lib -Ilib
-MTest::More
-e"0"
Can't load module List::Util, dynamic loading not available in this perl.
(You may need to build a new perl executable which either supports
dynamic loading or has the List::Util module statically linked into it.)
at lib/Scalar/Util.pm line 11.
Compilation failed in require at lib/Scalar/Util.pm line 11.
Compilation failed in require at
C:\sources\testmorenew\lib/Test/Stream/Util.pm line 5.
BEGIN failed--compilation aborted at
C:\sources\testmorenew\lib/Test/Stream/Util.pm line 5.
Compilation failed in require at C:\sources\testmorenew\lib/Test/Stream.pm
line 9.
BEGIN failed--compilation aborted at
C:\sources\testmorenew\lib/Test/Stream.pm line 9.
Compilation failed in require at C:\sources\testmorenew\lib/Test/More.pm
line 10.
BEGIN failed--compilation aborted at
C:\sources\testmorenew\lib/Test/More.pm line 10.
Compilation failed in require.
BEGIN failed--compilation aborted.
C:\perl521\srcnewb4opt>
---------------------------------------------------------
Legacy doesn't requre XS
---------------------------------------------------------
C:\perl521\srcnewb4opt>miniperl -Ilib -MTest::More -e"0"
C:\perl521\srcnewb4opt>
---------------------------------------------------------
So what is your (exodist) design rule, atleast for now, on XS and new
Test::Simple?

Ricardo Signes

2015-04-26 12:40:25 UTC

Permalink

Because of the rjbs post that supposedly (someone else should confirm this)
minitest does not use Test::*, new Test::Simple can use all the XS it wants.

Re-confirmation welcome! For the record: My first check was scanning all the
tests run by the minitest make target for 'use Test'. Then I ran minitest with
a dtrace probe for all files opened. Only two files matching /Test/ were
opened:

~$ grep Test dtrace.log | sort | uniq
0 151 open:entry miniperl ../TestInit.pm
1 151 open:entry miniperl ../TestInit.pm
2 151 open:entry miniperl ../TestInit.pm
3 151 open:entry miniperl ../TestInit.pm
3 151 open:entry miniperl ../lib/unicore/TestProp.pl

--
rjbs

Chad Granum

2015-04-26 14:02:34 UTC

Permalink

For now I am going to add the minimum necessary Scalar::Util version to the
requirements list in Makefile.PL. This should meet the toolchain
requirement of supporting back to 5.8.1 as Scalar::Util is on cpan for
install in perl versions where it is too old. It also sounds like this is
not a blocker for putting new Test-Simple into core.

As for people on perls older than 5.8.5 that cannot update Scalar::Util due
to XS restrictions, or people who use newer perls but cannot load
Scalar::Util, I suspect that in practice it is exceedingly rare that anyone
in this position would want to update Test-Simple, let alone need to. For
now I am not going to put effort into making Test-Simple work without
Scalar::Util. If anybody comes to me and tells me they are in this
position, and asks for my help, I would be happy to work on it.

-Chad

Post by bulk88

Post by bulk88
Because of the rjbs post that supposedly (someone else should confirm

this)

Post by bulk88
minitest does not use Test::*, new Test::Simple can use all the XS it

wants.
Re-confirmation welcome! For the record: My first check was scanning all the
tests run by the minitest make target for 'use Test'. Then I ran minitest with
a dtrace probe for all files opened. Only two files matching /Test/ were
~$ grep Test dtrace.log | sort | uniq
0 151 open:entry miniperl ../TestInit.pm
1 151 open:entry miniperl ../TestInit.pm
2 151 open:entry miniperl ../TestInit.pm
3 151 open:entry miniperl ../TestInit.pm
3 151 open:entry miniperl
../lib/unicore/TestProp.pl
--
rjbs

James E Keenan

2015-04-25 23:10:19 UTC

Permalink

Post by bulk88
Since Test::Stream::Context's core design uses weaken() everywhere, that
means new Test::Simple always requires XS building (for Scalar::Util),
which means it will never run on miniperl, and hence, new Test::Simple
will never be in Perl core, is my understanding correct?
I am asking this since if new Test::Simple is not PP clean and never
will be, there are couple places I would like to throw XS at (the
accessors in Test::Stream::HashBase::Meta in particular).

The use of functions defined in Scalar-List-Util and implemented via XS
inside the definitions of functions in other libraries which are
distributed with Perl 5 core happens not to be frequent but is not,
AFAIK, forbidden.

From the top of a checkout of the core distro (Linux syntax for xargs):

find cpan/ -type f -name '*.xs' | \
sort | \
cut -d '/' -f1,2 | \
xargs -I % find % -type f -name '*.pm' | \
xargs grep -n -E '(use|require)\s+(Scalar|List)::Util' | \
grep -v Scalar-List-Utils

This shows that Compress-Raw-Bzip2's test suite uses
Scalar::Util::dualvar. Scalar::Uti::dualvar is also used inside Socket
to formulate an error message in one situation.

Scalar-List-Utils's Changes file states that 'weaken()' was added at
some point before version 1.14. Scalar-List-Utils v1.14 was first
distributed with a *production* version of Perl 5 in v5.8.5. Hence,
there might be a limit to the backwards compatibility of a new
Test-Simple distribution.

Thank you very much.
Jim Keenan

Leon Timmermans

2015-04-25 23:10:13 UTC

Permalink

That is so mauveâŠ

Leon

Ricardo Signes

2015-04-25 23:51:59 UTC

Permalink

"make minitest" does not load Test::*. I don't see how using weaken blocks
putting Test-Simple v1.3.x into core. Am I missing something?

--
rjbs