[c++] Simplifying test environments and running tests on installations

[Alan Conway-2]

For tests that run executables and scripts (tests that are bash scripts, python
scripts) we currently are setting a lot of ad-hoc environment variables so those
tests can locate exes and libraries.

I propose to rationalize this as follows:

1. Rewrite the tests with NO paths on executables or libraries in --load-module
options.

2. Write a testenv script to set up PATH and LD_LIBRARY_PATH with paths where
exes, scripts and libraries are found in a build at the front. Source these
scripts before running the tests.

The benefits are:
- simpler scripts, fewer env vars to think about
- can write additional env scripts to run tests in other settings e.g..
  - run tests against installed qpidd
  - run tests against store module and qpid built in 2 separate build trees.

The only downside I can think of is that if an exe fails to build and qpid is
installed you might run with installed exe or lib by mistake. However this
presupposes you didn't notice that your build failed so the risk seems minor
compared to the benefits.

Are there any other problems I've missed? Will the same approach work on windows
with .bat files instead of sh scripts?

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@...

[Steve Huston-2]

Hi Alan,

> For tests that run executables and scripts (tests that are
> bash scripts, python
> scripts) we currently are setting a lot of ad-hoc environment
> variables so those tests can locate exes and libraries.

Yes, I've been struggling with translating that for Cmake on both
Linux and Windows.

> I propose to rationalize this as follows:
>
> 1. Rewrite the tests with NO paths on executables or
> libraries in --load-module options.
>
> 2. Write a testenv script to set up PATH and LD_LIBRARY_PATH
> with paths where
> exes, scripts and libraries are found in a build at the
> front. Source these scripts before running the tests.

Ok.

> The benefits are:
> - simpler scripts, fewer env vars to think about
> - can write additional env scripts to run tests in other
> settings e.g..
>   - run tests against installed qpidd
>   - run tests against store module and qpid built in 2
> separate build trees.

Cool.

> The only downside I can think of is that if an exe fails to
> build and qpid is
> installed you might run with installed exe or lib by mistake.
> However this
> presupposes you didn't notice that your build failed so the
> risk seems minor compared to the benefits.

Yes, I agree.

> Are there any other problems I've missed?

Possibly with different locations of the resultant libs, etc. on
different platforms, but this should be workable.

It may be an idea to make install (and the equiv on Windows) before
running the tests. Then things could be in a predictable place and
more closely mimic what will be in a user site.

> Will the same
> approach work on windows with .bat files instead of sh scripts?

Windows PowerShell scripts, yes.

-Steve


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@...

[Alan Conway-2]

On 10/30/2009 11:03 AM, Steve Huston wrote:

> Hi Alan,
>
>> For tests that run executables and scripts (tests that are
>> bash scripts, python
>> scripts) we currently are setting a lot of ad-hoc environment
>> variables so those tests can locate exes and libraries.
>
> Yes, I've been struggling with translating that for Cmake on both
> Linux and Windows.
>
>> I propose to rationalize this as follows:
>>
>> 1. Rewrite the tests with NO paths on executables or
>> libraries in --load-module options.
>>
>> 2. Write a testenv script to set up PATH and LD_LIBRARY_PATH
>> with paths where
>> exes, scripts and libraries are found in a build at the
>> front. Source these scripts before running the tests.
>
> Ok.
>
>> The benefits are:
>> - simpler scripts, fewer env vars to think about
>> - can write additional env scripts to run tests in other
>> settings e.g..
>>    - run tests against installed qpidd
>>    - run tests against store module and qpid built in 2
>> separate build trees.
>
> Cool.
>
>> The only downside I can think of is that if an exe fails to
>> build and qpid is
>> installed you might run with installed exe or lib by mistake.
>> However this
>> presupposes you didn't notice that your build failed so the
>> risk seems minor compared to the benefits.
>
> Yes, I agree.
>
>> Are there any other problems I've missed?
>
> Possibly with different locations of the resultant libs, etc. on
> different platforms, but this should be workable.

Yes, I was thinking there would be a separate env script per platform. That
doesn't affect the test scripts since the env script would be run by make,
outside the test scripts.

>
> It may be an idea to make install (and the equiv on Windows) before
> running the tests. Then things could be in a predictable place and
> more closely mimic what will be in a user site.
>

Interesting idea - do an install in a subdivide of tests as part of make check.
  I'm not sure if automake is up to the job, as the install prefix is set as
part of configure. Maybe it can be over-ridden, I'll poke around. Does cmake
support that type of install as part of the test phase?

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@...

[Ján Sáreník]

Hello!

On Fri, Oct 30, 2009 at 11:03:46AM -0400, Steve Huston wrote:
> It may be an idea to make install (and the equiv on Windows) before
> running the tests. Then things could be in a predictable place and
> more closely mimic what will be in a user site.

This can possibly lead into hardwired code in the future
which would not be even noticed...

IMHO, make check running in not so predictable place
maintains vigilance.

  Jasan

--
Red Hat Czech, MRG Quality Assurance Associate

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@...