recursive configure and some $ac_highest_top_srcdir?

[Steffen Dettmer-2]

Hi,

when using recursive configure via AC_CONFIG_SUBDIRS, how to know
(in a sub-configure) where the top directory is?

I mean, something like automakes $(top_srcdir) inside configure.

NB: This is not $ac_top_srcdir which (as far as I understood)
    should always point to the directory where the running
    configure is (but not the highest parent). For some reason in
    my test configure.in $ac_top_srcdir even is empty but anyway.

I'm afraid I cannot explain my need well...
So I ask again differently:
Is there any way to pass some "--topsrcdir=../$topsrcdir" to the
subconfigures?  topdir would start with "." if not set and
extended in each level. Of course it would be more complicated
for things like AC_CONFIG_SUBDIRS([two/dir_levels]).

This also would be needed by configure to find config.cache, but
this is solved by "--cache_file=../config.cache", which does not
help much in the "general case" (like custom global files etc).

I understand that such an $ac_highest_top_srcdir would make it
complicated (or even impossible) to run a sub-configure directly.

We could "count double-dot levels" in --srcdir or "detect" the
depth by counting dots in --cache_file and prepend it to
$ac_top_srcdir, but this sucks and I guess it is likely to cause
other trouble.

I have no idea how to address this in a not-to-bad-way.
Where should I look to? Any pointers/proposals/recommendations?



I also try to explain why (I think) we need this.

We usually have a structure like

configuration1/configure
configuration1/package1/configure
configuration1/package2/configure
configuration1/package3/configure

configuration2/configure
configuration2/package1/configure
configuration2/package2/configure
configuration2/package4/configure

(package1 and package2 are reused/shared).

Lets assume package4 depends on package2. Now we have a mechanism
that package4 has some --with-package2 defaulting to
"../package2".

This breaks when needing:
configuration2/configure
configuration2/package1/configure
configuration2/package2/configure
configuration2/package4/configure
configuration2/package5/configure
configuration2/package5/subdir1/configure

with package5 depending on package4 and thus package2. Includes
for example would be something like "-I$(top_srcdir)/../package2/src"
works with automake in package5 but not in package5/subdir1,
because here "-I$(top_srcdir)/../../package2/src" would be
needed.

Additionally, it is impossible to use something like
--with-package2=../../package2, because then it would work in
package5/subdir1 but not in package5 itself (--with options are
same for all configures).

Absolute paths in --with-me fail at least when using WINE.

NB: Using absolute paths works on unix, but causes tears and
    endless trouble with cygwin or MSYS windows drive letters and
    when using WINE. Since we have some DOS-only compilers, we
    need that and cannot use absolute paths.

Another (at the moment less important) aspect is when package2
has linker wrappers and if they are needed in configure itself.

NB: It already is impossible to AC_CHECK_XXX for things from package
    (it would be cool to be able to check if package2 has this
    and that header or some FUNC, but this won't work, because
    -I$(top_srcdir)/../package2/src needs "make" to be evaluated,
    so compiler would try a directory "$(top_srcdir)", which does
    not exist, and thus never finds the header of course).
    Because of the same problem with compiler wrappers
    ("CCLD=$(top_srcdir)/../package2/applinkwrap.sh" etc) we
    cannot check if the wrappers do work.


Sorry for the long big text and thanks for reading it anyway :-)

oki,

Steffen


_______________________________________________
Autoconf mailing list
Autoconf@...
http://lists.gnu.org/mailman/listinfo/autoconf

[Ralf Wildenhues]

* Steffen Dettmer wrote on Wed, Oct 21, 2009 at 11:26:22AM CEST:
> when using recursive configure via AC_CONFIG_SUBDIRS, how to know
> (in a sub-configure) where the top directory is?

Good question.  If your project structure has a variable level of upper
configure scripts then, as you describe, you cannot hard-code this.

However, if we follow the ideas behind configure recursion, you also
should not expect that above the top-most level configure script is one
that comes from your project: one goal of the uniform argument scheme
of configure scripts is that I could take your project, plus several
others, put them side by side in subdirectories, and write an upper
level configure.ac:
  AC_INIT([super-package])
  AC_CONFIG_SUBDIRS([your-package other-package ...])
  AC_OUTPUT

and your package should cope with that.

This explains why $ac_highest_top_srcdir should not come from Autoconf.

For absolute names, you can easily solve this issue in your own package
by adding something like
  if test "${highest_abs_top_srcdir+set}" != set; then
    highest_abs_top_srcdir=`cd "$srcdir" && pwd`
    export highest_abs_top_srcdir
  fi

to each configure.ac that could potentially serve as the highest one in
your package tree.  (Note you still have to propagate the value from
configure to makefiles etc.)

Dealing with not-necessarily-absolute names looks like it needs Autoconf
support.  But even then it seems quite tricky.

It would be easy to have an Autoconf API that lets configure correct a
variables or a command-line argument by the reverse path
($ac_top_builddir_sub) upon configure recursion in _AC_OUTPUT_SUBDIRS.

But would this be elegant, too?  Ideally, it should be possible to
transport a
  --with-foo-package=../foo

through a couple of sub configure scripts that each don't care about the
--with-foo-package switch at all, thus won't have any annotation for it
either.  OTOH, we certainly cannot correct all --with-* switches that
way; while it would have been cool to specify such semantics from the
beginning, it's far too late to do so now.

Maybe you can live with something like this: pass absolute directories
to the --with-* flags.  In the macros that deal with their values, try
to relativize the directories.  There exist a few ways to do that; they
typically have problems with symlinks though, and might need adjustment
for w32-like drive specs.  The one I know is in gnulib-tool, but I think
I saw a perl version somewhere too.

> Is there any way to pass some "--topsrcdir=../$topsrcdir" to the
> subconfigures?  topdir would start with "." if not set and
> extended in each level.

Not currently, but as I said, something like that could easily be added.

> Of course it would be more complicated
> for things like AC_CONFIG_SUBDIRS([two/dir_levels]).

Not really.  If you look at the code generated for a AC_CONFIG_SUBDIRS,
there is a $ac_top_build_prefix computed on the fly which computes the
needed string.  It's just that you couldn't get at it currently.

> This also would be needed by configure to find config.cache, but
> this is solved by "--cache_file=../config.cache", which does not
> help much in the "general case" (like custom global files etc).

Yep.

> I understand that such an $ac_highest_top_srcdir would make it
> complicated (or even impossible) to run a sub-configure directly.

No, I don't see that.

> We could "count double-dot levels" in --srcdir or "detect" the
> depth by counting dots in --cache_file and prepend it to
> $ac_top_srcdir, but this sucks and I guess it is likely to cause
> other trouble.

That won't work if I run this with the toplevel:
  mkdir build
  cd build
  ../configure
  ^^^
(one more level than you'd think).

> Absolute paths in --with-me fail at least when using WINE.

Is that some issue in Wine or in Autoconf, or elsewhere?
The typical thing to write is
  case $file in
  [\\/]* | ?:[\\/]* ) $file is absolute ...;;
  *) $file is relative ...;;
  esac

> NB: It already is impossible to AC_CHECK_XXX for things from package
>     (it would be cool to be able to check if package2 has this
>     and that header or some FUNC, but this won't work, because
>     -I$(top_srcdir)/../package2/src needs "make" to be evaluated,
>     so compiler would try a directory "$(top_srcdir)", which does
>     not exist, and thus never finds the header of course).
>     Because of the same problem with compiler wrappers
>     ("CCLD=$(top_srcdir)/../package2/applinkwrap.sh" etc) we
>     cannot check if the wrappers do work.

But shouldn't absolute names work as arguments to -I, or in CCLD
settings?

Cheers,
Ralf


_______________________________________________
Autoconf mailing list
Autoconf@...
http://lists.gnu.org/mailman/listinfo/autoconf

[Steffen Dettmer-2]

Hi,

thank you for your great detailed reply!

On Wed, Oct 21, 2009 at 10:08 PM, Ralf Wildenhues wrote:

> * Steffen Dettmer wrote on Wed, Oct 21, 2009 at 11:26:22AM CEST:
>> when using recursive configure via AC_CONFIG_SUBDIRS, how to know
>> (in a sub-configure) where the top directory is?
>
> Good question.  If your project structure has a variable level of upper
> configure scripts then, as you describe, you cannot hard-code this.
>
> However, if we follow the ideas behind configure recursion, you also
> should not expect that above the top-most level configure script is one
> that comes from your project: one goal of the uniform argument scheme
> of configure scripts is that I could take your project, plus several
> others, put them side by side in subdirectories, and write an upper
> level configure.ac:
>  AC_INIT([super-package])
>  AC_CONFIG_SUBDIRS([your-package other-package ...])
>  AC_OUTPUT
>
> and your package should cope with that.

Yes, I think this wouldn't be a problem, as long as the packages
needed by "your-package" are available too (and/or specified by
--with-neededpgk).

> This explains why $ac_highest_top_srcdir should not come from Autoconf.

Ahh yes, the packages needed by "your-package" in this case would
not be "super-package/your-package/package1" but
"super-package/package1" and --with-package1=../package1 would
work. But whoever creates super-package would need to know and do
this, right.

> For absolute names, you can easily solve this issue in your own package
> by adding something like
>  if test "${highest_abs_top_srcdir+set}" != set; then
>    highest_abs_top_srcdir=`cd "$srcdir" && pwd`
>    export highest_abs_top_srcdir
>  fi
>
> to each configure.ac that could potentially serve as the highest one in
> your package tree.  (Note you still have to propagate the value from
> configure to makefiles etc.)

Beside that this is absolute (instead of relative), would this
work? I would expect that ./config.status --recheck, which is
called by (auto-)make won't work then, because
highest_abs_top_srcdir would be set differently that in the first
run?

> Dealing with not-necessarily-absolute names looks like it needs
> Autoconf support.  But even then it seems quite tricky.
>
> It would be easy to have an Autoconf API that lets configure correct a
> variables or a command-line argument by the reverse path
> ($ac_top_builddir_sub) upon configure recursion in _AC_OUTPUT_SUBDIRS.

I'm sorry, I don't understand this.
It is easy to correct a command-line argument automatically (i.e.
by prepending a number of "../" prefixes)?

> But would this be elegant, too?  Ideally, it should be possible to
> transport a
>  --with-foo-package=../foo
>
> through a couple of sub configure scripts that each don't care about the
> --with-foo-package switch at all, thus won't have any annotation for it
> either.  OTOH, we certainly cannot correct all --with-* switches that
> way; while it would have been cool to specify such semantics from the
> beginning, it's far too late to do so now.

It would be sufficient to have one single option transformed in
this way. Our stuff around --with-package uses the parameter
../foo as search directory. This code instead could do to use
$transformed_directory/../foo instead.

So with --with-parent (defaulting to ".") beeing transformed to
--with-parent=.. and --with-parent=../.. or alike would be
sufficient.

Is there a way for a toplevel configure to pass a different
option automatically to a sub-configure? Internally it is,
because by some way --cache-file works, could we create something
similar?

This would solve our problem: we would initiate in our highest
level configure (regardless if called by an even higher one or
not, it would work) and all subconfigures would get this option
(and ignore it if not "our" package) but evaluate it in our code
around --with-package.

It would be great if you have any pointers where I can learn
more, please!

> Maybe you can live with something like this: pass absolute directories
> to the --with-* flags.  In the macros that deal with their values, try
> to relativize the directories.  There exist a few ways to do that; they
> typically have problems with symlinks though, and might need adjustment
> for w32-like drive specs.  The one I know is in gnulib-tool, but I think
> I saw a perl version somewhere too.

We had several attempts to map paths from relative to absolute
and vice versa and to "canonify" (cut dir/../dir to dir and
remove "././." etc). By the time, we had to replace all of them
by relative paths because we were unable to get it working for
all cases where we need it. On linux, of course everything works
all the time, but cygwin, MSYS and WINE are troublesome. Usually
it worked only for two of the three...

>> Is there any way to pass some "--topsrcdir=../$topsrcdir" to the
>> subconfigures?  topdir would start with "." if not set and
>> extended in each level.
>
> Not currently, but as I said, something like that could easily be added.

Do you have some key words I could search for to learn more
about this?

>> Of course it would be more complicated
>> for things like AC_CONFIG_SUBDIRS([two/dir_levels]).
>
> Not really.  If you look at the code generated for a AC_CONFIG_SUBDIRS,
> there is a $ac_top_build_prefix computed on the fly which computes the
> needed string.  It's just that you couldn't get at it currently.

Where do I find this code?

steffen@desupr-dev:/local/tmp/steffen/systest_exp # grep
ac_top_build_prefix ./configure one/configure two/configure
reveals nothing.

>> I understand that such an $ac_highest_top_srcdir would make it
>> complicated (or even impossible) to run a sub-configure directly.
>
> No, I don't see that.

ok, if there is some --with-parent option, user would just need
to set it correctly, otherwise it would default to . but in this
cause it could be needed to have .. or `../..'. But
--with-parent=.. would solve this. Of course ./config.status
--recheck would need to remember the --with-parent passed by the
top level configure (I know it will remember if passed by the
user, because AFAIK it simply remembers all user-passed
parameters, right?).

>> We could "count double-dot levels" in --srcdir or "detect" the
>> depth by counting dots in --cache_file and prepend it to
>> $ac_top_srcdir, but this sucks and I guess it is likely to cause
>> other trouble.
>
> That won't work if I run this with the toplevel:
>  mkdir build
>  cd build
>  ../configure
>  ^^^
> (one more level than you'd think).

why? --cache-file would be build/config.cache and in a sub
configure, lets say packge1, would configure in build/package1
and have --cache-file=../config.cache - and from package1 srcdir
.. is the top sourcedir, so it would be correct I think?
(But anyway, this seems to be a bad option).

>> Absolute paths in --with-me fail at least when using WINE.
>
> Is that some issue in Wine or in Autoconf, or elsewhere?
> The typical thing to write is
>  case $file in
>  [\\/]* | ?:[\\/]* ) $file is absolute ...;;
>  *) $file is relative ...;;
>  esac

All of them, including elsewhere :)
(in autoconf, AFAIK, no real issue just complexities with
quoting).

Some notes from our bug tracker:

> > WINE must not have a drive letter mapped to a directory above
> > the current workdirectory (otherwise, it uses this for the
> > compiler process as cwd, and then the absolute path is "too
> > long" and does not work).
> > When this happens also the dependencies are wrong because
> > including driver letters breaking Makefile where : is special
> > cygwin needs IC_CANONICAL_PATH because the CL.EXE compiler
> > does not understand /cygdrive/c, resulting in C: which in
> > turn is not understood by Makefile so no depencies work.
> > with tcc it works under cygwin when both relative and
> > absolute pathes are used, because when first the relatives
> > are in command line, they are used for deps which accidently
> > works.
> > with cygwin-gcc under cygwin it works.  An idea was to use
> > relatives both with and without $(top_srcdir) but this fails
> > for armlink -libpath seem not to work.
> > update: can we fix the drive-letter-in-deps by extending depcom_arm
> > ohh, now there is something about that limits.h cannot be
> > found by tcc, check if you have a WINE driver letter shortcut
> > mapping such as ~/.wine/dosdevices/x: -> /tmp which fails
> > when you are working in /tmp!

For WINE a problem is that configure runs under linux and sees
lets say /tmp and finds files there and stuff, but compiler sees
some X:\TMP only.

Another problem is when user has mappings
/ -> Z:
/tmp -> X:
and then does some
$ cd /tmp/myworkdir/build
$ ../configure
than absolute paths would be Z:/tmp/myworkdir but WINE will not
set pwd to Z:/tmp/myworkdir but automagically the shorter
X:/myworkdir.

Compiler requires to get a windows-path but make (dependencies,
include ../mymak.mak etc) does not understand things with driver
letters.

On cygwin, this works a bit more when mapping via cygpath
from /cygdrive/c to c: and vice versa, but at least it is
complicated (for me). Also great is handling c:\Program Files\
due to its lovely blank space :(

To run java tools we ended up with something like
steffen@desupr-dev:/local/tmp/steffen/.../src # make echo-java
"/usr/lib/java/bin/java" "-DILOG_CONSOLE_COLOR=$ILOG_CONSOLE_COLOR"
-cp "`cygpath -w -m -p
../lib/junit.jar:../lib/rxtxcomm.jar:../classes`"
"-Djava.library.path=../lib"

I tried a similar thing for compiler wrappers, but I though its
complexity exeeds everything :)

>> NB: It already is impossible to AC_CHECK_XXX for things from package
>>     (it would be cool to be able to check if package2 has this
>>     and that header or some FUNC, but this won't work, because
>>     -I$(top_srcdir)/../package2/src needs "make" to be evaluated,
>>     so compiler would try a directory "$(top_srcdir)", which does
>>     not exist, and thus never finds the header of course).
>>     Because of the same problem with compiler wrappers
>>     ("CCLD=$(top_srcdir)/../package2/applinkwrap.sh" etc) we
>>     cannot check if the wrappers do work.
>
> But shouldn't absolute names work as arguments to -I, or in CCLD
> settings?

On linux (without WINE), it works well (as long the build
directory is not moved) yes. In WINE, the absolute path may not
exist or work - with my buggy scripts at least. Probably we do
several things wrong or bad...
I think in theory of course there should be some way to map
everything everywhere with stacks of wrappers, but I fear its
complexity would not be maintainable.

Well, thanks for your reply and help so far!

Have a great weekend,

Steffen


_______________________________________________
Autoconf mailing list
Autoconf@...
http://lists.gnu.org/mailman/listinfo/autoconf