Choosing system GL vs. mangled Mesa at runtime

[tom fogal-3]

This is a continuation of the thread archived at:

  http://www.nabble.com/glewContextInit-fails:-glXGetCurrentDisplay-is-NULL-td20906479.html

Quick summary: I need to choose between `system' GL or mangled Mesa
at runtime, sometime after startup but thankfully before glewInit.
Current GLEW sources always load GL functions from the library it was
compiled against (the `system' GL, unless one hacks makefiles), which
is of course too early for my purposes.


I managed to sink some time into this over the past week(end), so I
thought I'd try to summarize my proposed approach and see if I could
get any comments.  Plus I need to back-burner it again soon, so I want
a record of where I'm at when I context switch back ;)

My idea for a solution to this issue involves a couple GLEW changes.
One is a new function in the API, currently

  GLvoid glewMangled(GLboolean m);

which sets internal state in GLEW.  This must be called before
glewInit.  glewGetProcAddress is setup to use a new variable,
__glewXGetProcAddress, instead of always using glXGetProcAddressARB.
There is a new short initialization stage which sets __glewX GPA to
either glXGetProcAddress or mglXGetProcAddress; this keys into the
aforementioned internal state.

There's an issue with symbol resolution here.  To reference
mglXGetProcAddress at compile time I have to declare it extern.  Then
this (of course) causes a link error when applications use GLEW but
don't also link in mangled Mesa.  I solve this by marking mglXGPA as a
weak symbol using gcc attributes.  Still, this approach is problematic
[1].

There's a completely different approach one might think of, which is to
load the correct GetProcAddress symbol by doing a dlopen() + dlsym(),
then we can load the rest of the functions using the now-loaded GPA.  I
haven't tried this yet.  There are some issues here, but it seems more
attractive the more I think about it.

Sample code is online [3].  Comments, ideas on alternative approaches
welcome.  Thanks,

-tom

[1] According to the ELF spec, unresolved weak symbols are set to NULL
    [2].  I use this in a standalone (no GLEW) test program to test if
    Mesa was linked into the executable, e.g.:

      if(use mangled mesa command line option specified) {
        if(mglXGetProcAddress == NULL) {
          fprintf(... error ...); abort();
        }
        my_loader = mglXGetProcAddress;
        XQueryExtension = my_loader("mglXGetQueryExtension");
        ... etc. ...
      }

    The same approach in GLEW causes a segmentation fault when
    referencing mglXGetProcAddress.  The same place in the standard
    [2] also says that global symbols won't be extracted to define
    undefined weak symbols.  So I'm not even sure which behavior is
    correct.  I suspect I'll have to do a good read-through of the `ld'
    documentation; maybe there's a relevant option here.

[2] http://www.skyfree.org/linux/references/ELF_Format.pdf
    See section 1, page 18, 2nd bullet.
[3] View online at:
      http://www.sci.utah.edu/~tfogal/glx/glxnull/
    or clone the small repo:
      http://www.sci.utah.edu/~tfogal/glx/glxnull.git/

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
glew-coders mailing list
glew-coders@...
https://lists.sourceforge.net/lists/listinfo/glew-coders

[Nigel Stewart-2]

Tom,

> Quick summary: I need to choose between `system' GL or mangled Mesa
> at runtime, sometime after startup but thankfully before glewInit.

>   GLvoid glewMangled(GLboolean m);

   How about:
     GLenum glewInitLibrary(const char *library);

   Which is an alternative to glewInit that takes a full-path
   argument of the specific OpenGL shared library to be loaded.

   This has the advantage of being more portable, I think.

- Nigel


-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
glew-coders mailing list
glew-coders@...
https://lists.sourceforge.net/lists/listinfo/glew-coders

[tom fogal-3]

Nigel Stewart <nstewart@...> writes:

> > Quick summary: I need to choose between `system' GL or mangled Mesa
> > at runtime, sometime after startup but thankfully before glewInit.
>
> >   GLvoid glewMangled(GLboolean m);
>
>    How about:
>      GLenum glewInitLibrary(const char *library);
>
>    Which is an alternative to glewInit that takes a full-path
>    argument of the specific OpenGL shared library to be loaded.
>
>    This has the advantage of being more portable, I think.

Okay -- this implies to me that dlopening the lib is fine, which is
great from my vantage point :)

However I think it needs to be a bit more complicated.  When loading
mangled symbols, we need to prepend "m" to all the function names.  If
the only bit of state GLEW is given is the name of a library, there's
not an easy way to figure out if we should prepend that "m".

Thus I propose a slight modification:
  GLenum glewInitLibrary(const char *library, GLboolean mangled);

Sound good?

-tom

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
glew-coders mailing list
glew-coders@...
https://lists.sourceforge.net/lists/listinfo/glew-coders

[Nigel Stewart-2]

>>>   GLvoid glewMangled(GLboolean m);
 >>

>>    How about:
>>      GLenum glewInitLibrary(const char *library);
>>
>>    Which is an alternative to glewInit that takes a full-path
>>    argument of the specific OpenGL shared library to be loaded.
>>
>
> Okay -- this implies to me that dlopening the lib is fine, which is
> great from my vantage point :)
>
> However I think it needs to be a bit more complicated.
>
> Thus I propose a slight modification:
>   GLenum glewInitLibrary(const char *library, GLboolean mangled);

        How about when we dlopen the library we first look for an
        unmangled symbol, and if that fails, then try the corresponding
        Mesa-mangled symbol, and set a flag accordingly.  This way
        seems a bit more open-ended than a Mesa-specific API, I think.

- Nigel

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
glew-coders mailing list
glew-coders@...
https://lists.sourceforge.net/lists/listinfo/glew-coders

[tom fogal-3]

Nigel Stewart <nstewart@...> writes:

> >>>   GLvoid glewMangled(GLboolean m);
>  >>
> >>    How about:
> >>      GLenum glewInitLibrary(const char *library);
> >>
> >>    Which is an alternative to glewInit that takes a full-path
> >>    argument of the specific OpenGL shared library to be loaded.
> >
> > [...] I think it needs to be a bit more complicated.
> >
> > Thus I propose a slight modification:
> >   GLenum glewInitLibrary(const char *library, GLboolean mangled);
>
> How about when we dlopen the library we first look for an
> unmangled symbol, and if that fails, then try the corresponding
> Mesa-mangled symbol, and set a flag accordingly.  This way
> seems a bit more open-ended than a Mesa-specific API, I think.

I thought about this a bit, but I was a little worried about what would
happen if somebody (my app, one of dependent libs, etc.) dlopen'd libGL
in the same process, using RTLD_GLOBAL.  That is, if somebody does:

  dlopen("/usr/lib64/libGL.so", RTLD_NOW | RTLD_GLOBAL);

and then I pass GLEW my mangled lib:

  glewInitLibrary("/path/to/my/libMesaGL.so"); /* mangled */
which in turn does:
  dlopen("/path/to/my/libMesaGL.so", RTLD_NOW | RTLD_LOCAL);

then I believe a dlsym of a GL symbol which does not exist in libMesaGL
can still be resolved, because of the previous global load (untested!).

Further, regardless of GLOBAL / LOCAL flags on the dlopen, if the
mangled Mesa lib was linked against the system GL lib [1], the first
dlsym would succeed even though I, as an application developer, really
wanted to force the mangled symbol to get loaded.

The worst part about loading the wrong symbol is that it's terribly
difficult to debug.  If I try to dlsym "mglXGetProcAddress" and it
gives back NULL, that's pretty obvious, and probably causes a segfault
close to the source.  Whereas if I load the wrong symbol, I probably
don't realize it until calling glXGetCurrentDisplay -- which I'm
probably doing through a function pointer -- and noticing it returns
NULL.  The latter case is likely to be far removed from the dlsym.

I would be more amenable to the approach if a patch adding
debugging support would also be accepted.  For example, if
getenv("GLEW_FORCE_MANGLED") was non-null, follow a separate path that
abort()s if loading `mglXGetProcAddress' fails.

-tom

[1] Admittedly fairly silly, in my opinion, but it's still a case out
    of my control as a developer, and there might even be a valid use
    case for such a setup that I'm not thinking of.

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
glew-coders mailing list
glew-coders@...
https://lists.sourceforge.net/lists/listinfo/glew-coders

[Nigel Stewart-2]

Tom,

> ...then I believe a dlsym of a GL symbol which does not exist in libMesaGL
> can still be resolved, because of the previous global load (untested!).

My reading of the dlsym manpage for Linux suggests that unless RTLD_DEFAULT
is used with dlsym, only the specified library will be searched for
symbols, along with dependencies.  But I don't see why we'd be using
RTLD_DEFAULT, unless we _wanted_ to bind to whatever OpenGL is already
"in play".

http://linux.die.net/man/3/dlsym

> Further, regardless of GLOBAL / LOCAL flags on the dlopen, if the
> mangled Mesa lib was linked against the system GL lib [1], the first
> dlsym would succeed even though I, as an application developer, really
> wanted to force the mangled symbol to get loaded.

So we should check for the mesa symbol first.  We don't really
expect to find that in a vendor-provided GL?

> I would be more amenable to the approach if a patch adding
> debugging support would also be accepted.  For example, if
> getenv("GLEW_FORCE_MANGLED") was non-null, follow a separate path that
> abort()s if loading `mglXGetProcAddress' fails.

How about a bitmask of acceptable naming conventions to bind
to:

#define GLEW_NAME_CONVENTION_NONE 0
#define GLEW_NAME_CONVENTION_GL   1
#define GLEW_NAME_CONVENTION_MESA 2
#define GLEW_NAME_CONVENTION_ALL  ~0

       GLenum glewInitLibrary(const char *library, GLenum nameConvention);

That way, if you really only want one the other possibilities are ignored.

- Nigel



-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
glew-coders mailing list
glew-coders@...
https://lists.sourceforge.net/lists/listinfo/glew-coders

[tom fogal-3]

Nigel Stewart <nstewart@...> writes:
> > ...then I believe a dlsym of a GL symbol which does not exist in libMesaGL
> > can still be resolved, because of the previous global load (untested!).
>
> My reading of the dlsym manpage for Linux suggests that unless RTLD_DEFAULT
> is used with dlsym, only the specified library will be searched for
> symbols, along with dependencies

I think this is a moot issue given the discussion below, but I was
referring to the RTLD_GLOBAL flag for dlopen, as opposed to dlsym:

       RTLD_GLOBAL
              The symbols defined by this library will be made available for symbol
              resolution of subsequently loaded libraries.

> > if mangled Mesa lib was linked against the system GL lib [1],
> > the first dlsym would succeed even though I, as an application
> > developer, really wanted to force the mangled symbol to get loaded.
>
> So we should check for the mesa symbol first.  We don't really
> expect to find that in a vendor-provided GL?

True.  It sounds fine for my current use case.

Still, in general I think we're essentially trying to guess what the
client wants here, and I don't think we have enough information to do
that 100%.  In the above case, maybe they wanted to init GLEW with the
system GL, and they have both linked into their app, so we'd do the
wrong thing.  We could do a little better by checking to see if the
loaded library has a valid glX context, but then client-level bugs
where the glX init is done in the wrong library would be harder to
figure out.

> > I would be more amenable to the approach if a patch adding
> > debugging support would also be accepted.
>
> How about a bitmask of acceptable naming conventions to bind to:
>
> #define GLEW_NAME_CONVENTION_NONE 0
[snip]
>        GLenum glewInitLibrary(const char *library, GLenum nameConvention);

I like this.  Notably, it gives that 100% information I think we're
lacking above.

I'll provide a patch `Real Soon Now' ... I don't have any time this
weekend, but maybe the week/weekend after.

Thanks!  Best,

-tom

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
glew-coders mailing list
glew-coders@...
https://lists.sourceforge.net/lists/listinfo/glew-coders

[Nigel Stewart-2]

Tom,

> I think this is a moot issue given the discussion below

Yes indeed.  But I do worry how anything can work if your
general concern turns out to be correct. :-)

>      GLenum glewInitLibrary(const char *library, GLenum nameConvention);

> I like this.  Notably, it gives that 100% information I think we're
> lacking above.

Actually, I'd like to back away from the bitmask idea and simply
say that a particular naming convention will be tried, either...

#define GLEW_NAME_CONVENTION_GL   0
#define GLEW_NAME_CONVENTION_MESA 1

And there is no issue of finding a library that happens to support
multiple naming conventions.

Another way to formulate this without adding an entry point is to
add variables for library and nameConvention.  I guess that depends
on what we'd prefer to happen with an existing version of GLEW.

> I'll provide a patch `Real Soon Now' ... I don't have any time this
> weekend, but maybe the week/weekend after.

Great!

- Nigel

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
glew-coders mailing list
glew-coders@...
https://lists.sourceforge.net/lists/listinfo/glew-coders