Namespaces and inter-library private symbols

[Greg Hudson]

I'd like to get a consensus on how we should handle library functions
which are not part of the public API but are used by other parts of
the tree.  Currently, we inconsistently do one of three things:

1. Use the accessor.  (Examples: ASN.1 encoders as used by the pkinit
module; HMAC as used by gss-krb5.)

2. Name the function krb5_foo, export it, but prototype it in k5-int.h
instead of krb5.h.  (Examples: krb5_get_cred_from_kdc as used by ksu;
krb5_free_realm_tree as used by kdc.)

3. Name the function krb5int_foo, export it, prototype it in k5-int.h,
and use it directly from the other part of the tree.  (Examples:
krb5int_cc_default as used by gss-krb5; krb5int_init_context_kdc as
used by kdc and libkadm5.)

I would like to better understand why we have the accessor and when it
should be used, if ever.  I consider it an overly heavyweight solution
to the problem.  Tom tells me that the initial justification had
something to do with Windows, but he didn't have details.

I think we have a partial consensus that (2) is bad, because those
symbols can show up as available in an autoconf test even if they
aren't part of the API in that version of Kerberos.  We've continued
to add new examples of (2) in recent work, however--examples are
krb5_c_weak_enctype (recently renamed to krb5int_c_weak enctype) and
krb5_get_credentials_for_proxy.

I would like to see us move towards using (3) uniformly.  If I receive
no feedback, I will document this as the preferred practice in
http://k5wiki.kerberos.org/wiki/Coding_style/Practices#Namespaces and
start converting cases to that form as I run into them.

A fourth option is to invent a new namespace for inter-library private
symbols, so that we can easily differentiate between "internal to this
library" and "internal to the krb5 tree."  That might result in a lot
of symbol renaming over time, though.
_______________________________________________
krbdev mailing list             krbdev@...
https://mailman.mit.edu/mailman/listinfo/krbdev

[Nicolas Williams]

On Tue, Nov 10, 2009 at 02:08:18PM -0500, ghudson@... wrote:
> I'd like to get a consensus on how we should handle library functions
> which are not part of the public API but are used by other parts of
> the tree.  Currently, we inconsistently do one of three things:
>
> 1. Use the accessor.  (Examples: ASN.1 encoders as used by the pkinit
> module; HMAC as used by gss-krb5.)

I don't mind this, but it'd be nice to have a set of macros to hide the
indirection.

> 2. Name the function krb5_foo, export it, but prototype it in k5-int.h
> instead of krb5.h.  (Examples: krb5_get_cred_from_kdc as used by ksu;
> krb5_free_realm_tree as used by kdc.)

I don't mind this either.

> 3. Name the function krb5int_foo, export it, prototype it in k5-int.h,
> and use it directly from the other part of the tree.  (Examples:
> krb5int_cc_default as used by gss-krb5; krb5int_init_context_kdc as
> used by kdc and libkadm5.)

Nor this.  But I prefer (2), slightly.

> I would like to better understand why we have the accessor and when it
> should be used, if ever.  I consider it an overly heavyweight solution
> to the problem.  Tom tells me that the initial justification had
> something to do with Windows, but he didn't have details.

*shrug*.  It's not heavyweight, but, rather, a way of building your own
light-weight run-time linker.  It's... OK, not great, but there's
precedent (e.g., PKCS#11 is meant to be used in a similar way).

What you call the "accessor" approach may be particularly useful when
you might risk interposition at run-time and you want to ensure that you
don't (you dlopen() the exact object you want, dlsym() each function
from it, or a single function that outputs a pointer to a struct
containing the necessary function pointers).

> I think we have a partial consensus that (2) is bad, because those
> symbols can show up as available in an autoconf test even if they
> aren't part of the API in that version of Kerberos.  We've continued
> to add new examples of (2) in recent work, however--examples are
> krb5_c_weak_enctype (recently renamed to krb5int_c_weak enctype) and
> krb5_get_credentials_for_proxy.

I don't think (2) is bad.  Here's why: you must export symbols as far as
the _linker_ goes, no matter which of (1), (2), or (3) you pick, and in
_all_ cases you depend on a private header to obtain access to the
necessary C function prototype declarations -- (2) fits the bill with
minimal impact on the existing source.

The prefix that indicates "internal" is somewhat useful -- you know
instantly by looking at a function if it's internal -- but, what is
"internal"?  Internal to a single library?  To all of MIT krb5?  To
third-party code that has an "interface contract" with MIT krb5?  How
many prefixes do you actually need?

> I would like to see us move towards using (3) uniformly.  If I receive
> no feedback, I will document this as the preferred practice in
> http://k5wiki.kerberos.org/wiki/Coding_style/Practices#Namespaces and
> start converting cases to that form as I run into them.

Certainly having a single style adopted uniformly would be good.

> A fourth option is to invent a new namespace for inter-library private
> symbols, so that we can easily differentiate between "internal to this
> library" and "internal to the krb5 tree."  That might result in a lot
> of symbol renaming over time, though.

You can't invent a new namespace, only a new naming convention.  I
recommend against it.  I recommend that you concentrate efforts on
actual, useful features and bug fixes over mass symbol rename
excercises.

Nico
--
_______________________________________________
krbdev mailing list             krbdev@...
https://mailman.mit.edu/mailman/listinfo/krbdev

[Tom Yu]

"ghudson@..." <ghudson@...> writes:

> I'd like to get a consensus on how we should handle library functions
> which are not part of the public API but are used by other parts of
> the tree.  Currently, we inconsistently do one of three things:
>
> 1. Use the accessor.  (Examples: ASN.1 encoders as used by the pkinit
> module; HMAC as used by gss-krb5.)
>
> 2. Name the function krb5_foo, export it, but prototype it in k5-int.h
> instead of krb5.h.  (Examples: krb5_get_cred_from_kdc as used by ksu;
> krb5_free_realm_tree as used by kdc.)

Some of these may have been prototyped in krb5.h at some point in the
past, and then moved to k5-int.h.  If that is true, we should avoid
renaming them, in order to preserve ABI compatibility.
_______________________________________________
krbdev mailing list             krbdev@...
https://mailman.mit.edu/mailman/listinfo/krbdev

[Nicolas Williams]

On Tue, Nov 10, 2009 at 02:08:18PM -0500, ghudson@... wrote:
> I would like to better understand why we have the accessor and when it
> should be used, if ever.  I consider it an overly heavyweight solution
> to the problem.  Tom tells me that the initial justification had
> something to do with Windows, but he didn't have details.

Ah, I bet the accessor is for pre-auth plugins / plugins in general, to
help avoid DLL hell.  I think this is best kept separate.

So we're really just talking about naming conventions for non-public
interfaces.  My preference is to not have a separate naming convention
for pubic vs. private symbols.

Vendors should just not ship k5-int.h and that's that.  Developers
should have enough warning from the fact that they cannot find C
prototype declarations for functions that they think might be useful to
them but which ware internal.

Nico
--
_______________________________________________
krbdev mailing list             krbdev@...
https://mailman.mit.edu/mailman/listinfo/krbdev

[Jeffrey Altman-2]

ghudson@... wrote:

> I would like to better understand why we have the accessors and when it
> should be used, if ever.  I consider it an overly heavyweight solution
> to the problem.  Tom tells me that the initial justification had
> something to do with Windows, but he didn't have details.

I don't remember the introduction of the accessors as having anything to
do with Windows in particular.  The problem was the pollution of the
export name space from shared libraries and the desire to be able to
change the names and parameters of truly internal private functions that
must be used by gssapi without being forced to change the signature of
the of the library.

Symbols that are exported by libraries are inevitably used even when
they are declared private.  The only way to ensure that third parties do
not place dependencies on a function is to not export it.

Jeffrey Altman


_______________________________________________
krbdev mailing list             krbdev@...
https://mailman.mit.edu/mailman/listinfo/krbdev

[Sam Hartman]

>>>>> "Tom" == Tom Yu <tlyu@...> writes:

    Tom> Some of these may have been prototyped in krb5.h at some
    Tom> point in the past, and then moved to k5-int.h.  If that is
    Tom> true, we should avoid renaming them, in order to preserve ABI
    Tom> compatibility.


I strongly disagree.
We defined a stable public ABI with 1.2.5.
Any symbol not part of that ABI should be renamed if it is changed and probably should be renamed anyway.

See my feedback from Samba; they would prefer that if things are removed from krb5.h they be renamed.
(Note all these symbols are presumably under KRB5_PRIVATE)
_______________________________________________
krbdev mailing list             krbdev@...
https://mailman.mit.edu/mailman/listinfo/krbdev

[Greg Hudson]

On Tue, 2009-11-10 at 16:48 -0500, Nicolas Williams wrote:
> Vendors should just not ship k5-int.h and that's that.  Developers
> should have enough warning from the fact that they cannot find C
> prototype declarations for functions that they think might be useful to
> them but which ware internal.

Jeff Altman wrote:
> Symbols that are exported by libraries are inevitably used even when
> they are declared private.  The only way to ensure that third parties
> do not place dependencies on a function is to not export it.

So, those are the two extremes.  I think if we name a function
krb5int_foo, that's a pretty clear indication that it's not part of the
stable API or ABI.  Adding an extra layer of hassle (such as the
accessor) isn't going to stop a truly determined application writer from
using internal functionality.

Merely failing to prototype a function in krb5.h is not, in my opinion,
a clear indication.  It also runs afoul of autoconf tests for functions
entering the public API in later versions of krb5.  For instance, Luke
chose not to make krb5_get_credentials_for_proxy part of the public API
for 1.8 so that we can retain the freedom to change it.  But suppose we
change the function signature and then make it part of the public API in
1.9.  How can an application tell whether krb5_get_credentials_for_proxy
is available in the API of the libkrb5 it is building against?  The
usual AC_CHECK_FUNC autoconf test will find the symbol in krb5 1.8, even
though it's not part of that version's API.

I am awaiting clarification on an offlist comment about the accessor,
and I'll probably bring this up again with the MIT staff, but my
expectation is that we will standardize on using exported symbols with
the krb5int_ prefix for tree-internal inter-library functions.

Nico wrote:
> I recommend that you concentrate efforts on actual, useful features
> and bug fixes over mass symbol rename excercises.

For the moment, I'm just interested in standardizing a practice for
future work, although it's possible we will rename some newer non-public
krb5_ symbols to krb5int_ to avoid the autoconf problem described above.


_______________________________________________
krbdev mailing list             krbdev@...
https://mailman.mit.edu/mailman/listinfo/krbdev

[Nicolas Williams]

On Fri, Nov 13, 2009 at 01:13:59AM -0500, Greg Hudson wrote:

> On Tue, 2009-11-10 at 16:48 -0500, Nicolas Williams wrote:
> > Vendors should just not ship k5-int.h and that's that.  Developers
> > should have enough warning from the fact that they cannot find C
> > prototype declarations for functions that they think might be useful to
> > them but which ware internal.
>
> Jeff Altman wrote:
> > Symbols that are exported by libraries are inevitably used even when
> > they are declared private.  The only way to ensure that third parties
> > do not place dependencies on a function is to not export it.

If you don't declare them in public/installed/shipped headers, then they
can only be used by folks who are willing to dig into the source to find
declarations.  If you don't export them then they can't be used period.

I definitely recommend not exporting symbols that need not be used
outside libkrb5 (and plugins, via the accessor).

Are there symbols which must be exported but aren't public?  (What we
might call "consolidation private" or "contracted private" in the Sun
world of development).  If not, then I strongly recommend Jeff's
proposal.  If there are, can you remove or promote them all so that you
only have libkrb5-private and public interfaces only?

> So, those are the two extremes.  I think if we name a function

I fully agree with Jeff though, but only for symbols that are not
"contracted" to other parties (see above).

> krb5int_foo, that's a pretty clear indication that it's not part of the
> stable API or ABI.  Adding an extra layer of hassle (such as the
> accessor) isn't going to stop a truly determined application writer from
> using internal functionality.

To be clear, I don't _mind_ the 'int' suffix to the krb5 prefix.  I just
don't think it's necessary: either way you're relying on convention
(header naming vs. symbol naming) or on not making relevant declarations
available (by not installing/shipping k5-int.h) to get third parties to
stay away from private symbols.

> Merely failing to prototype a function in krb5.h is not, in my opinion,
> a clear indication.  It also runs afoul of autoconf tests for functions
> entering the public API in later versions of krb5.  For instance, Luke

It depends on the autoconf test.  If it's looking only for a symbol in a
shared object's exported symbol table, then yes, autoconf will find it,
but any source code actually trying to use it will fail to compile.

> chose not to make krb5_get_credentials_for_proxy part of the public API
> for 1.8 so that we can retain the freedom to change it.  But suppose we
> change the function signature and then make it part of the public API in
> 1.9.  How can an application tell whether krb5_get_credentials_for_proxy
> is available in the API of the libkrb5 it is building against?  The

It can't have used krb5_get_credentials_for_proxy when built against
1.8.

> usual AC_CHECK_FUNC autoconf test will find the symbol in krb5 1.8, even
> though it's not part of that version's API.
>
> I am awaiting clarification on an offlist comment about the accessor,
> and I'll probably bring this up again with the MIT staff, but my
> expectation is that we will standardize on using exported symbols with
> the krb5int_ prefix for tree-internal inter-library functions.

The accessor is important for plug-in access (to public and private
interfaces both) as a way to avoid DLL hell.  I'm OK with the accessor
being used to access private functions, provided there's convenience
macros for using them (but then too you'll still be relying on
convention to keep third parties from using private interfaces).

> Nico wrote:
> > I recommend that you concentrate efforts on actual, useful features
> > and bug fixes over mass symbol rename excercises.
>
> For the moment, I'm just interested in standardizing a practice for
> future work, although it's possible we will rename some newer non-public
> krb5_ symbols to krb5int_ to avoid the autoconf problem described above.

The autoconf problem is a non-problem.  The key, above all, is to not
install private headers while defining private symbols in private
headers.

C doesn't have a neat way to deal with private vs. public interfaces
other than by declaration segregation through headers.  That's the
least, and also the most that can be done.  The rest is lexical
convention -- how desirable that is will vary with your p.o.v., but it's
certainly not the main thing to do, nor does guarantee anything, it's
just convention.

Nico
--
_______________________________________________
krbdev mailing list             krbdev@...
https://mailman.mit.edu/mailman/listinfo/krbdev

[Greg Hudson]

On Fri, 2009-11-13 at 11:43 -0500, Nicolas Williams wrote:
> Are there symbols which must be exported but aren't public?

Uh, yes.  The subject of this thread is "inter-library private symbols."
The opening message of this thread lists numerous examples of non-public
libkrb5 symbols used by other parts of the tree.  We have dozens and
dozens of cases where we currently do this.

Symbols used only within a library are easy to handle, and are not at
issue in this thread.

> It depends on the autoconf test.  If it's looking only for a symbol in a
> shared object's exported symbol table, then yes, autoconf will find it,
> but any source code actually trying to use it will fail to compile.

Source code typically does not fail to compile when using a
non-prototyped function; at most, you generally get a warning, and often
only with certain compiler flags.

The normal method of checking libraries for symbols in autoconf is to
attempt to link a program using that function (un-prototyped) and see if
you succeed, which is equivalent to looking for the symbol in the shared
object's exported symbol table.  There is no easy way to use autoconf
tests which check for a prototype, because there is no portable compiler
behavior for detecting the use of a non-prototyped function.

> The accessor is important for plug-in access (to public and private
> interfaces both) as a way to avoid DLL hell.

The accessor is only used for private interfaces, and use of the
accessor begins by invoking a private libkrb5 function
(krb5int_accessor), so I believe this explanation is incorrect.


_______________________________________________
krbdev mailing list             krbdev@...
https://mailman.mit.edu/mailman/listinfo/krbdev

[Nicolas Williams]

On Fri, Nov 13, 2009 at 12:17:46PM -0500, Greg Hudson wrote:
> On Fri, 2009-11-13 at 11:43 -0500, Nicolas Williams wrote:
> > Are there symbols which must be exported but aren't public?
>
> Uh, yes.  The subject of this thread is "inter-library private symbols."

In my defense, I hadn't finished my morning coffee :(

> The opening message of this thread lists numerous examples of non-public
> libkrb5 symbols used by other parts of the tree.  We have dozens and
> dozens of cases where we currently do this.

Can you eliminate them?

> > It depends on the autoconf test.  If it's looking only for a symbol in a
> > shared object's exported symbol table, then yes, autoconf will find it,
> > but any source code actually trying to use it will fail to compile.
>
> Source code typically does not fail to compile when using a
> non-prototyped function; at most, you generally get a warning, and often
> only with certain compiler flags.

Sorry, I think I'm assuming C99 too much.  But you could require C99...

> > The accessor is important for plug-in access (to public and private
> > interfaces both) as a way to avoid DLL hell.
>
> The accessor is only used for private interfaces, and use of the
> accessor begins by invoking a private libkrb5 function
> (krb5int_accessor), so I believe this explanation is incorrect.

Why would the accessor have come to happen if there was already a
convention for private symbols?  I glanced last week and IIRC I saw only
accessor uses by plug-ins.  DLL hell avoidance immediately came to mind.
I think it's the most natural explanation.  You might have to ask
whoever added the accessor though.

Nico
--
_______________________________________________
krbdev mailing list             krbdev@...
https://mailman.mit.edu/mailman/listinfo/krbdev

[Ken Raeburn]

On Nov 13, 2009, at 11:43, Nicolas Williams wrote:
> If you don't declare them in public/installed/shipped headers, then  
> they
> can only be used by folks who are willing to dig into the source to  
> find
> declarations.  If you don't export them then they can't be used  
> period.

There are still C compilers that will silently assume an int-returning  
function of unspecified arguments, if it hasn't seen a declaration.  
If after looking at some of our sources to see how something was done,  
someone tries adding a call to their code and it compiles, they may  
not think about it any further.

On the other hand, an interface that requires the use of a structure  
that's not in the installed headers, that's a somewhat bigger stick to  
beat them over the head with.  If they are using our sources, they can  
grab the internal headers, but they know (better) that they're doing it.

> Are there symbols which must be exported but aren't public?  (What we
> might call "consolidation private" or "contracted private" in the Sun
> world of development).  If not, then I strongly recommend Jeff's
> proposal.  If there are, can you remove or promote them all so that  
> you
> only have libkrb5-private and public interfaces only?

I think there are things we're doing that we want to use in multiple  
places internally, but don't want to be stuck maintaining indefinitely  
for random programs that ought not to be messing around with such  
things.  For example, the "send a message to service X in realm Y and  
wait for a response" code, or the wrappers around mutexes, or our  
getaddrinfo wrapper.  Or anything that lets us peek at internal state  
for debugging, printing values for test vectors, etc.

Ken
_______________________________________________
krbdev mailing list             krbdev@...
https://mailman.mit.edu/mailman/listinfo/krbdev

[Nicolas Williams]

On Wed, Nov 11, 2009 at 10:11:55PM -0500, Jeffrey Altman wrote:
> I don't remember the introduction of the accessors as having anything to
> do with Windows in particular.  The problem was the pollution of the
> export name space from shared libraries and the desire to be able to
> change the names and parameters of truly internal private functions that
> must be used by gssapi without being forced to change the signature of
> the of the library.

If a run-time linker encodes prototype information into its shared
object outputs, then I agree that exporting private symbols becomes a
problem, and the accessor is probably the best solution.

Nico
--
_______________________________________________
krbdev mailing list             krbdev@...
https://mailman.mit.edu/mailman/listinfo/krbdev

[Greg Hudson]

On Fri, 2009-11-13 at 13:25 -0500, Nicolas Williams wrote:
> If a run-time linker encodes prototype information into its shared
> object outputs, then I agree that exporting private symbols becomes a
> problem, and the accessor is probably the best solution.

No, I think encoding prototype information into shared object outputs is
highly unusual.  What's possibly confusing you is that C compilers
typically allow an application to invoke a function with no prototype
information, and will simply assume that the function returns int and
needs no argument conversions.  (This is true of gcc even with -std=c99,
although that flag does cause gcc to issue a warning.)


_______________________________________________
krbdev mailing list             krbdev@...
https://mailman.mit.edu/mailman/listinfo/krbdev