Re: [DISCUSS] BOM and supported platforms for Bigtop 0.4.0

[owen.omalley]

On Mon, Apr 30, 2012 at 9:39 PM, Roman Shaposhnik <rvs@...> wrote:
> On Mon, Apr 30, 2012 at 9:13 PM, Eric Baldeschwieler
> <eric14@...> wrote:
>> So you are suggesting expanding the charter to include projects not hosted at Apache?
>
> I don't think this is what Bruno suggested. Personally I can't find
> any reference in Bigtop
> charter that restricts it to Projects that belong to Apache Software
> Foundation.

As a mentor of the Bigtop project, I don't see it as acceptable for an
Apache project to distribute binaries of non-Apache software. If the
owners of the Hue project decide to donate it to Apache and it had
been released by Apache, then it would be acceptable. I'm strictly -1
on releasing any version of Bigtop with Hue or any other non-Apache
software as part of the release.

-- Owen

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Roman Shaposhnik-2]

On Thu, May 3, 2012 at 8:06 AM, Owen O'Malley <omalley@...> wrote:

> On Mon, Apr 30, 2012 at 9:39 PM, Roman Shaposhnik <rvs@...> wrote:
>> On Mon, Apr 30, 2012 at 9:13 PM, Eric Baldeschwieler
>> <eric14@...> wrote:
>>> So you are suggesting expanding the charter to include projects not hosted at Apache?
>>
>> I don't think this is what Bruno suggested. Personally I can't find
>> any reference in Bigtop
>> charter that restricts it to Projects that belong to Apache Software
>> Foundation.
>
> As a mentor of the Bigtop project, I don't see it as acceptable for an
> Apache project to distribute binaries of non-Apache software.

In that case, I suggest you start with filing upstream JIRAs for pretty much
all of the Hadoop ecosystem projects kindly asking them to remove
dependencies on non-ASF (but APL!) projects like Guava libraries. Until
that happens there's very little Bigtop can do.

> If the owners of the Hue project decide to donate it to Apache and it had
> been released by Apache, then it would be acceptable. I'm strictly -1
> on releasing any version of Bigtop with Hue or any other non-Apache
> software as part of the release.

I would like to point out, that your -1 on Hue is highly inconsistent for the
reason I mentioned above. I'm not sure what rules ASF has for gratuitous
-1 votes devoid of clear technical reasoning, but I'm sure as a project
mentor you can help us find out what that policy is.

Thanks,
Roman.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Alan Gates-2]

Roman,

I see your point that many Apache projects include non-Apache code in their binary distributions.  But there is a distinction here.  In the case of Hadoop and other projects, they bring things such Guava along because they need them, not for the express purpose of distributing those artifacts.  Bigtop, by its nature, is different because it provides artifacts for users to download regardless of what other components they need.  It is the difference between "we include this because we need it" and "we include this because you might want it".  

My concern is that this is a slippery slope.  There are lots of other things people use with Hadoop (Ganglia for monitoring, Postgres for their Hive metastore, Cascading, etc.).  Would we want Bigtop distributing those?  This would consume a lot of Apache resources to host these things on the download servers.  

Additionally, we need to think about maintaining Apache's brand.  When we redistribute Apache binaries, we know those have gone through an established release process.  With non-Apache binaries, even those that are APL, we know nothing of their releases processes, code quality, etc.  I do not mean this as a slight to Hue nor any of the projects mentioned above.  But if we let one in we will have to let others in.  Again this is important because we would be opening ourselves up as a distribution point for those projects independent of their usage in other Apache projects.

By drawing the line at distributing only Apache projects we protect Apache both in terms of server resource usage and in branding.

As Bruno pointed out in the thread on bigtop-dev (http://mail-archives.apache.org/mod_mbox/incubator-bigtop-dev/201205.mbox/%3C4F9F4F6C.4000404%40apache.org%3E ) this does limit Bigtop, so I understand the motivation to do it.  But before we take this step it merits discussion in the community.

Finally, a comment on the role of mentors.  You were concerned that Owen was vetoing this for non-technical reasons.  Your mentors are not here to guide the project just, or even primarily, technically.  We are here to help the project learn the Apache way.  It is perfectly legitimate, even expected, for a mentor to raise non-techincal concerns such as these.

Alan.

On May 3, 2012, at 8:23 AM, Roman Shaposhnik wrote:

> On Thu, May 3, 2012 at 8:06 AM, Owen O'Malley <omalley@...> wrote:
>> On Mon, Apr 30, 2012 at 9:39 PM, Roman Shaposhnik <rvs@...> wrote:
>>> On Mon, Apr 30, 2012 at 9:13 PM, Eric Baldeschwieler
>>> <eric14@...> wrote:
>>>> So you are suggesting expanding the charter to include projects not hosted at Apache?
>>>
>>> I don't think this is what Bruno suggested. Personally I can't find
>>> any reference in Bigtop
>>> charter that restricts it to Projects that belong to Apache Software
>>> Foundation.
>>
>> As a mentor of the Bigtop project, I don't see it as acceptable for an
>> Apache project to distribute binaries of non-Apache software.
>
> In that case, I suggest you start with filing upstream JIRAs for pretty much
> all of the Hadoop ecosystem projects kindly asking them to remove
> dependencies on non-ASF (but APL!) projects like Guava libraries. Until
> that happens there's very little Bigtop can do.
>
>> If the owners of the Hue project decide to donate it to Apache and it had
>> been released by Apache, then it would be acceptable. I'm strictly -1
>> on releasing any version of Bigtop with Hue or any other non-Apache
>> software as part of the release.
>
> I would like to point out, that your -1 on Hue is highly inconsistent for the
> reason I mentioned above. I'm not sure what rules ASF has for gratuitous
> -1 votes devoid of clear technical reasoning, but I'm sure as a project
> mentor you can help us find out what that policy is.
>
> Thanks,
> Roman.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@...
> For additional commands, e-mail: general-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Alan Gates-2]

Roman,

I see your point that many Apache projects include non-Apache code in their binary distributions.  But there is a distinction here.  In the case of Hadoop and other projects, they bring things such Guava along because they need them, not for the express purpose of distributing those artifacts.  Bigtop, by its nature, is different because it provides artifacts for users to download regardless of what other components they need.  It is the difference between "we include this because we need it" and "we include this because you might want it".  

My concern is that this is a slippery slope.  There are lots of other things people use with Hadoop (Ganglia for monitoring, Postgres for their Hive metastore, Cascading, etc.).  Would we want Bigtop distributing those?  This would consume a lot of Apache resources to host these things on the download servers.  

Additionally, we need to think about maintaining Apache's brand.  When we redistribute Apache binaries, we know those have gone through an established release process.  With non-Apache binaries, even those that are APL, we know nothing of their releases processes, code quality, etc.  I do not mean this as a slight to Hue nor any of the projects mentioned above.  But if we let one in we will have to let others in.  Again this is important because we would be opening ourselves up as a distribution point for those projects independent of their usage in other Apache projects.

By drawing the line at distributing only Apache projects we protect Apache both in terms of server resource usage and in branding.

As Bruno pointed out in the thread on bigtop-dev (http://mail-archives.apache.org/mod_mbox/incubator-bigtop-dev/201205.mbox/%3C4F9F4F6C.4000404%40apache.org%3E ) this does limit Bigtop, so I understand the motivation to do it.  But before we take this step it merits discussion in the community.

Finally, a comment on the role of mentors.  You were concerned that Owen was vetoing this for non-technical reasons.  Your mentors are not here to guide the project just, or even primarily, technically.  We are here to help the project learn the Apache way.  It is perfectly legitimate, even expected, for a mentor to raise non-techincal concerns such as these.

Alan.

On May 3, 2012, at 8:23 AM, Roman Shaposhnik wrote:

> On Thu, May 3, 2012 at 8:06 AM, Owen O'Malley <omalley@...> wrote:
>> On Mon, Apr 30, 2012 at 9:39 PM, Roman Shaposhnik <rvs@...> wrote:
>>> On Mon, Apr 30, 2012 at 9:13 PM, Eric Baldeschwieler
>>> <eric14@...> wrote:
>>>> So you are suggesting expanding the charter to include projects not hosted at Apache?
>>>
>>> I don't think this is what Bruno suggested. Personally I can't find
>>> any reference in Bigtop
>>> charter that restricts it to Projects that belong to Apache Software
>>> Foundation.
>>
>> As a mentor of the Bigtop project, I don't see it as acceptable for an
>> Apache project to distribute binaries of non-Apache software.
>
> In that case, I suggest you start with filing upstream JIRAs for pretty much
> all of the Hadoop ecosystem projects kindly asking them to remove
> dependencies on non-ASF (but APL!) projects like Guava libraries. Until
> that happens there's very little Bigtop can do.
>
>> If the owners of the Hue project decide to donate it to Apache and it had
>> been released by Apache, then it would be acceptable. I'm strictly -1
>> on releasing any version of Bigtop with Hue or any other non-Apache
>> software as part of the release.
>
> I would like to point out, that your -1 on Hue is highly inconsistent for the
> reason I mentioned above. I'm not sure what rules ASF has for gratuitous
> -1 votes devoid of clear technical reasoning, but I'm sure as a project
> mentor you can help us find out what that policy is.
>
> Thanks,
> Roman.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@...
> For additional commands, e-mail: general-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Bruno Mahé-2]

On 05/03/2012 08:06 AM, Owen O'Malley wrote:

> On Mon, Apr 30, 2012 at 9:39 PM, Roman Shaposhnik <rvs@...> wrote:
>> On Mon, Apr 30, 2012 at 9:13 PM, Eric Baldeschwieler
>> <eric14@...> wrote:
>>> So you are suggesting expanding the charter to include projects not hosted at Apache?
>>
>> I don't think this is what Bruno suggested. Personally I can't find
>> any reference in Bigtop
>> charter that restricts it to Projects that belong to Apache Software
>> Foundation.
>
> As a mentor of the Bigtop project, I don't see it as acceptable for an
> Apache project to distribute binaries of non-Apache software. If the
> owners of the Hue project decide to donate it to Apache and it had
> been released by Apache, then it would be acceptable. I'm strictly -1
> on releasing any version of Bigtop with Hue or any other non-Apache
> software as part of the release.
>
> -- Owen

As part of mentoring Apache Bigtop (incubating) project, it would also
be greatly appreciated if you would explain why this -1.

Apache Bigtop (incubating) does not and will not include anything that
does not belong to the Apache Foundation.
So I am really confused as to why this strong reaction.

The convenience artefact may pull Hue in, but this is in no way
different from Apache Hadoop pulling in Google protocol buffer or Google
guava. So again, how is this different? Is Apache Hadoop going to
avandon Google Protocolbuffer?

Furthermore as I just stated above, Apache Bigtop (incubating) does not
and will not include any source of Hue.
I am also confused on your definition of release since from my
understanding, only source releases are voted on. And they won't contain
Hue or any code that does not belong to the Apache Foundation. We don't
check in any of our dependency, so I am confused about your statements.
All of this is really no different from Apache Hadoop which pulls some
non-Apache Foundation dependencies such as Google Protocolbuffer.


Thanks,
Bruno


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Bruno Mahé-2]

On 05/03/2012 11:46 AM, Alan Gates wrote:

> Roman,
>
> I see your point that many Apache projects include non-Apache code in their binary distributions.  But there is a distinction here.  In the case of Hadoop and other projects, they bring things such Guava along because they need them, not for the express purpose of distributing those artifacts.  Bigtop, by its nature, is different because it provides artifacts for users to download regardless of what other components they need.  It is the difference between "we include this because we need it" and "we include this because you might want it".  
>
> My concern is that this is a slippery slope.  There are lots of other things people use with Hadoop (Ganglia for monitoring, Postgres for their Hive metastore, Cascading, etc.).  Would we want Bigtop distributing those?  This would consume a lot of Apache resources to host these things on the download servers.  
>
> Additionally, we need to think about maintaining Apache's brand.  When we redistribute Apache binaries, we know those have gone through an established release process.  With non-Apache binaries, even those that are APL, we know nothing of their releases processes, code quality, etc.  I do not mean this as a slight to Hue nor any of the projects mentioned above.  But if we let one in we will have to let others in.  Again this is important because we would be opening ourselves up as a distribution point for those projects independent of their usage in other Apache projects.
>
> By drawing the line at distributing only Apache projects we protect Apache both in terms of server resource usage and in branding.
>
> As Bruno pointed out in the thread on bigtop-dev (http://mail-archives.apache.org/mod_mbox/incubator-bigtop-dev/201205.mbox/%3C4F9F4F6C.4000404%40apache.org%3E ) this does limit Bigtop, so I understand the motivation to do it.  But before we take this step it merits discussion in the community.
>
> Finally, a comment on the role of mentors.  You were concerned that Owen was vetoing this for non-technical reasons.  Your mentors are not here to guide the project just, or even primarily, technically.  We are here to help the project learn the Apache way.  It is perfectly legitimate, even expected, for a mentor to raise non-techincal concerns such as these.
>
> Alan.
>

Apache Bigtop (incubating) releases don't distribute any binary. They
only contain Apache Bigtop (incubating) source code.

Convenience binaries are just convenience as far as I understand. So
from what you are saying, your issues have nothing to do with Apache
Bigtop (incubating) releases, but the convenience binaries. Therefore I
don't see the need to suddenly -1 releases of Apache Bigtop (incubating).


Given that we already have to host our own build machines because the
Apache infrastructure is not suited to this sort of work, we could also
host the artefacts on our jenkins instance and save them some burded?
But even though, this is for a discussion with Apache Infrastructure.
And also I wouldn't see any point in distributing Ganglia or Postgres,
even though I see your point.

Or we could restrict the convenience binaries to be just of those
projects belonging to the Apache Foundation. So it would not limit the
scope and abilities of Apache Bigtop (incubating) so much.


Regarding the fact we don't consider these as dependencies, I disagree.
Apache Bigtop (incubating) is an integration point for any project
related to Apache Hadoop. So we produce recipes for a consistent stack.
So of course convenience binaries will include packages, but Apache
Bigtop (incubating) goes way beyond that. Anyone can build their own
stack, with their own version on their favorite GNU/Linux distribution.
There are all sorts of tests (integration, performance, package),
recipes to build your own virtual machine, bootable image and even
deployment recipes for puppet.
So packages are not an end but only a part of this story. The end
product is a deployable stack of Apache Hadoop ecosystem and these
projects related to Apache Hadoop are just used as dependencies.

Thanks,
Bruno

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Roman Shaposhnik-2]

Hi Alan!

On Thu, May 3, 2012 at 11:46 AM, Alan Gates <gates@...> wrote:
> Bigtop, by its nature, is different because it provides artifacts for users to download
> regardless of what other components they need.
> It is the difference between "we include this because we need it" and "we include this because you might want it".

I think what needs to be kept in mind is that at the end of the day,
Apache Foundation Projects release source code. Everything else
is inconsequential binary artifacts.

For some projects binary artifacts hardly matter (e.g. Apache HTTPD
server), for some others they get a lot of attention and consume
a lot of precious Apache INFRA bandwidth (e.g. Apache OpenOffice
[incubating], Apache Bigtop [incubating]).

Yet, I would dare to say that from ASF perspective even OpenOffice
is all about source code releases.

From that perspective, Bigtop's ultimate goal is to be an ASF project that
provides an integration, validation and deployment framework for
Bigdata management stacks based on Apache Hadoop. In that respect
Bigtop is very close to OpenEmbedded/OpenWRT projects in how
they provide you with a tools to create custom linux distributions.

> My concern is that this is a slippery slope.  There are lots of other things people
> use with Hadoop (Ganglia for monitoring, Postgres for their Hive metastore,
> Cascading, etc.).  Would we want Bigtop distributing those?

Including support for them into the framework -- sure. Why not? If folks
find it useful to manage all of the above via the Bigtop framework -- where's
harm in that? As long as the license is compatible with APL -- my answer
would be a positive one. In fact, if you look closely you'll see that Bigtop
already supports Kerberos, MySQL and hopefully will soon support
Ganglia via Puppet code.

Now, if you're asking a question of whether we should be including all these
dependencies into the binary convenience artifacts that we publish, then
my answer is predicated on two things:
   #1 I would like to steer clear from distributing anything that is
not APL licensed
        using Apache infrastructure.
   #2 I would like to get an explicit Apache INFRA team blessing that they are
        comfortable with the storage and bandwidth requirements.

> Additionally, we need to think about maintaining Apache's brand.
> When we redistribute Apache binaries, we know those have gone
> through an established release process.  With non-Apache binaries,
> even those that are APL, we know nothing of their releases processes,
> code quality, etc.  I do not mean this as a slight to Hue nor any of the
> projects mentioned above.  But if we let one in we will have to let others in.
> Again this is important because we would be opening ourselves up as a
> distribution point for those projects independent of their usage in other
> Apache projects.

I don't think I agree with your line of thought as far as "known
release process"
is concerned. The same argument could be applied to every project that
depends on Guava -- it is something we know nothing about, yet we're perefctly
willing to make it part of the binary artifacts for Hadoop and quite a few other
projects.

I do, however, agree with your concern as far as "distribution point"
goes. Indeed,
it is pretty outlandish to imagine somebody downloading Hadoop binary artifacts
just to extract Guava and use it from there. Thus, by any stretch of
imagination,
Hadoop is not a distribution point for Guava, but if we start
packaging it in Bigtop's
binary convenience artifact -- we will be. This is a valid concern.

Now, personally, I think that this is much more a concern for the
project itself,
rather that Apache, but I'd be curious to hear what other think about it.

> Finally, a comment on the role of mentors.  You were concerned that Owen
> was vetoing this for non-technical reasons.  Your mentors are not here to
> guide the project just, or even primarily, technically.  We are here to help the
> project learn the Apache way.  It is perfectly legitimate, even expected, for a
> mentor to raise non-techincal concerns such as these.

Good point. However, as a mentee I expect the level of discourse to be
appropriate.
I have no problem with your reply and careful explanation of the issues at hand.
In fact, I welcome it very much. Owen's reply lacked any of the nuance and felt
as a "my way or the highway" kind of -1. I don't think there's
anything to be learned
from that type of mentoring approach.

Thanks,
Roman.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[owen.omalley]

On Thu, May 3, 2012 at 12:01 PM, Bruno Mahé <bmahe@...> wrote:

>>
>> As a mentor of the Bigtop project, I don't see it as acceptable for an
>> Apache project to distribute binaries of non-Apache software. If the
>> owners of the Hue project decide to donate it to Apache and it had
>> been released by Apache, then it would be acceptable. I'm strictly -1
>> on releasing any version of Bigtop with Hue or any other non-Apache
>> software as part of the release.
>>
>> -- Owen
>
>
> As part of mentoring Apache Bigtop (incubating) project, it would also
> be greatly appreciated if you would explain why this -1.
>
> Apache Bigtop (incubating) does not and will not include anything that
> does not belong to the Apache Foundation.
> So I am really confused as to why this strong reaction.

The strong reaction is because Roman was proposing a Bigtop release
with rpms and debs for non-Apache projects. That is a non-starter.
Apache will not distribute non-Apache projects. Saying that Bigtop
does not release the projects that it incorporates is not justified
given the fact that Bigtop is putting rpms of each of the incorporated
projects into /dist/incubator/bigtop. The 2.9GB size of the latest
Bigtop release has already caused infrastructure significant
headaches.

> The convenience artefact may pull Hue in, but this is in no way
> different from Apache Hadoop pulling in Google protocol buffer or Google
> guava. So again, how is this different? Is Apache Hadoop going to
> avandon Google Protocolbuffer?

There is a big difference between referencing external projects that
are required for your project's functionality and incorporating
non-Apache projects into your project and publishing releases of them
using independent artifacts. When the user installs a Hadoop rpm, the
protobuf.jar is there under the hood, but is considered an
implementation detail that is required for Hadoop to run.

I'd complain similarly if Hadoop was downloading protobuf tarballs,
making changes to protobuf, making protobuf rpms with those changes,
and publishing those rpms on Apache's servers.

However, it goes deeper than than that. If the user installs Bigtop's
rpms and hits a bug do they contact Hue or Bigtop? Furthermore, I'm
sure the links that are displayed when you run Bigtop's Hue point off
to Cloudera's bug and support system. That kind of branding is not ok
for an Apache project.

Even with Apache projects, Bigtop may become problematic. Look at the
mess that happened when Lucene made a bugfix release of Apache Commons
CSV. Lucene needed a bugfix release of CSV, didn't wait for CSV to
release, and instead released it themselves. Needless to say Apache
Commons didn't like that result. Bigtop is overriding decisions made
by the upstream projects about things like the way the launching
scripts operate and where the configuration directory is. When asked
to correct it, they complain about compatibility for their users
rather than compatibility for Hadoop's users.

-- Owen

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Bruno Mahé-3]

Hi,

Please see my reply inline.

On 05/03/2012 04:00 PM, Owen O'Malley wrote:

> On Thu, May 3, 2012 at 12:01 PM, Bruno Mahé <bmahe@...> wrote:
>>>
>>> As a mentor of the Bigtop project, I don't see it as acceptable for an
>>> Apache project to distribute binaries of non-Apache software. If the
>>> owners of the Hue project decide to donate it to Apache and it had
>>> been released by Apache, then it would be acceptable. I'm strictly -1
>>> on releasing any version of Bigtop with Hue or any other non-Apache
>>> software as part of the release.
>>>
>>> -- Owen
>>
>>
>> As part of mentoring Apache Bigtop (incubating) project, it would also
>> be greatly appreciated if you would explain why this -1.
>>
>> Apache Bigtop (incubating) does not and will not include anything that
>> does not belong to the Apache Foundation.
>> So I am really confused as to why this strong reaction.
>
> The strong reaction is because Roman was proposing a Bigtop release
> with rpms and debs for non-Apache projects. That is a non-starter.
> Apache will not distribute non-Apache projects. Saying that Bigtop
> does not release the projects that it incorporates is not justified
> given the fact that Bigtop is putting rpms of each of the incorporated
> projects into /dist/incubator/bigtop. The 2.9GB size of the latest
> Bigtop release has already caused infrastructure significant
> headaches.
>

Seems like you are still (this is not the first time this is explained
to you) conflating Apache releases on which members vote on, with
convenience artefacts.
Apache Bigtop (incubating) releases are not the packages. Apache Bigtop
(incubating) releases are Apache Bigtop (incubating) source code.
RPMs and DEBs are convenience artefact.
If they are not that convenient to the Apache Foundation, I don't see
the issue with not distributing the ones that are not convenient.

As far as I know, Apache Infra was only asking for heads up. Which we
will provide and we will pay attention to work more closely with them.
I also fail to see the relationship between the size of the convenience
artefacts and the bill of materials for the coming release of Apache
Bigtop (incubating), which I repeat only contains Apache Bigtop
(incubating) source code.

So now we have establish that you issue is about the convenience
artefacts, I don't see any remaining issue with Apache Bigtop
(incubating) releases.

>> The convenience artefact may pull Hue in, but this is in no way
>> different from Apache Hadoop pulling in Google protocol buffer or Google
>> guava. So again, how is this different? Is Apache Hadoop going to
>> avandon Google Protocolbuffer?
>
> There is a big difference between referencing external projects that
> are required for your project's functionality and incorporating
> non-Apache projects into your project and publishing releases of them
> using independent artifacts. When the user installs a Hadoop rpm, the
> protobuf.jar is there under the hood, but is considered an
> implementation detail that is required for Hadoop to run.
>
> I'd complain similarly if Hadoop was downloading protobuf tarballs,
> making changes to protobuf, making protobuf rpms with those changes,
> and publishing those rpms on Apache's servers.
>

In any case, there is still distribution of a non-Apache project's
artefacts by both projects.
You either distribute artefacts of it, or you don't. Here the end goal
is not to provide packages, but a deployable big data stack. Packages
are just a mean to an end.
We don't distribute upstream projects, they are dependencies.


> However, it goes deeper than than that. If the user installs Bigtop's
> rpms and hits a bug do they contact Hue or Bigtop? Furthermore, I'm
> sure the links that are displayed when you run Bigtop's Hue point off
> to Cloudera's bug and support system. That kind of branding is not ok
> for an Apache project.
>

Hue is not even integrated into Apache Bigtop (incubating). So let's
cross that bridge when we get there. And in any case, this is an issue
that can be fixed trivially, so I wouldn't make it a blocker.

But beyond that, we don't patch anything. So any product issue would
come from the product. Any integration issue would be an Apache Bigtop
(incubating) issue. The same way with Apache Hadoop.
No matter what you do, educating users will be the most important part.

> Even with Apache projects, Bigtop may become problematic. Look at the
> mess that happened when Lucene made a bugfix release of Apache Commons
> CSV. Lucene needed a bugfix release of CSV, didn't wait for CSV to
> release, and instead released it themselves. Needless to say Apache
> Commons didn't like that result. Bigtop is overriding decisions made
> by the upstream projects about things like the way the launching
> scripts operate and where the configuration directory is. When asked
> to correct it, they complain about compatibility for their users
> rather than compatibility for Hadoop's users.
>

How is this related with Hue?
I am also not sure about the point of complaining about this here on
general@. This is a direction taken by the Apache Bigtop (incubating)
community at large. You may not agree, but there is a consensus on that
part. Unless there is something in the Apache Foundation bylaws that
would force an upstream project to force decisions on other communities?

And beyond that, this is not new. As it was explained in an other thread
to Matt on Apache Bigtop (incubating) mailing list:
Apache Bigtop (incubating) has made the choice from the very beginning
to be close to what GNU/Linux distributions have been doing and what
sysadmins have been used to. This can differ with the experience one may
have with a tarball development of Apache Hadoop.

Keep in mind also that each component has its own way of doing things.
Each one having its own issues.
Apache Bigtop (incubating) smooth this up and provide an easy and
unified experience to users. Furthermore, we cannot satisfy any whim of
every single upstream components.

In order to use pristine Apache Hadoop, one would have to be familiar
with its usage and its configuration. Apache Hadoop experience is only
familiar to people *already* familiar with Apache Hadoop.

So using upstream Apache Hadoop will imply a lot of reading through
forums, documentation and frustration.
For instance, Apache Bigtop (incubating) will pre-set the ulimits for
you, will set up the logging to well-known locations, provide init
scripts and make a pseudo-configuration available to users.

In a word, Apache Bigtop (incubating) has a different use case than
upstream Apache Hadoop and therefore will be different in some areas.
Or put it differently, upstream projects should focus on what they do
best, which is to work on delivering awesome projects, while in Apache
Bigtop (incubating) we focus on what we do best, which is making a
production quality/ready big data stack.

And I don't see the issue of caring about compatibility of Apache Bigtop
(incubating) users within the Apache Bigtop (incubating) project.

But again, all of this is unrelated to this thread. So you may want to
move that discussion to another thread.

Thanks,
Bruno

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Patrick Hunt-3]

It's not the job of the incubator to create new rules, but rather to
help podlings to graduation while following existing Apache
guidelines. It's very clear from
http://www.apache.org/legal/resolved.html that what has been proposed
is acceptable under existing Apache rules. Bigtop is building
on/around ASL licensed software (packaging, iteroperability tests,
utilities, etc...), much like many other Apache projects, much like
other distributions do with Apache's own projects/software. I don't
see any reason to create new rules which limit the podling's ability
to include compatibly licensed software in their releases. (as long as
they follow the licenses of said software)

Patrick

On Thu, May 3, 2012 at 4:30 PM, Bruno Mahé <bruno@...> wrote:

> Hi,
>
> Please see my reply inline.
>
> On 05/03/2012 04:00 PM, Owen O'Malley wrote:
>> On Thu, May 3, 2012 at 12:01 PM, Bruno Mahé <bmahe@...> wrote:
>>>>
>>>> As a mentor of the Bigtop project, I don't see it as acceptable for an
>>>> Apache project to distribute binaries of non-Apache software. If the
>>>> owners of the Hue project decide to donate it to Apache and it had
>>>> been released by Apache, then it would be acceptable. I'm strictly -1
>>>> on releasing any version of Bigtop with Hue or any other non-Apache
>>>> software as part of the release.
>>>>
>>>> -- Owen
>>>
>>>
>>> As part of mentoring Apache Bigtop (incubating) project, it would also
>>> be greatly appreciated if you would explain why this -1.
>>>
>>> Apache Bigtop (incubating) does not and will not include anything that
>>> does not belong to the Apache Foundation.
>>> So I am really confused as to why this strong reaction.
>>
>> The strong reaction is because Roman was proposing a Bigtop release
>> with rpms and debs for non-Apache projects. That is a non-starter.
>> Apache will not distribute non-Apache projects. Saying that Bigtop
>> does not release the projects that it incorporates is not justified
>> given the fact that Bigtop is putting rpms of each of the incorporated
>> projects into /dist/incubator/bigtop. The 2.9GB size of the latest
>> Bigtop release has already caused infrastructure significant
>> headaches.
>>
>
> Seems like you are still (this is not the first time this is explained
> to you) conflating Apache releases on which members vote on, with
> convenience artefacts.
> Apache Bigtop (incubating) releases are not the packages. Apache Bigtop
> (incubating) releases are Apache Bigtop (incubating) source code.
> RPMs and DEBs are convenience artefact.
> If they are not that convenient to the Apache Foundation, I don't see
> the issue with not distributing the ones that are not convenient.
>
> As far as I know, Apache Infra was only asking for heads up. Which we
> will provide and we will pay attention to work more closely with them.
> I also fail to see the relationship between the size of the convenience
> artefacts and the bill of materials for the coming release of Apache
> Bigtop (incubating), which I repeat only contains Apache Bigtop
> (incubating) source code.
>
> So now we have establish that you issue is about the convenience
> artefacts, I don't see any remaining issue with Apache Bigtop
> (incubating) releases.
>
>
>>> The convenience artefact may pull Hue in, but this is in no way
>>> different from Apache Hadoop pulling in Google protocol buffer or Google
>>> guava. So again, how is this different? Is Apache Hadoop going to
>>> avandon Google Protocolbuffer?
>>
>> There is a big difference between referencing external projects that
>> are required for your project's functionality and incorporating
>> non-Apache projects into your project and publishing releases of them
>> using independent artifacts. When the user installs a Hadoop rpm, the
>> protobuf.jar is there under the hood, but is considered an
>> implementation detail that is required for Hadoop to run.
>>
>> I'd complain similarly if Hadoop was downloading protobuf tarballs,
>> making changes to protobuf, making protobuf rpms with those changes,
>> and publishing those rpms on Apache's servers.
>>
>
> In any case, there is still distribution of a non-Apache project's
> artefacts by both projects.
> You either distribute artefacts of it, or you don't. Here the end goal
> is not to provide packages, but a deployable big data stack. Packages
> are just a mean to an end.
> We don't distribute upstream projects, they are dependencies.
>
>
>> However, it goes deeper than than that. If the user installs Bigtop's
>> rpms and hits a bug do they contact Hue or Bigtop? Furthermore, I'm
>> sure the links that are displayed when you run Bigtop's Hue point off
>> to Cloudera's bug and support system. That kind of branding is not ok
>> for an Apache project.
>>
>
> Hue is not even integrated into Apache Bigtop (incubating). So let's
> cross that bridge when we get there. And in any case, this is an issue
> that can be fixed trivially, so I wouldn't make it a blocker.
>
> But beyond that, we don't patch anything. So any product issue would
> come from the product. Any integration issue would be an Apache Bigtop
> (incubating) issue. The same way with Apache Hadoop.
> No matter what you do, educating users will be the most important part.
>
>
>> Even with Apache projects, Bigtop may become problematic. Look at the
>> mess that happened when Lucene made a bugfix release of Apache Commons
>> CSV. Lucene needed a bugfix release of CSV, didn't wait for CSV to
>> release, and instead released it themselves. Needless to say Apache
>> Commons didn't like that result. Bigtop is overriding decisions made
>> by the upstream projects about things like the way the launching
>> scripts operate and where the configuration directory is. When asked
>> to correct it, they complain about compatibility for their users
>> rather than compatibility for Hadoop's users.
>>
>
> How is this related with Hue?
> I am also not sure about the point of complaining about this here on
> general@. This is a direction taken by the Apache Bigtop (incubating)
> community at large. You may not agree, but there is a consensus on that
> part. Unless there is something in the Apache Foundation bylaws that
> would force an upstream project to force decisions on other communities?
>
> And beyond that, this is not new. As it was explained in an other thread
> to Matt on Apache Bigtop (incubating) mailing list:
> Apache Bigtop (incubating) has made the choice from the very beginning
> to be close to what GNU/Linux distributions have been doing and what
> sysadmins have been used to. This can differ with the experience one may
> have with a tarball development of Apache Hadoop.
>
> Keep in mind also that each component has its own way of doing things.
> Each one having its own issues.
> Apache Bigtop (incubating) smooth this up and provide an easy and
> unified experience to users. Furthermore, we cannot satisfy any whim of
> every single upstream components.
>
> In order to use pristine Apache Hadoop, one would have to be familiar
> with its usage and its configuration. Apache Hadoop experience is only
> familiar to people *already* familiar with Apache Hadoop.
>
> So using upstream Apache Hadoop will imply a lot of reading through
> forums, documentation and frustration.
> For instance, Apache Bigtop (incubating) will pre-set the ulimits for
> you, will set up the logging to well-known locations, provide init
> scripts and make a pseudo-configuration available to users.
>
> In a word, Apache Bigtop (incubating) has a different use case than
> upstream Apache Hadoop and therefore will be different in some areas.
> Or put it differently, upstream projects should focus on what they do
> best, which is to work on delivering awesome projects, while in Apache
> Bigtop (incubating) we focus on what we do best, which is making a
> production quality/ready big data stack.
>
> And I don't see the issue of caring about compatibility of Apache Bigtop
> (incubating) users within the Apache Bigtop (incubating) project.
>
> But again, all of this is unrelated to this thread. So you may want to
> move that discussion to another thread.
>
> Thanks,
> Bruno
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@...
> For additional commands, e-mail: general-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[owen.omalley]

On Fri, May 4, 2012 at 8:58 AM, Patrick Hunt <phunt@...> wrote:
> It's not the job of the incubator to create new rules, but rather to
> help podlings to graduation while following existing Apache
> guidelines.

We aren't making new rules. We are trying to help the Bigtop project
understand the rules about not releasing non-Apache software. There is
a huge difference between depending on an artifact from another
project and building and distributing non-Apache rpms in the project's
/dist directory.

> It's very clear from
> http://www.apache.org/legal/resolved.html that what has been proposed
> is acceptable under existing Apache rules.

Can you find a single instance other than the disagreement between
Apache Lucene and Apache Commons where one project is distributing
another project's rpms? Are there any other non-Apache rpms in /dist?
Clearly the answer is a resounding NO. It would be a huge violation of
the trust the incubator is putting in me as a mentor if I didn't block
Bigtop's plan to do so.

-- Owen

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Patrick Hunt-3]

On Fri, May 4, 2012 at 9:47 AM, Owen O'Malley <omalley@...> wrote:

> On Fri, May 4, 2012 at 8:58 AM, Patrick Hunt <phunt@...> wrote:
>> It's not the job of the incubator to create new rules, but rather to
>> help podlings to graduation while following existing Apache
>> guidelines.
>
> We aren't making new rules. We are trying to help the Bigtop project
> understand the rules about not releasing non-Apache software. There is
> a huge difference between depending on an artifact from another
> project and building and distributing non-Apache rpms in the project's
> /dist directory.

They are not releasing non-Apache software. They are not forking an
existing project. Bigtop's release artifact will contain packaging
code which allows users to compile packages (deb, rpm, etc...) for
this ASL licensed component, not the source/binaries of the component
itself.

>
>> It's very clear from
>> http://www.apache.org/legal/resolved.html that what has been proposed
>> is acceptable under existing Apache rules.
>
> Can you find a single instance other than the disagreement between
> Apache Lucene and Apache Commons where one project is distributing
> another project's rpms? Are there any other non-Apache rpms in /dist?
> Clearly the answer is a resounding NO. It would be a huge violation of
> the trust the incubator is putting in me as a mentor if I didn't block
> Bigtop's plan to do so.

If the component made an objection to being included in Bigtop then I
could see an argument to be made, that's not the case here. The
opposite is true from what I've seen -- people want their software to
be included so that users can more easily consume it. That's why they
released their software under a less restrictive license in the first
place.

EOD existing Apache rules/license make no such distinction. "Works
under the following licenses may be included within Apache products"
(includes ASL).

Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Greg Stein-4]

On May 4, 2012 2:03 PM, "Patrick Hunt" <phunt@...> wrote:
>...
> EOD existing Apache rules/license make no such distinction. "Works
> under the following licenses may be included within Apache products"
> (includes ASL).

Can people please stop using "ASL" or "APL"? No such thing. It is the
Apache License. AL for short, or even ALv2.

Thanks,
-g

[Jukka Zitting]

Hi,

I don't understand the BigTop use cases and release model in too much
detail to have very specific or hard opinions on this, but here's a
few high-level observations that hopefully are useful for this
discussion:

* AFAICT there's no immediate release that's being blocked by this
discussion, so everyone can calm down. An issue was brought up, it's
being discussed and I'm sure we'll soon enough have a solution that
everyone is happy about.

* It sounds like BigTop is doing something that few Apache projects
have done before. Thus it's fine to question whether and to what
extent existing rules apply. However, at the same time it's good to
acknowledge that new rules and consensus on a new interpretation of
existing rules may be needed for something like this. It's natural for
this process to take some time and involve some misunderstandings
along the way.

* The convenience binaries many projects are providing are normally
the result of building the respective source release (together with
any required third party dependencies). As a general rule it should be
possible for anyone to reproduce equivalent binaries by following the
build instructions included in the source release.

* As a recent addition, some projects have started providing also
convenience packages containing such dependencies required by the
source build as described above. In both cases the contents of all
such binary packages should be properly signed and contain appropriate
LICENSE and NOTICE files.

* As far as I can tell from the discussion, the BigTop repos directory
[1] doesn't neatly fit into either of the above categories. I guess
the key question here is whether the purpose of BigTop is to be a
particular, tested combination of upstream projects or rather a tool
for testing and building such combinations. (Or perhaps something else
entirely?)

* If the former, then each subdirectory of [1] falls fairly
conveniently into the traditional concept of convenience binaries
built from the source release. The only extra thing you'd need is a
proper set of license metadata and signatures for the binaries.

* If the latter, it seems to me that it isn't BigTop that should be
distributing the packages in [1]. Instead each upstream project should
using BigTop as a tool to produce such packages as a part of their own
release processes.

* In that case there might still be a role for BigTop to provide a
central repository for such easily consumable upstream releases. This
would be somewhat similar to the discussions that took place a few
years ago about whether and how the ASF could host something like the
central Maven repository.

[1] http://www.apache.org/dist/incubator/bigtop/bigtop-0.3.0-incubating/repos/

BR,

Jukka Zitting

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Patrick Hunt-3]

On Fri, May 4, 2012 at 11:54 AM, Greg Stein <gstein@...> wrote:
> On May 4, 2012 2:03 PM, "Patrick Hunt" <phunt@...> wrote:
>>...
>> EOD existing Apache rules/license make no such distinction. "Works
>> under the following licenses may be included within Apache products"
>> (includes ASL).
>
> Can people please stop using "ASL" or "APL"? No such thing. It is the
> Apache License. AL for short, or even ALv2.

Sorry for the incorrect tla usage. Will do.

Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Roman Shaposhnik-2]

Hi Jukka!

Thanks a million for chiming in.

On Fri, May 4, 2012 at 12:52 PM, Jukka Zitting <jukka.zitting@...> wrote:
> Hi,
>
> I don't understand the BigTop use cases

Perhaps this preso can help a bit:
    http://people.apache.org/~rvs/apache-bigtop2.pdf

> * If the former, then each subdirectory of [1] falls fairly
> conveniently into the traditional concept of convenience binaries
> built from the source release. The only extra thing you'd need is a
> proper set of license metadata and signatures for the binaries.

This is, in fact, the process we've been following with our convenience
artifacts for as long as we've had releases. We're signing every single
binary and are pretty pedantic about providing LICENSE and NOTICE files.
All of the distributed convenience artifacts have Apache License and this
policy will remain. Staring from Bigtop 0.4.0 release we are going to also
take great care in coordinating release of our convenience artifacts with
Apache Software Foundation's Infrastructure team in order to avoid
surprises and reduce strain on the mirroring infrastructure.

Please let me know if, from your point of view, the above alleviates the
concerns that have been expressed on this list.

Thanks,
Roman.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Alan Gates-2]

Jukka,

Thanks for your response, this is very helpful.  I have a couple of questions/clarifications inlined.


On May 4, 2012, at 12:52 PM, Jukka Zitting wrote:

>
>
> * As far as I can tell from the discussion, the BigTop repos directory
> [1] doesn't neatly fit into either of the above categories. I guess
> the key question here is whether the purpose of BigTop is to be a
> particular, tested combination of upstream projects or rather a tool
> for testing and building such combinations. (Or perhaps something else
> entirely?)
>
> * If the former, then each subdirectory of [1] falls fairly
> conveniently into the traditional concept of convenience binaries
> built from the source release. The only extra thing you'd need is a
> proper set of license metadata and signatures for the binaries.

My question here was whether this concept of convenience binaries should extend beyond ASF owned packages.  I realize that many existing convenience binaries contain non-ASF jars, etc.  But taking the next step of explicitly distributing non-ASF binaries on their own concerns me.

>
> * If the latter, it seems to me that it isn't BigTop that should be
> distributing the packages in [1]. Instead each upstream project should
> using BigTop as a tool to produce such packages as a part of their own
> release processes.
>
> * In that case there might still be a role for BigTop to provide a
> central repository for such easily consumable upstream releases. This
> would be somewhat similar to the discussions that took place a few
> years ago about whether and how the ASF could host something like the
> central Maven repository.

Do you know what list that discussion took place on and a general time frame?  Reading through that would be very helpful for my thinking on this topic.

Alan.

>
> [1] http://www.apache.org/dist/incubator/bigtop/bigtop-0.3.0-incubating/repos/
>
> BR,
>
> Jukka Zitting
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@...
> For additional commands, e-mail: general-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Jukka Zitting]

Hi,

On Fri, May 4, 2012 at 11:00 PM, Roman Shaposhnik <rvs@...> wrote:
> Perhaps this preso can help a bit:
>    http://people.apache.org/~rvs/apache-bigtop2.pdf

Perfect, thanks!

>> * If the former, then each subdirectory of [1] falls fairly
>> conveniently into the traditional concept of convenience binaries
>> built from the source release. The only extra thing you'd need is a
>> proper set of license metadata and signatures for the binaries.
>
> This is, in fact, the process we've been following with our convenience
> artifacts for as long as we've had releases. We're signing every single
> binary and are pretty pedantic about providing LICENSE and NOTICE files.

Sounds good.

I'm just wondering about where do I find for example the licensing
metadata for example for the files in
http://www.apache.org/dist/incubator/bigtop/bigtop-0.3.0-incubating/repos/fedora16/hive/

Or am I just missing something obvious? Like that you're using RPM
conventions for those bits, similar to how binary jars typically have
their licensing metadata in META-INF/{LICENSE,NOTICE}.

BR,

Jukka Zitting

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[Jukka Zitting]

Hi,

On Sat, May 5, 2012 at 1:34 AM, Alan Gates <gates@...> wrote:
> My question here was whether this concept of convenience binaries
> should extend beyond ASF owned packages.  I realize that many existing
> convenience binaries contain non-ASF jars, etc.  But taking the next
> step of explicitly distributing non-ASF binaries on their own concerns me.

The normal guidelines of what kind of third-party bits can be
redistributed by Apache projects are in
http://www.apache.org/legal/resolved.html. As long as all the
components included in a "BigTop x.y" installation meet those
guidelines or are system dependencies like "Fedora 16" that don't come
form the ASF, then the ASF policy side of things should be fine.

A somewhat similar example from another domain is Apache Tika, which
binds together a lot of upstream libraries from both within and
outside the ASF and makes them available as a single integrated and
tested package. AFAIUI the main difference here is that unlike in
Tika, BigTop doesn't have a programming API for accessing the included
components. Instead, IIUC, the "BigTop API" is a deployment/testing
one with stuff like "yum install", etc.

Now (again IIUC) the interesting bit is whether it's better for BigTop
to be repackaging and -distributing upstream components by itself, or
if it would in fact be better for BigTop to simply provide something
like "bigtop-x.y.rpm" and "bigtop-x.y.deb" packages that just declare
dependencies to specific, integration-tested versions of upstream
packages.

To do this, BigTop would need to work with the upstream projects to
help them produce the appropriate deployment packages as a part of
their normal release processes. And BigTop could also team up with
Infrastructure to maintain the kind of repository structure and
download service expected by deployment tools like yum and apt, a bit
like what Maven projects have in https://repository.apache.org/.

This is in fact a bit like what Tika has recently been considering
(see http://markmail.org/message/wj4xkoax2ojnqlht) with it's upstream
components. Instead of repackaging and -distributing them directly as
a part of a Tika release, we're looking at ways to add the required
extra bits to  the upstream releases so that Tika could just consume
them as normal dependencies.

>> * In that case there might still be a role for BigTop to provide a
>> central repository for such easily consumable upstream releases. This
>> would be somewhat similar to the discussions that took place a few
>> years ago about whether and how the ASF could host something like the
>> central Maven repository.
>
> Do you know what list that discussion took place on and a general time frame?
> Reading through that would be very helpful for my thinking on this topic.

See January 2007 on board@ and infrastructure@.

BR,

Jukka Zitting

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...

[struberg]

spreading the world since years, and it's always funny what new combinations come up:

https://twitter.com/#!/struberg/status/197003905027149824

:)


LieGrue,
strub

>________________________________
> From: Greg Stein <gstein@...>
>To: general@...
>Cc: bigtop-dev@...
>Sent: Friday, May 4, 2012 8:54 PM
>Subject: Re: [DISCUSS] BOM and supported platforms for Bigtop 0.4.0
>
>On May 4, 2012 2:03 PM, "Patrick Hunt" <phunt@...> wrote:
>>...
>> EOD existing Apache rules/license make no such distinction. "Works
>> under the following licenses may be included within Apache products"
>> (includes ASL).
>
>Can people please stop using "ASL" or "APL"? No such thing. It is the
>Apache License. AL for short, or even ALv2.
>
>Thanks,
>-g
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@...
For additional commands, e-mail: general-help@...