Nabble - archiva-dev

Re: seeking review on plan/work for MRM-1025

2009-11-20T01:47:13Z

In addition, this is the content model I was thinking of: http://svn.apache.org/repos/asf/archiva/branches/MRM-1025/archiva-modules/metadata/content-model.txt (represented as a tree structure, and can be rendered in XML, json, etc.)

Somewhat simpler but still as flexible as the one previously on the wiki.

- Brett

On 20/11/2009, at 12:38 PM, Brett Porter wrote:

> Hi,
>
> I have again picked up the task of removing the requirement of the database from Archiva (seeking to push it over to either an optional add-on, or replace it with an alternative). We use very little of the data in there for core functionality at the moment.
>
> The main reasoning is to clean up the architecture though and remove a certain classification of bugs that we get on a regular basis.
>
> I'd like to make sure the work I'm doing is reviewed though, even though it's on a branch. If you get time to review the code, please do. Short of that, please sanity check the current plan: http://jira.codehaus.org/browse/MRM-1025
>
> If you have any questions, comments, or would like to help, just shout!
>
> - Brett

seeking review on plan/work for MRM-1025

2009-11-19T17:38:26Z

Hi,

I have again picked up the task of removing the requirement of the database from Archiva (seeking to push it over to either an optional add-on, or replace it with an alternative). We use very little of the data in there for core functionality at the moment.

The main reasoning is to clean up the architecture though and remove a certain classification of bugs that we get on a regular basis.

I'd like to make sure the work I'm doing is reviewed though, even though it's on a branch. If you get time to review the code, please do. Short of that, please sanity check the current plan: http://jira.codehaus.org/browse/MRM-1025

If you have any questions, comments, or would like to help, just shout!

- Brett

Re: svn commit: r882207 [1/3] - in /archiva/branches/MRM-1025: ./ archiva-modules/ archiva-modules/archiva-base/archiva-consumers/archiva-lucene-consumers/ archiva-modules/archiva-base/archiva-consumers/archiva-lucene-consumers/src/main/java/org/apac

2009-11-19T15:17:16Z

Sorry, I didn't see that the issue key was in the path! Just a knee
jerk reaction to big commit without an issue. :) -Wendy

On Thu, Nov 19, 2009 at 4:05 PM, Brett Porter <brett@...> wrote:

> Thanks for the reminder - I'll put the issue tag in all the commit messages.
>
> This all falls under the branch issue (MRM-1025). This particular change could actually be ported back to the mainline as it stands alone - but I don't want to disturb the 1.2.3 work. To some extent, I'm still playing around with what will work before proposing anything more formal :)
>
> Cheers,
> Brett
>
> On 20/11/2009, at 4:24 AM, Wendy Smoak wrote:
>
>> Seems like a big change, is there a JIRA issue with details? -Wendy
>>
>> On Thu, Nov 19, 2009 at 10:16 AM, <brett@...> wrote:
>>> Author: brett
>>> Date: Thu Nov 19 17:16:20 2009
>>> New Revision: 882207
>>>
>>> URL: http://svn.apache.org/viewvc?rev=882207&view=rev
>>> Log:
>>> split the scheduler into modules to isolate database and indexer dependent code
>> ...
>
>

Re: svn commit: r882207 [1/3] - in /archiva/branches/MRM-1025: ./ archiva-modules/ archiva-modules/archiva-base/archiva-consumers/archiva-lucene-consumers/ archiva-modules/archiva-base/archiva-consumers/archiva-lucene-consumers/src/main/java/org/apac

2009-11-19T15:12:41Z

Actually, to provide more detail...

There was no functional change here, just a move to reduce the complexity of the transitive closure. I do think in other instances where functionality is enhanced / modified to remove the database, it is important to create separate issues (the next steps are to flesh out the metadata repository, which is probably a candidate for this).

Hope that makes sense!

- Brett

On 20/11/2009, at 10:05 AM, Brett Porter wrote:

Re: svn commit: r882207 [1/3] - in /archiva/branches/MRM-1025: ./ archiva-modules/ archiva-modules/archiva-base/archiva-consumers/archiva-lucene-consumers/ archiva-modules/archiva-base/archiva-consumers/archiva-lucene-consumers/src/main/java/org/apac

2009-11-19T15:05:49Z

Thanks for the reminder - I'll put the issue tag in all the commit messages.

This all falls under the branch issue (MRM-1025). This particular change could actually be ported back to the mainline as it stands alone - but I don't want to disturb the 1.2.3 work. To some extent, I'm still playing around with what will work before proposing anything more formal :)

Cheers,
Brett

On 20/11/2009, at 4:24 AM, Wendy Smoak wrote:

> Seems like a big change, is there a JIRA issue with details? -Wendy
>
> On Thu, Nov 19, 2009 at 10:16 AM, <brett@...> wrote:
>> Author: brett
>> Date: Thu Nov 19 17:16:20 2009
>> New Revision: 882207
>>
>> URL: http://svn.apache.org/viewvc?rev=882207&view=rev
>> Log:
>> split the scheduler into modules to isolate database and indexer dependent code
> ...

Re: svn commit: r882207 [1/3] - in /archiva/branches/MRM-1025: ./ archiva-modules/ archiva-modules/archiva-base/archiva-consumers/archiva-lucene-consumers/ archiva-modules/archiva-base/archiva-consumers/archiva-lucene-consumers/src/main/java/org/apac

2009-11-19T09:24:46Z

Seems like a big change, is there a JIRA issue with details? -Wendy

On Thu, Nov 19, 2009 at 10:16 AM, <brett@...> wrote:
> Author: brett
> Date: Thu Nov 19 17:16:20 2009
> New Revision: 882207
>
> URL: http://svn.apache.org/viewvc?rev=882207&view=rev
> Log:
> split the scheduler into modules to isolate database and indexer dependent code
...

Re: svn commit: r834415 - in /archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test: ./ ReadMe.txt pom.xml

2009-11-11T15:09:11Z

Thanks, sorry about that!

- Brett

On 11/11/2009, at 5:02 PM, Deng Ching wrote:

> Fixed now, it just needed the ${basedir} in the path :)
>
> On Wed, Nov 11, 2009 at 11:52 AM, Deng Ching <oching@...>
> wrote:
>
>> Hi Brett,
>>
>> This fails when running the webapp-tests as part of the build (e.g.
>> mvn
>> clean install -Pit from the root), it looks like the Tomcat
>> installation
>> isn't unzipped in the correct path..
>>
>> Thanks,
>> Deng
>>
>>
>> On Tue, Nov 10, 2009 at 6:26 PM, <brett@...> wrote:
>>
>>> Author: brett
>>> Date: Tue Nov 10 10:26:23 2009
>>> New Revision: 834415
>>>
>>> URL: http://svn.apache.org/viewvc?rev=834415&view=rev
>>> Log:
>>> store cargo installations outside target to avoid re-downloading
>>>
>>> Modified:
>>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/
>>> (props
>>> changed)
>>>
>>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/
>>> ReadMe.txt
>>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/
>>> pom.xml
>>>
>>> Propchange: archiva/trunk/archiva-modules/archiva-web/archiva-
>>> webapp-test/
>>>
>>> ------------------------------------------------------------------------------
>>> --- svn:ignore (original)
>>> +++ svn:ignore Tue Nov 10 10:26:23 2009
>>> @@ -6,3 +6,4 @@
>>> .settings
>>> .classpath
>>> .project
>>> +cargo-installs
>>>
>>> Modified:
>>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/
>>> ReadMe.txt
>>> URL:
>>> http://svn.apache.org/viewvc/archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/ReadMe.txt?rev=834415&r1=834414&r2=834415&view=diff
>>>
>>> =
>>> =
>>> =
>>> =
>>> =
>>> =
>>> =
>>> =
>>> =
>>> =
>>> ====================================================================
>>> ---
>>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/
>>> ReadMe.txt
>>> (original)
>>> +++
>>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/
>>> ReadMe.txt Tue
>>> Nov 10 10:26:23 2009
>>> @@ -9,7 +9,11 @@
>>> - modify src/test/resources/testng.properties as needed
>>> - mvn clean install -Dcontainer-existing
>>>
>>> +The Cargo installations are stored outside of target to avoid
>>> multiple
>>> downloads.
>>> +To remove the Cargo installations and re-download them next run,
>>> use:
>>> + - mvn -Pclean-cargo clean
>>> +
>>> For the default values in the scripts, to pass all the tests, you
>>> need to
>>> add an artifact in internal repository.
>>>
>>> Run Selenium tests in src/test/it with Maven and JUnit
>>> - - mvn clean install -f junit-pom.xml
>>> \ No newline at end of file
>>> + - mvn clean install -f junit-pom.xml
>>>
>>> Modified:
>>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/
>>> pom.xml
>>> URL:
>>> http://svn.apache.org/viewvc/archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/pom.xml?rev=834415&r1=834414&r2=834415&view=diff
>>>
>>> =
>>> =
>>> =
>>> =
>>> =
>>> =
>>> =
>>> =
>>> =
>>> =
>>> ====================================================================
>>> --- archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/
>>> pom.xml
>>> (original)
>>> +++ archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/
>>> pom.xml
>>> Tue Nov 10 10:26:23 2009
>>> @@ -265,7 +265,7 @@
>>> <fileset dir="src/test/${container.name}" />
>>> </copy>
>>> <copy
>>> - todir="${project.build.directory}/installs/${
>>> container.name
>>> }/apache-tomcat-${tomcat5x.version}/apache-tomcat-$
>>> {tomcat5x.version}/common/lib">
>>> + todir="${cargo.install.dir}/${container.name
>>> }/apache-tomcat-${tomcat5x.version}/apache-tomcat-$
>>> {tomcat5x.version}/common/lib">
>>> <fileset
>>> dir="${project.build.directory}/providedDependencies">
>>> <include name="**/*.jar" />
>>> </fileset>
>>> @@ -317,7 +317,7 @@
>>> <containerId>${container.name}</containerId>
>>> <zipUrlInstaller>
>>> <url>${container.url}</url>
>>> - <installDir>${project.build.directory}/installs/
>>> ${
>>> container.name}</installDir>
>>> + <installDir>${cargo.install.dir}/${container.name
>>> }</installDir>
>>> </zipUrlInstaller>
>>> <log>${project.build.directory}/logs/${container.name
>>> }.log</log>
>>> <output>${project.build.directory}/logs/$
>>> {container.name
>>> }.out</output>
>>> @@ -443,6 +443,24 @@
>>> </properties>
>>> </profile>
>>> <profile>
>>> + <id>clean-cargo</id>
>>> + <build>
>>> + <plugins>
>>> + <plugin>
>>> + <groupId>org.apache.maven.plugins</groupId>
>>> + <artifactId>maven-clean-plugin</artifactId>
>>> + <configuration>
>>> + <filesets>
>>> + <fileset>
>>> + <directory>${cargo.install.dir}</directory>
>>> + </fileset>
>>> + </filesets>
>>> + </configuration>
>>> + </plugin>
>>> + </plugins>
>>> + </build>
>>> + </profile>
>>> + <profile>
>>> <id>headless</id>
>>> <build>
>>> <plugins>
>>> @@ -466,5 +484,6 @@
>>> <properties>
>>> <tomcat5x.version>5.5.27</tomcat5x.version>
>>> <cargo.wait>false</cargo.wait>
>>> + <cargo.install.dir>cargo-installs</cargo.install.dir>
>>> </properties>
>>> </project>
>>>
>>>
>>>
>>

Re: svn commit: r834415 - in /archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test: ./ ReadMe.txt pom.xml

2009-11-10T22:02:42Z

Fixed now, it just needed the ${basedir} in the path :)

On Wed, Nov 11, 2009 at 11:52 AM, Deng Ching <oching@...> wrote:

> Hi Brett,
>
> This fails when running the webapp-tests as part of the build (e.g. mvn
> clean install -Pit from the root), it looks like the Tomcat installation
> isn't unzipped in the correct path..
>
> Thanks,
> Deng
>
>
> On Tue, Nov 10, 2009 at 6:26 PM, <brett@...> wrote:
>
>> Author: brett
>> Date: Tue Nov 10 10:26:23 2009
>> New Revision: 834415
>>
>> URL: http://svn.apache.org/viewvc?rev=834415&view=rev
>> Log:
>> store cargo installations outside target to avoid re-downloading
>>
>> Modified:
>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/ (props
>> changed)
>>
>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/ReadMe.txt
>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/pom.xml
>>
>> Propchange: archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/
>>
>> ------------------------------------------------------------------------------
>> --- svn:ignore (original)
>> +++ svn:ignore Tue Nov 10 10:26:23 2009
>> @@ -6,3 +6,4 @@
>> .settings
>> .classpath
>> .project
>> +cargo-installs
>>
>> Modified:
>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/ReadMe.txt
>> URL:
>> http://svn.apache.org/viewvc/archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/ReadMe.txt?rev=834415&r1=834414&r2=834415&view=diff
>>
>> ==============================================================================
>> ---
>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/ReadMe.txt
>> (original)
>> +++
>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/ReadMe.txt Tue
>> Nov 10 10:26:23 2009
>> @@ -9,7 +9,11 @@
>> - modify src/test/resources/testng.properties as needed
>> - mvn clean install -Dcontainer-existing
>>
>> +The Cargo installations are stored outside of target to avoid multiple
>> downloads.
>> +To remove the Cargo installations and re-download them next run, use:
>> + - mvn -Pclean-cargo clean
>> +
>> For the default values in the scripts, to pass all the tests, you need to
>> add an artifact in internal repository.
>>
>> Run Selenium tests in src/test/it with Maven and JUnit
>> - - mvn clean install -f junit-pom.xml
>> \ No newline at end of file
>> + - mvn clean install -f junit-pom.xml
>>
>> Modified:
>> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/pom.xml
>> URL:
>> http://svn.apache.org/viewvc/archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/pom.xml?rev=834415&r1=834414&r2=834415&view=diff
>>
>> ==============================================================================
>> --- archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/pom.xml
>> (original)
>> +++ archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/pom.xml
>> Tue Nov 10 10:26:23 2009
>> @@ -265,7 +265,7 @@
>> <fileset dir="src/test/${container.name}" />
>> </copy>
>> <copy
>> - todir="${project.build.directory}/installs/${
>> container.name
>> }/apache-tomcat-${tomcat5x.version}/apache-tomcat-${tomcat5x.version}/common/lib">
>> + todir="${cargo.install.dir}/${container.name
>> }/apache-tomcat-${tomcat5x.version}/apache-tomcat-${tomcat5x.version}/common/lib">
>> <fileset
>> dir="${project.build.directory}/providedDependencies">
>> <include name="**/*.jar" />
>> </fileset>
>> @@ -317,7 +317,7 @@
>> <containerId>${container.name}</containerId>
>> <zipUrlInstaller>
>> <url>${container.url}</url>
>> - <installDir>${project.build.directory}/installs/${
>> container.name}</installDir>
>> + <installDir>${cargo.install.dir}/${container.name
>> }</installDir>
>> </zipUrlInstaller>
>> <log>${project.build.directory}/logs/${container.name
>> }.log</log>
>> <output>${project.build.directory}/logs/${container.name
>> }.out</output>
>> @@ -443,6 +443,24 @@
>> </properties>
>> </profile>
>> <profile>
>> + <id>clean-cargo</id>
>> + <build>
>> + <plugins>
>> + <plugin>
>> + <groupId>org.apache.maven.plugins</groupId>
>> + <artifactId>maven-clean-plugin</artifactId>
>> + <configuration>
>> + <filesets>
>> + <fileset>
>> + <directory>${cargo.install.dir}</directory>
>> + </fileset>
>> + </filesets>
>> + </configuration>
>> + </plugin>
>> + </plugins>
>> + </build>
>> + </profile>
>> + <profile>
>> <id>headless</id>
>> <build>
>> <plugins>
>> @@ -466,5 +484,6 @@
>> <properties>
>> <tomcat5x.version>5.5.27</tomcat5x.version>
>> <cargo.wait>false</cargo.wait>
>> + <cargo.install.dir>cargo-installs</cargo.install.dir>
>> </properties>
>> </project>
>>
>>
>>
>

Re: svn commit: r834415 - in /archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test: ./ ReadMe.txt pom.xml

2009-11-10T19:52:47Z

Hi Brett,

This fails when running the webapp-tests as part of the build (e.g. mvn
clean install -Pit from the root), it looks like the Tomcat installation
isn't unzipped in the correct path..

Thanks,
Deng

On Tue, Nov 10, 2009 at 6:26 PM, <brett@...> wrote:

> Author: brett
> Date: Tue Nov 10 10:26:23 2009
> New Revision: 834415
>
> URL: http://svn.apache.org/viewvc?rev=834415&view=rev
> Log:
> store cargo installations outside target to avoid re-downloading
>
> Modified:
> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/ (props
> changed)
> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/ReadMe.txt
> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/pom.xml
>
> Propchange: archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/
>
> ------------------------------------------------------------------------------
> --- svn:ignore (original)
> +++ svn:ignore Tue Nov 10 10:26:23 2009
> @@ -6,3 +6,4 @@
> .settings
> .classpath
> .project
> +cargo-installs
>
> Modified:
> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/ReadMe.txt
> URL:
> http://svn.apache.org/viewvc/archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/ReadMe.txt?rev=834415&r1=834414&r2=834415&view=diff
>
> ==============================================================================
> ---
> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/ReadMe.txt
> (original)
> +++
> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/ReadMe.txt Tue
> Nov 10 10:26:23 2009
> @@ -9,7 +9,11 @@
> - modify src/test/resources/testng.properties as needed
> - mvn clean install -Dcontainer-existing
>
> +The Cargo installations are stored outside of target to avoid multiple
> downloads.
> +To remove the Cargo installations and re-download them next run, use:
> + - mvn -Pclean-cargo clean
> +
> For the default values in the scripts, to pass all the tests, you need to
> add an artifact in internal repository.
>
> Run Selenium tests in src/test/it with Maven and JUnit
> - - mvn clean install -f junit-pom.xml
> \ No newline at end of file
> + - mvn clean install -f junit-pom.xml
>
> Modified:
> archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/pom.xml
> URL:
> http://svn.apache.org/viewvc/archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/pom.xml?rev=834415&r1=834414&r2=834415&view=diff
>
> ==============================================================================
> --- archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/pom.xml
> (original)
> +++ archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test/pom.xml
> Tue Nov 10 10:26:23 2009
> @@ -265,7 +265,7 @@
> <fileset dir="src/test/${container.name}" />
> </copy>
> <copy
> - todir="${project.build.directory}/installs/${
> container.name
> }/apache-tomcat-${tomcat5x.version}/apache-tomcat-${tomcat5x.version}/common/lib">
> + todir="${cargo.install.dir}/${container.name
> }/apache-tomcat-${tomcat5x.version}/apache-tomcat-${tomcat5x.version}/common/lib">
> <fileset
> dir="${project.build.directory}/providedDependencies">
> <include name="**/*.jar" />
> </fileset>
> @@ -317,7 +317,7 @@
> <containerId>${container.name}</containerId>
> <zipUrlInstaller>
> <url>${container.url}</url>
> - <installDir>${project.build.directory}/installs/${
> container.name}</installDir>
> + <installDir>${cargo.install.dir}/${container.name
> }</installDir>
> </zipUrlInstaller>
> <log>${project.build.directory}/logs/${container.name
> }.log</log>
> <output>${project.build.directory}/logs/${container.name
> }.out</output>
> @@ -443,6 +443,24 @@
> </properties>
> </profile>
> <profile>
> + <id>clean-cargo</id>
> + <build>
> + <plugins>
> + <plugin>
> + <groupId>org.apache.maven.plugins</groupId>
> + <artifactId>maven-clean-plugin</artifactId>
> + <configuration>
> + <filesets>
> + <fileset>
> + <directory>${cargo.install.dir}</directory>
> + </fileset>
> + </filesets>
> + </configuration>
> + </plugin>
> + </plugins>
> + </build>
> + </profile>
> + <profile>
> <id>headless</id>
> <build>
> <plugins>
> @@ -466,5 +484,6 @@
> <properties>
> <tomcat5x.version>5.5.27</tomcat5x.version>
> <cargo.wait>false</cargo.wait>
> + <cargo.install.dir>cargo-installs</cargo.install.dir>
> </properties>
> </project>
>
>
>

Re: Failing unit tests on Windows

2009-11-02T17:55:43Z

On Mon, Nov 2, 2009 at 5:45 PM, Deng Ching <oching@...> wrote:
> Hi Martin,
>
> I'll look into these failing tests.

Thanks!

> Do you know which module who's tests are
> failing due to checking for the day of the month?

The problem is in AbstractUpdatePolicy, in the DAILY section of
applyPolicy. There's an adjustment subtracting 1 from DAY_OF_MONTH
that seems to do strange things on the 1st of the month.

> (I think this calls for a Windows build agent in our CI server (Continuum)
> in vmbuild so we could keep track of these..)

Funny - that's what Brett said a few minutes ago. :-)

> Thanks for the heads up! :)

You're welcome. :-)

--
Martin Cooper

> -Deng
>
> On Tue, Nov 3, 2009 at 8:21 AM, Martin Cooper <martinc@...> wrote:
>
>> I'm trying to build Archiva for the first time in a long time, and am
>> running into some unit test failures that are quite probably Windows
>> only. The problem is that trying to track these down quickly leads
>> into Nexus code and thence into Lucene code, so I'm looking for ideas
>> from someone who knows the code better than I do (which is probably
>> pretty much everyone else here).
>>
>> The (first) problem occurs in ArchivaIndexingTaskExecutorTest, in the
>> tearDown method. Here's the relevant chunk of code:
>>
>> context.close( false );
>>
>> // delete created index in the repository
>> File indexDir = new File( repositoryConfig.getLocation(), ".indexer"
>> );
>> FileUtils.deleteDirectory( indexDir );
>> assertFalse( indexDir.exists() );
>>
>> The deleteDirectory() call throws an exception because it cannot
>> delete a file (_1.cfs) in that directory, which would happen when the
>> file is still open. This means that either the context.close() method
>> didn't do its thing properly or something earlier on didn't clean up
>> after itself. Unfortunately, that context.close() call leads directly
>> into Nexus, and that in turn leads quickly into Lucene. I'm not about
>> to start debugging either of those. (Debugging this is hard enough
>> when I can't generate the IDEA project because I can't build Archiva
>> in the first place!)
>>
>> I'm not sure where to go from here. Skipping the deletion causes
>> subsequent failures, and since the problem is an unclosed file,
>> there's a bug somewhere that needs to be fixed. Almost certainly,
>> other people are not seeing it because *nix has different ideas about
>> deleting open files.
>>
>> Oh - there are other "nearby" tests that are failing, but I'm not sure
>> if they're related to this problem or not. Also, there are other
>> non-nearby test failures that I was trying to track down yesterday,
>> but those are not showing up today because it is no longer the 1st of
>> the month. (Yes, I'm serious.)
>>
>> Thoughts?
>>
>> --
>> Martin Cooper
>>
>

Re: Failing unit tests on Windows

2009-11-02T17:45:32Z

Hi Martin,

I'll look into these failing tests. Do you know which module who's tests are
failing due to checking for the day of the month?
(I think this calls for a Windows build agent in our CI server (Continuum)
in vmbuild so we could keep track of these..)

Thanks for the heads up! :)

-Deng

On Tue, Nov 3, 2009 at 8:21 AM, Martin Cooper <martinc@...> wrote:

> I'm trying to build Archiva for the first time in a long time, and am
> running into some unit test failures that are quite probably Windows
> only. The problem is that trying to track these down quickly leads
> into Nexus code and thence into Lucene code, so I'm looking for ideas
> from someone who knows the code better than I do (which is probably
> pretty much everyone else here).
>
> The (first) problem occurs in ArchivaIndexingTaskExecutorTest, in the
> tearDown method. Here's the relevant chunk of code:
>
> context.close( false );
>
> // delete created index in the repository
> File indexDir = new File( repositoryConfig.getLocation(), ".indexer"
> );
> FileUtils.deleteDirectory( indexDir );
> assertFalse( indexDir.exists() );
>
> The deleteDirectory() call throws an exception because it cannot
> delete a file (_1.cfs) in that directory, which would happen when the
> file is still open. This means that either the context.close() method
> didn't do its thing properly or something earlier on didn't clean up
> after itself. Unfortunately, that context.close() call leads directly
> into Nexus, and that in turn leads quickly into Lucene. I'm not about
> to start debugging either of those. (Debugging this is hard enough
> when I can't generate the IDEA project because I can't build Archiva
> in the first place!)
>
> I'm not sure where to go from here. Skipping the deletion causes
> subsequent failures, and since the problem is an unclosed file,
> there's a bug somewhere that needs to be fixed. Almost certainly,
> other people are not seeing it because *nix has different ideas about
> deleting open files.
>
> Oh - there are other "nearby" tests that are failing, but I'm not sure
> if they're related to this problem or not. Also, there are other
> non-nearby test failures that I was trying to track down yesterday,
> but those are not showing up today because it is no longer the 1st of
> the month. (Yes, I'm serious.)
>
> Thoughts?
>
> --
> Martin Cooper
>

Failing unit tests on Windows

2009-11-02T16:21:14Z

I'm trying to build Archiva for the first time in a long time, and am
running into some unit test failures that are quite probably Windows
only. The problem is that trying to track these down quickly leads
into Nexus code and thence into Lucene code, so I'm looking for ideas
from someone who knows the code better than I do (which is probably
pretty much everyone else here).

The (first) problem occurs in ArchivaIndexingTaskExecutorTest, in the
tearDown method. Here's the relevant chunk of code:

context.close( false );

// delete created index in the repository
File indexDir = new File( repositoryConfig.getLocation(), ".indexer" );
FileUtils.deleteDirectory( indexDir );
assertFalse( indexDir.exists() );

The deleteDirectory() call throws an exception because it cannot
delete a file (_1.cfs) in that directory, which would happen when the
file is still open. This means that either the context.close() method
didn't do its thing properly or something earlier on didn't clean up
after itself. Unfortunately, that context.close() call leads directly
into Nexus, and that in turn leads quickly into Lucene. I'm not about
to start debugging either of those. (Debugging this is hard enough
when I can't generate the IDEA project because I can't build Archiva
in the first place!)

I'm not sure where to go from here. Skipping the deletion causes
subsequent failures, and since the problem is an unclosed file,
there's a bug somewhere that needs to be fixed. Almost certainly,
other people are not seeing it because *nix has different ideas about
deleting open files.

Oh - there are other "nearby" tests that are failing, but I'm not sure
if they're related to this problem or not. Also, there are other
non-nearby test failures that I was trying to track down yesterday,
but those are not showing up today because it is no longer the 1st of
the month. (Yes, I'm serious.)

Thoughts?

--
Martin Cooper

Re: Proposal: concurrent remote-requests / "ASF Certified Maven Repository"

2009-10-18T23:43:56Z

On 15/10/2009, at 7:00 PM, Marc Lustig wrote:

> For companies, this would be a compelling feature! I (working for
> insurances
> and banks) often hear the argument "of boy - they are downloading
> software
> from some obscure server from russia". Having the label "Certified
> Maven
> Repository" would surely make those noises more silent :-)
> The ASF could release a rule-set that the Maven-repo must conform to
> in
> order to get the "certified" label.

This isn't really in the ASF's mission to provide. Everyone is going
to have their own rules for what is certified - there are varying
levels of trust, even if you verify it comes from the project itself
(for example, see Eclipse's IP verification process).

In this case you are better off having a dedicated group of people
approving third party artifacts to arrive into Archiva for use by
others, and limiting proxy access outside. You can obviously do this
manually in Archiva now, though ideally you want a "quarantine" area
where they can be retrieved and await approval with a decent workflow
for moving them into an accessible repository.

- Brett

Re: Proposal: concurrent remote-requests / "ASF Certified Maven Repository"

2009-10-15T01:00:35Z

thanks Brett for the input.
I can confirm that using black and white lists the case is rather rare when all remote-repos are searched sequentially and the artifact is not found in the end. However it is typical for some scenarios e. g. when you enable the source-jars to get downloaded for a project. From 40 deps, maybe 5 will have source-jars available. In that way a simple mvn-goal takes 30 minutes or more.

I mentioned the timeout just to have a maximum value. Of course usually the requests don't run in a timeout (except when the repo is down) - the average response time is maybe 3-4 secs (for our installation).

Also it is clear that the first-serve concept conflicts with the existing concept of an (ordered) list of repos that is searched for.
Can we not assume that artifacts with a given spec. are identical from whatever repo they come, provided the hash is matching?

Btw., this brings up another idea: could the ASF possibly grant "official" certificates for remote-repos?
In that way, Archiva could distinguish between trusted and non-trusted repos.
For companies, this would be a compelling feature! I (working for insurances and banks) often hear the argument "of boy - they are downloading software from some obscure server from russia". Having the label "Certified Maven Repository" would surely make those noises more silent :-)
The ASF could release a rule-set that the Maven-repo must conform to in order to get the "certified" label.
Or even better, the ASF could offer a VMware-image that includes all the software ready to run the Maven-repo - including some logic to verify that known artifacts are mirrored correctly. A total control of repos is not possible, of course. But the contract between Archiva and the remote repo could be tightened pretty much.

Back to the concurrent requests idea: sending the HEAD request before the actual GET is surely a good idea. Archiva could decide to which repo to send the GET based on the shortest response-time.
Anyway, this feature needs more brainstorming...

brettporter wrote:

On 15/10/2009, at 12:06 AM, Marc Lustig wrote:

>
> Hi all,
>
> we have configured about 25 remote-repos for our public-artifacts
> managed
> repo.
> In certain cases, black and white lists don't help and a request is
> proxied
> to all the 20 remote-repos _sequentially_. Even thou we have
> configured a
> short timeout of 5 secs, this takes 125 secs in case the artifacts
> doesn't
> exist in any remote-repo - per artifact!
>
> So I was wondering if it would make sense to send requests to all of
> the
> remote-repos _concurrently_.
> The first thread that find the artifacts could cause all the other
> threads
> to cancel the http-request.
> The total request time would reduce from 100 secs++ to merely 5 secs.
> Tremendous win or?
>
> Has this been discussed before?

I think this is a pretty unusual case. I don't quite understand why
you are hitting the timeout limit on the remote repo - if they are up
they should be fast. Also, "first that finds" is different to the
current rule since it's first that appears in the list. I worry that
in this set up you're not entirely sure which repository the artifacts
are meant to be coming from, so maybe it points to another problem.

> Is there an argument against this strategy?

Particularly if we turned on streaming of the proxied download to the
client (which is intended) - we couldn't do so if they were pooled
like this, unless we accepted the "first found rule".

That said, this might speed up requests with a long list of proxies,
even if they are functioning properly. So it might be reasonable as an
optional capability. One thing to consider would be doing a HEAD
request instead of a GET for all the remotes first to select where to
download from, then execute the GET from the desired one.

- Brett

Re: svn commit: r824677 - in /archiva/trunk/archiva-modules/archiva-web/archiva-webdav/src: main/java/org/apache/maven/archiva/webdav/ test/java/org/apache/maven/archiva/webdav/ test/resources/

2009-10-15T00:08:41Z

On Wed, Oct 14, 2009 at 11:40 PM, Brett Porter <brett@...> wrote:

>
>
>>> // MRM-872 : merge all available metadata
>>>
>>>> // merge metadata only when requested via the repo group
>>>> - if ( ( repositoryRequest.isMetadata( requestedResource ) || (
>>>> requestedResource.endsWith( "metadata.xml.sha1" ) ||
>>>> requestedResource.endsWith( "metadata.xml.md5" ) ) )
>>>> - && repoGroupConfig != null )
>>>> + if ( ( repositoryRequest.isMetadata( requestedResource ) || (
>>>> requestedResource.endsWith( "metadata.xml.sha1" ) ||
>>>> requestedResource.endsWith( "metadata.xml.md5" ) ) ) &&
>>>> + repoGroupConfig != null )
>>>>
>>>>
>>> Should this use "isSupportFile" like below? That will cover the two
>>> metadata checksums
>>>
>>
>>
>> .. but it will also get the other non-metadata checksum files so I don't
>> think we can use isSupportFile(..) here
>>
>>
> ok - could the check be moved to the repository request (eg,
> isMetadataSupportFile), so that it is all in one spot?

ok, will move this one over..

>
>
>
>>
>>>
>>> @@ -482,6 +496,35 @@
>>>
>>>>
>>>> if ( request.getMethod().equals( HTTP_PUT_METHOD ) )
>>>> {
>>>> + String resourcePath = logicalResource.getPath();
>>>> +
>>>> + // check if target repo is enabled for releases
>>>> + // we suppose that release-artifacts can deployed only
>>>> to
>>>> repos enabled for releases
>>>> + if ( managedRepository.getRepository().isReleases() &&
>>>> !repositoryRequest.isMetadata( resourcePath ) &&
>>>> + !repositoryRequest.isSupportFile( resourcePath ) )
>>>> + {
>>>> + ArtifactReference artifact = null;
>>>> + try
>>>> + {
>>>> + artifact =
>>>> managedRepository.toArtifactReference(
>>>> resourcePath );
>>>> + }
>>>> + catch ( LayoutException e )
>>>> + {
>>>> + throw new DavException(
>>>> HttpServletResponse.SC_BAD_REQUEST, e );
>>>> + }
>>>> +
>>>> + if ( !VersionUtil.isSnapshot( artifact.getVersion()
>>>> )
>>>> )
>>>> + {
>>>> + // check if artifact already exists
>>>> + if ( managedRepository.hasContent( artifact ) )
>>>> + {
>>>> + log.warn( "Overwriting released artifacts
>>>> is
>>>> not allowed." );
>>>> + throw new
>>>> ReleaseArtifactAlreadyExistsException( managedRepository.getId(),
>>>> +
>>>> "Overwriting released artifacts is not allowed." );
>>>> + }
>>>> + }
>>>> + }
>>>> +
>>>>
>>>>
>>> Is it necessarily a bad request if the reference can't be derived, or
>>> should the check just be skipped?
>>>
>>>
>> I don't think this is just a check though but it's for getting the
>> artifact
>> object and its coordinates. Maybe we could add a fall back for getting the
>> artifact obj & its coordinates when a LayoutException is thrown instead of
>> immediately propagating it as a bad request error?
>>
>
> The check I meant was the VersionUtil bit.
>
> Say you are trying to store /foo.txt using webdav, which is not a valid
> artifact. I believe this was still allowed previously (though I might be
> wrong) - but now it will be a bad request because of the artifact path, when
> all it needs that for is to check if it is a release. Does that make sense?

Ok, I get it now :) I think we still need to check if the version is a
SNAPSHOT or not though, and block deployment if it is a released artifact or
an invalid artifact (LayoutException is thrown) as long as it already exists
in the repository. We're also using the artifact reference to check if it is
in the repository so it's not just the version we're using. I'll see if
there's another way to do this.

Thanks,
Deng

Re: Proposal: concurrent remote-requests

2009-10-14T16:02:30Z

On 15/10/2009, at 12:06 AM, Marc Lustig wrote:

>
> Hi all,
>
> we have configured about 25 remote-repos for our public-artifacts
> managed
> repo.
> In certain cases, black and white lists don't help and a request is
> proxied
> to all the 20 remote-repos _sequentially_. Even thou we have
> configured a
> short timeout of 5 secs, this takes 125 secs in case the artifacts
> doesn't
> exist in any remote-repo - per artifact!
>
> So I was wondering if it would make sense to send requests to all of
> the
> remote-repos _concurrently_.
> The first thread that find the artifacts could cause all the other
> threads
> to cancel the http-request.
> The total request time would reduce from 100 secs++ to merely 5 secs.
> Tremendous win or?
>
> Has this been discussed before?

I think this is a pretty unusual case. I don't quite understand why
you are hitting the timeout limit on the remote repo - if they are up
they should be fast. Also, "first that finds" is different to the
current rule since it's first that appears in the list. I worry that
in this set up you're not entirely sure which repository the artifacts
are meant to be coming from, so maybe it points to another problem.

> Is there an argument against this strategy?

Particularly if we turned on streaming of the proxied download to the
client (which is intended) - we couldn't do so if they were pooled
like this, unless we accepted the "first found rule".

That said, this might speed up requests with a long list of proxies,
even if they are functioning properly. So it might be reasonable as an
optional capability. One thing to consider would be doing a HEAD
request instead of a GET for all the remotes first to select where to
download from, then execute the GET from the desired one.

- Brett

Re: svn commit: r824677 - in /archiva/trunk/archiva-modules/archiva-web/archiva-webdav/src: main/java/org/apache/maven/archiva/webdav/ test/java/org/apache/maven/archiva/webdav/ test/resources/

2009-10-14T08:40:59Z

On 14/10/2009, at 7:03 PM, Deng Ching wrote:

> Hi Brett,
>
> On Tue, Oct 13, 2009 at 8:49 PM, Brett Porter <brett@...>
> wrote:
>
>>
>> On 13/10/2009, at 9:36 PM, oching@... wrote:
>>
>> - resource =
>>> - processRepositoryGroup( request,
>>> archivaLocator,
>>> repoGroupConfig.getRepositories(),
>>> - activePrincipal,
>>> resourcesInAbsolutePath );
>>> + try
>>> + {
>>> + resource =
>>> + processRepositoryGroup( request,
>>> archivaLocator,
>>> repoGroupConfig.getRepositories(),
>>> + activePrincipal,
>>> resourcesInAbsolutePath );
>>> + }
>>> + catch ( ReleaseArtifactAlreadyExistsException e )
>>> + {
>>> + throw new DavException(
>>> HttpServletResponse.SC_CONFLICT );
>>> + }
>>>
>>
>>
>> it might make more sense just to throw this at the source and
>> eliminate the
>> exception, since the result is always the same?
>
>
> agreed :)
>
>
>>
>> // MRM-872 : merge all available metadata
>>> // merge metadata only when requested via the repo group
>>> - if ( ( repositoryRequest.isMetadata( requestedResource )
>>> || (
>>> requestedResource.endsWith( "metadata.xml.sha1" ) ||
>>> requestedResource.endsWith( "metadata.xml.md5" ) ) )
>>> - && repoGroupConfig != null )
>>> + if ( ( repositoryRequest.isMetadata( requestedResource )
>>> || (
>>> requestedResource.endsWith( "metadata.xml.sha1" ) ||
>>> requestedResource.endsWith( "metadata.xml.md5" ) ) ) &&
>>> + repoGroupConfig != null )
>>>
>>
>> Should this use "isSupportFile" like below? That will cover the two
>> metadata checksums
>
>
> .. but it will also get the other non-metadata checksum files so I
> don't
> think we can use isSupportFile(..) here
>

ok - could the check be moved to the repository request (eg,
isMetadataSupportFile), so that it is all in one spot?

>
>>
>>
>> @@ -482,6 +496,35 @@
>>>
>>> if ( request.getMethod().equals( HTTP_PUT_METHOD ) )
>>> {
>>> + String resourcePath = logicalResource.getPath();
>>> +
>>> + // check if target repo is enabled for releases
>>> + // we suppose that release-artifacts can deployed
>>> only to
>>> repos enabled for releases
>>> + if ( managedRepository.getRepository().isReleases
>>> () &&
>>> !repositoryRequest.isMetadata( resourcePath ) &&
>>> + !repositoryRequest.isSupportFile
>>> ( resourcePath ) )
>>> + {
>>> + ArtifactReference artifact = null;
>>> + try
>>> + {
>>> + artifact =
>>> managedRepository.toArtifactReference(
>>> resourcePath );
>>> + }
>>> + catch ( LayoutException e )
>>> + {
>>> + throw new DavException(
>>> HttpServletResponse.SC_BAD_REQUEST, e );
>>> + }
>>> +
>>> + if ( !VersionUtil.isSnapshot
>>> ( artifact.getVersion() )
>>> )
>>> + {
>>> + // check if artifact already exists
>>> + if ( managedRepository.hasContent
>>> ( artifact ) )
>>> + {
>>> + log.warn( "Overwriting released
>>> artifacts is
>>> not allowed." );
>>> + throw new
>>> ReleaseArtifactAlreadyExistsException( managedRepository.getId(),
>>> +
>>> "Overwriting released artifacts is not allowed." );
>>> + }
>>> + }
>>> + }
>>> +
>>>
>>
>> Is it necessarily a bad request if the reference can't be derived, or
>> should the check just be skipped?
>>
>
> I don't think this is just a check though but it's for getting the
> artifact
> object and its coordinates. Maybe we could add a fall back for
> getting the
> artifact obj & its coordinates when a LayoutException is thrown
> instead of
> immediately propagating it as a bad request error?

The check I meant was the VersionUtil bit.

Say you are trying to store /foo.txt using webdav, which is not a
valid artifact. I believe this was still allowed previously (though I
might be wrong) - but now it will be a bad request because of the
artifact path, when all it needs that for is to check if it is a
release. Does that make sense?

>
>
>> Also, given this is a point release (1.2.3), I don't think this
>> kind of
>> functionality change should be imposed on users - can we offer a
>> configuration option?
>>
>
> +1
> I've left the issue open yesterday since this functionality also
> needs to be
> applied to the web upload.
>
> Thanks,
> Deng
>

Thanks!

- Brett

Proposal: concurrent remote-requests

2009-10-14T06:06:18Z

Hi all,

we have configured about 25 remote-repos for our public-artifacts managed repo.
In certain cases, black and white lists don't help and a request is proxied to all the 20 remote-repos _sequentially_. Even thou we have configured a short timeout of 5 secs, this takes 125 secs in case the artifacts doesn't exist in any remote-repo - per artifact!

So I was wondering if it would make sense to send requests to all of the remote-repos _concurrently_.
The first thread that find the artifacts could cause all the other threads to cancel the http-request.
The total request time would reduce from 100 secs++ to merely 5 secs. Tremendous win or?

Has this been discussed before?
Is there an argument against this strategy?

The implementation could be based on a thread-pool, or rather a pool of thread-pools.

greetings
Marc

Re: svn commit: r824677 - in /archiva/trunk/archiva-modules/archiva-web/archiva-webdav/src: main/java/org/apache/maven/archiva/webdav/ test/java/org/apache/maven/archiva/webdav/ test/resources/

2009-10-14T01:03:00Z

Hi Brett,

On Tue, Oct 13, 2009 at 8:49 PM, Brett Porter <brett@...> wrote:

>
> On 13/10/2009, at 9:36 PM, oching@... wrote:
>
> - resource =
>> - processRepositoryGroup( request, archivaLocator,
>> repoGroupConfig.getRepositories(),
>> - activePrincipal,
>> resourcesInAbsolutePath );
>> + try
>> + {
>> + resource =
>> + processRepositoryGroup( request, archivaLocator,
>> repoGroupConfig.getRepositories(),
>> + activePrincipal,
>> resourcesInAbsolutePath );
>> + }
>> + catch ( ReleaseArtifactAlreadyExistsException e )
>> + {
>> + throw new DavException(
>> HttpServletResponse.SC_CONFLICT );
>> + }
>>
>
>
> it might make more sense just to throw this at the source and eliminate the
> exception, since the result is always the same?

agreed :)

>
> // MRM-872 : merge all available metadata
>> // merge metadata only when requested via the repo group
>> - if ( ( repositoryRequest.isMetadata( requestedResource ) || (
>> requestedResource.endsWith( "metadata.xml.sha1" ) ||
>> requestedResource.endsWith( "metadata.xml.md5" ) ) )
>> - && repoGroupConfig != null )
>> + if ( ( repositoryRequest.isMetadata( requestedResource ) || (
>> requestedResource.endsWith( "metadata.xml.sha1" ) ||
>> requestedResource.endsWith( "metadata.xml.md5" ) ) ) &&
>> + repoGroupConfig != null )
>>
>
> Should this use "isSupportFile" like below? That will cover the two
> metadata checksums

.. but it will also get the other non-metadata checksum files so I don't
think we can use isSupportFile(..) here

>
>
> @@ -482,6 +496,35 @@
>>
>> if ( request.getMethod().equals( HTTP_PUT_METHOD ) )
>> {
>> + String resourcePath = logicalResource.getPath();
>> +
>> + // check if target repo is enabled for releases
>> + // we suppose that release-artifacts can deployed only to
>> repos enabled for releases
>> + if ( managedRepository.getRepository().isReleases() &&
>> !repositoryRequest.isMetadata( resourcePath ) &&
>> + !repositoryRequest.isSupportFile( resourcePath ) )
>> + {
>> + ArtifactReference artifact = null;
>> + try
>> + {
>> + artifact = managedRepository.toArtifactReference(
>> resourcePath );
>> + }
>> + catch ( LayoutException e )
>> + {
>> + throw new DavException(
>> HttpServletResponse.SC_BAD_REQUEST, e );
>> + }
>> +
>> + if ( !VersionUtil.isSnapshot( artifact.getVersion() )
>> )
>> + {
>> + // check if artifact already exists
>> + if ( managedRepository.hasContent( artifact ) )
>> + {
>> + log.warn( "Overwriting released artifacts is
>> not allowed." );
>> + throw new
>> ReleaseArtifactAlreadyExistsException( managedRepository.getId(),
>> +
>> "Overwriting released artifacts is not allowed." );
>> + }
>> + }
>> + }
>> +
>>
>
> Is it necessarily a bad request if the reference can't be derived, or
> should the check just be skipped?
>

I don't think this is just a check though but it's for getting the artifact
object and its coordinates. Maybe we could add a fall back for getting the
artifact obj & its coordinates when a LayoutException is thrown instead of
immediately propagating it as a bad request error?

> Also, given this is a point release (1.2.3), I don't think this kind of
> functionality change should be imposed on users - can we offer a
> configuration option?
>

+1
I've left the issue open yesterday since this functionality also needs to be
applied to the web upload.

Thanks,
Deng

> Cheers,
> Brett
>
>

Re: svn commit: r824677 - in /archiva/trunk/archiva-modules/archiva-web/archiva-webdav/src: main/java/org/apache/maven/archiva/webdav/ test/java/org/apache/maven/archiva/webdav/ test/resources/

2009-10-13T05:49:17Z

On 13/10/2009, at 9:36 PM, oching@... wrote:

> - resource =
> - processRepositoryGroup( request,
> archivaLocator, repoGroupConfig.getRepositories(),
> - activePrincipal,
> resourcesInAbsolutePath );
> + try
> + {
> + resource =
> + processRepositoryGroup( request,
> archivaLocator, repoGroupConfig.getRepositories(),
> + activePrincipal,
> resourcesInAbsolutePath );
> + }
> + catch ( ReleaseArtifactAlreadyExistsException e )
> + {
> + throw new DavException
> ( HttpServletResponse.SC_CONFLICT );
> + }

it might make more sense just to throw this at the source and
eliminate the exception, since the result is always the same?

> // MRM-872 : merge all available metadata
> // merge metadata only when requested via the repo group
> - if ( ( repositoryRequest.isMetadata( requestedResource ) ||
> ( requestedResource.endsWith( "metadata.xml.sha1" ) ||
> requestedResource.endsWith( "metadata.xml.md5" ) ) )
> - && repoGroupConfig != null )
> + if ( ( repositoryRequest.isMetadata( requestedResource ) ||
> ( requestedResource.endsWith( "metadata.xml.sha1" ) ||
> requestedResource.endsWith( "metadata.xml.md5" ) ) ) &&
> + repoGroupConfig != null )

Should this use "isSupportFile" like below? That will cover the two
metadata checksums

> @@ -482,6 +496,35 @@
>
> if ( request.getMethod().equals( HTTP_PUT_METHOD ) )
> {
> + String resourcePath = logicalResource.getPath();
> +
> + // check if target repo is enabled for releases
> + // we suppose that release-artifacts can deployed
> only to repos enabled for releases
> + if ( managedRepository.getRepository().isReleases()
> && !repositoryRequest.isMetadata( resourcePath ) &&
> + !repositoryRequest.isSupportFile
> ( resourcePath ) )
> + {
> + ArtifactReference artifact = null;
> + try
> + {
> + artifact =
> managedRepository.toArtifactReference( resourcePath );
> + }
> + catch ( LayoutException e )
> + {
> + throw new DavException
> ( HttpServletResponse.SC_BAD_REQUEST, e );
> + }
> +
> + if ( !VersionUtil.isSnapshot
> ( artifact.getVersion() ) )
> + {
> + // check if artifact already exists
> + if ( managedRepository.hasContent
> ( artifact ) )
> + {
> + log.warn( "Overwriting released
> artifacts is not allowed." );
> + throw new
> ReleaseArtifactAlreadyExistsException( managedRepository.getId(),
> +
> "Overwriting
> released artifacts is not allowed." );
> + }
> + }
> + }
> +

Is it necessarily a bad request if the reference can't be derived, or
should the check just be skipped?

Also, given this is a point release (1.2.3), I don't think this kind
of functionality change should be imposed on users - can we offer a
configuration option?

Cheers,
Brett

Re: helping to implement MRM-1256

2009-10-13T05:31:34Z

On 13/10/2009, at 1:02 AM, Marc Lustig wrote:

>
>
>
> brettporter wrote:
>>
>>
>> On 13/10/2009, at 12:25 AM, Marc Lustig wrote:
>>
>>>
>>> can I help to implement this?
>>
>> of course!
>>
>>>
>>> What "event ideas for 1.3" was Brett refering to ?
>>> any existing issue?
>>
>> http://cwiki.apache.org/confluence/display/ARCHIVA/Event+Framework
>> http://jira.codehaus.org/browse/MRM-976
>> http://svn.apache.org/viewvc/archiva/branches/MRM-976/branch-working-notes.txt?revision=706263&view=markup
>>
>> Unfortunately work on this stalled some time back.
>>
>>>
>>> I would prefer a generic solution to implement extension-point, in
>>> the way I
>>> proposed it.
>>> One or another consumer may be used perhaps for one or another task.
>>> But consumers run only on repo-scan, right?
>>> Action events triggered on artifact-deployment are a different thing
>>> and not
>>> less important.
>>
>> I'd say if you've got an idea to go ahead with that - we can look at
>> it early and often and look to incorporate it. The above, if picked
>> up
>> again, can still support it afterwards. Thanks!
>>
>> Cheers,
>> Brett
>>
>
> thanks Brett for the references.
> So you mean the consumers-architecture could be and should be
> replaced by a
> generic event-framework, that also provides other extension-points
> like the
> ones I suggested?

That's my impression, yes.

> That sounds pretty cool to me - but it's also a severe change -
> looks like a
> bunch of work ;-)

Yes, and no. On the database refactoring branch I put in something
very basic - I think if we start to introduce a central point to
gather the handling, and introduce them in specific cases like the one
you want, we can gradually introduce it.

> Perhaps we can share the work load? Would you like to go ahead and
> coordinate it?

I'm a bit swamped up until ApacheCon next month, but I'm happy to help
you get started here, and I'm sure others are interested. If you
wanted to share some ideas for how it might work I'm sure we can get
something happening.

Cheers,
Brett

Re: svn commit: r824220 - in /archiva/trunk/archiva-modules/archiva-base/archiva-repository-layer/src: main/java/org/apache/maven/archiva/repository/scanner/functors/ test/java/org/apache/maven/archiva/repository/metadata/ test/java/org/apache/maven/

2009-10-12T21:08:31Z

On 13/10/2009, at 2:52 PM, Jevica Arianne Zurbano wrote:

> On Tue, Oct 13, 2009 at 8:12 AM, Brett Porter <brett@...>
> wrote:
>
>>
>> On 12/10/2009, at 3:29 PM, jzurbano@... wrote:
>>
>> -
>>> /**
>>> * ConsumerWantsFilePredicate
>>> *
>>> @@ -62,8 +63,19 @@
>>> // Timestamp finished points to the last
>>> successful
>>> scan, not this current one.
>>> if ( basefile.lastModified() < changesSince )
>>> {
>>> - // Skip file as no change has occured.
>>> - satisfies = false;
>>> + // MRM-1246
>>> + // compares the lastModified of the
>>> version-level
>>> (basefile) and the project-level (parent) metadata
>>> + File parent =
>>> basefile.getParentFile().getParentFile();
>>> +
>>> + if ( parent.lastModified() >
>>> basefile.lastModified() )
>>> + {
>>> + satisfies = true;
>>> + }
>>> + else
>>> + {
>>> + // Skip file as no change has occurred.
>>> + satisfies = false;
>>> + }
>>> }
>>>
>>
>> is this doing what it intends? It seems to be comparing the
>> timestamp of
>> the parent directory, not the parent metadata file, and not all files
>> passing through the predicate are metadata files.
>>
>
> It does update the parent metadata file. But I missed that not all
> are of
> metadata files.
> Will make modifications for this to check the consumer type.

but basefile.getParentFile().getParentFile() is ".." for "maven-
metadata.xml", ie a directory. I don't think it is checking the right
timestamp. I also don't think building knowledge of the repository
format into this level of abstraction is appropriate.

>
>
>>
>> Does disabling this change cause the tests to fail to verify they are
>> working?
>>
>
> I added a test to check the metadata before and after it has been
> updated.
>
> But I haven't added a test on the fix yet. I am still figuring out
> where to
> put the test since
> the test should include scanning with the consumer.
>
> However, the repo scanning tests are ran before the consumers are
> built.
>
> Any thoughts on this?

Usually you'd use a mock consumer to test whether it is called when
expected.

- Brett

Re: svn commit: r824220 - in /archiva/trunk/archiva-modules/archiva-base/archiva-repository-layer/src: main/java/org/apache/maven/archiva/repository/scanner/functors/ test/java/org/apache/maven/archiva/repository/metadata/ test/java/org/apache/maven/

2009-10-12T20:52:26Z

On Tue, Oct 13, 2009 at 8:12 AM, Brett Porter <brett@...> wrote:

>
> On 12/10/2009, at 3:29 PM, jzurbano@... wrote:
>
> -
>> /**
>> * ConsumerWantsFilePredicate
>> *
>> @@ -62,8 +63,19 @@
>> // Timestamp finished points to the last successful
>> scan, not this current one.
>> if ( basefile.lastModified() < changesSince )
>> {
>> - // Skip file as no change has occured.
>> - satisfies = false;
>> + // MRM-1246
>> + // compares the lastModified of the version-level
>> (basefile) and the project-level (parent) metadata
>> + File parent =
>> basefile.getParentFile().getParentFile();
>> +
>> + if ( parent.lastModified() >
>> basefile.lastModified() )
>> + {
>> + satisfies = true;
>> + }
>> + else
>> + {
>> + // Skip file as no change has occurred.
>> + satisfies = false;
>> + }
>> }
>>
>
> is this doing what it intends? It seems to be comparing the timestamp of
> the parent directory, not the parent metadata file, and not all files
> passing through the predicate are metadata files.
>

It does update the parent metadata file. But I missed that not all are of
metadata files.
Will make modifications for this to check the consumer type.

>
> Does disabling this change cause the tests to fail to verify they are
> working?
>

I added a test to check the metadata before and after it has been updated.

But I haven't added a test on the fix yet. I am still figuring out where to
put the test since
the test should include scanning with the consumer.

However, the repo scanning tests are ran before the consumers are built.

Any thoughts on this?

>
> - Brett
>
>

Re: svn commit: r824220 - in /archiva/trunk/archiva-modules/archiva-base/archiva-repository-layer/src: main/java/org/apache/maven/archiva/repository/scanner/functors/ test/java/org/apache/maven/archiva/repository/metadata/ test/java/org/apache/maven/archiv...

2009-10-12T17:12:39Z

On 12/10/2009, at 3:29 PM, jzurbano@... wrote:

> -
> /**
> * ConsumerWantsFilePredicate
> *
> @@ -62,8 +63,19 @@
> // Timestamp finished points to the last
> successful scan, not this current one.
> if ( basefile.lastModified() < changesSince )
> {
> - // Skip file as no change has occured.
> - satisfies = false;
> + // MRM-1246
> + // compares the lastModified of the version-
> level (basefile) and the project-level (parent) metadata
> + File parent = basefile.getParentFile
> ().getParentFile();
> +
> + if ( parent.lastModified() >
> basefile.lastModified() )
> + {
> + satisfies = true;
> + }
> + else
> + {
> + // Skip file as no change has occurred.
> + satisfies = false;
> + }
> }

is this doing what it intends? It seems to be comparing the timestamp
of the parent directory, not the parent metadata file, and not all
files passing through the predicate are metadata files.

Does disabling this change cause the tests to fail to verify they are
working?

- Brett

Re: SvnPubSub websites -- need more volunteers

2009-10-12T15:11:23Z

Anyone?

On 01/10/2009, at 5:02 PM, Brett Porter wrote:

> do we want to try this?
>
> Begin forwarded message:
>
>> From: Paul Querna <paul@...>
>> Date: 1 October 2009 4:46:17 PM AEST
>> To: infrastructure-dev@...
>> Subject: SvnPubSub websites -- need more volunteers
>> Reply-To: infrastructure-dev@...
>> list-id: <infrastructure-dev.apache.org>
>> message-id: <4239a4320909302346m7d76c134j3efeea0ff816119e@...
>> >
>>
>> Hi,
>>
>> apr.apache.org is now managed by SvnPubSub.
>>
>> I think we are ready to add more.
>>
>> Any volunteer TLPs?
>>
>> Once its done, any commit to your SVN repo for your site will be
>> automatically propagated to the live servers within seconds.
>>
>> Things i need to know:
>> - Path to Website Checkout
>> - If applicable, do you need a dev/dist directory
>> - Do you want a $tlp.staging.apache.org (Just maps to another SVN
>> url, like $tlp/site/branche/staging)
>>
>>
>> Notes to self current process:
>> - Add new paths to /etc/svnwcsub.conf
>> - Checkout svn paths to /x1/tmp
>> - mv /x1/www/$tlp old-site && mv /x1/tmp/$tlp /x1/www/$tlp
>> - chown -R svnwc /x1/www/$tlp
>> - /etc/init.d/svnwcsub restart
>

Re: helping to implement MRM-1256

2009-10-12T07:02:21Z

brettporter wrote:

On 13/10/2009, at 12:25 AM, Marc Lustig wrote:

>
> can I help to implement this?

of course!

>
> What "event ideas for 1.3" was Brett refering to ?
> any existing issue?

http://cwiki.apache.org/confluence/display/ARCHIVA/Event+Framework
http://jira.codehaus.org/browse/MRM-976
http://svn.apache.org/viewvc/archiva/branches/MRM-976/branch-working-notes.txt?revision=706263&view=markup

Unfortunately work on this stalled some time back.

>
> I would prefer a generic solution to implement extension-point, in
> the way I
> proposed it.
> One or another consumer may be used perhaps for one or another task.
> But consumers run only on repo-scan, right?
> Action events triggered on artifact-deployment are a different thing
> and not
> less important.

I'd say if you've got an idea to go ahead with that - we can look at
it early and often and look to incorporate it. The above, if picked up
again, can still support it afterwards. Thanks!

Cheers,
Brett

thanks Brett for the references.
So you mean the consumers-architecture could be and should be replaced by a generic event-framework, that also provides other extension-points like the ones I suggested?
That sounds pretty cool to me - but it's also a severe change - looks like a bunch of work ;-)
Perhaps we can share the work load? Would you like to go ahead and coordinate it?

Re: helping to implement MRM-1256

2009-10-12T06:36:53Z

On 13/10/2009, at 12:25 AM, Marc Lustig wrote:

>
> can I help to implement this?

of course!

>
> What "event ideas for 1.3" was Brett refering to ?
> any existing issue?

http://cwiki.apache.org/confluence/display/ARCHIVA/Event+Framework
http://jira.codehaus.org/browse/MRM-976
http://svn.apache.org/viewvc/archiva/branches/MRM-976/branch-working-notes.txt?revision=706263&view=markup

Unfortunately work on this stalled some time back.

>
> I would prefer a generic solution to implement extension-point, in
> the way I
> proposed it.
> One or another consumer may be used perhaps for one or another task.
> But consumers run only on repo-scan, right?
> Action events triggered on artifact-deployment are a different thing
> and not
> less important.

I'd say if you've got an idea to go ahead with that - we can look at
it early and often and look to incorporate it. The above, if picked up
again, can still support it afterwards. Thanks!

Cheers,
Brett

helping to implement MRM-1256

2009-10-12T06:25:00Z

can I help to implement this?

What "event ideas for 1.3" was Brett refering to ?
any existing issue?

I would prefer a generic solution to implement extension-point, in the way I proposed it.
One or another consumer may be used perhaps for one or another task.
But consumers run only on repo-scan, right?
Action events triggered on artifact-deployment are a different thing and not less important.

Re: logic to prevent overwriting release-artifacts

2009-10-09T00:41:08Z

Hi Marc,

I think you can pattern and add your test to the ones for the
RepositoryServlet where all the webdav tests are. They're sort of an
integration test for the webdav components.

Thanks,
Deng

On Fri, Oct 9, 2009 at 2:59 PM, Marc Lustig <ml@...> wrote:

>
> To implement the unit-test it would be quite helpful to execute the
> unit-test
> based on a runtime-environment. Does the Archivas build-process allow to
> execute unit-tests in a runtime-environment?
> Otherwise you will have to mock all kinds of objects, which doesn't reflect
> the real scenario in the end and also is a lot of work.
> --
> View this message in context:
> http://www.nabble.com/logic-to-prevent-overwriting-release-artifacts-tp25564416p25816355.html
> Sent from the archiva-dev mailing list archive at Nabble.com.
>
>

Re: logic to prevent overwriting release-artifacts

2009-10-08T23:59:16Z

To implement the unit-test it would be quite helpful to execute the unit-test based on a runtime-environment. Does the Archivas build-process allow to execute unit-tests in a runtime-environment?
Otherwise you will have to mock all kinds of objects, which doesn't reflect the real scenario in the end and also is a lot of work.

[Results] [VOTE] Re-release Archiva 1.2.2 source binaries

2009-10-08T23:03:27Z

The vote has passed with 4 binding votes:

+1 - Brett, James, Emmanuel, Deng
No 0s and -1s so I'll finalize the release..

Thanks,
Deng

On Wed, Sep 30, 2009 at 10:29 AM, Deng Ching <oching@...> wrote:

> The source binaries of the recently released Archiva 1.2.2 is broken. You
> can see the thread about the corrupt source at
> http://www.nabble.com/Fwd%3A-Corrupt-Source-Zip--td25638356.html
>
> Because of this, the Archiva 1.2.2 source binaries has been re-built again
> from the tag and is staged at:
>
> http://people.apache.org/~oching/archiva-1.2.2-sources/<http://people.apache.org/%7Eoching/archiva-1.2.2-sources/>
>
> Everyone is encouraged to vote and give their feedback..
>
> [ ] +1 Release it!
> [ ] 0
> [ ] -1 Don't release it, because...
>
> The vote will be open for 72 hours.
> Thanks,
> Deng
>
>
>
>

Re: logic to prevent overwriting release-artifacts

2009-10-08T07:06:49Z

Hi all,

please find the patch here: http://jira.codehaus.org/browse/MRM-747

For us it would be important to get this integrated into the trunk asap, so that 1.2.3 will already contain it.
Can I please ask to review the patch and let me know if something should be changed.

thanks
Marc

Deng Ching-2 wrote:

Hi Marc,

Sorry for the confusion, I think the blocking of overwriting released
artifacts should be placed in the ArchivaDavResourceFactory instead of the
RepositoryServlet or RepositoryContentConsumer since that handles the
resource requests and that's where the artifact data is determined. You
might want to take a look at the processRepository(..) method where the PUT
requests are handled.

There's also the upload from the web UI to consider for this feature as it
is entirely separate from the DAV deployment. This is in turn handled by the
UploadAction in archiva-webapp :)

Thanks,
Deng

On Mon, Oct 5, 2009 at 11:58 PM, Marc Lustig <ml@marclustig.com> wrote:

>
> Ok, thanks, so if I understand you correct, we should not the
> RepositoryServlet, but the repo-consumer that you proposed, right?
>
>
> Deng Ching-2 wrote:
> >
> > Hi Marc,
> >
> > Please see in-line comments below :)
> >
> > On Mon, Oct 5, 2009 at 7:32 PM, Marc Lustig <ml@marclustig.com> wrote:
> >
> >>
> >> Deng, Brett, et al
> >>
> >> if you give me some instructions to implement this straight away, I will
> >> be
> >> glad to do that.
> >>
> >>
> >>
> >> Marc Lustig wrote:
> >> >
> >> > yes please - the initial call to deploy the artifact is done in #124
> in
> >> > RepositoryServlet right?
> >> > (Deng mentioned that the repo-consumer is called immediately after
> >> > deploying the artifact, but I guess the check should be done before
> the
> >> > deployment is triggered.)
> >> >
> >> > So could you give me a hint how to retrieve the information necessary
> >> to
> >> > do the check:
> >> > - target repo and location in the repo in the fs (?)
> >>
> >
> > You can get this from ArchivaConfiguration, I think this is already a
> > component of the repo-consumer.
> >
> >
> >> > - artifact data (groupId, artifactId, ...)
> >>
> >
> > This is usually passed in to the repo consumer. If you want to check for
> > the
> > existing artifact in the database, you can get it through the
> > ArtifactDAO..
> >
> >
> >> >
> >> > What is the proper way to do that lookup? via File.exists(), or not
> >> better
> >> > using some Archiva-method like
> >> > SomeStaticSingleton. getRepo(reponame).artfifactExists(groupId,
> >> > artifactId, ....)
> >> >
> >>
> >
> > I think there is a component for this in the repository-layer module,
> I'll
> > get back to you on this one :)
> >
> > Thanks,
> > Deng
> >
> >
> >> > That would be better software design than looking up in the fs...
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > brettporter wrote:
> >> >>
> >> >>
> >> >> On 25/09/2009, at 7:56 PM, Marc Lustig wrote:
> >> >>
> >> >>>
> >> >>> OK, I agree that instead of adding a permission for overwriting
> >> >>> artifacts it
> >> >>> should be sufficient to delete that particlar artifact. The case
> >> >>> when you
> >> >>> have to overwrite a whole bunch of artifacts in once should be
> >> >>> rather rare.
> >> >>>
> >> >>> Yeah, I already filed that a while ago as MRM-992.
> >> >>> And even that was a duplicate for 747.
> >> >>> OK, I will see if I find the time to fix it. The first barrier is to
> >> >>> get
> >> >>> acqainted with the code...
> >> >>
> >> >> Let us know how we can help!
> >> >>
> >> >> - Brett
> >> >>
> >> >>
> >> >>
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/logic-to-prevent-overwriting-release-artifacts-tp25564416p25749141.html
> >> Sent from the archiva-dev mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/logic-to-prevent-overwriting-release-artifacts-tp25564416p25753420.html
> Sent from the archiva-dev mailing list archive at Nabble.com.
>
>

Re: [VOTE] Re-release Archiva 1.2.2 source binaries

2009-10-07T05:27:50Z

+1

Emmanuel

On Mon, Oct 5, 2009 at 3:58 AM, Deng Ching <oching@...> wrote:

> ping?
>
> On Wed, Sep 30, 2009 at 10:29 AM, Deng Ching <oching@...> wrote:
>
> > The source binaries of the recently released Archiva 1.2.2 is broken. You
> > can see the thread about the corrupt source at
> > http://www.nabble.com/Fwd%3A-Corrupt-Source-Zip--td25638356.html
> >
> > Because of this, the Archiva 1.2.2 source binaries has been re-built
> again
> > from the tag and is staged at:
> >
> > http://people.apache.org/~oching/archiva-1.2.2-sources/<http://people.apache.org/%7Eoching/archiva-1.2.2-sources/>
> <http://people.apache.org/%7Eoching/archiva-1.2.2-sources/>
> >
> > Everyone is encouraged to vote and give their feedback..
> >
> > [ ] +1 Release it!
> > [ ] 0
> > [ ] -1 Don't release it, because...
> >
> > The vote will be open for 72 hours.
> > Thanks,
> > Deng
> >
> >
> >
> >
>

Re: [VOTE] Re-release Archiva 1.2.2 source binaries

2009-10-07T04:24:36Z

+1

Sent from my iPhone

On 07/10/2009, at 10:21 PM, Brett Porter <brett@...> wrote:

> +1, built and running just fine
>
> On 05/10/2009, at 12:58 PM, Deng Ching wrote:
>
>> ping?
>>
>> On Wed, Sep 30, 2009 at 10:29 AM, Deng Ching <oching@...>
>> wrote:
>>
>>> The source binaries of the recently released Archiva 1.2.2 is
>>> broken. You
>>> can see the thread about the corrupt source at
>>> http://www.nabble.com/Fwd%3A-Corrupt-Source-Zip--td25638356.html
>>>
>>> Because of this, the Archiva 1.2.2 source binaries has been re-
>>> built again
>>> from the tag and is staged at:
>>>
>>> http://people.apache.org/~oching/archiva-1.2.2-sources/<http://people.apache.org/%7Eoching/archiva-1.2.2-sources/
>>> >
>>>
>>> Everyone is encouraged to vote and give their feedback..
>>>
>>> [ ] +1 Release it!
>>> [ ] 0
>>> [ ] -1 Don't release it, because...
>>>
>>> The vote will be open for 72 hours.
>>> Thanks,
>>> Deng
>>>
>>>
>>>
>>>
>

Re: [VOTE] Re-release Archiva 1.2.2 source binaries

2009-10-07T04:21:12Z

+1, built and running just fine

On 05/10/2009, at 12:58 PM, Deng Ching wrote:

> ping?
>
> On Wed, Sep 30, 2009 at 10:29 AM, Deng Ching <oching@...>
> wrote:
>
>> The source binaries of the recently released Archiva 1.2.2 is
>> broken. You
>> can see the thread about the corrupt source at
>> http://www.nabble.com/Fwd%3A-Corrupt-Source-Zip--td25638356.html
>>
>> Because of this, the Archiva 1.2.2 source binaries has been re-
>> built again
>> from the tag and is staged at:
>>
>> http://people.apache.org/~oching/archiva-1.2.2-sources/<http://people.apache.org/%7Eoching/archiva-1.2.2-sources/
>> >
>>
>> Everyone is encouraged to vote and give their feedback..
>>
>> [ ] +1 Release it!
>> [ ] 0
>> [ ] -1 Don't release it, because...
>>
>> The vote will be open for 72 hours.
>> Thanks,
>> Deng
>>
>>
>>
>>