Tuning our Artifactory

[sverhagen]

The promise of Artifactory is great, it looks like the right tool for the job, but we keep on disappointing ourselves by having it all not quite configured as good as we'd want to. Simply because we're just not as good Linux system admins as we would want to be. So I'm about to phrase some questions here that may have more to do with Linux than with Artifactory, but I still hope y'all are willing to help me, so here goes nothing!

- Running Artifactory 2.0.5 (r4903)
- Tomcat 5.5, Apache2, AJP:

ProxyPass        /repo ajp://localhost:6009/artifactory
ProxyPassReverse /repo ajp://localhost:6009/artifactory

- Environment: Red Hat Enterprise Linux Server release 5.3 (Tikanga)
- Something dual quadcore, Xeon, 4GB RAM, Blade server
- Shares the machine with one other application that's idle most of the time
- We've a few backup windows, where we've configured Artifactory to export data. During this time developers can go for a walk: they can not use the repository *at all*
- Outside the backup windows, the repository is generally available, but we're still experiencing a lot of failures, such as those listed at the end of my post
- Our Continuous Integration build fails about 20% of the time just due to Artifactory not being properly available
- No other backups (than the one mentioned before) running (e.g. "in the background") at the times mentioned here

Is there any environment optimisation that is generally recommended to get Artifactory running well? Any help is much appreciated. I can provide more information if there's anything specific you'd need to know!

Best regards, Sander.


(1)
Error installing artifact's metadata: repository metadata for: 'artifact com.company:project' could not be retrieved from repository: companycentral due to an error: Error transferring file
Server returned HTTP response code: 503 for URL: https://url/artifactory/libs-snapshots/com/company/project/maven-metadata.xml

(2)
[INFO] Error deploying artifact: Failed to transfer file: https://url/artifactory/libs-snapshots/com/company/reporting/reporting/0.0.26-SNAPSHOT/reporting-0.0.26-20091027.132606-593.war. Return code is: 503

(3)
[INFO] Error retrieving previous build number for artifact 'com.company:project:jar': repository metadata for: 'snapshot com.company:project:0.0.15-SNAPSHOT' could not be retrieved from repository: companycentral due to an error: Error transferring file
Server returned HTTP response code: 503 for URL: https://url/artifactory/libs-snapshots/com/company/project/0.0.15-SNAPSHOT/maven-metadata.xml

(4)
A lot of "blacklisted" of our repository in our command-line builds.

[Yoav Landman-3]

It looks like you are experiencing two main problems:
- A connectivity problem between Apache and Artifactory
- A performance issue with backups

For the second problem (backups) - please make sure you do not backup to zip archives since it can be extremely wasteful in terms of CPU usage. Also, you may want to try the incremental backups, which are ultra-fast and lightweight.
Please also upgrade to the latest stable release. At present it's either 2.1.1, or at least 2.0.8.
Another common cause of slowdowns is using a slow network storage - a slow network disk or a slow network between the machine running Artifactory and the network disk.

For the connectivity issue, you are getting a lot of 503 errors, indicating that the connection between Apache and Artifactory cannot be reliably established. This can be caused by many reasons.
One reason is poorly tuned Artifatcory VM - in this case you should see errors in the Artifactory log, usually VM related exceptions, such as out of perm gen memory errors. The backup problems may also be related to this if they make the whole VM unresponsive, leading to connection timeouts.
The more common reason is poorly tuned AJP connector settings - check what errors you get in the Apache log file. When using AJP you have to make sure the Apache settings and the Tomcat AJP connector settings (in server.xml) match in terms of number of connections/threads, buffer size, connection timeout etc. otherwise it can lead to poor performance. The first thing to check is that Tomcat AJP connector can serve enough requests (check threads and backlog) and is not overloaded with requests from Apache. For the sake of identifying if the problem is with your connection setup, you can also try to switch temporarily to http: proxying (instead of ajp:), which is much less effective, but is also less prone to configuration tuning problems.

Hopefully that helps putting you in the right direction,

--
Yoav Landman
Artifactory Founder
Co-founder and Products Development Lead
JFrog Ltd.
http://www.jfrog.org/

On Thu, Oct 29, 2009 at 2:53 AM, sverhagen <sverhagen@...> wrote:

The promise of Artifactory is great, it looks like the right tool for the
job, but we keep on disappointing ourselves by having it all not quite
configured as good as we'd want to. Simply because we're just not as good
Linux system admins as we would want to be. So I'm about to phrase some
questions here that may have more to do with Linux than with Artifactory,
but I still hope y'all are willing to help me, so here goes nothing!

- Running Artifactory 2.0.5 (r4903)
- Tomcat 5.5, Apache2, AJP:

ProxyPass        /repo ajp://localhost:6009/artifactory
ProxyPassReverse /repo ajp://localhost:6009/artifactory

- Environment: Red Hat Enterprise Linux Server release 5.3 (Tikanga)
- Something dual quadcore, Xeon, 4GB RAM, Blade server
- Shares the machine with one other application that's idle most of the time
- We've a few backup windows, where we've configured Artifactory to export
data. During this time developers can go for a walk: they can not use the
repository *at all*
- Outside the backup windows, the repository is generally available, but
we're still experiencing a lot of failures, such as those listed at the end
of my post
- Our Continuous Integration build fails about 20% of the time just due to
Artifactory not being properly available
- No other backups (than the one mentioned before) running (e.g. "in the
background") at the times mentioned here

Is there any environment optimisation that is generally recommended to get
Artifactory running well? Any help is much appreciated. I can provide more
information if there's anything specific you'd need to know!

Best regards, Sander.


(1)
Error installing artifact's metadata: repository metadata for: 'artifact
com.company:project' could not be retrieved from repository: companycentral
due to an error: Error transferring file
Server returned HTTP response code: 503 for URL:
https://url/artifactory/libs-snapshots/com/company/project/maven-metadata.xml

(2)
[INFO] Error deploying artifact: Failed to transfer file:
https://url/artifactory/libs-snapshots/com/company/reporting/reporting/0.0.26-SNAPSHOT/reporting-0.0.26-20091027.132606-593.war.
Return code is: 503

(3)
[INFO] Error retrieving previous build number for artifact
'com.company:project:jar': repository metadata for: 'snapshot
com.company:project:0.0.15-SNAPSHOT' could not be retrieved from repository:
companycentral due to an error: Error transferring file
Server returned HTTP response code: 503 for URL:
https://url/artifactory/libs-snapshots/com/company/project/0.0.15-SNAPSHOT/maven-metadata.xml

(4)
A lot of "blacklisted" of our repository in our command-line builds.
--
View this message in context: http://www.nabble.com/Tuning-our-Artifactory-tp26104589p26104589.html
Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

[sverhagen]

Thanks for that.

I've looked into things again. Upgraded to 2.1.1 (put just the new WAR in place) with basically the same situation still in effect.

Yoav Landman-3 wrote:

One reason is poorly tuned Artifatcory VM - in this case you should see
errors in the Artifactory log, usually VM related exceptions, such as out of
perm gen memory errors.

Seeing no errors in the ${artifactory.home}/logs whatsoever. The Artifactory VM... that would be Tomcat then. All default configuration: I've this one running dedicatedly for Tomcat.

Yoav Landman-3 wrote:

The more common reason is poorly tuned AJP connector settings - check what
errors you get in the Apache log file.

What settings? :-)

[Fri Oct 30 15:25:11 2009] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Oct 30 15:25:11 2009] [error] (120006)APR does not understand this error code: proxy: read response failed from 127.0.0.1:6009 (localhost)

Yoav Landman-3 wrote:

When using AJP you have to make sure
the Apache settings and the Tomcat AJP connector settings (in server.xml)
match in terms of number of connections/threads, buffer size, connection
timeout etc. otherwise it can lead to poor performance.

That will definitely be a problem then. But it appears that not only my knowledge is limited on this matter, but also my ability to compose the right Google queries for it :-(

I'm using mod_proxy_ajp and I'd be very happy if you could share your settings with me for this. A nice example of what works for someone else on both ends would be so great for me!

Thanks already again!!!

[Yoav Landman-3]

Sorry for the late reply.

The specific settings are in Tomcat's 'server.xml' file on the Ajp13
connector section and sometimes on the Apache side itself. The
property names are different between Tomcat versions (e.g. 5 and 6).
At the minimum, you need to verify that the connector on the Tomcat
side has enough processors to handle the connections from Apache (the
numbers should be equal, at least). Also, turning off lookups is a
good idea. The exact values depend on the average load on your system
of course, so there is no single "good configuration", like there is
none for Apache.

Googling for "tuning tomcat ajp13" should give you a good heads up.
Just make sure you adjust the property names according to your Tomcat
version.

We can give you better help if you share your specific environment
setup and usage patterns with us -  I think you may want to contact us
directly for this.

Yoav

On Fri, Oct 30, 2009 at 7:30 PM, sverhagen <sverhagen@...> wrote:

>
> Thanks for that.
>
> I've looked into things again. Upgraded to 2.1.1 (put just the new WAR in
> place) with basically the same situation still in effect.
>
>
> Yoav Landman-3 wrote:
>>
>> One reason is poorly tuned Artifatcory VM - in this case you should see
>> errors in the Artifactory log, usually VM related exceptions, such as out
>> of
>> perm gen memory errors.
>>
>
> Seeing no errors in the ${artifactory.home}/logs whatsoever. The Artifactory
> VM... that would be Tomcat then. All default configuration: I've this one
> running dedicatedly for Tomcat.
>
>
> Yoav Landman-3 wrote:
>>
>> The more common reason is poorly tuned AJP connector settings - check what
>> errors you get in the Apache log file.
>>
>
> What settings? :-)
>
> [Fri Oct 30 15:25:11 2009] [error] ajp_read_header: ajp_ilink_receive failed
> [Fri Oct 30 15:25:11 2009] [error] (120006)APR does not understand this
> error code: proxy: read response failed from 127.0.0.1:6009 (localhost)
>
>
> Yoav Landman-3 wrote:
>>
>> When using AJP you have to make sure
>> the Apache settings and the Tomcat AJP connector settings (in server.xml)
>> match in terms of number of connections/threads, buffer size, connection
>> timeout etc. otherwise it can lead to poor performance.
>>
>
> That will definitely be a problem then. But it appears that not only my
> knowledge is limited on this matter, but also my ability to compose the
> right Google queries for it :-(
>
> I'm using mod_proxy_ajp and I'd be very happy if you could share your
> settings with me for this. A nice example of what works for someone else on
> both ends would be so great for me!
>
> Thanks already again!!!
> --
> View this message in context: http://old.nabble.com/Tuning-our-Artifactory-tp26104589p26134048.html
> Sent from the Artifactory-Users mailing list archive at Nabble.com.
>
>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
> _______________________________________________
> Artifactory-users mailing list
> Artifactory-users@...
> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

[sverhagen]

Yoav Landman-3 wrote:

Googling for "tuning tomcat ajp13" should give you a good heads up.
Just make sure you adjust the property names according to your Tomcat
version.

You'd say, but tuning of mod_proxy_ajp is hard to come by. Maybe I just don't know how to use Google. I had someone set up mod_jk instead, and things are working well now, apart from what's being discussed in http://issues.jfrog.org/jira/browse/RTFACT-1314 .

Regards, Sander.