artifactory 2.0.6 speed problems after upgrade

[damonjebb]

I upgraded from artifactory v1.2.5-rc5 to 2.0.6 recently by exporting the original system (using artadmin) then re-importing.  Since then we have seen periods when the response from artifactory is extremely slow to respond to requests, though it eventually does work.

There's very little in the artifactory log files - no errors and nothing unusual.  top indicates that there is a significant bottle neck somewhere because when artifactory is slow it show wait states in the >50% range.

the server in use is the same as the one that the previous version was on, so I am thinking that there is soemthing about the new version of artifactory, perhaps memory usage, that is causing this.  However, top indicates that artifactory is using about 15% of the 4GB available on the machine, so it's not using an excessive amount.

Does this new version  (2.0.6) require or use more memory than 1.2.5rc5 by default?

Thanks
Damon

[freddy33]

A lot of things changed in 2.0 :), but to make sure:
- Which OS are you using?
- Do you use a Web container or the standalone Jetty?
- Do you use Derby DB or an external DB?
- Is the artifactory home (especially the data folder) on a remote disk or local?
- How much JVM heap size did you configured (default is 1Gb when installing the Unix service: /etc/artifactory/default )

Thanks.

On Tue, Jun 30, 2009 at 1:29 PM, damonjebb <djebb@...> wrote:

I upgraded from artifactory v1.2.5-rc5 to 2.0.6 recently by exporting the
original system (using artadmin) then re-importing.  Since then we have seen
periods when the response from artifactory is extremely slow to respond to
requests, though it eventually does work.

There's very little in the artifactory log files - no errors and nothing
unusual.  top indicates that there is a significant bottle neck somewhere
because when artifactory is slow it show wait states in the >50% range.

the server in use is the same as the one that the previous version was on,
so I am thinking that there is soemthing about the new version of
artifactory, perhaps memory usage, that is causing this.  However, top
indicates that artifactory is using about 15% of the 4GB available on the
machine, so it's not using an excessive amount.

Does this new version  (2.0.6) require or use more memory than 1.2.5rc5 by
default?

Thanks
Damon
--
View this message in context: http://www.nabble.com/artifactory-2.0.6-speed-problems-after-upgrade-tp24269554p24269554.html
Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

--
JFrog Ltd
5 Habonim st., P.O.Box 8187
Netanya, Israel 42504.
Tel: [phone]    
Fax: [phone]
http://www.jfrog.org/
http://freddy33.blogspot.com/
http://nothingisinfinite.blogspot.com/

------------------------------------------------------------------------------

_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

[damonjebb]

freddy33 wrote:

A lot of things changed in 2.0 :), but to make sure:
- Which OS are you using?

Redhat EL 4.

freddy33 wrote:

- Do you use a Web container or the standalone Jetty?

I'm running a tomcat container behind a httpd (apache).  Our previous configuration used a 'worker' configuration pointed at the ajp connector on tomcat.  The httpd is on a separate server from the artifactory one.

freddy33 wrote:

- Do you use Derby DB or an external DB?

Using the derby database

freddy33 wrote:

- Is the artifactory home (especially the data folder) on a remote disk or
local?

Local disk

freddy33 wrote:

- How much JVM heap size did you configured (default is 1Gb when installing
the Unix service: /etc/artifactory/default )

I haven't configured any special options for the JVM that's running the tomcat - which is exactly the same as was working ok with the previous version.  I'm not seeing any log messages that indicate a problem with the memory available to artifactory (or anything else)

I wanted to make the minimum changes before looking at any optimisations for the new version, which is why the configuratin, jvm setup, etc. have not changed yet.

Since posting I have tried to use the instructions in the artifactory manual for running behind a httpd using proxying, but they do not work for me (I get 'permission denied' errors for all pages).

I can connect directly to the artifactory server on the http port configured for the tomcat container - this works in much the same way as the one behind apache on the other server and has been seen to 'hang' too.

I've looked more closely at the memory usage and do not think that there is a significant problem with memory - I had assumed that exceeding physical memory and using the swap space was the most likely cause of wait-states, but there is little evidence for this now.  Free looks like this...

# free -m
             total       used       free     shared    buffers     cached
Mem:          4057       4031         25          0          8       3416
-/+ buffers/cache:        607       3449
Swap:         2047          0       2047

seems like 3/4 of the memory is currently available and we still get slow downs and high percentages of waits.

Any suggestions for things that I could look at to see what could be bottlenecking the system would be gratefully received!

Thanks.
Damon

[Yoav Landman]

Hi Damon,

Can you post the output of running the following once you experience slowdowns + specify the exact JVM version you are using:

jmap -heap <artifactory-pid>

Thanks,

Yoav

On Tue, Jun 30, 2009 at 2:25 PM, damonjebb <djebb@...> wrote:

freddy33 wrote:
>
> A lot of things changed in 2.0 :), but to make sure:
> - Which OS are you using?
>

Redhat EL 4.

freddy33 wrote:
>
> - Do you use a Web container or the standalone Jetty?
>

I'm running a tomcat container behind a httpd (apache).  Our previous
configuration used a 'worker' configuration pointed at the ajp connector on
tomcat.  The httpd is on a separate server from the artifactory one.

freddy33 wrote:
>
> - Do you use Derby DB or an external DB?
>

Using the derby database

freddy33 wrote:
>
> - Is the artifactory home (especially the data folder) on a remote disk or
> local?
>

Local disk

freddy33 wrote:
>
> - How much JVM heap size did you configured (default is 1Gb when
> installing
> the Unix service: /etc/artifactory/default )
>

I haven't configured any special options for the JVM that's running the
tomcat - which is exactly the same as was working ok with the previous
version.  I'm not seeing any log messages that indicate a problem with the
memory available to artifactory (or anything else)

I wanted to make the minimum changes before looking at any optimisations for
the new version, which is why the configuratin, jvm setup, etc. have not
changed yet.

Since posting I have tried to use the instructions in the artifactory manual
for running behind a httpd using proxying, but they do not work for me (I
get 'permission denied' errors for all pages).

I can connect directly to the artifactory server on the http port configured
for the tomcat container - this works in much the same way as the one behind
apache on the other server and has been seen to 'hang' too.

I've looked more closely at the memory usage and do not think that there is
a significant problem with memory - I had assumed that exceeding physical
memory and using the swap space was the most likely cause of wait-states,
but there is little evidence for this now.  Free looks like this...

# free -m
            total       used       free     shared    buffers     cached
Mem:          4057       4031         25          0          8       3416
-/+ buffers/cache:        607       3449
Swap:         2047          0       2047

seems like 3/4 of the memory is currently available and we still get slow
downs and high percentages of waits.

Any suggestions for things that I could look at to see what could be
bottlenecking the system would be gratefully received!

Thanks.
Damon

--
View this message in context: http://www.nabble.com/artifactory-2.0.6-speed-problems-after-upgrade-tp24269554p24270973.html

Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

------------------------------------------------------------------------------

_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

[damonjebb]

Hi Yoav, sorry about the delay, some more urgent issues arose.

Yoav Landman wrote:

Can you post the output of running the following once you experience
slowdowns + specify the exact JVM version you are using:

jmap -heap <artifactory-pid>

The JVM used is …

# java -version
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
Java HotSpot(TM) Server VM (build 1.5.0_16-b02, mixed mode)


the output from jmap I obtained by searching for the process ID (grepping ps for artifactory') because I couldn't locate an artifactory-pid file anywhere...

# jmap -heap  24557
Attaching to process ID 24557, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.5.0_16-b02

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 1065353216 (1016.0MB)
   NewSize          = 655360 (0.625MB)
   MaxNewSize       = 4294901760 (4095.9375MB)
   OldSize          = 1441792 (1.375MB)
   NewRatio         = 8
   SurvivorRatio    = 8
   PermSize         = 16777216 (16.0MB)
   MaxPermSize      = 67108864 (64.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 21823488 (20.8125MB)
   used     = 388640 (0.370635986328125MB)
   free     = 21434848 (20.441864013671875MB)
   1.780833567942943% used
From Space:
   capacity = 10158080 (9.6875MB)
   used     = 931328 (0.88818359375MB)
   free     = 9226752 (8.79931640625MB)
   9.168346774193548% used
To Space:
   capacity = 10616832 (10.125MB)
   used     = 0 (0.0MB)
   free     = 10616832 (10.125MB)
   0.0% used
PS Old Generation
   capacity = 270794752 (258.25MB)
   used     = 164454496 (156.83602905273438MB)
   free     = 106340256 (101.41397094726562MB)
   60.73031134665416% used
PS Perm Generation
   capacity = 45744128 (43.625MB)
   used     = 45345680 (43.24501037597656MB)
   free     = 398448 (0.3799896240234375MB)
   99.12896361255372% used

I have noticed another thing.  The garbage collection routine seems to be triggering on an hourly basis and taking about 30 minutes to run...

2009-07-07 15:46:14,907 [INFO ] (o.a.j.j.ArtifactoryGarbageCollector:332) - Artifactory Jackrabbit's datastore garbage collector report:
Total execution: 1768280ms :
Data Store Query: 1118620ms
Binary Properties Query: 9680ms
Total Scanning: 1768280ms
Element count: 19067
Total size: 41719982031 bytes

Our monitoring systems are showing a very consistent 'tick' of failures (timeout getting to the login page) on an hourly basis.  Could this garbage collection have anything to do with these issues?

Thanks
Damon

[Yoav Landman]

Ok, looks like it's the perm-gen which is running out of space.
Try increasing it to, say 128M, and also put a minimum barrier for it, by adding the following to your Tomcat JVM options:

-XX:PermSize=128m -XX:MaxPermSize=128m

HTH,

Yoav

On Tue, Jul 7, 2009 at 6:05 PM, damonjebb <djebb@...> wrote:

Hi Yoav, sorry about the delay, some more urgent issues arose.

Yoav  Landman wrote:
>
> Can you post the output of running the following once you experience
> slowdowns + specify the exact JVM version you are using:
>
> jmap -heap <artifactory-pid>
>

The JVM used is …

# java -version
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
Java HotSpot(TM) Server VM (build 1.5.0_16-b02, mixed mode)


the output from jmap I obtained by searching for the process ID (grepping ps
for artifactory') because I couldn't locate an artifactory-pid file
anywhere...

# jmap -heap  24557
Attaching to process ID 24557, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.5.0_16-b02

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
  MinHeapFreeRatio = 40
  MaxHeapFreeRatio = 70
  MaxHeapSize      = 1065353216 (1016.0MB)
  NewSize          = 655360 (0.625MB)
  MaxNewSize       = 4294901760 (4095.9375MB)
  OldSize          = 1441792 (1.375MB)
  NewRatio         = 8
  SurvivorRatio    = 8
  PermSize         = 16777216 (16.0MB)
  MaxPermSize      = 67108864 (64.0MB)

Heap Usage:
PS Young Generation
Eden Space:
  capacity = 21823488 (20.8125MB)
  used     = 388640 (0.370635986328125MB)
  free     = 21434848 (20.441864013671875MB)
  1.780833567942943% used
>From Space:
  capacity = 10158080 (9.6875MB)
  used     = 931328 (0.88818359375MB)
  free     = 9226752 (8.79931640625MB)
  9.168346774193548% used
To Space:
  capacity = 10616832 (10.125MB)
  used     = 0 (0.0MB)
  free     = 10616832 (10.125MB)
  0.0% used
PS Old Generation
  capacity = 270794752 (258.25MB)
  used     = 164454496 (156.83602905273438MB)
  free     = 106340256 (101.41397094726562MB)
  60.73031134665416% used
PS Perm Generation
  capacity = 45744128 (43.625MB)
  used     = 45345680 (43.24501037597656MB)
  free     = 398448 (0.3799896240234375MB)
  99.12896361255372% used

I have noticed another thing.  The garbage collection routine seems to be
triggering on an hourly basis and taking about 30 minutes to run...

2009-07-07 15:46:14,907 [INFO ] (o.a.j.j.ArtifactoryGarbageCollector:332) -
Artifactory Jackrabbit's datastore garbage collector report:
Total execution: 1768280ms :
Data Store Query: 1118620ms
Binary Properties Query: 9680ms
Total Scanning: 1768280ms
Element count: 19067
Total size: 41719982031 bytes

Our monitoring systems are showing a very consistent 'tick' of failures
(timeout getting to the login page) on an hourly basis.  Could this garbage
collection have anything to do with these issues?

Thanks
Damon
--
View this message in context: http://www.nabble.com/artifactory-2.0.6-speed-problems-after-upgrade-tp24269554p24375051.html

Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------

Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/blackberry

_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/blackberry
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

[damonjebb]

Yoav Landman wrote:

Ok, looks like it's the perm-gen which is running out of space.
Try increasing it to, say 128M, and also put a minimum barrier for it, by
adding the following to your Tomcat JVM options:

-XX:PermSize=128m -XX:MaxPermSize=128m

I've made these changes and though the systems seems slightly more responsive we are still getting regular (twice hourly) alerts from the monitoring system for slow responses to display the home page.

The log is still showing that the garbage collection is taking 30 minutes in every hour.  Is this normal?

Thanks
Damon

[Yoav Landman]

The garbage collection you see is the storage garbage collection and it normally doesn't take more than a couple of seconds, even on a large repository like this one. We suspect that there are JVM major GCs happening while the storage GC is running, which slows it down dramatically. This can be verified by turning on -verbose:gc with -XX:+PrintGCDetails.

I would also suggest putting a minimum on the young gen size -XX:NewSize=128m, as well as trying to increase the total heap.

Having another jmap output to verify that the perm gen issue is fixed is also a good idea.

HTH,

Yoav

On Wed, Jul 8, 2009 at 9:21 AM, damonjebb <djebb@...> wrote:

Yoav  Landman wrote:
>
> Ok, looks like it's the perm-gen which is running out of space.
> Try increasing it to, say 128M, and also put a minimum barrier for it, by
> adding the following to your Tomcat JVM options:
>
> -XX:PermSize=128m -XX:MaxPermSize=128m
>
>

I've made these changes and though the systems seems slightly more
responsive we are still getting regular (twice hourly) alerts from the
monitoring system for slow responses to display the home page.

The log is still showing that the garbage collection is taking 30 minutes in
every hour.  Is this normal?

Thanks
Damon
--
View this message in context: http://www.nabble.com/artifactory-2.0.6-speed-problems-after-upgrade-tp24269554p24386031.html

Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize

details at: http://p.sf.net/sfu/Challenge

_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

[damonjebb]

Yoav Landman wrote:

The garbage collection you see is the storage garbage collection and it
normally doesn't take more than a couple of seconds, even on a large
repository like this one. We suspect that there are JVM major GCs happening
while the storage GC is running, which slows it down dramatically. This can
be verified by turning on -verbose:gc with -XX:+PrintGCDetails.

I have added these to the JVM parameters and am seeing log entries in dicating that it is taking <0.1 secs to run an individual GC activity ...

[GC [PSYoungGen: 77072K->22267K(83584K)] 192972K->138168K(226112K), 0.0733560 secs]
[GC [PSYoungGen: 83579K->24994K(88896K)] 199480K->140895K(231424K), 0.0799120 secs]
[GC [PSYoungGen: 76962K->26674K(78656K)] 192863K->142575K(221184K), 0.0799260 secs]
[GC [PSYoungGen: 78642K->26853K(85888K)] 194543K->142753K(228416K), 0.0833920 secs]

Yoav Landman wrote:

I would also suggest putting a minimum on the young gen size
-XX:NewSize=128m, as well as trying to increase the total heap.

Having another jmap output to verify that the perm gen issue is fixed is
also a good idea.

I have set the max heap size to 1024m (the box has 4GB total memory)

Here's the latest jmap...

jmap -heap 9294
Attaching to process ID 9294, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.5.0_16-b02

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 1073741824 (1024.0MB)
   NewSize          = 134217728 (128.0MB)
   MaxNewSize       = 4294901760 (4095.9375MB)
   OldSize          = 1441792 (1.375MB)
   NewRatio         = 8
   SurvivorRatio    = 8
   PermSize         = 134217728 (128.0MB)
   MaxPermSize      = 134217728 (128.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 47513600 (45.3125MB)
   used     = 32162872 (30.67290496826172MB)
   free     = 15350728 (14.639595031738281MB)
   67.69192820581897% used
From Space:
   capacity = 42401792 (40.4375MB)
   used     = 4734408 (4.515083312988281MB)
   free     = 37667384 (35.92241668701172MB)
   11.16558469981646% used
To Space:
   capacity = 40239104 (38.375MB)
   used     = 0 (0.0MB)
   free     = 40239104 (38.375MB)
   0.0% used
PS Old Generation
   capacity = 145948672 (139.1875MB)
   used     = 118682176 (113.18414306640625MB)
   free     = 27266496 (26.00335693359375MB)
   81.31774984564437% used
PS Perm Generation
   capacity = 134217728 (128.0MB)
   used     = 40296296 (38.429542541503906MB)
   free     = 93921432 (89.5704574584961MB)
   30.023080110549927% used

Thanks for your help with this
Damon

[Yoav Landman]

I was referring to the full GC cycles (of the old generation, not the young GCs which are very short) as a potential slowdown cause (typically, you'd see the word "Full" as part of the older generation collection log message).
One more thing that can drastically affect performance and is worth checking is the type of storage being used - are you using a remote network storage with Artifactory?

On Fri, Jul 10, 2009 at 4:21 PM, damonjebb <djebb@...> wrote:

Yoav  Landman wrote:
>
> The garbage collection you see is the storage garbage collection and it
> normally doesn't take more than a couple of seconds, even on a large
> repository like this one. We suspect that there are JVM major GCs
> happening
> while the storage GC is running, which slows it down dramatically. This
> can
> be verified by turning on -verbose:gc with -XX:+PrintGCDetails.
>
>

I have added these to the JVM parameters and am seeing log entries in
dicating that it is taking <0.1 secs to run an individual GC activity ...

[GC [PSYoungGen: 77072K->22267K(83584K)] 192972K->138168K(226112K),
0.0733560 secs]
[GC [PSYoungGen: 83579K->24994K(88896K)] 199480K->140895K(231424K),
0.0799120 secs]
[GC [PSYoungGen: 76962K->26674K(78656K)] 192863K->142575K(221184K),
0.0799260 secs]
[GC [PSYoungGen: 78642K->26853K(85888K)] 194543K->142753K(228416K),
0.0833920 secs]

Yoav  Landman wrote:
>
>
> I would also suggest putting a minimum on the young gen size
> -XX:NewSize=128m, as well as trying to increase the total heap.
>
> Having another jmap output to verify that the perm gen issue is fixed is
> also a good idea.
>
>

I have set the max heap size to 1024m (the box has 4GB total memory)

Here's the latest jmap...

jmap -heap 9294
Attaching to process ID 9294, please wait...

Debugger attached successfully.
Server compiler detected.
JVM version is 1.5.0_16-b02

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
  MinHeapFreeRatio = 40
  MaxHeapFreeRatio = 70

  MaxHeapSize      = 1073741824 (1024.0MB)
  NewSize          = 134217728 (128.0MB)

  MaxNewSize       = 4294901760 (4095.9375MB)
  OldSize          = 1441792 (1.375MB)
  NewRatio         = 8
  SurvivorRatio    = 8

  PermSize         = 134217728 (128.0MB)
  MaxPermSize      = 134217728 (128.0MB)

Heap Usage:
PS Young Generation
Eden Space:

  capacity = 47513600 (45.3125MB)
  used     = 32162872 (30.67290496826172MB)
  free     = 15350728 (14.639595031738281MB)
  67.69192820581897% used
>From Space:
  capacity = 42401792 (40.4375MB)
  used     = 4734408 (4.515083312988281MB)
  free     = 37667384 (35.92241668701172MB)
  11.16558469981646% used
To Space:
  capacity = 40239104 (38.375MB)
  used     = 0 (0.0MB)
  free     = 40239104 (38.375MB)

  0.0% used
PS Old Generation

  capacity = 145948672 (139.1875MB)
  used     = 118682176 (113.18414306640625MB)
  free     = 27266496 (26.00335693359375MB)
  81.31774984564437% used
PS Perm Generation
  capacity = 134217728 (128.0MB)
  used     = 40296296 (38.429542541503906MB)
  free     = 93921432 (89.5704574584961MB)
  30.023080110549927% used

Thanks for your help with this
Damon
--
View this message in context: http://www.nabble.com/artifactory-2.0.6-speed-problems-after-upgrade-tp24269554p24427490.html

Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users