Nabble - Unicore

UNICORE and PGI Standardization: Request for Comments

2009-12-02T03:01:33Z

Hi UNICORE team,

some of you might be aware that Shahbaz, Shiraz, and myself are working to get open standards in UNICORE used with real applications (e.g. OGSA-BES, JSDL, GLUE2, etc. in WISDOM, EUFORIA, VPH, etc.) that typically require the access to both HPC (via UNICORE) and HTC (via ARC, gLite, etc.).

While working in this field for a longer time now we clearly identified that OGSA-BES et al. standards are not sufficient to meet the high (but proprietary) quality standard interface that we all know as UNICORE Atomic Services (UAS).

Therefore, I created together with folks from ARC & gLite a spin-off group called Production Grid Infrastructure (PGI) working group out of the Grid Interoperation Now (GIN) community group and I personally chair both with great interest. While GIN is responsible to gain lessons learned and experiences in a wide variety of interop* activities, PGI should ensure that these valuable experiences find their ways in improved specifications/profiles in OGF.

In short, the aim of the PGI working group is to have a set of standard specifications that are augmented with lessons learned gained by the use of UNICORE (and other middleware like ARC, gLite, and GENESIS) having one goal in mind: Getting the standards really workable and thus usable in production interoperability use cases and beyond. We have active participants from ARC (deployed on NorduGrid), gLite (deployed on EGEE), GENESIS (deployed on Campus Grids), and EDGES (BOINC-based DesktopGrids). Being non-active the last couple of month because of the EU proposal period, we will start this week again with a TelCon on Friday.

More information about PGI on GridForge:
http://forge.gridforum.org/projects/pgi-wg

More information about GIN on GridForge:
http://forge.gridforum.org/projects/gin

So far, the PGI group focused on computation and information aspects in order to improve the joint use of evolutions of OGSA-BES, JSDL, and GLUE2 while SRM-based data-stagings are currently a little bit postponed, but will certainly play a role soon together with GridFTP, HTTP and such like technologies. The crucial challenge here is the agreement on security setups with having the overall goal of removing the Grid Security Infrastructure (GSI) dependency, but realizing also full dynamic delegation capabilities as required by middleware like ARC & gLite.

Although there is an initial work and agreement between ARC, gLite, and UNICORE, I'm looking forward to receive comments from the UNICORE community on the following document:
http://forge.gridforum.org/sf/go/doc15628?nav=1

FYI: An XML-based form of the evolution of JSDL can be found at:
http://forge.gridforum.org/sf/go/doc15737?nav=1

I welcome every participation from the UNICORE community in PGI, but also seek to gather our aligned visions and thoughts on this list before we actually give combined inputs to PGI.

Finally, please feel free to contact me any time if you have questions about interoperability and open standards within UNICORE or if you just need slides to present open standards in UNICORE at events.

Take care,
Morris

------------------------------------------------------------
Morris Riedel
SW - Engineer
Distributed Systems and Grid Computing Division
Jülich Supercomputing Centre (JSC)
Forschungszentrum Juelich
Wilhelm-Johnen-Str. 1
D - 52425 Juelich
Germany

Email: m.riedel@...
Info: http://www.fz-juelich.de/jsc/JSCPeople/riedel
Phone: +49 2461 61 - 3651
Fax: +49 2461 61 - 6656

Skype: MorrisRiedel

"We work to better ourselves, and the rest of humanity"

Sitz der Gesellschaft: Jülich
Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498
Vorsitzende des Aufsichtsrats: MinDirig'in Bärbel Brumme-Bothe
Vorstand: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender)

------------------------------------------------------------------------------
Join us December 9, 2009 for the Red Hat Virtual Experience,
a free event focused on virtualization and cloud computing.
Attend in-depth sessions from your desk. Your couch. Anywhere.
http://p.sf.net/sfu/redhat-sfdev2dev
_______________________________________________
Unicore-devel mailing list
Unicore-devel@...
https://lists.sourceforge.net/lists/listinfo/unicore-devel

smime.p7s (4K) Download Attachment

From forum: Unicore - Developers

Re: connection reset errors & unicore server 6.21

2009-11-27T04:10:15Z

hi,

I can confirm that version 6.22 is much better - my test went well after switching to it
looks like unicore server 6.22 i much more efficient than the previous versions

now I can only suggest to everybody - if you use older unicore server it's better to upgrade to 6.22

I also configured proxy support without any problems (I had some in 6.21 version)

Regards,
Dawid

dejw wrote:

ok then

I will have to switch to the 6.22 and I will see

I did a very simple test: just a loop which submits jsdl jobs one after another
after each submit I got the epr of the job and go further

in the mean time I have some poller thread which checks the state of the job till it is finished.
if completed I also get the url for output and error

I tried to push over 500 hundred jobs in this way and after a first 20-30 simultaneous (so quite fast) job run and status checking from time to time I got errors I wrote about
The job was simple like 'ls' or 'date'

Regards,
Dawid

Bernd Schuller wrote:

hi Dawid,

first of all, you should upgrade to 6.2.2 :-), we did lots of work
to improve scalability and throughput compared to 6.2.1

This should be better, but of course at some point you'll run
into scalability issues. Would be interesting what these are
in your case, what kind of load generation strategy you use, etc.

Best regards,
Bernd.

On Do, 2009-11-26 at 10:55 +0100, dejw wrote:

Hi,

I'm testing unicore server 6.21 (core server bundle with standard settings)

I tried to make some kind of jsdl job flood - and I can see taht from
time to time I had connection reset errors on the server side in gateway
log - job jobs are failed.

It could be due to wrong configuration or something. Is there any
official instruction how to set up environment to maximize throughout ?

Additionally I got weird nullpointer errors on the client side which
did not occur  at normal  rate while doing the same thing (job submit).
Errors are below.

Server side error:

2009-11-26 10:41:16,156 [1792194126@qtp-1323423210-26 -
/TEST-SITE/services/TargetSystemService?res=9e86f618-9f18-463f-a488-c5d2b2384b6c]
DEBUG JettyLogger  - Stack trace
com.ctc.wstx.exc.WstxIOException: Connection reset
        at
com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:548)
        at
com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:604)
        at
com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:629)
        at
com.ctc.wstx.stax.WstxInputFactory.createXMLStreamReader(WstxInputFactory.java:324)
        at eu.unicore.gateway.GatewayHandler.invoke(GatewayHandler.java:216)
        at eu.unicore.gateway.base.Servlet.doPost(Servlet.java:296)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at eu.unicore.gateway.base.Servlet.service(Servlet.java:95)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
        at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:380)
        at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
        at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
        at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:320)
        at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:535)
        at
eu.unicore.gateway.util.CustomHttpConnection.handleRequest(CustomHttpConnection.java:31)
        at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:880)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at
org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:636)
        at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at
com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
        at
com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
        at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:723)
        at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:680)
        at
com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
        at java.io.FilterInputStream.read(FilterInputStream.java:111)
        at
org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)
        at
java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:214)
        at
java.util.zip.InflaterInputStream.read(InflaterInputStream.java:134)
        at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:87)
        at java.io.FilterInputStream.read(FilterInputStream.java:90)
        at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
        at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
        at
com.ctc.wstx.io.ReaderBootstrapper.initialLoad(ReaderBootstrapper.java:245)
        at
com.ctc.wstx.io.ReaderBootstrapper.bootstrapInput(ReaderBootstrapper.java:132)
        at
com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:543)
        ... 23 more


Client side errors. (occurred only while many requests were sent)

java.lang.NullPointerException
    at de.fzj.unicore.uas.client.TSSClient.submit(TSSClient.java:91)
    .....................



client.StorageClient - Can't create import client.
java.lang.NullPointerException
    at
de.fzj.unicore.uas.client.StorageClient.getExport(StorageClient.java:433)
    at
de.fzj.unicore.uas.client.StorageClient.getExport(StorageClient.java:456)
    ...................

Regards,
Dawid

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Unicore-support mailing list
Unicore-support@...
https://lists.sourceforge.net/lists/listinfo/unicore-support

--
Dr. Bernd Schuller
Distributed Systems and Grid Computing
Juelich Supercomputing Centre, http://www.fz-juelich.de/jsc
Phone: +49 246161-8736 (fax -8556)
Personal blog: www.jroller.com/page/gridhaus


------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july


_______________________________________________
Unicore-support mailing list
Unicore-support@...
https://lists.sourceforge.net/lists/listinfo/unicore-support

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Unicore-support mailing list
Unicore-support@...
https://lists.sourceforge.net/lists/listinfo/unicore-support

Nabble - Unicore

UNICORE and PGI Standardization: Request for Comments

Re: connection reset errors & unicore server 6.21

Re: connection reset errors & unicore server 6.21

Re: connection reset errors & unicore server 6.21

Re: connection reset errors & unicore server 6.21

connection reset errors & unicore server 6.21

Re: problem with UNICORE6 workflow service

Re: problem with UNICORE6 workflow service

Re: problem with UNICORE6 workflow service

Re: problem with UNICORE6 workflow service

Re: problem with UNICORE6 workflow service

problem with UNICORE6 workflow service

RPM specfile for unicore-servers-6.2.2

Re: //Unicore 6.2.2 and WindowsXP x64

//Unicore 6.2.2 and WindowsXP x64

Re: XNJS SSL configuration

XNJS SSL configuration

Re: Client with TD

Client with TD

Re: rich client gridbeans

Re: rich client gridbeans

Re: rich client gridbeans

rich client gridbeans

Re: Where is the XUUDB?

Re: Where is the XUUDB?

Re: Where is the XUUDB?

Re: Where is the XUUDB?

Re: Where is the XUUDB?

Re: Where is the XUUDB?

Where is the XUUDB?

Re: wsrflite persistence layer (for 6.3 core servers)

UNICORE release feed

Re: Problems with BES submission after changing to newer Unicore server version

Fwd: Problems with BES submission after changing to newer Unicore server version

Re: Problems with BES submission after changing to newer Unicore server version