JNI for UDT library

[Dmitry Sheremetev]

Hello, Bernd!

I think, that at the first stage (before porting UDT library to Java) it is
possible to use JNI and create native Java-methods for direct call of UDT
С-functions.
Moreover, it is possible to call Java-methods (for example,
java.io.InputStream.read() or java.io.OutputStream.write()) directly from
the C-library.
At this time, direct calling of native methods from a modified UDT C-library
is much more preferable, than uncontrollable start of an external executable
file (fufex-send or fufex-recv). Yesterday I have compiled fufex-send.exe,
fufex-recv.exe and udt.dll under a Windows and have checked up them in the
"hand-held" mode (without Unicore) - they transfer the file perfectly even
under Win2k.
Next, we can make modified native UDT C-library, place it, for example, in a
"lib" directory with other JARs, and to specify this path for loading in the
"-Djava.library.path" property for java-VM.
For example, we make "libudtN.so" for Linux, "udtN.dll" for Windows, place
them in the "lib" directory, load library in Java simply with
System.loadLibrary("udtN") and use native UDT-methods directly from
java-code (and native UDT C-code will work with java InputStream or
OutputStream).

For this intermediate solution (UDT C-library with JNI) interaction between
the client and server can be next (for example, for "getFile" or
"putFile" operation):

a) client ask for a "getFile" or "putFile"
b) server responds him
c) client creates UDT-socket and reports his port to the server
d) server creates UDT-socket, starts listener and reports his port to client
e) client receives a port number of the server-side and starts UDT::connect
f) further, depending on a direction of a file transfer (readAllData or
write AllData), client and server can communicate about a file size
(UDT::send and UDT::recv) and starts filetransfer - UDT::recvfile or
UDT::sendfile.
g) client and server close UDT-sockets.

Also it is possible make so that the sender-server always knew from UDT
library about of transferred data size without of asking about it
receiving-side.

Here still a question about a file size - client in the method
writeAllData() does not know about a file size, it receives only InputStream
object.
After small modification of CSndBuffer::addBufferFromFile() it will be
possible to transfer files with unknown length, but receiving with unknown
size leads to "throw CUDTException(2, 1, 0)" in the CUDT::recvfile()...

But now I'm busy, and I can make such adapting of UDT4 for direct calling
from java-code only in December...


Best regards,
Dmitry Sheremetev, Gomel, Belarus


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Unicore-devel mailing list
Unicore-devel@...
https://lists.sourceforge.net/lists/listinfo/unicore-devel

[Bernd Schuller]

Hi Dimitry,

On Sa, 2009-10-10 at 15:51 +0200, Dmitry Sheremetev wrote:

> I think, that at the first stage (before porting UDT library to Java) it is
> possible to use JNI and create native Java-methods for direct call of UDT
> С-functions.
> Moreover, it is possible to call Java-methods (for example,
> java.io.InputStream.read() or java.io.OutputStream.write()) directly from
> the C-library.
> At this time, direct calling of native methods from a modified UDT C-library
> is much more preferable, than uncontrollable start of an external executable
> file (fufex-send or fufex-recv). Yesterday I have compiled fufex-send.exe,
> fufex-recv.exe and udt.dll under a Windows and have checked up them in the
> "hand-held" mode (without Unicore) - they transfer the file perfectly even
> under Win2k.

yes. The problems are more in the integration of the send/receive stuff
with UNICORE, this needs a bit more testing and bugfixing.

> Next, we can make modified native UDT C-library, place it, for example, in a
> "lib" directory with other JARs, and to specify this path for loading in the
> "-Djava.library.path" property for java-VM.
> For example, we make "libudtN.so" for Linux, "udtN.dll" for Windows, place
> them in the "lib" directory, load library in Java simply with
> System.loadLibrary("udtN") and use native UDT-methods directly from
> java-code (and native UDT C-code will work with java InputStream or
> OutputStream).

Have to say I don't like JNI, if you have problems in the native code
you may crash the Java VM.

In any case, since the target site and UNICORE/X are not necessarily the
same machine, a Java or JNI version of UDT will not be useful for
high-performance server-to-server transfer. You will only get the data
to UNICORE/X and have to transfer them to the target site, losing
performance. So you need a commandline tool for doing the transfer.

> For this intermediate solution (UDT C-library with JNI) interaction between
> the client and server can be next (for example, for "getFile" or
> "putFile" operation):
>
> a) client ask for a "getFile" or "putFile"
> b) server responds him
> c) client creates UDT-socket and reports his port to the server
> d) server creates UDT-socket, starts listener and reports his port to client
> e) client receives a port number of the server-side and starts UDT::connect
> f) further, depending on a direction of a file transfer (readAllData or
> write AllData), client and server can communicate about a file size
> (UDT::send and UDT::recv) and starts filetransfer - UDT::recvfile or
> UDT::sendfile.
> g) client and server close UDT-sockets.
>
> Also it is possible make so that the sender-server always knew from UDT
> library about of transferred data size without of asking about it
> receiving-side.
>
> Here still a question about a file size - client in the method
> writeAllData() does not know about a file size, it receives only InputStream
> object.
> After small modification of CSndBuffer::addBufferFromFile() it will be
> possible to transfer files with unknown length, but receiving with unknown
> size leads to "throw CUDTException(2, 1, 0)" in the CUDT::recvfile()...
>
> But now I'm busy, and I can make such adapting of UDT4 for direct calling
> from java-code only in December...
>
>

I don't really think it is worth the effort. If one wants to use UDT for
client-server transfer, a native Java UDT is much preferable than
JNI and C++, because it is much more portable and easy to install
client-side. In fact we have a diploma student right now on who is
working on native Java UDT.
For server-server, having native C++ executables makes more sense, as
explained above one needs commandline tools anyway, so using JNI does
not provide any benefit.

Sure, the current solution is not yet sufficiently stable, but I don't
think that it is a problem in principle, but something that can be
fixed.

Best regards,
Bernd.


--
Dr. Bernd Schuller
Distributed Systems and Grid Computing
Juelich Supercomputing Centre, http://www.fz-juelich.de/jsc
Phone: [phone] (fax -8556)
Personal blog: www.jroller.com/page/gridhaus


------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Unicore-devel mailing list
Unicore-devel@...
https://lists.sourceforge.net/lists/listinfo/unicore-devel