Linux server clustered HDFS: access from Windows eclipse Java application

[Alberto Forcén]

Hi all.

I'm running a clustering HDFS on linux and I need to access files (I/O) from eclipse Java application running on Windows. It seems simple, but is it possible?

I have write code using API but I have a problem: when code invokes DistributedFileSystem.initialize() method I receive an exception: java.net.SocketTimeoutException


[code]
String ipStr = "192.168.75.191";
String portStr = "9000";
String uriStr = "http://" + ipStr + ":" + portStr;

Configuration conf = new Configuration();
conf.set("hadoop.job.ugi", "user,group"); // Usuario y grupos a los que pertenece

DistributedFileSystem dfs = new DistributedFileSystem();
dfs.initialize(new URI(uriStr), conf);
[/code]

[trace]
Exception in thread "main" java.net.SocketTimeoutException: timed out waiting for rpc response
at org.apache.hadoop.ipc.Client.call(Client.java:559)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
at org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
at examples.HadoopDFS.main(HadoopDFS.java:153)
[/trace]


      ______________________________________________
Enviado desde Correo Yahoo! La bandeja de entrada más inteligente.

[Qin Gao-2]

I think IBM has a plugin that can access HDFS, I don't know whether it
contains source code, but maybe it helps.

www.alphaworks.*ibm*.com/tech/mapreducetools


On Tue, Aug 5, 2008 at 5:16 AM, Alberto Forcén <otrebla81@...> wrote:

> Hi all.
>
> I'm running a clustering HDFS on linux and I need to access files (I/O)
> from eclipse Java application running on Windows. It seems simple, but is it
> possible?
>
> I have write code using API but I have a problem: when code invokes
> DistributedFileSystem.initialize() method I receive an exception:
> java.net.SocketTimeoutException
>
>
> [code]
> String ipStr = "192.168.75.191";
> String portStr = "9000";
> String uriStr = "http://" + ipStr + ":" + portStr;
>
> Configuration conf = new Configuration();
> conf.set("hadoop.job.ugi", "user,group"); // Usuario y grupos a los que
> pertenece
>
> DistributedFileSystem dfs = new DistributedFileSystem();
> dfs.initialize(new URI(uriStr), conf);
> [/code]
>
> [trace]
> Exception in thread "main" java.net.SocketTimeoutException: timed out
> waiting for rpc response
> at org.apache.hadoop.ipc.Client.call(Client.java:559)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
> at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
> at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
> at
> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
> at examples.HadoopDFS.main(HadoopDFS.java:153)
> [/trace]
>
>
>      ______________________________________________
> Enviado desde Correo Yahoo! La bandeja de entrada más inteligente.
>

[Konstantin Shvachko]

I was wondering around Hadoop wiki and found this page dedicated to name-node failover.
http://wiki.apache.org/hadoop/NameNodeFailover

I think it is confusing, contradicts other documentation on the subject and contains incorrect facts. See
http://hadoop.apache.org/core/docs/current/hdfs_user_guide.html#Secondary+Namenode
http://wiki.apache.org/hadoop/FAQ#7

Besides it contains some kind of discussion.
It is not that I am against discussions, lets have them on this list.
But I was trying to understand were all the confusion about secondary-node issues comes from lately...

Imho we either need to correct it or remove.

Thanks,
--Konstantin

[Michael Bieniosek]

Are you sure both ends are running the same version of hadoop?

On 8/5/08 2:16 AM, "Alberto Forcén" <otrebla81@...> wrote:

Hi all.

I'm running a clustering HDFS on linux and I need to access files (I/O) from eclipse Java application running on Windows. It seems simple, but is it possible?

I have write code using API but I have a problem: when code invokes DistributedFileSystem.initialize() method I receive an exception: java.net.SocketTimeoutException


[code]
String ipStr = "192.168.75.191";
String portStr = "9000";
String uriStr = "http://" + ipStr + ":" + portStr;

Configuration conf = new Configuration();
conf.set("hadoop.job.ugi", "user,group"); // Usuario y grupos a los que pertenece

DistributedFileSystem dfs = new DistributedFileSystem();
dfs.initialize(new URI(uriStr), conf);
[/code]

[trace]
Exception in thread "main" java.net.SocketTimeoutException: timed out waiting for rpc response
at org.apache.hadoop.ipc.Client.call(Client.java:559)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
at org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
at examples.HadoopDFS.main(HadoopDFS.java:153)
[/trace]


      ______________________________________________
Enviado desde Correo Yahoo! La bandeja de entrada más inteligente.

[Doug Cutting-4]

Konstantin Shvachko wrote:
> Imho we either need to correct it or remove.

+1

Doug

[Steve Loughran]

Doug Cutting wrote:
> Konstantin Shvachko wrote:
>> Imho we either need to correct it or remove.
>
> +1
>
> Doug

I added some pages there on namenode/jobtracker, etc, linking to the
faiover doc, which I didnt compare to the svn docs to see what was
correct. Perhaps the failover page could be set up to say "you can do
some things here" and point to the full docs at SVN or the hadoop site

--
Steve Loughran                  http://www.1060.org/blogxter/publish/5
Author: Ant in Action           http://antbook.org/