Nabble - PelicanHPC

Re: networking

2009-10-30T23:26:07Z

Often, the dhcp client will take care of that. To do it manually, you can use kcontrol for a graphical frontend, or you can edit the file /etc/network/interfaces to set things the way you like. For setting up the netorking of the cluster, the pelican_setup script will do it for you.

networking

2009-10-28T15:28:58Z

Hello,
I'm new to the Debian flavor of Linux and haven't had much success in determining how to configure networking (setting the default gateway, ip address for one card, etc.).
Thanks,
-Tom

Re: newbie question

2009-10-19T08:59:28Z

No, it's not that simple, unfortunately. For parallelization using MPI, the source code has to include specific instructions. There are some compilers that can automatically parallelize certain types of code, but that is a different approach from what PelicanHPC does. Such compilers usually cost a good deal of money. To get a benefit from PelicanHPC, the application has to use MPI.

Re: newbie question

2009-10-19T04:52:29Z

Hi, as I am also a newbie and interested in the same thing, I'll post in here.

I run model simulations with Modelica, and as for some models it takes a lot of time I thought of using Pelican.

There is a linux version of Modelica, but I think it won't get parallelized.
Modelica is a language for modeling purposes, and to evaluate the models it compiles into C code and then executes.

I've been reading a lot in this forum, and, I have one question:
As modelica - openmodelica- is opensource, if I compile it somehow using mpicc or similar, it will get paralellized?

Thank you!

Re: starting problems with Pelican

2009-10-02T06:28:04Z

you,re right...

Timing results, second example
Number of nodes: 0 Time (sec.): 626.631167
Timing results, second example
Number of nodes: 1 Time (sec.): 311.761412
Timing results, second example
Number of nodes: 2 Time (sec.): 171.927231
Timing results, second example
Number of nodes: 3 Time (sec.): 113.922598
Timing results, second example
Number of nodes: 4 Time (sec.): 92.087825
Timing results, second example
Number of nodes: 5 Time (sec.): 73.063092
Timing results, second example
Number of nodes: 6 Time (sec.): 73.621497
Timing results, second example
Number of nodes: 7 Time (sec.): 54.708440
Timing results, second example
Number of nodes: 8 Time (sec.): 54.226124
octave:34>

It,s working great... thank you again for your help. PelicanHPC is a really nice linux distro to learn something about clusters. Now it,s working i,m going to read a little bit about MPI and try to start RAxML on the cluster.

Re: starting problems with Pelican

2009-10-02T05:41:19Z

Your cluster should do better than that for Monte Carlo. Try the code below. This uses a large number of reps, so that the compute nodes are busy for a little while. Also, the nodes do 1000 replications at a time, rather than 100, to lower network effects. Finally, and importantly, I moved "toc;" to avoid counting the time used to make the bar graph (stupid bug!). I think this last bug is what caused the problem.

########### cut here ###############

# this runs the test on several different configurations and reports timings.
# to see good speedups in parallel, you need to set "reps" to a fairly
# big number (say, 100000 or so).
outfile = "example2.out";
T = 1000;
dim = 5;
reps = 100000;

maxslaves = 8;
timings = zeros(maxslaves,1);
for i = 0:maxslaves;
# figure;
tic;
n_received = montecarlo("tracetest", {T,dim}, reps, outfile, i, 1000);
t = toc;
if i > 0
bar(0:i, n_received/reps);
xlabel("node");
grid on;
title("percentage of results received by node");
drawnow();
endif

timings(i+1,:) = t;
endfor

for i = 0:maxslaves;
printf("\n\nTiming results, second example\n");
printf("Number of nodes: %d Time (sec.): %f\n", i, timings(i+1,:));
endfor

Re: starting problems with Pelican

2009-10-02T05:28:13Z

We're hosting 9 similar machines, it was a gift from a offical company, all have the same mothoerboard and CPU and RAM. The montecarlo example 2 is nice... I think, maxslaves is the number of maximum nodes that should be used? So if I increase the number of reps to 10,000 - the time is going down for the first 2 nodes and then it stucks there, later the time increases also, like:

0 node - 65 sec
1 node - 45 sec
2 node - 25 sec
3 node - 25 sec
...
8 node - 26 sec

So this is a problem of our slow ethernet? The mobos have only a fast and the switch is a 100mbps full duplex switch. I read in the forum someting about benchmarking, but if I'm honest, I didn't understand how to optimize it. I'm now a little confused if what I'm trying is actually working - to compute my phyolgeny stuff over night instead of 1 week...

Re: starting problems with Pelican

2009-10-02T05:04:50Z

OK, I'm glad it's working. I guess that the node that has CPU usage at 90% has a slower CPU than those of the other nodes. They are mostly idle because they are waiting for the slow node to finish. The speedup you get depends on the relative speeds of the nodes, and on the latency and bandwidth of the networking that connects them. The results you obtain depend very much on the details of the cluster. Doing work on a single fast computer will often be faster than using a 2 node cluster made of the fast computer plus a slow node, unless you have load balancing built into the code. This is because if you just give the slow computer half of the computations, the power of the fast computer will not be fully used.

The pea_example code is fairly "tightly coupled" which means that it is sensitive to the latency and bandwidth of the network.

I have a paper out that discusses this example, at
http://www.springerlink.com/content/b2l43gg723781m87/

There are also some Monte Carlo examples using Octave that are not so sensitive to the networking hardware. Try running mc_example1 and mc_example2 while in Octave.

Re: starting problems with Pelican

2009-10-02T04:47:01Z

So, I edited the pea example to the count of nodes we do have and it's working. All nodes run a command "octave". funnily enough the cpu load of the nodes just lays between 25 - 30% - only node 1 has a cpu load over 90% - why come so? And I thought, the calculation is faster, like if you had 10 seconds for 60 iterations with 2 nodes, it's only 5 or someting with 8 nodes... but it isn't.

Re: starting problems with Pelican

2009-10-02T03:25:47Z

Thank you, I really appreciate your work and support. I'll try this and give you a feedback. We only tried to edit the kernel_example.

Re: starting problems with Pelican

2009-10-02T03:03:58Z

No, the examples don't necessarily use all available nodes, the number is often hardcoded. With pea_example, you can change the number of nodes by calling "edit pea_example" from inside octave, and increasing the number of compute nodes in the line

nodes = 1; # number of compute nodes to use (0 for serial)

If you increase that number, then you should see activity on more nodes.

Cheers, M.

Re: starting problems with Pelican

2009-10-02T02:53:39Z

Michael Creel wrote:

Well, it seems that things worked after the restart. Now that I think about it, this is a circumstance that does occur occasionally - if you click yes while a compute node is just finishing booting, sometimes you see what has happened to you. The solution is to just re-run pelican_restart_hpc. So this is normal. The way to never see this problem is to wait a bit longer before clicking yes, so that the compute nodes all finish booting. Perhaps I could add a little sleep time to the script to avoid this.

In general, I don't recommend using v1.99.0, unless you need the newer Linux kernel, or unless you really want to use kernel regression or density smoothing in Octave. v1.9.1 has had a lot more testing. I believe that newer Semprons are 64 bit, so it should work, if you decide to try it. It depends on whether or not the CPUs are 64 bit.

If 2 MPI ranks are used, it is normal that top only shows 1 running on the frontend, because the other is running on a compute node. To see it, ssh into the compute node and run top there (or set up a cluster monitor - see the homepage for a link how to do it)

Ok, I remember: somewhere it's written like "better use 1.9.1" - and actually I'm impressed how easy it was to setup a cluster with pelican. Now the only concern I do have is that not all nodes are computing. So I did what you told me and logged me into the nodes and run "top" - but only 1 node showed octave at the pea_example. I can ping and ssh the nodes. Interestingly sometimes if I start a example, the failban2server appears in top. Actually all examples should use all achievable nodes?

Re: starting problems with Pelican

2009-10-02T02:06:45Z

Well, it seems that things worked after the restart. Now that I think about it, this is a circumstance that does occur occasionally - if you click yes while a compute node is just finishing booting, sometimes you see what has happened to you. The solution is to just re-run pelican_restart_hpc. So this is normal. The way to never see this problem is to wait a bit longer before clicking yes, so that the compute nodes all finish booting. Perhaps I could add a little sleep time to the script to avoid this.

In general, I don't recommend using v1.99.0, unless you need the newer Linux kernel, or unless you really want to use kernel regression or density smoothing in Octave. v1.9.1 has had a lot more testing. I believe that newer Semprons are 64 bit, so it should work, if you decide to try it. It depends on whether or not the CPUs are 64 bit.

PelicanHPC should let you use any MPI programs that work with LAM/MPI or OpenMPI. It's main purpose is precisely to provide an easy to set up platform for working with MPI. I think that the code you mention should work fine, but I have no experience with it.

Cheers, M.

Re: starting problems with Pelican

2009-10-02T01:48:05Z

Hello Michael, thank you for the fast answer...

Michael Creel wrote:

So, the setup process goes as it appears in the Tutorial or the screencast, and seems to complete normally. The setup process reports finding the compute nodes, as in this shot:
but when you finish the setup process and call "lamnodes" is says that lamd is not running? Is that an accurate summary of the problem?

That's correct. The screens are the same like in the turtorial and at the end the screen says: Found 8 nodes (we're running 8 at the moment)

Michael Creel wrote:

What version of PelicanHPC are you using?

From this mirror: http://download.mi.hs-heilbronn.de/pelicanhpc/
pelicanhpc-v1.9.1-32bit.iso

Michael Creel wrote:

After you re-run pelican_restarthpc things seem to work correctly?

Interestingly it's "pelican_restart_hpc" (with underscore), but yes & no, I did the following:
1.) lamboot
2.) lamnodes shows only the host
3.) pelican_restart_hpc
4.) the host finds 8 nodes, but lamnodes has still only 1
5.) pelican_setup again
6.) the host finds 8 nodes, lamnodes shows all 8 IPs of the nodes
7.) startx and then I tried some examples

Michael Creel wrote:

kernel_example has a bug, thanks for the report. I'll fix that for the next release. (EDIT: actually, this is already fixed in the v1.99.0 release.). I think that pea_example is running fine. This is an indication that re-running pelican_restarthpc in fact worked. If 2 MPI ranks are used, it is normal that top only shows 1 running on the frontend, because the other is running on a compute node. To see it, ssh into the compute node and run top there (or set up a cluster monitor - see the homepage for a link how to do it).

Ok, here we go: v1.99 - We do have AMD Semprons 3000+ - and i'm not sure if they support 64bit - so maybe here settles the problem we do have? We tried running the v1.9.1-32bit.iso as I mentioned.

Michael Creel wrote:

sorry, I don't know anything about that.

No problem, but actually MPI Programs should be supported by Pelican? I couldn't find any "howto" for MPI Porgrams to run it on al cluster... it's quite new for me... maybe you have a hint.

Re: starting problems with Pelican

2009-10-02T01:13:35Z

meckart wrote:

wie tried to setup a cluster with Pelican hpc as in the tutroial. The head is running fine, the nodes are booting over lan. The setup script is finding the nodes, so we are pushing "yes" - but then if we execute "lamnodes" it says: lamd is not running, exceute lamboot or something... so we start lamboot, the pelican_restart_hpc, search for nodes and the lamnodes shows the 10.11.12.* - so we are thinking: ready to run.

So, the setup process goes as it appears in the Tutorial or the screencast, and seems to complete normally. The setup process reports finding the compute nodes, as in this shot:

but when you finish the setup process and call "lamnodes" is says that lamd is not running? Is that an accurate summary of the problem?

What version of PelicanHPC are you using?

After you re-run pelican_restarthpc things seem to work correctly?

meckart wrote:

Now the problems: if we're running "kernel_example" we've got errors in octave. If we're running "pea_example" it runs the example, but only with 2 nodes (it says 2 nodes in ocatve and if we're watching the nodes we find only in one top "octave".

So, what are we doing wrong? We stepped through the tutorial several times.

kernel_example has a bug, thanks for the report. I'll fix that for the next release. (EDIT: actually, this is already fixed in the v1.99.0 release.). I think that pea_example is running fine. This is an indication that re-running pelican_restarthpc in fact worked. If 2 MPI ranks are used, it is normal that top only shows 1 running on the frontend, because the other is running on a compute node. To see it, ssh into the compute node and run top there (or set up a cluster monitor - see the homepage for a link how to do it).

meckart wrote:

At the end we want to try · RAxML-VI-HPC (version 2.2.3)
http://icwww.epfl.ch/~stamatak/index-Dateien/Page443.htm

So, we figured out to make the executing - but it is only running on the head...
If sombody wants to help us, feel free :)

sorry, I don't know anything about that.

starting problems with Pelican

2009-10-01T06:43:00Z

Hello,

wie tried to setup a cluster with Pelican hpc as in the tutroial. The head is running fine, the nodes are booting over lan. The setup script is finding the nodes, so we are pushing "yes" - but then if we execute "lamnodes" it says: lamd is not running, exceute lamboot or something... so we start lamboot, the pelican_restart_hpc, search for nodes and the lamnodes shows the 10.11.12.* - so we are thinking: ready to run.

Now the problems: if we're running "kernel_example" we've got errors in octave. If we're running "pea_example" it runs the example, but only with 2 nodes (it says 2 nodes in ocatve and if we're watching the nodes we find only in one top "octave".

So, what are we doing wrong? We stepped through the tutorial several times.

At the end we want to try · RAxML-VI-HPC (version 2.2.3)
http://icwww.epfl.ch/~stamatak/index-Dateien/Page443.htm

So, we figured out to make the executing - but it is only running on the head...
If sombody wants to help us, feel free :)

Martin
University of Jena

Re: help me about creating cluster by pelican hpc iam beginer

2009-09-29T12:18:12Z

Often times to set up PXE boot you need to enable it in a few places in the bios. On the computer I've used Pelican on I needed to enable PXE boot on the ethernet card, enable the bios to boot from PXE and then set PXE to the first boot device...or some similar procedure...I haven't messed with it in a while, but if you hunt around in your bios long enough you should be able to figure it out. Hope this helps.

Re: Remaster with external storage

2009-09-27T06:37:02Z

Probably not, as it's intended for use by a single user. It runs well on hardware that often is used for servers, though. It also runs fine virtualized.

Re: Remaster with external storage

2009-09-26T08:48:03Z

Cool thanks.

Forgot to ask is Pelican suitable for server use?

Re: Remaster with external storage

2009-09-25T23:07:28Z

If a reboot occurs (power goes out, whatever) you won't loose whatever is on /home, if it's on permanent storage. You just need to specify its location again, the same as when you did the first time. The setup script will wait for you to do so. For this reason, I don't see any need to make a remaster that mounts a specific device. If you want to do so, search the make_pelican script for

mount /dev/\$HOMELOCATION /home

and replace $HOMELOCATION with the device name you want to use. That should do it. You might also want to comment out the dialog that asks you to input the home location device.

Remaster with external storage

2009-09-25T07:17:19Z

Hi, I've got a room full of old machines that I'm wanting to try setting up with Pelican. I see its possible to move the /home directory to an external source (ie HD, external HD etc) but can you remaster the cd so that it always points to that location? I'd like this to be as hands off as possible and if it reboots I'd hate to lose my way /home :p

Re: HOW TO: For those whose nodes don't support PXE booting

2009-09-23T16:05:46Z

Hi I found PXE Boot Images Generated on this
http://rom-o-matic.net/

An easy way to get PXE Boot Images for almost every use and Hardware.

Output Formats available:
- Floppy bootable ROM Image .dsk
- Binary ROM Image .rom
- ISO bootable image .iso
- ISO bootable image with legacy floppy emulation .liso
- LILO/GRUB/SYSLINUX loadable Linux kernel format .lkrn
- PXE bootstrap loader format ROM Image .pxe
- USB Keychain Disk Image / bootable HD image .usb

Re: help me about creating cluster by pelican hpc iam beginer

2009-09-21T01:54:56Z

Sorry, offhand I don't know what the process name is when the atftpd daemon is started. Whatever it is, it is running after you do pelican_setup. I went back and looked at your original message, and I see you're using a hub. Why don't you try to connect the frontend and a single compute node using a crossover cable? If the hub is managed, it can mess things up by handing out IP addresses. An unmanaged switch is what I use.

Re: help me about creating cluster by pelican hpc iam beginer

2009-09-21T01:34:28Z

but if tftp not running how i could to get pxelinux.0 in
/var/lib/tftpboot with tftp client on WindowsXp .

Re: help me about creating cluster by pelican hpc iam beginer

2009-09-21T01:29:19Z

#Hi
this is very very odd to me
i understood that tftp is not run .
i test this :
# /etc/init.d/atftpd start then i do #ps -aux but i don't saw tftp or ftp or atftp
then i act this
#sudo ps -aux but again i dont see tftp proccess
then i change the 'USE_INETD=true' to 'USE_INETD=false' in '/etc/init.d/atftpd'
and then /etc/init.d/atftpd start then i get #ps but again not exist.
at last i test this /usr/sbin/atftpd --daemon /var/lib/tftpboot
and again i get #ps -aux but tftp process not exist .

this odd why tftp process not run?
well when i run commands for example :
#/usr/sbin/atftpd --daemon /var/lib/tftpboot
i see command prompt means ' # ' and this is no error.
what happened ? what i should to do ?
i use live cd linux
best regard
Hogat

Re: Using C on PelicanHPC

2009-09-21T00:47:06Z

OK, the problem with running this from /media/disk is that the executable is available only to the frontend, but not the compute nodes. Doing it from /home/user solves the problem, because that directory is NFS exported.

For your work, I don't see any reason why PelicanHPC shouldn't work well. You will definitely want to mount external storage (during the initial boot up, replace "ram1" with something like "sda1", where that is the device name of a partition formatted as ext2 or ext3. For speeding up the work, I have a couple of ideas. If it is possible to generate all the information needed to plot a frame ahead of time, do that, and then plot complete frames on nodes. If you absolutely need to have the .bmp file (and not just the input parameters) before the information needed to plot the next frame can be computed, then slice the frames into blocks, and compute blocks on nodes. This might be complicated if the pixels on either size of a slice are not independent. That's the reason I would try to do the first approach.

Plotting and video rendering is pretty widespread, but I don't know much about it. You might try searching for some topic-specific mail lists to get advice.

Cheers, M.

Re: Using C on PelicanHPC

2009-09-20T10:45:17Z

Thanks for the quick response. I was unfortunately sick/studying for most of the week and was unable to try out your solution right away.

I was actually in the /media/disk directory when trying to execute "hello". That gave the error that was described previously. As you suggested, I moved the source code to /home/user, and this time "hello" executed successfully with a "Hello World" output.

I'm not sure if anyone has done this before on Pelican.. but I'm building this cluster to process optical simulations for a professor of mine. The optical simulations, based on a C source, generate a frame-by-frame throughput of light on a nano-scale basis. Each frame consists of a bmp file(14MB each) and usually the simulation is run until gigabytes of data are outputted. From what I read online, Pelican seems to be able to do such a task but I'm just not sure how to modify the C source code to do so..

Could you please offer some general guidance on how to split the bitmap generation task to all the nodes in a cluster? I can give you the source code (totalling 62KB) if you are interested.

Thanks alot Michael.

Re: v1.9.1 release candidate is available

2009-09-17T00:46:04Z

Given that you're using the scripts in a way they're not designed for, the results are a little unpredictable. The whole set of script is intended to be used only on a PelicanHPC cluster, so I don't consider this to be a bug. If you find out the cause, I would be interested to know. Thanks, M.

Re: help me about creating cluster by pelican hpc iam beginer

2009-09-17T00:38:36Z

Sorry, but I don't know. I attempt to support hardware that does PXE boot, but I don't have the resources to try to support work arounds like gpxe. I would be happy to receive a HOWTO document that describes how to do it, but I can't try to figure it out myself - too many other things to do.

Re: help me about creating cluster by pelican hpc iam beginer

2009-09-16T16:36:35Z

hi Michael
On Wed, Sep 16, 2009 at 10:00 PM, hogat allah shahamiri
<hogat.sh@gmail.com> wrote:
> hi
> i think i forget to write two slash after tftp:
> please think that is right
> and the line is : tftp://10.11.12.1/pxelinux.0 is TRUE then what is problem?
> if no this wrong and problem is in tftp:10.11.12.1/pxelinux.0
> what i should to do exactly?
> i want cluster computers in university by pelican hpc 1.8
> but i have above problem
> i am in wait
> best regard
> Hogat
>
> On 9/16/09, Miller, Shao <Shao.Miller@yrdsb.edu.on.ca> wrote:
>> Hello again Hogat,
>>
>> Your DHCP filename should be simply:
>>
>> pxelinux.0
>>
>> or
>>
>> tftp://10.11.12.1/pxelinux.0
>>
>> But not:
>>
>> tftp:10.11.12.1/pxelinux.0
>>
>> If this change does not fix your problem, please describe what your DHCP
>> server's filename option is set to.
>>
>> - Shao Miller
>>

Hi
The error code 0x4c126035 is a TFTP timeout.
Maybe your TFTP server IP is wrong(the 'next-server' option handed out
by gPXE) or the filename is not correct, or the TFTP server simply
isn't running at the server.

Thomas

if thomas say correct how solve it in pelican?
thanks

Re: v1.9.1 release candidate is available

2009-09-16T08:39:54Z

my main cluster is not based on pelican. i use pelican to resize'em (add tmp nodes). pelican-booted-nodes tell me smthng about blacklisted ssh key if i use ssh connection from main (not pelican) server.
eropp

Re: help me about creating cluster by pelican hpc iam beginer

2009-09-16T08:00:32Z

Hi Hogat,

I'm sorry, but I don't know what is the cause of this problem. I don't have experience with gpxe, so I really don't know how it will work. The fact that the compute node receives an IP means that the machines are connected properly. I don't know why tftp is timing out. Sorry, but I don't have any ideas.

One thing you might do is set up a virtual cluster using QEMU, Virtualbox or VMware server. This is a convenient way to learn, and it will solve your problems of netboot. The screencast video on the PelicanHPC homepage shows how that can be done.

Another solution is to buy an ethernet card that is equipped to do PXE. They are pretty cheap secondhand. Are you sure your cards won't do PXE? It is an option than is often turned off by default. You might need to activate it in your BIOS setup.

Michael

Re: help me about creating cluster by pelican hpc iam beginer

2009-09-16T00:48:24Z

Hi
at last i success to load by netboot
this all :
i insert pelican hpc into one pc and load it then
#pelican_setup
and i select select "eth0 network_device_eth0 " because pc have two Nic then
#pelican_restart_hpc
and press "NO"
and then insert cd gpxe into other pc and reboot it and in frontend node at #pelican_restart_hpc
i press no
but when pc's that gpxe cd in it , load
i see follow, what is problem?

net0:00:15:f2:08:a2:c0 no PCI 02:05.0 (open)[Link:up TX0 TXE:0 RX:0 RXE:0 ] waiting for Linkup on eth0 ... ok
net0:10.11.12.3/255.255.255.0 gw 0.0.0.0
Booting from filename "pxelinux.0"
tftp://10.11.12.1/pxelinux.0 ... connection time out (0x4c126035)
could not load tftp:10.11.12.1/pxelinux.0 :connection time out (0x4c126035)
no more Network Device

at the frontend node i have :
#cat ~/tmp/bohsts
10.11.12.1
#
what is my mistake ? what i should to do?
oh
Michael my english is very bad?
very thanks
best regard
Hogat

Re: Using C on PelicanHPC

2009-09-15T01:40:43Z

I would guess that you are not in /media/disk when you call mpirun. Try "ls", and make sure you see the executable "hello". If not, cd to /media/disk

Better yet, you probably want to put the executable in a place accessible to all nodes of the cluster. Please try this:

1. copy the source code from /media/disk to /home/user
2. cd to /home/user
3. compile and run from /home/user

Any luck?

Re: Using C on PelicanHPC

2009-09-15T00:30:14Z

Apologies for my ambiguity.

Here's an update:

I successfully compiled the following hello.c on /media/disk using "mpicc hello.c -o hello"
==============================
#include <stdio.h>
#include "mpi.h"
int main(int argc, char **argv)
{
MPI_Init(&argc, &argv);
printf("Hello world\n");
MPI_Finalize();
return 0;
}
=============================

When I try to run "mpirun C hello" or "mpirun N hello" or "mpirun -np 4 hello", I get the following error:
"mpirun: cannot start hello on n0: No such file or directory"

Does anyone know how to fix this error? Do I have to set the default directory to something other than n0 somehow?

Thanks.