how to scale into the cloud using process? example computing simple average

[Kid Erlang]

hi again everybody!  it has been a few months of using Erlang!  I have been porting my Perl scripts into Erlang and now I am almost done!!! thanks for your help before getting my scripts ported over.

my program is now in Erlang except I do not see how to make it scale into the cloud.  I have heard I need to use Erlang process to do this but I do not understand how this is works.  I have tried using spawn command to create a processes but it does not seem to do anything.  Where is good documentation for how to use spawn?

I try to write simple example with spawn.  I want to make simple average of these numbers as example, but calculate in paralell so it can automagically scale into the cloud!  Here is my program:

-module(averager).
-compile(export_all).

paralell_average(List)->
  N=0,lists:foreach(fun(X)-> spawn(fun()-> N+X end)end,List),N/length(List).

but it always returns 0.0 no mater what List is!  what am I doing wrong? :(  I have tried to use N+=X like in Perl but I think this does not work because of the single assignment?  same with N=N+X.  how can I make processes make the value of N go up?  I don't get it :(

I have also read that processes must be in recursive?  How would I write this process in recursive?

and as soon as my program is made of processes where is good documentation for making it scale into the cloud?

- Kid Erlang
_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Bengt Kleberg]

Greetings,

The return value of spawn/1 is the (Erlang) process identity. If you
want data from a process you have to send it (using !) and receive it
(using receive).

It would be a good idea to read one of the Erlang books or the online
documentation.


bengt

On Tue, 2009-05-19 at 01:55 -0600, Kid Erlang wrote:

> hi again everybody!  it has been a few months of using Erlang!  I have
> been porting my Perl scripts into Erlang and now I am almost done!!!
> thanks for your help before getting my scripts ported over.
>
> my program is now in Erlang except I do not see how to make it scale
> into the cloud.  I have heard I need to use Erlang process to do this
> but I do not understand how this is works.  I have tried using spawn
> command to create a processes but it does not seem to do anything.
> Where is good documentation for how to use spawn?
>
> I try to write simple example with spawn.  I want to make simple
> average of these numbers as example, but calculate in paralell so it
> can automagically scale into the cloud!  Here is my program:
>
> -module(averager).
> -compile(export_all).
>
> paralell_average(List)->
>   N=0,lists:foreach(fun(X)-> spawn(fun()-> N+X
> end)end,List),N/length(List).
>
> but it always returns 0.0 no mater what List is!  what am I doing
> wrong? :(  I have tried to use N+=X like in Perl but I think this does
> not work because of the single assignment?  same with N=N+X.  how can
> I make processes make the value of N go up?  I don't get it :(
>
> I have also read that processes must be in recursive?  How would I
> write this process in recursive?
>
> and as soon as my program is made of processes where is good
> documentation for making it scale into the cloud?
>
> - Kid Erlang
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@...
> http://www.erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Kid Erlang]

On Tue, May 19, 2009 at 2:26 AM, Bengt Kleberg <bengt.kleberg@...> wrote:

The return value of spawn/1 is the (Erlang) process identity. If you
want data from a process you have to send it (using !) and receive it
(using receive).

okay I have rewrite my project to use receive with spawn, but I still do not get how to handle errors?  especially in the cloud?
 

It would be a good idea to read one of the Erlang books or the online
documentation.

what book is best for me to read?  and I cannot find online documentation for how to use spawn + receive + messages to scale into the cloud?

that is my main worry: say I have something that uses spawn + receive + messages.  how do I make it scale into the cloud?  where can I find online documentation on this?  i want my erlang program to run on as many systems as I need it to for however many users.  how do I do that?

- Kid Erlang



_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Ulf Wiger-3]

Kid Erlang wrote:

>
> I try to write simple example with spawn.  I want to make simple average
> of these numbers as example, but calculate in paralell so it can
> automagically scale into the cloud!  Here is my program:
>
> -module(averager).
> -compile(export_all).
>
> paralell_average(List)->
>   N=0,lists:foreach(fun(X)-> spawn(fun()-> N+X end)end,List),N/length(List).
>
> but it always returns 0.0 no mater what List is!  what am I doing wrong?

When you spawn processes, they will have their own memory and
their own scope. The original process will have no idea what
they do unless they send a message back. In your program above,
the spawned processes add to N, but this has no effect whatsoever
on the original N on the heap of the parent process. Even if it
would, remember that the value of a variable cannot change.
You have declared that N=0, so N/length(List) can never be
anything other than 0 (within the scope of this function.)

Specifically, your spawned processes will begin life by executing
the given fun, which adds X to N, returning the value... in this
case to nothing, since there's no calling function to receive it.
Thus, the return value of the fun is discarded, and the process,
having no more work to perform, dies. Since it doesn't communicate
with anyone, and no other process is monitoring it, its death
goes unnoticed.

A function like parallel_average will not scale in this manner,
since the operation of spawning processes, lightweight as it may
be, is heavier than the operation of calculating X+N.

BR,
Ulf W
--
Ulf Wiger
CTO, Erlang Training & Consulting Ltd
http://www.erlang-consulting.com
_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Torben Hoffmann-2]

I think you are taking a too big bite to start with, but it is good that your interest is driven by a concrete need and not just technology frenzy!

http://heroku.com/ uses Erlang for their routing mesh.
http://www.infoq.com/news/2008/06/introducing-vertebra is built with Erlang + Ruby.

As Bengt said: get a book and then go to one of the Erlang Factory conferences - or have a look at the presentations from the previous ones. That should give you some ideas on how to proceed.

Cheers,
Torben

On Tue, May 26, 2009 at 8:01 AM, Kid Erlang <kiderlang@...> wrote:

On Tue, May 19, 2009 at 2:26 AM, Bengt Kleberg <bengt.kleberg@...> wrote:

The return value of spawn/1 is the (Erlang) process identity. If you
want data from a process you have to send it (using !) and receive it
(using receive).

okay I have rewrite my project to use receive with spawn, but I still do not get how to handle errors?  especially in the cloud?
 

It would be a good idea to read one of the Erlang books or the online
documentation.

what book is best for me to read?  and I cannot find online documentation for how to use spawn + receive + messages to scale into the cloud?

that is my main worry: say I have something that uses spawn + receive + messages.  how do I make it scale into the cloud?  where can I find online documentation on this?  i want my erlang program to run on as many systems as I need it to for however many users.  how do I do that?

- Kid Erlang



_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Joe Armstrong-2]

On Tue, May 26, 2009 at 8:01 AM, Kid Erlang <kiderlang@...> wrote:

> On Tue, May 19, 2009 at 2:26 AM, Bengt Kleberg <bengt.kleberg@...>
> wrote:
>>
>> The return value of spawn/1 is the (Erlang) process identity. If you
>> want data from a process you have to send it (using !) and receive it
>> (using receive).
>
> okay I have rewrite my project to use receive with spawn, but I still do not
> get how to handle errors?  especially in the cloud?
>
>>
>> It would be a good idea to read one of the Erlang books or the online
>> documentation.
>
> what book is best for me to read?

Modesty prevents me from recommending "Programming Erlang".

> and I cannot find online documentation

http://www.erlang.org/doc.html

> for how to use spawn + receive + messages to scale into the cloud?

What is a cloud?

<< seriously! - Just what is a cloud? -

   I asked myself just this question the other evening -

 Now I did actually go to "a glory glory hallelujah, the cloud is COOL
and WOW - MAN - TWITTER ..." meeting in London - each participant was
given 45 seconds to explain their next greatest "WOW COOL" cloud
application to an assembly of what appeared to be thirteen year old
programmers (the kind that think that PHP is cool) and 25+ ish wannabe
business types
in shiny suits" - I stood at the back for listening for a few minutes
with a group of like-minded souls - after 10 minutes we could take it
no more - we retired to the pub.

   So the the cloud appears to be "a computer at the other end of a
bit of wire, that
somebody else will setup and manage so that you can just pay the bills
and use the thing
and have non of the pain and hassle of actually owning and running a
pile of junk that might on a good day be called a computer."

   This is actually nothing new - we had things called "servers" for a
while now - and
network computing and clusters and virtual machines.

   What seems to be pertinent is management - from my point of view
(and I'm thinking Erlang
here) what I want is a distributed Erlang node up and running on a
remote machine and
I don't care zoot dinkels about how it got there. Since management is
"out of band" I could
happily manage this through a web interface - all need is a form
saying "create N erlang nodes"
I click on "doit" and N erlang nodes are are installed an the
cheapest, lowest-latency, largest
storage, fastest bandwidth kick-arse machine in the universe (has
anybody written this?)

But I digress ...

Tentative answer

     cloud = "a set of abstractions"

     so what are the abstractions?

    The Amazon Web Services says essentially

     Cloud = "CPU + Storage + Queues + management"

     (where CPU = EC2, storage = simple DB etc ...)

     They also have an abstraction "elastic map reduce" (interesting)

     Basing things on messages and queue is of course "the erlang way"
     (ie you send a message to a process mailbox, in order to get it
to do something)
     but for reliable large-scale computing we would need persistent queues with
     reliable delivery etc.

     I guess a good start point for an erlang cloud would be "a set of
erlang nodes"
     (that's the "CPU") - Rabbit MQ (for queues) and a database
back-end of your choice.

     Then you have to manage the system - create all the nodes etc,
make it secure etc.
     This seems to be a mess - how do I create 1000 EC2 instances each
with an Erlang node?
     I just want Erlang, but am forced to install a host OS just for
the purpose. What is the
     minimal OS I need to do this?

     ...


    I'll stop here

>>

>
> that is my main worry: say I have something that uses spawn + receive +
> messages.  how do I make it scale into the cloud?  where can I find online
> documentation on this?  i want my erlang program to run on as many systems
> as I need it to for however many users.  how do I do that?
>
> - Kid Erlang
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@...
> http://www.erlang.org/mailman/listinfo/erlang-questions
>

_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Steve Davis-5]

I'll have a stab,,,

"cloud" the mechanism by which application processing and data is
_automatically distributed_ across hardware resources to meet user
demand.

...so I'm not sure that one exists.

/s


On May 26, 3:00 am, Joe Armstrong <erl...@...> wrote:

> On Tue, May 26, 2009 at 8:01 AM, Kid Erlang <kiderl...@...> wrote:
> > On Tue, May 19, 2009 at 2:26 AM, Bengt Kleberg <bengt.kleb...@...>
> > wrote:
>
> >> The return value of spawn/1 is the (Erlang) process identity. If you
> >> want data from a process you have to send it (using !) and receive it
> >> (using receive).
>
> > okay I have rewrite my project to use receive with spawn, but I still do not
> > get how to handle errors?  especially in the cloud?
>
> >> It would be a good idea to read one of the Erlang books or the online
> >> documentation.
>
> > what book is best for me to read?
>
> Modesty prevents me from recommending "Programming Erlang".
>
> > and I cannot find online documentation
>
> http://www.erlang.org/doc.html
>
> > for how to use spawn + receive + messages to scale into the cloud?
>
> What is a cloud?
>
> << seriously! - Just what is a cloud? -
>
>    I asked myself just this question the other evening -
>
>  Now I did actually go to "a glory glory hallelujah, the cloud is COOL
> and WOW - MAN - TWITTER ..." meeting in London - each participant was
> given 45 seconds to explain their next greatest "WOW COOL" cloud
> application to an assembly of what appeared to be thirteen year old
> programmers (the kind that think that PHP is cool) and 25+ ish wannabe
> business types
> in shiny suits" - I stood at the back for listening for a few minutes
> with a group of like-minded souls - after 10 minutes we could take it
> no more - we retired to the pub.
>
>    So the the cloud appears to be "a computer at the other end of a
> bit of wire, that
> somebody else will setup and manage so that you can just pay the bills
> and use the thing
> and have non of the pain and hassle of actually owning and running a
> pile of junk that might on a good day be called a computer."
>
>    This is actually nothing new - we had things called "servers" for a
> while now - and
> network computing and clusters and virtual machines.
>
>    What seems to be pertinent is management - from my point of view
> (and I'm thinking Erlang
> here) what I want is a distributed Erlang node up and running on a
> remote machine and
> I don't care zoot dinkels about how it got there. Since management is
> "out of band" I could
> happily manage this through a web interface - all need is a form
> saying "create N erlang nodes"
> I click on "doit" and N erlang nodes are are installed an the
> cheapest, lowest-latency, largest
> storage, fastest bandwidth kick-arse machine in the universe (has
> anybody written this?)
>
> But I digress ...
>
> Tentative answer
>
>      cloud = "a set of abstractions"
>
>      so what are the abstractions?
>
>     The Amazon Web Services says essentially
>
>      Cloud = "CPU + Storage + Queues + management"
>
>      (where CPU = EC2, storage = simple DB etc ...)
>
>      They also have an abstraction "elastic map reduce" (interesting)
>
>      Basing things on messages and queue is of course "the erlang way"
>      (ie you send a message to a process mailbox, in order to get it
> to do something)
>      but for reliable large-scale computing we would need persistent queues with
>      reliable delivery etc.
>
>      I guess a good start point for an erlang cloud would be "a set of
> erlang nodes"
>      (that's the "CPU") - Rabbit MQ (for queues) and a database
> back-end of your choice.
>
>      Then you have to manage the system - create all the nodes etc,
> make it secure etc.
>      This seems to be a mess - how do I create 1000 EC2 instances each
> with an Erlang node?
>      I just want Erlang, but am forced to install a host OS just for
> the purpose. What is the
>      minimal OS I need to do this?
>
>      ...
>
>     I'll stop here
>
>
>
>
>
>
>
> > that is my main worry: say I have something that uses spawn + receive +
> > messages.  how do I make it scale into the cloud?  where can I find online
> > documentation on this?  i want my erlang program to run on as many systems
> > as I need it to for however many users.  how do I do that?
>
> > - Kid Erlang
>
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questi...@...
> >http://www.erlang.org/mailman/listinfo/erlang-questions
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questi...@...://www.erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Ulf Wiger-3]

Steve Davis wrote:
> I'll have a stab,,,
>
> "cloud" the mechanism by which application processing and data is
> _automatically distributed_ across hardware resources to meet user
> demand.
>
> ...so I'm not sure that one exists.
>
> /s

So, in the old days, when computers were physical, steam-powered
beasts that you could actually see and touch, I believe this
would have been called a 'cluster'.

According to Wikipedia:
"A computer cluster is a group of linked computers, working together
closely so that in many respects they form a single computer."

By comparison, Wikipedia describes Cloud Computing as
"a style of computing in which dynamically scalable and often
virtualized resources are provided as a service over the Internet"

One concrete difference then, would be that you'd expect far
more control over a cluster - 100% attention from each CPU,
and a fast interconnect with predictable characteristics.

Computing clouds, by contrast, seem much more 'best effort', where
you will statistically get at least as much CPU capacity as you've
paid for, and the communication backbone will exhibit some average
level of 'goodness' (hopefully).

A traditional problem with cluster-based applications has been that
managing state becomes terribly difficult - you have to worry about
locality of state and timing-dependent behaviour, in the form of
accidental sequencing of messages, etc. I think many of these
problems ought to be aggravated if available CPU and network
capacity become less predictable.

Relatively speaking, it is easier to avoid these problems in
Erlang than in most other programming languages. Still, having
spent years discussing the scary issues with "cluster computing",
I don't necessarily see "cloud computing" as a step in the right
direction in all respects.

BR,
Ulf W
--
Ulf Wiger
CTO, Erlang Training & Consulting Ltd
http://www.erlang-consulting.com
_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Tim Fletcher-5]

> "cloud" the mechanism by which application processing and data is
> _automatically distributed_ across hardware resources to meet user
> demand.
>
> ...so I'm not sure that one exists.

For Ruby: http://heroku.com/

For Python/JVM: http://code.google.com/appengine/

For Javascript: http://reasonablysmart.com/

All in their infancy. AFAIK no such "auto scaling" service currently
exists for Erlang applications.
_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Joe Armstrong-2]

On Tue, May 26, 2009 at 2:46 PM, Tim Fletcher <twoggle@...> wrote:
>> "cloud" the mechanism by which application processing and data is
>> _automatically distributed_ across hardware resources to meet user
>> demand.
>>
>> ...so I'm not sure that one exists.
>
> For Ruby: http://heroku.com/


http://highscalability.com/heroku-simultaneously-develop-and-deploy-automatically-scalable-rails-applications-cloud

Suggest the fun bit is implemented In erlang. This being the case
one wonders if heroku might provide a hook to the underlying
Erlang bit since they seem to have thought about the problem of
creating large numbers of EC2 instances

/J

>
> For Python/JVM: http://code.google.com/appengine/
>
> For Javascript: http://reasonablysmart.com/
>
> All in their infancy. AFAIK no such "auto scaling" service currently
> exists for Erlang applications.
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@...
> http://www.erlang.org/mailman/listinfo/erlang-questions
>

_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Tony Arcieri]

On Tue, May 26, 2009 at 2:00 AM, Joe Armstrong <erlang@...> wrote:

What is a cloud?

It's basically the same as any other distributed environment, but there's a central authority you can go to and ask "I needs me more computars ktks" and it's like "ok thar you be bai" and you can get as many computars as you need to get job X done.

That's really the only difference, except for the fact that when you're done the central authority is like "I need to eat the whole state of your instance in order to survive" and maybe you're like "no!" but it's like "I must I'm huuungary" and maybe with an environment like Amazon EC2 they have Elastic Block Store so you can rip out the guts of your state and stash them somewhere before the central authority beast eats up the state and resources of your instance and recycles them into new instances.

--
Tony Arcieri
medioh.com

_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Joe Armstrong-2]

So it sounds like a cloud is a dynamic pool to which you can independently add
workers and jobs. The jobs "just get done" and are allocated to the
best workers for
the job - there is a management interface so you can ask how the system is
doing, get statistics on the jobs etc.

What (Erlang) abstractions do we need?

Here's a suggestion for a cloud API

    cloud:add_job(Job) -> JobRef
          Job is a fun/0 that evaluates to result()
    cloud:status(JobRef) -> queued | {running, Where} | done | {suspended, Why}
          Query the status of the job
    cloud:result(JobRef) -> result().
           Can be called when status= done to get the result
    cloud:delete_job(JobRef) -> true
           remove all traces of the job

And the managment API

      cloud:add_node(Host, Port) -> NodeRef
      cloud:remove_node(NodeRef)
      cloud:nodes() -> [NodeRef]
      cloud:info() -> job_and_node_statistics()

I guess we might need a persistent storage abstraction to aid writing the
jobs. Lets assume the job refererences are used as keys into a persistent
fault-tolerent store

Then we need

     cloud:mutate_store(Fun/0) -> commit | fail

Fun0 is a function which can call

     cloud:store(JobRef, Key, Val)
     cloud:fetch(JobRef, key)

This is done with transaction semantics.

The API is intentionally minimal - How could this be implemented?
a layer on top of scalaris or Rabbit MQ?

Is this API complete?

Anybody want to build a prototype?

/Joe






On Wed, May 27, 2009 at 8:34 AM, Tony Arcieri <tony@...> wrote:

> On Tue, May 26, 2009 at 2:00 AM, Joe Armstrong <erlang@...> wrote:
>>
>> What is a cloud?
>
> It's basically the same as any other distributed environment, but there's a
> central authority you can go to and ask "I needs me more computars ktks" and
> it's like "ok thar you be bai" and you can get as many computars as you need
> to get job X done.
>
> That's really the only difference, except for the fact that when you're done
> the central authority is like "I need to eat the whole state of your
> instance in order to survive" and maybe you're like "no!" but it's like "I
> must I'm huuungary" and maybe with an environment like Amazon EC2 they have
> Elastic Block Store so you can rip out the guts of your state and stash them
> somewhere before the central authority beast eats up the state and resources
> of your instance and recycles them into new instances.
>
> --
> Tony Arcieri
> medioh.com
>

_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Zvi]

Joe Armstrong-2 wrote:

What is a cloud?

Cloud - is a Datacenter with API!

Zvi

[Kid Erlang]

hi everyone.  i am still confused?  is there a good book on ERLANG to read
on how to do multi process cloud scaling?

i do not understand your answers

i want to scale into the cloud to generate bigger sets of numbers and be
able to run multiple copies of my scripts which already run in a collection.

how do I get erlang to communicate across different computers in the cloud?
do I need to use sockets?  sockets are confusing.  right now I am using
pipes for my scripts which is very easy.  I can't figure out how to do pipes
in erlang.  I create process and try to write to its pipe but then it
doesn't seem to do anything.

here is my source code program:

-module(number_adder).
-compile(export_all).

add_up_numbers(NumberArray)->
  [Number|OtherNumbers]=NumberArray,add_number(Number,OtherNumbers).
add_number(Number, OtherNumbers) ->
  [NextNumber|StillToBeAdded]=OtherNumbers,
  spawn(fun()->number_adder(Number, NextNumber)end),
  receive ReturnValue-> ReturnValue+add_up_numbers(StillToBeAdded)end.
number_adder(Number1,Number2)->Number1+Number2.

try it out it just hangs for some reason and I can't figure out why.  I am
trying to add up the numbers in paralell

- Kid Erlang

On Wed, May 27, 2009 at 12:09 PM, Steve Davis <
steven.charles.davis@...> wrote:

>
>
> On May 26, 4:58 am, Ulf Wiger <ulf.wi...@...> wrote:
> > Steve Davis wrote:
> > > I'll have a stab,,,
> >
> > > "cloud" the mechanism by which application processing and data is
> > > _automatically distributed_ across hardware resources to meet user
> > > demand.
> >
> > > ...so I'm not sure that one exists.
> >
> > > /s
> >
> > So, in the old days, when computers were physical, steam-powered
> > beasts that you could actually see and touch, I believe this
> > would have been called a 'cluster'.
> >
>
> On May 26, 4:58 am, Ulf Wiger <ulf.wi...@...> wrote:
> >
> > So, in the old days, when computers were physical, steam-powered
> > beasts that you could actually see and touch, I believe this
> > would have been called a 'cluster'.
> >
>
> Perhaps the definition should read "to meet unlimited user demand".
>
> I suspect that the big issue with doing this is probably not the
> processing part but the (persistent) data part.
>
> BigTable and S3 solve this by accepting "eventual
> consistency" (generally a few seconds but can be more). For many
> applications this is enough (e.g. google search), since the now-ness/
> consistency of the data is not mission critical.
>
> Where it is mission critical, even a clustered transactional/
> relational layer will eventually suffer unacceptable performance
> degradation.
>
> I'm not sure I have seen a solution to that, and I'm not even sure
> that it is physically possible. Of course, just because I personally
> cannot see how, that doesn't mean it isn't possible.
>
> /s
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>
>

[Matthew Palmer-2]

On Sat, May 30, 2009 at 02:08:18AM -0600, Kid Erlang wrote:
> hi everyone.  i am still confused?  is there a good book on ERLANG to read
> on how to do multi process cloud scaling?
>
> i do not understand your answers
>
> i want to scale into the cloud

*facepalm*

The point of the answers you've been given is that the phrase "scale into
the cloud" is basically gibberish.  There's no clear definition of "the
cloud", and depending on what you think "the cloud" actually is, you could
require very different solutions.

Practically, what you probably want is a way to create lots of Erlang nodes
running a very simple program which can receive code updates and run them.
There's some examples of this sort of thing in /Programming Erlang/, and
probably lots of other places, too.

- Matt

--
"A cat spends her life conflicted between a deep, passionate and profound
desire for fish and an equally deep, passionate and profound desire to avoid
getting wet.  This is the defining metaphor of my life right now."
                -- Unknown, Seen on the 'net

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Ulf Wiger-3]

----- "Kid Erlang" <kiderlang@...> wrote:

> hi everyone.  i am still confused?  is there a good book on
> ERLANG to read on how to do multi process cloud scaling?
>
> i do not understand your answers

Apologies for the somewhat existential rants.

You should read Joe Armstrong's book, which explains a lot
of things, including how to scale with Distributed Erlang
and multicore.

If you use Distributed Erlang, that is, many Erlang nodes
connected in the most natural Erlang way, you will simply
use processes and message passing. When using sockets, you
can write a module similar to the rpc module, but which
calls term_to_binary(Data) before sending the request to
the socket, and binary_to_term(Bin) on the response coming
back (and vice versa on the other end). It is reasonably
straightforward. Joe's book has a chapter (ch 14) on
socket programming, which pretty much spells out how to
do this.

One of the new problems with clouds is how host names can
come and go, making it difficult to have the kind of static
configuration of the members as one usually has in a cluster
(where one usually has full control over the IP addresses,
host names, etc.)

Joel Reymont used to have a blog article about setting up
mnesia for EC2, which AFAIR addressed these issues, but
the article seems to have rotted away.

I found this:
http://blog.onclearlake.ca/2008/03/setup-ec2-instance-with-erlang-this.html


> i want to scale into the cloud to generate bigger sets of
> numbers and be able to run multiple copies of my scripts which
> already run in a collection.

In your example, you are basically doing addition. You should
be aware that this form of work cannot be sped up even by
parallelizing on multicore. Just running through the list and
spawning a process (or even sending a message) is more work than
just performing the addition.

I showed an example of that in my Erlang Factory presentation:
http://www.erlang-factory.com/upload/presentations/56/UlfWiger_ErlangMulticoreEF.pdf
(slide 9 shows some simple benchmark figures on two different
types of jobs being parallelized.)

BR,
Ulf W
--
Ulf Wiger
CTO, Erlang Training & Consulting Ltd.
http://www.erlang-consulting.com

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Joel Reymont]

On May 30, 2009, at 9:48 AM, Ulf Wiger wrote:

> Joel Reymont used to have a blog article about setting up
> mnesia for EC2, which AFAIR addressed these issues, but
> the article seems to have rotted away.

The articles are at http://thinkerlang.com and I plan to add to the  
collection now that I'm scaling RabbitMQ, ejabberd, etc. on Amazon EC2.

Tying the new EC2 load-balancing, scaling and monitoring services into  
Erlang should work particularly well!

The intractable (I think) problem is using Mnesia as a backend for  
internet services. I don't think it can be used since there's now way  
to automatically sync up the various database instances after a  
network split.

Imagine customers transacting with two separate Mnesia db instances  
without without replication happening between them. Horrors and  
balances gone awry!

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Ulf Wiger-3]

----- "Joel Reymont" <joelr1@...> wrote:

> The intractable (I think) problem is using Mnesia as a backend for  
> internet services. I don't think it can be used since there's now way
>
> to automatically sync up the various database instances after a  
> network split.

I think it ought to be fairly straightforward to design
an arbitrator similar to the one used by MySQL Cluster.
The tools needed to detect partitioned network and to
resolve it are there - the thing that mnesia doesn't
provide is a default resolution algorithm. If the
algorithm used by MySQL Cluster is acceptable, I see
no reason why it couldn't be applied to Mnesia as well.

(For those who don't want to go chasing for it,
the MySQL Arbitrator basically selects the biggest
group of connected nodes, or the group that reported
to the Arbitrator last. This is from memory. The details
may be wrong.

FWIW, the AXD 301 used a somewhat similar arbitration
mechanism: each node tried to ping itself through the
switch core. If it couldn't, it had no means to control
the HW resources and had to reboot; otherwise, the
node that had a stable copy of the traffic handling
application became the master; if both nodes were
running traffic handling (which could happen if
the outage lasted a few seconds), I believe the one
hard-wired as Control Processor 1 was designated
master.)

Bottom line: if you can pick an arbitration algorithm
that offers suitable recovery behaviour, I believe you
can implement it on top of what mnesia provides.
One could imagine a peer-to-peer resolution protocol
(difficult to make generic) or a 'central observer',
which is the MySQL strategy.

Or are you referring to the lack of a 'merge' of
two table copies? This is a more difficult thing to
add. Right now, that would have to be done by
extracting records from the non-master, have it
restart and load tables from the master, then
re-inserting the records. Since this cannot be done
within a larger transaction, it might be necessary to
delay requests from the application layer during this
time.

Which DBMSes have a good solution to this problem?
How do they do it? And what prevents using the same
strategy with mnesia?

(Mnesia's supposed lack of support for handling
partitioned networks has more or less become an
urban legend. Everyone assumes that this is a
major weakness in mnesia, but there is very little
detail about how the competing alternatives
supposedly handle this in a much better way.
I would love to be enlightened about this.)

BR,
Ulf W
--
Ulf Wiger
CTO, Erlang Training & Consulting Ltd.
http://www.erlang-consulting.com

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Ulf Wiger-3]

----- "Ulf Wiger" <ulf.wiger@...> wrote:

> FWIW, the AXD 301 used a somewhat similar arbitration
> mechanism: each node tried to ping itself through the
> switch core.

For newcomers who haven't studied the ancient scrolls
about the AXD 301, one particular 'feature' of that
system was that the control processors communicated
via the very ATM switch that they were controlling.
As a consequence, if the ATM device boards were unstable
(which could happen during some phases of development),
partitioned network could occur daily, or even
many times a day. Since I was in charge of writing the
cluster controller and solving the issues with mnesia
during that time, I recall those periods vividly.

The AXD 301 *did* recover automatically from partitioned
networks, and had to, since it was usually installed in
very remote locations (still is, I should say, since it
is being sold and used to this day.)

I submit this as proof that it is *not* impossible to
recover automatically from partitioned networks when
using mnesia. I think that there should be some behaviour
for resolving it using an observer, and a default
algorithm for those who don't want to roll their own.

But there's no need for hacking mnesia in order to provide
that.

BR,
Ulf W
--
Ulf Wiger
CTO, Erlang Training & Consulting Ltd.
http://www.erlang-consulting.com

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Kid Erlang]

where can I get joe armstrongs book?

i was not trying to speed up adding by paralell it was just trivial example
to see if I could use erlang to paralell up my program.  and my original
program was trying to compute average which may work paralell better than
adding

why is it not working? :(

On Sat, May 30, 2009 at 2:48 AM, Ulf Wiger
<ulf.wiger@...>wrote:

>
> ----- "Kid Erlang" <kiderlang@...> wrote:
>
> > hi everyone.  i am still confused?  is there a good book on
> > ERLANG to read on how to do multi process cloud scaling?
> >
> > i do not understand your answers
>
> Apologies for the somewhat existential rants.
>
> You should read Joe Armstrong's book, which explains a lot
> of things, including how to scale with Distributed Erlang
> and multicore.
>
> If you use Distributed Erlang, that is, many Erlang nodes
> connected in the most natural Erlang way, you will simply
> use processes and message passing. When using sockets, you
> can write a module similar to the rpc module, but which
> calls term_to_binary(Data) before sending the request to
> the socket, and binary_to_term(Bin) on the response coming
> back (and vice versa on the other end). It is reasonably
> straightforward. Joe's book has a chapter (ch 14) on
> socket programming, which pretty much spells out how to
> do this.
>
> One of the new problems with clouds is how host names can
> come and go, making it difficult to have the kind of static
> configuration of the members as one usually has in a cluster
> (where one usually has full control over the IP addresses,
> host names, etc.)
>
> Joel Reymont used to have a blog article about setting up
> mnesia for EC2, which AFAIR addressed these issues, but
> the article seems to have rotted away.
>
> I found this:
> http://blog.onclearlake.ca/2008/03/setup-ec2-instance-with-erlang-this.html
>
>
> > i want to scale into the cloud to generate bigger sets of
> > numbers and be able to run multiple copies of my scripts which
> > already run in a collection.
>
> In your example, you are basically doing addition. You should
> be aware that this form of work cannot be sped up even by
> parallelizing on multicore. Just running through the list and
> spawning a process (or even sending a message) is more work than
> just performing the addition.
>
> I showed an example of that in my Erlang Factory presentation:
>
> http://www.erlang-factory.com/upload/presentations/56/UlfWiger_ErlangMulticoreEF.pdf
> (slide 9 shows some simple benchmark figures on two different
> types of jobs being parallelized.)
>
> BR,
> Ulf W
> --
> Ulf Wiger
> CTO, Erlang Training & Consulting Ltd.
> http://www.erlang-consulting.com
>