how to scale into the cloud using process? example computing simple average

[Ulf Wiger-3]

----- "Kid Erlang" <kiderlang@...> wrote:

> where can I get joe armstrongs book?

http://www.pragprog.com/titles/jaerlang/programming-erlang

(If you're in a real hurry, they sell it as PDF.)

You can also get it from Amazon, and many decent book stores
in the real world (or AFK*, as one is apparently supposed to
say nowadays, according to the defendants in the Pirate Bay
trial.)

* Away From Keyboard, as "the internet is real".

BR,
Ulf W
--
Ulf Wiger
CTO, Erlang Training & Consulting Ltd.
http://www.erlang-consulting.com

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Steve Davis-5]

I too got triggered off by the mention of that word "cloud".

Note that each process that you spawn never sends a message back with
the result!

Have a good read about spawning processes and particularly about inter-
process messaging with send and receive in Joe's book. You can get the
PDF immediately from...

http://www.pragprog.com/titles/jaerlang/programming-erlang

regs,
/s



On May 30, 12:42 pm, Kid Erlang <kiderl...@...> wrote:

> where can I get joe armstrongs book?
>
> i was not trying to speed up adding by paralell it was just trivial example
> to see if I could use erlang to paralell up my program.  and my original
> program was trying to compute average which may work paralell better than
> adding
>
> why is it not working? :(
>
> On Sat, May 30, 2009 at 2:48 AM, Ulf Wiger
> <ulf.wi...@...>wrote:
>
>
>
>
>
> > ----- "Kid Erlang" <kiderl...@...> wrote:
>
> > > hi everyone.  i am still confused?  is there a good book on
> > > ERLANG to read on how to do multi process cloud scaling?
>
> > > i do not understand your answers
>
> > Apologies for the somewhat existential rants.
>
> > You should read Joe Armstrong's book, which explains a lot
> > of things, including how to scale with Distributed Erlang
> > and multicore.
>
> > If you use Distributed Erlang, that is, many Erlang nodes
> > connected in the most natural Erlang way, you will simply
> > use processes and message passing. When using sockets, you
> > can write a module similar to the rpc module, but which
> > calls term_to_binary(Data) before sending the request to
> > the socket, and binary_to_term(Bin) on the response coming
> > back (and vice versa on the other end). It is reasonably
> > straightforward. Joe's book has a chapter (ch 14) on
> > socket programming, which pretty much spells out how to
> > do this.
>
> > One of the new problems with clouds is how host names can
> > come and go, making it difficult to have the kind of static
> > configuration of the members as one usually has in a cluster
> > (where one usually has full control over the IP addresses,
> > host names, etc.)
>
> > Joel Reymont used to have a blog article about setting up
> > mnesia for EC2, which AFAIR addressed these issues, but
> > the article seems to have rotted away.
>
> > I found this:
> >http://blog.onclearlake.ca/2008/03/setup-ec2-instance-with-erlang-thi...
>
> > > i want to scale into the cloud to generate bigger sets of
> > > numbers and be able to run multiple copies of my scripts which
> > > already run in a collection.
>
> > In your example, you are basically doing addition. You should
> > be aware that this form of work cannot be sped up even by
> > parallelizing on multicore. Just running through the list and
> > spawning a process (or even sending a message) is more work than
> > just performing the addition.
>
> > I showed an example of that in my Erlang Factory presentation:
>
> >http://www.erlang-factory.com/upload/presentations/56/UlfWiger_Erlang...
> > (slide 9 shows some simple benchmark figures on two different
> > types of jobs being parallelized.)
>
> > BR,
> > Ulf W
> > --
> > Ulf Wiger
> > CTO, Erlang Training & Consulting Ltd.
> >http://www.erlang-consulting.com

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Joel Reymont]

On May 30, 2009, at 1:44 PM, Ulf Wiger wrote:

> Or are you referring to the lack of a 'merge' of
> two table copies?

Yes.

> This is a more difficult thing to
> add. Right now, that would have to be done by
> extracting records from the non-master, have it
> restart and load tables from the master, then
> re-inserting the records. Since this cannot be done
> within a larger transaction, it might be necessary to
> delay requests from the application layer during this
> time.

I think the only way to solve this would be to keep a separate  
transaction log. Then you could try to merge the logs from  
disconnected nodes and re-apply transactions in different order to  
build a merged database.

Then again, who knows what the right order is?

Either way, it does not bode well to keeping "account balances" in  
Mnesia for example, since you could easily overdraw your account if  
you manage to hit two disconnected nodes.

> Which DBMSes have a good solution to this problem?
> How do they do it? And what prevents using the same
> strategy with mnesia?

I think Oracle has a solution. I heard that they use 3 copies of the  
database to settle on a good one.


> (Mnesia's supposed lack of support for handling
> partitioned networks has more or less become an
> urban legend.

An urban legend is a story that's untrue ;-).

> Everyone assumes that this is a
> major weakness in mnesia, but there is very little
> detail about how the competing alternatives
> supposedly handle this in a much better way.

This is mostly because almost no competing alternative tries to build  
a multi-master cluster with real-time replication.

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Ulf Wiger-3]

Joel Reymont wrote:
>
> On May 30, 2009, at 1:44 PM, Ulf Wiger wrote:
>
> I think the only way to solve this would be to keep a separate
> transaction log. Then you could try to merge the logs from
> disconnected nodes and re-apply transactions in different order to
> build a merged database.
>
> Then again, who knows what the right order is?

This is one of the reasons why it doesn't happen
automatically. Merging of records after a split-brain
condition is an application-specific problem.
Personal Information Managers have had this problem for
ages, and usually resort to presenting the user with a
list of records that couldn't be deconflicted automatically.

Source code merging is another problem with corner cases
requiring human intervention.


> Either way, it does not bode well to keeping "account balances" in
> Mnesia for example, since you could easily overdraw your account if
> you manage to hit two disconnected nodes.

...and when building the AXD 301, we were not thrilled by
the prospect of having two control processors simultaneously
controlling the resources in the ATM switch fabric, so we
designed checks to make sure that the processors would be
able to determine the source of the problem by asking a third
party (the switch).

>> Which DBMSes have a good solution to this problem? How do they do
>> it? And what prevents using the same strategy with mnesia?
>
> I think Oracle has a solution. I heard that they use 3 copies of the
>  database to settle on a good one.

What prevents you from doing something similar with mnesia?

>> (Mnesia's supposed lack of support for handling partitioned
>> networks has more or less become an urban legend.
>
> An urban legend is a story that's untrue ;-).

Exactly. :)

>> Everyone assumes that this is a major weakness in mnesia, but there
>> is very little detail about how the competing alternatives
>> supposedly handle this in a much better way.
>
> This is mostly because almost no competing alternative tries to build
> a multi-master cluster with real-time replication.

As far as I know, just about all competing alternatives offer
multimaster replication: Oracle, Ingres, MySQL, PostgreSQL,
Sybase, SQL Server and DB2 come to mind.

At least in one or some of those, if memory serves, the way
to recover from partitioned network was to initiate disaster
recovery and restart from a stable backup. In one study, this
was noted as an example of the type of automatic handling of
partitioned networks that mnesia supposedly was unable to
provide.

Someone with plenty of time on their hands could perhaps
inventory the strategies of these products and identify
which of these strategies could not be implemented on top
of mnesia without major surgery.

BR,
Ulf W
--
Ulf Wiger
CTO, Erlang Training & Consulting Ltd.
http://www.erlang-consulting.com

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org