tag:old.nabble.com,2006:forum-14095Nabble - Erlang2009-11-09T09:42:39ZErlang is a programming language which has many features more commonly associated with an operating system than with a programming language: concurrent processes, scheduling, memory management, distribution, networking, etc. Erlang home is <a href="http://www.erlang.org/" target="_top" rel="nofollow">here</a>.tag:old.nabble.com,2006:post-26271203Re: Why Beam.smp crashes when memory is over?2009-11-09T09:42:39Z2009-11-09T09:42:39ZJayson Vantuyl-2The problem is that there is no way for the VM to really know which
<br>process is leaking. The Linux OOM-killer has gone through a bunch of
<br>rewrites and it still doesn't always get it right.
<br><br>It might be possible to have a threshold trigger that would hit some
<br>sort of global alarm that could be application specific. That's
<br>probably the closest you could get though.
<br><br>On Nov 8, 2009, at 9:08 PM, Max Lapshin wrote:
<br><div class='shrinkable-quote'><br>> Look, I have a program, that reads MPEG TS stream from network. There
<br>> is some bug in my code
<br>> and my ts_lander crashes beam. How is it possible to store state of
<br>> external HTTP stream??
<br>>
<br>> If Erlang VM wouldn't crash, but only kill the leaking process, I
<br>> could buffer stream in other process and feed
<br>> new mpeg ts lander after restart. But it is impossible =(
<br>>
<br>> ________________________________________________________________
<br>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> erlang-questions (at) erlang.org
<br>>
</div><br><br><br>--
<br>Jayson Vantuyl
<br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26271203&i=0" target="_top" rel="nofollow">kagato@...</a>
<br><br><br><br><br><br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26271159Re: Why Beam.smp crashes when memory is over?2009-11-09T09:40:55Z2009-11-09T09:40:55ZJayson Vantuyl-2You could run two Erlang VMs (thus, keeping the HTTP connection and
<br>its state separate). Then the crasher wouldn't take the other one
<br>down. One with the processing and one with the networking, and have
<br>them communicate via distributed Erlang. You should still ulimit beam
<br>to keep the other one from getting OOM-killed though.
<br><br>Assuming you can get the throughput you need, this also would
<br>decompose some of the components, which might helps scale horizontally
<br>later.
<br><br>On Nov 8, 2009, at 9:08 PM, Max Lapshin wrote:
<br><div class='shrinkable-quote'><br>> Look, I have a program, that reads MPEG TS stream from network. There
<br>> is some bug in my code
<br>> and my ts_lander crashes beam. How is it possible to store state of
<br>> external HTTP stream??
<br>>
<br>> If Erlang VM wouldn't crash, but only kill the leaking process, I
<br>> could buffer stream in other process and feed
<br>> new mpeg ts lander after restart. But it is impossible =(
<br>>
<br>> ________________________________________________________________
<br>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> erlang-questions (at) erlang.org
<br>>
</div><br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26270548Re: AW: Why Beam.smp crashes when memory is over?2009-11-09T09:38:59Z2009-11-09T09:38:59ZJayson Vantuyl-2This would be a very useful tunable.
<br><br>On Nov 9, 2009, at 2:31 AM, Decker, Nils wrote:
<br><div class='shrinkable-quote'><br>> Hello,
<br>>
<br>> I had a few occasions during developing that a runaway process
<br>> caused my machine to swap and gind to a halt. (i should have used
<br>> ulimit on beam)
<br>>
<br>> I wondered why there is no way to limit the size of a single
<br>> process. It could be a simple option to spawn to limit the heap size
<br>> of a process. (like ulimit)
<br>> The process gets killed if it ever grows beyond the limit. A limit
<br>> for the size of the message queue would be nice too, because a
<br>> process with a few thousand entries is (most of the time) not going
<br>> to cope with it anyway.
<br>> A similar limit for ETS tables would be possible too. A memory limit
<br>> for shared resources (large binaries) would be more difficult.
<br>>
<br>> Nils Decker
</div><br><br><br>--
<br>Jayson Vantuyl
<br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26270548&i=0" target="_top" rel="nofollow">kagato@...</a>
<br><br><br><br><br><br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26269216Re: Why Beam.smp crashes when memory is over?2009-11-09T08:21:41Z2009-11-09T08:21:41ZAngel Alvarez<br>El Lunes, 9 de Noviembre de 2009 Tony Rogvall escribió:
<div class='shrinkable-quote'><br>> Hi!
<br>>
<br>> On 9 nov 2009, at 14.54, Angel Alvarez wrote:
<br>>
<br>> > Well still there are many issues with this new approach
<br>> >
<br>> Yes! But it does not scare me ;-)
<br>>
<br>> > Where are the maibox of processes located?
<br>> >
<br>> > With a heap pre process...
<br>>
<br>> Depends on the implementation. But in general you could do something
<br>> like, If the the data is shared then
<br>> you split the share (memory_size/ref_count). If the data is copied
<br>> then you must count it in.
<br>>
<br>> >
<br>> > Couldnt you trigger a memory exception on a remote process by just
<br>> > sending one message
<br>> > when the process is almost consuming its reserved memory?
<br>> >
<br>> Yes. But that is the point. If it pass the limit the process will die.
<br>> There are many special cases where you could think of using the memory
<br>> in a better and more optimal way.
<br>> Lets say you are reaching the memory limit you may switch to a
<br>> compression algorithm for heap memory !?
<br>> But lets keep it simple for the prototype, and see if it useful.
<br>>
<br>>
<br>> > Systems other than embedded erlangs deploy ( form de current erlang
<br>> > movement as a server/desktop plataform)
<br>> > will suffer from resource contention beetween erlang VM and other OS
<br>> > processes.
<br>> >
<br>> > Port programs also need system resources...
<br>> >
<br>> For loadable drivers using driver_alloc, one could possibly do
<br>> something, otherwise it will be
<br>> up to the driver designer to handle it. There is a max_ports in the
<br>> prototype that limits number of
<br>> open_ports. If sockets/files are mapped to single ports then it may
<br>> help a bit.
<br>>
<br>>
<br>> > Well in the end your approach is still very interesting as a
<br>> > framework for continous erlang VM innovations...
<br>> >
<br>> Thanks.
<br>>
<br>> > but please correct me if im wrong but I saw that memory carriers
<br>> > allowed to set several options on erlang VM start-up so,
<br>> >
<br>> I am not sure what you mean here?
</div><br>Well some bits can be controlled as you show on spawn_opt. In the other hand i mean that VM Safe memory
<br>as stated by others (Andrew, Tom ...) should be controlled on memory carriers: ( erts_alloc(3) )
<br><br>In esence
<br><br>- A new emergency memory alocator in the alloc_util framework just in case the VM need memory to recover from a OOM.
<br><br>or perhaps
<br><br><WARNING speculative mode ON>
<br><br>- Using the segment allocator on mmap supported arquitectures to allow fast recovery for the full erlang vm (sort of checkpointing)
<br>using a special BIF you could instruct memory carriers to checkpoint the entire VM usining this allocator just in case the VM crashes.
<br>so on next run the enrire VM can be recovered parsing the last checkpint
<br><br></WARNING>
<br><br><br>Still i think is better and more clean having two o more instances that bulletprooffing the VM
<br><br>/Angel
<br><div class='shrinkable-quote'><br>>
<br>> /Tony
<br>>
<br>> > is stil posible to pacth those carriers to allow a safe memory
<br>> > reservation to let de VM manage properly a memory full
<br>> > condition by killing the offending process (sort of a OOM killer for
<br>> > the VM)?
<br>> >
<br>> > Just telling the VM not to "kill system process" and let the
<br>> > supervisors do the work...
<br>> >
<br>>
<br>>
<br>>
<br>> > /Angel
<br>> >
<br>> >
<br>> >
<br>> > El Lunes, 9 de Noviembre de 2009 Tony Rogvall escribió:
<br>> >>
<br>> >> It is still the same "let it crash" concept using the resource limit
<br>> >> system I am designing.
<br>> >> But you can limit the crash in a more controlled way. Also you will
<br>> >> be
<br>> >> able to report
<br>> >> interesting information about what is crashing and when.
<br>> >>
<br>> >> There is sometimes an issue when big systems crash. The restart may
<br>> >> take a lot of time.
<br>> >> Nodes must be synchronised, database tables must be repaired etc etc.
<br>> >> I guess you can design this to be light and easy, but it is not
<br>> >> always
<br>> >> the case.
<br>> >>
<br>> >> /Tony
<br>> >>
<br>> >>
<br>> >>
<br>> >> On 9 nov 2009, at 12.43, Angel Alvarez wrote:
<br>> >>
<br>> >>>
<br>> >>> Well please let me say something
<br>> >>>
<br>> >>> I'm plain new but some things are pretty clear for me.
<br>> >>>
<br>> >>>
<br>> >>> The beauty of the erlang concept is "let it crash" , "don't program
<br>> >>> defensively"
<br>> >>> so the VM and the underlaying hardware are entities that can fail,
<br>> >>> that's it.
<br>> >>>
<br>> >>> What's the problem so?
<br>> >>>
<br>> >>> Joe said...
<br>> >>>
<br>> >>> If you want failure tolerance you need at least two nodes...
<br>> >>>
<br>> >>> From J.A thesis
<br>> >>> " ...Schneider [60, 59] answered this question by giving three
<br>> >>> properties
<br>> >>> that he thought a hardware system should have in order to be
<br>> >>> suitable for
<br>> >>> programming a fault-tolerant system. These properties Schneider
<br>> >>> called:..."
<br>> >>>
<br>> >>> 1. Halt on failure — in the event of an error a processor should
<br>> >>> halt
<br>> >>> instead of performing a possibly erroneous operation.2
<br>> >>>
<br>> >>> So on memory exhaustation the VM has to die and other node (erlang)
<br>> >>> will do the recovery.
<br>> >>>
<br>> >>> that's the distrirbution role, no only to span computations over
<br>> >>> several nodes to enhance performance
<br>> >>> but to provide resilence in the presence of fatal errors (non
<br>> >>> correctable).
<br>> >>>
<br>> >>> As a OS process the VM has to compete with other OS processes so in
<br>> >>> a shared deployment (a VM running
<br>> >>> on a server or a desktop) you cant be safe agaisnt a OOM trigered by
<br>> >>> other entities.
<br>> >>>
<br>> >>> Such resource control thing wil only augment process overhead and
<br>> >>> context switching in the VM.
<br>> >>>
<br>> >>> People new to erlang will be atracted to this hierarquical
<br>> >>> decomposition of tasks as joe stated in his thesis
<br>> >>> "If you cant run certaing task try doing something simpler"
<br>> >>>
<br>> >>> Many languages and VM's are incorporating erlang's good "multicore"
<br>> >>> features but not the erlang powerfull error handling concept
<br>> >>> and you guys want to kill the simpliticy incorporating many
<br>> >>> defensive capabilities to avoid fatality instead of just organize
<br>> >>> code to
<br>> >>> handle such fatality.
<br>> >>>
<br>> >>> ¿whats next?, ¿A mailbox maximum message queue control?
<br>> >>>
<br>> >>>
<br>> >>> Well, that's all i have to say about that, Forrest Gump.
<br>> >>>
<br>> >>>
<br>> >>> El Lunes, 9 de Noviembre de 2009 09:45:10 Tony Rogvall escribió:
<br>> >>>> Interesting discussion!
<br>> >>>>
<br>> >>>> I have been working on a resource system for Erlang for nearly two
<br>> >>>> years now.
<br>> >>>> I have a working (tm) prototype where you can set resource limits
<br>> >>>> like
<br>> >>>> max_processes/max_ports/max_memory/max_time/max_reductions ...
<br>> >>>> The limits are passed with spawn_opt and are inherited by the
<br>> >>>> processes spawned.
<br>> >>>> This means that if you spawn_opt(M,F,A[{max_memory, 1024*1024}])
<br>> >>>> the
<br>> >>>> process
<br>> >>>> will be able to use 1M words for it self and it's "subprocesses".
<br>> >>>> This
<br>> >>>> also means
<br>> >>>> that the spawner will get 1M less to use (as designed right now).
<br>> >>>> If a
<br>> >>>> resource limit
<br>> >>>> is reached the process crash with system_limt reason.
<br>> >>>>
<br>> >>>> There are still some details to work out before a release, but I
<br>> >>>> will
<br>> >>>> try to get it ready before
<br>> >>>> end of this year.
<br>> >>>>
<br>> >>>> /Tony
<br>> >>>>
<br>> >>>>
<br>> >>>>
<br>> >>>> On 9 nov 2009, at 09.16, Robert Virding wrote:
<br>> >>>>
<br>> >>>>> No.
<br>> >>>>>
<br>> >>>>> There is a major difference between handling OOM in an OS and in
<br>> >>>>> the
<br>> >>>>> BEAM.
<br>> >>>>> In an OS it usually at a per process level that memory runs out so
<br>> >>>>> it is
<br>> >>>>> easy to decide which process to kill so that the OS can
<br>> >>>>> continue. In
<br>> >>>>> the
<br>> >>>>> BEAM, however, it is the VM as a whole which has run out of memory
<br>> >>>>> not a
<br>> >>>>> specific, it is. therefore, much more difficult to work out which
<br>> >>>>> process is
<br>> >>>>> the culprit and to decide what to do. For example it might be that
<br>> >>>>> the
<br>> >>>>> process which causes the OOM is not the actual problem process, it
<br>> >>>>> might
<br>> >>>>> just the last straw. Or the actual cause may that the whole app
<br>> >>>>> might be
<br>> >>>>> generating large binaries too quickly. Or it might be that the
<br>> >>>>> whole
<br>> >>>>> app is
<br>> >>>>> spawning to many processes without any one process being the
<br>> >>>>> cause.
<br>> >>>>> Or ...
<br>> >>>>> In all these cases killing the process which triggered the OOM
<br>> >>>>> would
<br>> >>>>> be the
<br>> >>>>> Wrong Thing. We found that it was difficult to work out a
<br>> >>>>> reasonable
<br>> >>>>> strategy to handle the actual cause so we decided not handle it.
<br>> >>>>>
<br>> >>>>> "Don't catch an error which you can't handle" as the bard put it.
<br>> >>>>>
<br>> >>>>> Robert
<br>> >>>>>
<br>> >>>>> 2009/11/9 Max Lapshin <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26269216&i=0" target="_top" rel="nofollow">max.lapshin@...</a>>
<br>> >>>>>
<br>> >>>>>> Yes, there are techniques to write watchdogs, but my question
<br>> >>>>>> was: is
<br>> >>>>>> it possible to prevent Erlang VM from crash?
<br>> >>>>>>
<br>> >>>>>> ________________________________________________________________
<br>> >>>>>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> >>>>>> erlang-questions (at) erlang.org
<br>> >>>>>>
<br>> >>>>>>
<br>> >>>>
<br>> >>>>
<br>> >>>> ________________________________________________________________
<br>> >>>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> >>>> erlang-questions (at) erlang.org
<br>> >>>>
<br>> >>>>
<br>> >>>
<br>> >>>
<br>> >>> Este correo no tiene dibujos. Las formas extrañas en la pantalla son
<br>> >>> letras.
<br>> >>> __________________________________________
<br>> >>>
<br>> >>> Clist UAH a.k.a Angel
<br>> >>> __________________________________________
<br>> >>> Warning: Microsoft_bribery.ISO contains OOXML code.
<br>> >>>
<br>> >>> ________________________________________________________________
<br>> >>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> >>> erlang-questions (at) erlang.org
<br>> >>>
<br>> >>
<br>> >>
<br>> >
<br>> >
<br>> >
<br>> > --
<br>> > No imprima este correo si no es necesario. El medio ambiente está en
<br>> > nuestras manos.
<br>> > __________________________________________
<br>> >
<br>> > Clist UAH a.k.a Angel
<br>> > __________________________________________
<br>> > China 'limpia' el Tibet para las Olimpiadas.
<br>>
<br>>
</div><br><br><br>--
<br>Este correo no tiene dibujos. Las formas extrañas en la pantalla son letras.
<br>__________________________________________
<br><br>Clist UAH a.k.a Angel
<br>__________________________________________
<br>Artista -- (internet) --> Usuario final. Así los artistas cobran más y dicen menos paridas sobre lo que creen que es la piratería.
<br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26267513Re: Why Beam.smp crashes when memory is over?2009-11-09T06:38:02Z2009-11-09T06:38:02ZTony Rogvall-2Hi!
<br><br>On 9 nov 2009, at 14.54, Angel Alvarez wrote:
<br><br>> Well still there are many issues with this new approach
<br>>
<br>Yes! But it does not scare me ;-)
<br><br>> Where are the maibox of processes located?
<br>>
<br>> With a heap pre process...
<br><br>Depends on the implementation. But in general you could do something
<br>like, If the the data is shared then
<br>you split the share (memory_size/ref_count). If the data is copied
<br>then you must count it in.
<br><br>>
<br>> Couldnt you trigger a memory exception on a remote process by just
<br>> sending one message
<br>> when the process is almost consuming its reserved memory?
<br>>
<br>Yes. But that is the point. If it pass the limit the process will die.
<br>There are many special cases where you could think of using the memory
<br>in a better and more optimal way.
<br>Lets say you are reaching the memory limit you may switch to a
<br>compression algorithm for heap memory !?
<br>But lets keep it simple for the prototype, and see if it useful.
<br><br><br>> Systems other than embedded erlangs deploy ( form de current erlang
<br>> movement as a server/desktop plataform)
<br>> will suffer from resource contention beetween erlang VM and other OS
<br>> processes.
<br>>
<br>> Port programs also need system resources...
<br>>
<br>For loadable drivers using driver_alloc, one could possibly do
<br>something, otherwise it will be
<br>up to the driver designer to handle it. There is a max_ports in the
<br>prototype that limits number of
<br>open_ports. If sockets/files are mapped to single ports then it may
<br>help a bit.
<br><br><br>> Well in the end your approach is still very interesting as a
<br>> framework for continous erlang VM innovations...
<br>>
<br>Thanks.
<br><br>> but please correct me if im wrong but I saw that memory carriers
<br>> allowed to set several options on erlang VM start-up so,
<br>>
<br>I am not sure what you mean here?
<br><br>/Tony
<br><br>> is stil posible to pacth those carriers to allow a safe memory
<br>> reservation to let de VM manage properly a memory full
<br>> condition by killing the offending process (sort of a OOM killer for
<br>> the VM)?
<br>>
<br>> Just telling the VM not to "kill system process" and let the
<br>> supervisors do the work...
<br>>
<br><br><br><div class='shrinkable-quote'><br>> /Angel
<br>>
<br>>
<br>>
<br>> El Lunes, 9 de Noviembre de 2009 Tony Rogvall escribió:
<br>>>
<br>>> It is still the same "let it crash" concept using the resource limit
<br>>> system I am designing.
<br>>> But you can limit the crash in a more controlled way. Also you will
<br>>> be
<br>>> able to report
<br>>> interesting information about what is crashing and when.
<br>>>
<br>>> There is sometimes an issue when big systems crash. The restart may
<br>>> take a lot of time.
<br>>> Nodes must be synchronised, database tables must be repaired etc etc.
<br>>> I guess you can design this to be light and easy, but it is not
<br>>> always
<br>>> the case.
<br>>>
<br>>> /Tony
<br>>>
<br>>>
<br>>>
<br>>> On 9 nov 2009, at 12.43, Angel Alvarez wrote:
<br>>>
<br>>>>
<br>>>> Well please let me say something
<br>>>>
<br>>>> I'm plain new but some things are pretty clear for me.
<br>>>>
<br>>>>
<br>>>> The beauty of the erlang concept is "let it crash" , "don't program
<br>>>> defensively"
<br>>>> so the VM and the underlaying hardware are entities that can fail,
<br>>>> that's it.
<br>>>>
<br>>>> What's the problem so?
<br>>>>
<br>>>> Joe said...
<br>>>>
<br>>>> If you want failure tolerance you need at least two nodes...
<br>>>>
<br>>>> From J.A thesis
<br>>>> " ...Schneider [60, 59] answered this question by giving three
<br>>>> properties
<br>>>> that he thought a hardware system should have in order to be
<br>>>> suitable for
<br>>>> programming a fault-tolerant system. These properties Schneider
<br>>>> called:..."
<br>>>>
<br>>>> 1. Halt on failure — in the event of an error a processor should
<br>>>> halt
<br>>>> instead of performing a possibly erroneous operation.2
<br>>>>
<br>>>> So on memory exhaustation the VM has to die and other node (erlang)
<br>>>> will do the recovery.
<br>>>>
<br>>>> that's the distrirbution role, no only to span computations over
<br>>>> several nodes to enhance performance
<br>>>> but to provide resilence in the presence of fatal errors (non
<br>>>> correctable).
<br>>>>
<br>>>> As a OS process the VM has to compete with other OS processes so in
<br>>>> a shared deployment (a VM running
<br>>>> on a server or a desktop) you cant be safe agaisnt a OOM trigered by
<br>>>> other entities.
<br>>>>
<br>>>> Such resource control thing wil only augment process overhead and
<br>>>> context switching in the VM.
<br>>>>
<br>>>> People new to erlang will be atracted to this hierarquical
<br>>>> decomposition of tasks as joe stated in his thesis
<br>>>> "If you cant run certaing task try doing something simpler"
<br>>>>
<br>>>> Many languages and VM's are incorporating erlang's good "multicore"
<br>>>> features but not the erlang powerfull error handling concept
<br>>>> and you guys want to kill the simpliticy incorporating many
<br>>>> defensive capabilities to avoid fatality instead of just organize
<br>>>> code to
<br>>>> handle such fatality.
<br>>>>
<br>>>> ¿whats next?, ¿A mailbox maximum message queue control?
<br>>>>
<br>>>>
<br>>>> Well, that's all i have to say about that, Forrest Gump.
<br>>>>
<br>>>>
<br>>>> El Lunes, 9 de Noviembre de 2009 09:45:10 Tony Rogvall escribió:
<br>>>>> Interesting discussion!
<br>>>>>
<br>>>>> I have been working on a resource system for Erlang for nearly two
<br>>>>> years now.
<br>>>>> I have a working (tm) prototype where you can set resource limits
<br>>>>> like
<br>>>>> max_processes/max_ports/max_memory/max_time/max_reductions ...
<br>>>>> The limits are passed with spawn_opt and are inherited by the
<br>>>>> processes spawned.
<br>>>>> This means that if you spawn_opt(M,F,A[{max_memory, 1024*1024}])
<br>>>>> the
<br>>>>> process
<br>>>>> will be able to use 1M words for it self and it's "subprocesses".
<br>>>>> This
<br>>>>> also means
<br>>>>> that the spawner will get 1M less to use (as designed right now).
<br>>>>> If a
<br>>>>> resource limit
<br>>>>> is reached the process crash with system_limt reason.
<br>>>>>
<br>>>>> There are still some details to work out before a release, but I
<br>>>>> will
<br>>>>> try to get it ready before
<br>>>>> end of this year.
<br>>>>>
<br>>>>> /Tony
<br>>>>>
<br>>>>>
<br>>>>>
<br>>>>> On 9 nov 2009, at 09.16, Robert Virding wrote:
<br>>>>>
<br>>>>>> No.
<br>>>>>>
<br>>>>>> There is a major difference between handling OOM in an OS and in
<br>>>>>> the
<br>>>>>> BEAM.
<br>>>>>> In an OS it usually at a per process level that memory runs out so
<br>>>>>> it is
<br>>>>>> easy to decide which process to kill so that the OS can
<br>>>>>> continue. In
<br>>>>>> the
<br>>>>>> BEAM, however, it is the VM as a whole which has run out of memory
<br>>>>>> not a
<br>>>>>> specific, it is. therefore, much more difficult to work out which
<br>>>>>> process is
<br>>>>>> the culprit and to decide what to do. For example it might be that
<br>>>>>> the
<br>>>>>> process which causes the OOM is not the actual problem process, it
<br>>>>>> might
<br>>>>>> just the last straw. Or the actual cause may that the whole app
<br>>>>>> might be
<br>>>>>> generating large binaries too quickly. Or it might be that the
<br>>>>>> whole
<br>>>>>> app is
<br>>>>>> spawning to many processes without any one process being the
<br>>>>>> cause.
<br>>>>>> Or ...
<br>>>>>> In all these cases killing the process which triggered the OOM
<br>>>>>> would
<br>>>>>> be the
<br>>>>>> Wrong Thing. We found that it was difficult to work out a
<br>>>>>> reasonable
<br>>>>>> strategy to handle the actual cause so we decided not handle it.
<br>>>>>>
<br>>>>>> "Don't catch an error which you can't handle" as the bard put it.
<br>>>>>>
<br>>>>>> Robert
<br>>>>>>
<br>>>>>> 2009/11/9 Max Lapshin <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26267513&i=0" target="_top" rel="nofollow">max.lapshin@...</a>>
<br>>>>>>
<br>>>>>>> Yes, there are techniques to write watchdogs, but my question
<br>>>>>>> was: is
<br>>>>>>> it possible to prevent Erlang VM from crash?
<br>>>>>>>
<br>>>>>>> ________________________________________________________________
<br>>>>>>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>>>>>>> erlang-questions (at) erlang.org
<br>>>>>>>
<br>>>>>>>
<br>>>>>
<br>>>>>
<br>>>>> ________________________________________________________________
<br>>>>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>>>>> erlang-questions (at) erlang.org
<br>>>>>
<br>>>>>
<br>>>>
<br>>>>
<br>>>> Este correo no tiene dibujos. Las formas extrañas en la pantalla son
<br>>>> letras.
<br>>>> __________________________________________
<br>>>>
<br>>>> Clist UAH a.k.a Angel
<br>>>> __________________________________________
<br>>>> Warning: Microsoft_bribery.ISO contains OOXML code.
<br>>>>
<br>>>> ________________________________________________________________
<br>>>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>>>> erlang-questions (at) erlang.org
<br>>>>
<br>>>
<br>>>
<br>>
<br>>
<br>>
<br>> --
<br>> No imprima este correo si no es necesario. El medio ambiente está en
<br>> nuestras manos.
<br>> __________________________________________
<br>>
<br>> Clist UAH a.k.a Angel
<br>> __________________________________________
<br>> China 'limpia' el Tibet para las Olimpiadas.
</div><br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26266807Re: Why Beam.smp crashes when memory is over?2009-11-09T05:54:10Z2009-11-09T05:54:10ZAngel AlvarezWell still there are many issues with this new approach
<br><br>Where are the maibox of processes located?
<br><br>With a heap pre process...
<br><br> Couldnt you trigger a memory exception on a remote process by just sending one message
<br>when the process is almost consuming its reserved memory?
<br><br>Systems other than embedded erlangs deploy ( form de current erlang movement as a server/desktop plataform)
<br>will suffer from resource contention beetween erlang VM and other OS processes.
<br><br>Port programs also need system resources...
<br><br>Well in the end your approach is still very interesting as a framework for continous erlang VM innovations...
<br><br>but please correct me if im wrong but I saw that memory carriers allowed to set several options on erlang VM start-up so,
<br><br>is stil posible to pacth those carriers to allow a safe memory reservation to let de VM manage properly a memory full
<br>condition by killing the offending process (sort of a OOM killer for the VM)?
<br><br>Just telling the VM not to "kill system process" and let the supervisors do the work...
<br><br>/Angel
<br><br><br><br>El Lunes, 9 de Noviembre de 2009 Tony Rogvall escribió:
<div class='shrinkable-quote'><br>>
<br>> It is still the same "let it crash" concept using the resource limit
<br>> system I am designing.
<br>> But you can limit the crash in a more controlled way. Also you will be
<br>> able to report
<br>> interesting information about what is crashing and when.
<br>>
<br>> There is sometimes an issue when big systems crash. The restart may
<br>> take a lot of time.
<br>> Nodes must be synchronised, database tables must be repaired etc etc.
<br>> I guess you can design this to be light and easy, but it is not always
<br>> the case.
<br>>
<br>> /Tony
<br>>
<br>>
<br>>
<br>> On 9 nov 2009, at 12.43, Angel Alvarez wrote:
<br>>
<br>> >
<br>> > Well please let me say something
<br>> >
<br>> > I'm plain new but some things are pretty clear for me.
<br>> >
<br>> >
<br>> > The beauty of the erlang concept is "let it crash" , "don't program
<br>> > defensively"
<br>> > so the VM and the underlaying hardware are entities that can fail,
<br>> > that's it.
<br>> >
<br>> > What's the problem so?
<br>> >
<br>> > Joe said...
<br>> >
<br>> > If you want failure tolerance you need at least two nodes...
<br>> >
<br>> > From J.A thesis
<br>> > " ...Schneider [60, 59] answered this question by giving three
<br>> > properties
<br>> > that he thought a hardware system should have in order to be
<br>> > suitable for
<br>> > programming a fault-tolerant system. These properties Schneider
<br>> > called:..."
<br>> >
<br>> > 1. Halt on failure — in the event of an error a processor should halt
<br>> > instead of performing a possibly erroneous operation.2
<br>> >
<br>> > So on memory exhaustation the VM has to die and other node (erlang)
<br>> > will do the recovery.
<br>> >
<br>> > that's the distrirbution role, no only to span computations over
<br>> > several nodes to enhance performance
<br>> > but to provide resilence in the presence of fatal errors (non
<br>> > correctable).
<br>> >
<br>> > As a OS process the VM has to compete with other OS processes so in
<br>> > a shared deployment (a VM running
<br>> > on a server or a desktop) you cant be safe agaisnt a OOM trigered by
<br>> > other entities.
<br>> >
<br>> > Such resource control thing wil only augment process overhead and
<br>> > context switching in the VM.
<br>> >
<br>> > People new to erlang will be atracted to this hierarquical
<br>> > decomposition of tasks as joe stated in his thesis
<br>> > "If you cant run certaing task try doing something simpler"
<br>> >
<br>> > Many languages and VM's are incorporating erlang's good "multicore"
<br>> > features but not the erlang powerfull error handling concept
<br>> > and you guys want to kill the simpliticy incorporating many
<br>> > defensive capabilities to avoid fatality instead of just organize
<br>> > code to
<br>> > handle such fatality.
<br>> >
<br>> > ¿whats next?, ¿A mailbox maximum message queue control?
<br>> >
<br>> >
<br>> > Well, that's all i have to say about that, Forrest Gump.
<br>> >
<br>> >
<br>> > El Lunes, 9 de Noviembre de 2009 09:45:10 Tony Rogvall escribió:
<br>> >> Interesting discussion!
<br>> >>
<br>> >> I have been working on a resource system for Erlang for nearly two
<br>> >> years now.
<br>> >> I have a working (tm) prototype where you can set resource limits
<br>> >> like
<br>> >> max_processes/max_ports/max_memory/max_time/max_reductions ...
<br>> >> The limits are passed with spawn_opt and are inherited by the
<br>> >> processes spawned.
<br>> >> This means that if you spawn_opt(M,F,A[{max_memory, 1024*1024}]) the
<br>> >> process
<br>> >> will be able to use 1M words for it self and it's "subprocesses".
<br>> >> This
<br>> >> also means
<br>> >> that the spawner will get 1M less to use (as designed right now).
<br>> >> If a
<br>> >> resource limit
<br>> >> is reached the process crash with system_limt reason.
<br>> >>
<br>> >> There are still some details to work out before a release, but I will
<br>> >> try to get it ready before
<br>> >> end of this year.
<br>> >>
<br>> >> /Tony
<br>> >>
<br>> >>
<br>> >>
<br>> >> On 9 nov 2009, at 09.16, Robert Virding wrote:
<br>> >>
<br>> >>> No.
<br>> >>>
<br>> >>> There is a major difference between handling OOM in an OS and in the
<br>> >>> BEAM.
<br>> >>> In an OS it usually at a per process level that memory runs out so
<br>> >>> it is
<br>> >>> easy to decide which process to kill so that the OS can continue. In
<br>> >>> the
<br>> >>> BEAM, however, it is the VM as a whole which has run out of memory
<br>> >>> not a
<br>> >>> specific, it is. therefore, much more difficult to work out which
<br>> >>> process is
<br>> >>> the culprit and to decide what to do. For example it might be that
<br>> >>> the
<br>> >>> process which causes the OOM is not the actual problem process, it
<br>> >>> might
<br>> >>> just the last straw. Or the actual cause may that the whole app
<br>> >>> might be
<br>> >>> generating large binaries too quickly. Or it might be that the whole
<br>> >>> app is
<br>> >>> spawning to many processes without any one process being the cause.
<br>> >>> Or ...
<br>> >>> In all these cases killing the process which triggered the OOM would
<br>> >>> be the
<br>> >>> Wrong Thing. We found that it was difficult to work out a reasonable
<br>> >>> strategy to handle the actual cause so we decided not handle it.
<br>> >>>
<br>> >>> "Don't catch an error which you can't handle" as the bard put it.
<br>> >>>
<br>> >>> Robert
<br>> >>>
<br>> >>> 2009/11/9 Max Lapshin <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26266807&i=0" target="_top" rel="nofollow">max.lapshin@...</a>>
<br>> >>>
<br>> >>>> Yes, there are techniques to write watchdogs, but my question
<br>> >>>> was: is
<br>> >>>> it possible to prevent Erlang VM from crash?
<br>> >>>>
<br>> >>>> ________________________________________________________________
<br>> >>>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> >>>> erlang-questions (at) erlang.org
<br>> >>>>
<br>> >>>>
<br>> >>
<br>> >>
<br>> >> ________________________________________________________________
<br>> >> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> >> erlang-questions (at) erlang.org
<br>> >>
<br>> >>
<br>> >
<br>> >
<br>> > Este correo no tiene dibujos. Las formas extrañas en la pantalla son
<br>> > letras.
<br>> > __________________________________________
<br>> >
<br>> > Clist UAH a.k.a Angel
<br>> > __________________________________________
<br>> > Warning: Microsoft_bribery.ISO contains OOXML code.
<br>> >
<br>> > ________________________________________________________________
<br>> > erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> > erlang-questions (at) erlang.org
<br>> >
<br>>
<br>>
</div><br><br><br>--
<br>No imprima este correo si no es necesario. El medio ambiente está en nuestras manos.
<br>__________________________________________
<br><br>Clist UAH a.k.a Angel
<br>__________________________________________
<br>China 'limpia' el Tibet para las Olimpiadas.
<br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26267321Re: Why Beam.smp crashes when memory is over?2009-11-09T05:25:09Z2009-11-09T05:25:09ZThomas Lindgren<br><br><br><br>----- Original Message ----
<div class='shrinkable-quote'><br>> From: Andrew Thompson <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26267321&i=0" target="_top" rel="nofollow">andrew@...</a>>
<br>> ... it'd be nice to have an
<br>> alternative in cases where nuking the VM and having to recover or simply
<br>> discard a lot of state is unpleasant.
<br>>
<br>> In an ideal world I'd like to see an *optional* OOM behaviour similar to
<br>> the following:
<br>>
<br>> On startup, the VM pre-allocates enough memory to be able to attempt OOM
<br>> recovery without additional malloc calls. If and when an OOM condition
<br>> is detected, freeze the VM, iterate the process list and simply
<br>> exit(FatPid, oom) the process with the most memory allocated (obviously
<br>> this strategy is full of holes, epecially with regard to off-heap
<br>> binaries), but *sometimes* it's better than just blowing away the whole
<br>> thing. Then, if the malloc fails again, revert to the current behaviour.
</div><br><br>Another way of doing that might be to hand the decision to (a modified version of) the supervisor hierarchy (if any). The supervisor could get info on sibling priority (if any) and/or memory usage as part of the request perhaps. Restart up the supervisor hierarchy to suit. That might give more structure to the restarting.
<br><br>Or you could do the OOM-killing in distributed erlang + unix instead. Run the buggy, memory-hungry stuff in a separate VM with a memory limit on the process, and manage the work from a "supervisor VM" using monitoring and rpc. Not quite as elegant, of course.
<br><br>Best,
<br>Thomas
<br><br><br><br>
<br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26266198Re: Why Beam.smp crashes when memory is over?2009-11-09T05:11:34Z2009-11-09T05:11:34ZTony Rogvall-2<br>It is still the same "let it crash" concept using the resource limit
<br>system I am designing.
<br>But you can limit the crash in a more controlled way. Also you will be
<br>able to report
<br>interesting information about what is crashing and when.
<br><br>There is sometimes an issue when big systems crash. The restart may
<br>take a lot of time.
<br>Nodes must be synchronised, database tables must be repaired etc etc.
<br>I guess you can design this to be light and easy, but it is not always
<br>the case.
<br><br>/Tony
<br><br><br><br>On 9 nov 2009, at 12.43, Angel Alvarez wrote:
<br><div class='shrinkable-quote'><br>>
<br>> Well please let me say something
<br>>
<br>> I'm plain new but some things are pretty clear for me.
<br>>
<br>>
<br>> The beauty of the erlang concept is "let it crash" , "don't program
<br>> defensively"
<br>> so the VM and the underlaying hardware are entities that can fail,
<br>> that's it.
<br>>
<br>> What's the problem so?
<br>>
<br>> Joe said...
<br>>
<br>> If you want failure tolerance you need at least two nodes...
<br>>
<br>> From J.A thesis
<br>> " ...Schneider [60, 59] answered this question by giving three
<br>> properties
<br>> that he thought a hardware system should have in order to be
<br>> suitable for
<br>> programming a fault-tolerant system. These properties Schneider
<br>> called:..."
<br>>
<br>> 1. Halt on failure — in the event of an error a processor should halt
<br>> instead of performing a possibly erroneous operation.2
<br>>
<br>> So on memory exhaustation the VM has to die and other node (erlang)
<br>> will do the recovery.
<br>>
<br>> that's the distrirbution role, no only to span computations over
<br>> several nodes to enhance performance
<br>> but to provide resilence in the presence of fatal errors (non
<br>> correctable).
<br>>
<br>> As a OS process the VM has to compete with other OS processes so in
<br>> a shared deployment (a VM running
<br>> on a server or a desktop) you cant be safe agaisnt a OOM trigered by
<br>> other entities.
<br>>
<br>> Such resource control thing wil only augment process overhead and
<br>> context switching in the VM.
<br>>
<br>> People new to erlang will be atracted to this hierarquical
<br>> decomposition of tasks as joe stated in his thesis
<br>> "If you cant run certaing task try doing something simpler"
<br>>
<br>> Many languages and VM's are incorporating erlang's good "multicore"
<br>> features but not the erlang powerfull error handling concept
<br>> and you guys want to kill the simpliticy incorporating many
<br>> defensive capabilities to avoid fatality instead of just organize
<br>> code to
<br>> handle such fatality.
<br>>
<br>> ¿whats next?, ¿A mailbox maximum message queue control?
<br>>
<br>>
<br>> Well, that's all i have to say about that, Forrest Gump.
<br>>
<br>>
<br>> El Lunes, 9 de Noviembre de 2009 09:45:10 Tony Rogvall escribió:
<br>>> Interesting discussion!
<br>>>
<br>>> I have been working on a resource system for Erlang for nearly two
<br>>> years now.
<br>>> I have a working (tm) prototype where you can set resource limits
<br>>> like
<br>>> max_processes/max_ports/max_memory/max_time/max_reductions ...
<br>>> The limits are passed with spawn_opt and are inherited by the
<br>>> processes spawned.
<br>>> This means that if you spawn_opt(M,F,A[{max_memory, 1024*1024}]) the
<br>>> process
<br>>> will be able to use 1M words for it self and it's "subprocesses".
<br>>> This
<br>>> also means
<br>>> that the spawner will get 1M less to use (as designed right now).
<br>>> If a
<br>>> resource limit
<br>>> is reached the process crash with system_limt reason.
<br>>>
<br>>> There are still some details to work out before a release, but I will
<br>>> try to get it ready before
<br>>> end of this year.
<br>>>
<br>>> /Tony
<br>>>
<br>>>
<br>>>
<br>>> On 9 nov 2009, at 09.16, Robert Virding wrote:
<br>>>
<br>>>> No.
<br>>>>
<br>>>> There is a major difference between handling OOM in an OS and in the
<br>>>> BEAM.
<br>>>> In an OS it usually at a per process level that memory runs out so
<br>>>> it is
<br>>>> easy to decide which process to kill so that the OS can continue. In
<br>>>> the
<br>>>> BEAM, however, it is the VM as a whole which has run out of memory
<br>>>> not a
<br>>>> specific, it is. therefore, much more difficult to work out which
<br>>>> process is
<br>>>> the culprit and to decide what to do. For example it might be that
<br>>>> the
<br>>>> process which causes the OOM is not the actual problem process, it
<br>>>> might
<br>>>> just the last straw. Or the actual cause may that the whole app
<br>>>> might be
<br>>>> generating large binaries too quickly. Or it might be that the whole
<br>>>> app is
<br>>>> spawning to many processes without any one process being the cause.
<br>>>> Or ...
<br>>>> In all these cases killing the process which triggered the OOM would
<br>>>> be the
<br>>>> Wrong Thing. We found that it was difficult to work out a reasonable
<br>>>> strategy to handle the actual cause so we decided not handle it.
<br>>>>
<br>>>> "Don't catch an error which you can't handle" as the bard put it.
<br>>>>
<br>>>> Robert
<br>>>>
<br>>>> 2009/11/9 Max Lapshin <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26266198&i=0" target="_top" rel="nofollow">max.lapshin@...</a>>
<br>>>>
<br>>>>> Yes, there are techniques to write watchdogs, but my question
<br>>>>> was: is
<br>>>>> it possible to prevent Erlang VM from crash?
<br>>>>>
<br>>>>> ________________________________________________________________
<br>>>>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>>>>> erlang-questions (at) erlang.org
<br>>>>>
<br>>>>>
<br>>>
<br>>>
<br>>> ________________________________________________________________
<br>>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>>> erlang-questions (at) erlang.org
<br>>>
<br>>>
<br>>
<br>>
<br>> Este correo no tiene dibujos. Las formas extrañas en la pantalla son
<br>> letras.
<br>> __________________________________________
<br>>
<br>> Clist UAH a.k.a Angel
<br>> __________________________________________
<br>> Warning: Microsoft_bribery.ISO contains OOXML code.
<br>>
<br>> ________________________________________________________________
<br>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> erlang-questions (at) erlang.org
<br>>
</div><br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26265010Re: Why Beam.smp crashes when memory is over?2009-11-09T03:43:02Z2009-11-09T03:43:02ZAngel Alvarez<br>Well please let me say something
<br><br>I'm plain new but some things are pretty clear for me.
<br><br><br>The beauty of the erlang concept is "let it crash" , "don't program defensively"
<br>so the VM and the underlaying hardware are entities that can fail, that's it.
<br><br>What's the problem so?
<br><br>Joe said...
<br><br>If you want failure tolerance you need at least two nodes...
<br><br>From J.A thesis
<br> " ...Schneider [60, 59] answered this question by giving three properties
<br>that he thought a hardware system should have in order to be suitable for
<br>programming a fault-tolerant system. These properties Schneider called:..."
<br>
<br>1. Halt on failure — in the event of an error a processor should halt
<br> instead of performing a possibly erroneous operation.2
<br><br>So on memory exhaustation the VM has to die and other node (erlang) will do the recovery.
<br><br>that's the distrirbution role, no only to span computations over several nodes to enhance performance
<br>but to provide resilence in the presence of fatal errors (non correctable).
<br><br>As a OS process the VM has to compete with other OS processes so in a shared deployment (a VM running
<br>on a server or a desktop) you cant be safe agaisnt a OOM trigered by other entities.
<br><br>Such resource control thing wil only augment process overhead and context switching in the VM.
<br><br>People new to erlang will be atracted to this hierarquical decomposition of tasks as joe stated in his thesis
<br>"If you cant run certaing task try doing something simpler"
<br><br>Many languages and VM's are incorporating erlang's good "multicore" features but not the erlang powerfull error handling concept
<br>and you guys want to kill the simpliticy incorporating many defensive capabilities to avoid fatality instead of just organize code to
<br>handle such fatality.
<br><br>¿whats next?, ¿A mailbox maximum message queue control?
<br><br><br>Well, that's all i have to say about that, Forrest Gump.
<br><br><br>El Lunes, 9 de Noviembre de 2009 09:45:10 Tony Rogvall escribió:
<div class='shrinkable-quote'><br>> Interesting discussion!
<br>>
<br>> I have been working on a resource system for Erlang for nearly two
<br>> years now.
<br>> I have a working (tm) prototype where you can set resource limits like
<br>> max_processes/max_ports/max_memory/max_time/max_reductions ...
<br>> The limits are passed with spawn_opt and are inherited by the
<br>> processes spawned.
<br>> This means that if you spawn_opt(M,F,A[{max_memory, 1024*1024}]) the
<br>> process
<br>> will be able to use 1M words for it self and it's "subprocesses". This
<br>> also means
<br>> that the spawner will get 1M less to use (as designed right now). If a
<br>> resource limit
<br>> is reached the process crash with system_limt reason.
<br>>
<br>> There are still some details to work out before a release, but I will
<br>> try to get it ready before
<br>> end of this year.
<br>>
<br>> /Tony
<br>>
<br>>
<br>>
<br>> On 9 nov 2009, at 09.16, Robert Virding wrote:
<br>>
<br>> > No.
<br>> >
<br>> > There is a major difference between handling OOM in an OS and in the
<br>> > BEAM.
<br>> > In an OS it usually at a per process level that memory runs out so
<br>> > it is
<br>> > easy to decide which process to kill so that the OS can continue. In
<br>> > the
<br>> > BEAM, however, it is the VM as a whole which has run out of memory
<br>> > not a
<br>> > specific, it is. therefore, much more difficult to work out which
<br>> > process is
<br>> > the culprit and to decide what to do. For example it might be that the
<br>> > process which causes the OOM is not the actual problem process, it
<br>> > might
<br>> > just the last straw. Or the actual cause may that the whole app
<br>> > might be
<br>> > generating large binaries too quickly. Or it might be that the whole
<br>> > app is
<br>> > spawning to many processes without any one process being the cause.
<br>> > Or ...
<br>> > In all these cases killing the process which triggered the OOM would
<br>> > be the
<br>> > Wrong Thing. We found that it was difficult to work out a reasonable
<br>> > strategy to handle the actual cause so we decided not handle it.
<br>> >
<br>> > "Don't catch an error which you can't handle" as the bard put it.
<br>> >
<br>> > Robert
<br>> >
<br>> > 2009/11/9 Max Lapshin <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26265010&i=0" target="_top" rel="nofollow">max.lapshin@...</a>>
<br>> >
<br>> >> Yes, there are techniques to write watchdogs, but my question was: is
<br>> >> it possible to prevent Erlang VM from crash?
<br>> >>
<br>> >> ________________________________________________________________
<br>> >> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> >> erlang-questions (at) erlang.org
<br>> >>
<br>> >>
<br>>
<br>>
<br>> ________________________________________________________________
<br>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> erlang-questions (at) erlang.org
<br>>
<br>>
</div><br><br>Este correo no tiene dibujos. Las formas extrañas en la pantalla son letras.
<br>__________________________________________
<br><br>Clist UAH a.k.a Angel
<br>__________________________________________
<br>Warning: Microsoft_bribery.ISO contains OOXML code.
<br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26264924Re: Dialyzer problem on OSX 10.6 (Erlang R13B02-1)2009-11-09T03:35:18Z2009-11-09T03:35:18ZTamas Nagy-3Hi Tino,
<br><br>Are you using homebrew?
<br><br>So far it seems that homebrew applies some c compiler magic (e.g optimizations) when building erlang which somehow borks dialyzer (strange) as compiling from source with the same ./configure options homebrew uses results in an erlang installation which works like a charm.
<br><br>Regards,
<br> Tamas
<br>----- "Tino Breddin" <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26264924&i=0" target="_top" rel="nofollow">tino.breddin@...</a>> wrote:
<br><div class='shrinkable-quote'><br>> Hey Tamas,
<br>>
<br>> Here's the output for the command run on my machine (Macbook Pro Core
<br>>
<br>> Duo, running OSX 10.6.1 32-bit):
<br>>
<br>> dialyzer --build_plt -r /usr/local/Cellar/erlang/R13B02-1/lib/erlang/
<br>>
<br>> lib/erts-5.7.3/ebin/ /usr/local/Cellar/erlang/R13B02-1/lib/erlang/lib/
<br>>
<br>> kernel-2.13.3/ebin/ /usr/local/Cellar/erlang/R13B02-1/lib/erlang/lib/
<br>>
<br>> stdlib-1.16.3/ebin/
<br>> Creating PLT /Users/tino/.dialyzer_plt ...
<br>> Unknown functions:
<br>> compile:file/2
<br>> compile:forms/2
<br>> compile:noenv_forms/2
<br>> compile:output_generated/1
<br>> crypto:des3_cbc_decrypt/5
<br>> crypto:start/0
<br>> done in 46m22.52s
<br>> done (passed successfully)
<br>>
<br>> Apparently it takes some time to complete but you'll get there.
<br>>
<br>> Cheers,
<br>> Tino
<br>>
<br>> On Nov 8, 2009, at 11:35 PM, Tamas Nagy wrote:
<br>>
<br>> > Hi Kostis,
<br>> >
<br>> > This is what all I get:
<br>> > $ dialyzer --build_plt -r /usr/local/Cellar/erlang/R13B02-1/lib/
<br>> > erlang/lib/erts-5.7.3/ebin/ /usr/local/Cellar/erlang/R13B02-1/lib/
<br>> > erlang/lib/kernel-2.13.3/ebin/
<br>> /usr/local/Cellar/erlang/R13B02-1/lib/
<br>> > erlang/lib/stdlib-1.16.3/ebin/
<br>> > Creating PLT /Users/lestat/.dialyzer_plt ...
<br>> >
<br>> > I will try to rebuild erlang with disabling hipe to see if that
<br>> helps.
<br>> >
<br>> > I am happy to run some tests/traces you would like to see the
<br>> > results of to help.
<br>> >
<br>> > Regards,
<br>> > Tamas
<br>> >
<br>> > Tamas Nagy
<br>> > Erlang Training & Consulting
<br>> > <a href="http://www.erlang-consulting.com" target="_top" rel="nofollow">http://www.erlang-consulting.com</a><br>> >
<br>> > On 8 Nov 2009, at 19:52, Kostis Sagonas wrote:
<br>> >
<br>> >> Tamas Nagy wrote:
<br>> >>> Hi,
<br>> >>> Am I the only one who cannot build a plt under OSX 10.6 in
<br>> >>> R13B02-1?
<br>> >>> No matter what applications I specify for the initial plt it just
<br>>
<br>> >>> hangs eating processing power but not producing anything.
<br>> >>> Erlang was installed with homebrew, but that shouldn't matter. Or
<br>>
<br>> >>> maybe it does. (beam is compiled for 64 bit)
<br>> >>> Here is the command I used:
<br>> >>> dialyzer --build_plt -r /usr/local/Cellar/erlang/R13B02-1/lib/
<br>> >>> erlang/lib/erts-5.7.3/ebin/ /usr/local/Cellar/erlang/R13B02-1/lib/
<br>>
<br>> >>> erlang/lib/kernel-2.13.3/ebin/ /usr/local/Cellar/erlang/R13B02-1/
<br>>
<br>> >>> lib/erlang/lib/stdlib-1.16.3/ebin/
<br>> >>> I gave waiting for it to finish after ~1 hour (on an old type
<br>> >>> white MacBook (MacBook2,1) with 2GB of RAM).
<br>> >>
<br>> >> None of the dialyzer developers has access to such a machine to see
<br>>
<br>> >> what's happening. Perhaps somebody else in that list might want to
<br>>
<br>> >> report whether this is reproducible or not.
<br>> >>
<br>> >> However, perhaps the following might help: when building the PLT,
<br>>
<br>> >> do you see a message:
<br>> >>
<br>> >> Compiling some key modules to native code...
<br>> >>
<br>> >> If so, can you disable hipe (64-bit Mac is not a platform we have
<br>>
<br>> >> ever tested due to no access to such a machine) to see whether this
<br>>
<br>> >> may be the culprit?
<br>> >>
<br>> >> Kostis
<br>> >
<br>> >
<br>> > ________________________________________________________________
<br>> > erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> > erlang-questions (at) erlang.org
<br>> >
</div><br>--
<br>Tamas Nagy
<br>Erlang Training & Consulting
<br><a href="http://www.erlang-consulting.com" target="_top" rel="nofollow">http://www.erlang-consulting.com</a><br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26264551Re: Why Beam.smp crashes when memory is over?2009-11-09T03:05:41Z2009-11-09T03:05:41ZDmitry BelyaevWhat if you'll have different nodes for different tasks instead of
<br>processes?
<br><br>Start one node to read stream and one (or many) other node to process
<br>the stream part.
<br><br>On Mon, 2009-11-09 at 08:08 +0300, Max Lapshin wrote:
<div class='shrinkable-quote'><br>> Look, I have a program, that reads MPEG TS stream from network. There
<br>> is some bug in my code
<br>> and my ts_lander crashes beam. How is it possible to store state of
<br>> external HTTP stream??
<br>>
<br>> If Erlang VM wouldn't crash, but only kill the leaking process, I
<br>> could buffer stream in other process and feed
<br>> new mpeg ts lander after restart. But it is impossible =(
<br>>
<br>> ________________________________________________________________
<br>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> erlang-questions (at) erlang.org
<br>>
<br>>
</div><br><br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26264132AW: Why Beam.smp crashes when memory is over?2009-11-09T02:31:19Z2009-11-09T02:31:19ZDecker, NilsHello,
<br><br>I had a few occasions during developing that a runaway process caused my machine to swap and gind to a halt. (i should have used ulimit on beam)
<br><br>I wondered why there is no way to limit the size of a single process. It could be a simple option to spawn to limit the heap size of a process. (like ulimit)
<br>The process gets killed if it ever grows beyond the limit. A limit for the size of the message queue would be nice too, because a process with a few thousand entries is (most of the time) not going to cope with it anyway.
<br>A similar limit for ETS tables would be possible too. A memory limit for shared resources (large binaries) would be more difficult.
<br><br>Nils Decker
<br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26263677Re: Why Beam.smp crashes when memory is over?2009-11-09T01:53:10Z2009-11-09T01:53:10ZKenneth LundinOn Mon, Nov 9, 2009 at 9:16 AM, Robert Virding <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26263677&i=0" target="_top" rel="nofollow">rvirding@...</a>> wrote:
<div class='shrinkable-quote'><br>> No.
<br>>
<br>> There is a major difference between handling OOM in an OS and in the BEAM.
<br>> In an OS it usually at a per process level that memory runs out so it is
<br>> easy to decide which process to kill so that the OS can continue. In the
<br>> BEAM, however, it is the VM as a whole which has run out of memory not a
<br>> specific, it is. therefore, much more difficult to work out which process is
<br>> the culprit and to decide what to do. For example it might be that the
<br>> process which causes the OOM is not the actual problem process, it might
<br>> just the last straw. Or the actual cause may that the whole app might be
<br>> generating large binaries too quickly. Or it might be that the whole app is
<br>> spawning to many processes without any one process being the cause. Or ...
<br>> In all these cases killing the process which triggered the OOM would be the
<br>> Wrong Thing. We found that it was difficult to work out a reasonable
<br>> strategy to handle the actual cause so we decided not handle it.
<br>>
<br>> "Don't catch an error which you can't handle" as the bard put it.
<br>>
<br>> Robert
</div><br>We have discussed this many times and have always come to the same
<br>conclusion as Robert expains above, that we can't know the right thing
<br>to do so we just do nothing and terminate the VM.
<br><br>What we could do is to make it easier for the user to prevent OOM
<br>situations and also to
<br>let him take the decision when it occurs or rather before it occurs.
<br><br>One way would be to let the user set a memory quota on a process with
<br>options at spawn time. When the process reaches it quota it can be
<br>automatically killed or the user can
<br>be notified in some way and take actions.
<br><br>We have also thought about ways for the user to monitor memory consumption and
<br>by this taking actions before the VM runs out of memory.
<br><br>/Kenneth, Erlang/OTP Ericsson
<br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26263613Re: Why Beam.smp crashes when memory is over?2009-11-09T00:45:10Z2009-11-09T00:45:10ZTony Rogvall-2Interesting discussion!
<br><br>I have been working on a resource system for Erlang for nearly two
<br>years now.
<br>I have a working (tm) prototype where you can set resource limits like
<br>max_processes/max_ports/max_memory/max_time/max_reductions ...
<br>The limits are passed with spawn_opt and are inherited by the
<br>processes spawned.
<br>This means that if you spawn_opt(M,F,A[{max_memory, 1024*1024}]) the
<br>process
<br>will be able to use 1M words for it self and it's "subprocesses". This
<br>also means
<br>that the spawner will get 1M less to use (as designed right now). If a
<br>resource limit
<br>is reached the process crash with system_limt reason.
<br><br>There are still some details to work out before a release, but I will
<br>try to get it ready before
<br>end of this year.
<br><br>/Tony
<br><br><br><br>On 9 nov 2009, at 09.16, Robert Virding wrote:
<br><div class='shrinkable-quote'><br>> No.
<br>>
<br>> There is a major difference between handling OOM in an OS and in the
<br>> BEAM.
<br>> In an OS it usually at a per process level that memory runs out so
<br>> it is
<br>> easy to decide which process to kill so that the OS can continue. In
<br>> the
<br>> BEAM, however, it is the VM as a whole which has run out of memory
<br>> not a
<br>> specific, it is. therefore, much more difficult to work out which
<br>> process is
<br>> the culprit and to decide what to do. For example it might be that the
<br>> process which causes the OOM is not the actual problem process, it
<br>> might
<br>> just the last straw. Or the actual cause may that the whole app
<br>> might be
<br>> generating large binaries too quickly. Or it might be that the whole
<br>> app is
<br>> spawning to many processes without any one process being the cause.
<br>> Or ...
<br>> In all these cases killing the process which triggered the OOM would
<br>> be the
<br>> Wrong Thing. We found that it was difficult to work out a reasonable
<br>> strategy to handle the actual cause so we decided not handle it.
<br>>
<br>> "Don't catch an error which you can't handle" as the bard put it.
<br>>
<br>> Robert
<br>>
<br>> 2009/11/9 Max Lapshin <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26263613&i=0" target="_top" rel="nofollow">max.lapshin@...</a>>
<br>>
<br>>> Yes, there are techniques to write watchdogs, but my question was: is
<br>>> it possible to prevent Erlang VM from crash?
<br>>>
<br>>> ________________________________________________________________
<br>>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>>> erlang-questions (at) erlang.org
<br>>>
<br>>>
</div><br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26262889Re: Why Beam.smp crashes when memory is over?2009-11-09T00:42:03Z2009-11-09T00:42:03ZRobert VirdingThat would be a start, it very much depends on your app. I am not really the
<br>right person to ask about details on chasing the culprit, but I do remember
<br>the discussions we had long ago on trying to work out a good solution. Which
<br>we couldn't.
<br><br>2009/11/9 Max Lapshin <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26262889&i=0" target="_top" rel="nofollow">max.lapshin@...</a>>
<br><br>> Ok, so You advice is just to look after memory usage of "dangereous"
<br>> processes?
<br>>
<br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26262659Re: Why Beam.smp crashes when memory is over?2009-11-09T00:19:12Z2009-11-09T00:19:12ZMax Lapshin-2Ok, so You advice is just to look after memory usage of "dangereous" processes?
<br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26262631Re: Why Beam.smp crashes when memory is over?2009-11-09T00:16:44Z2009-11-09T00:16:44ZRobert VirdingNo.
<br><br>There is a major difference between handling OOM in an OS and in the BEAM.
<br>In an OS it usually at a per process level that memory runs out so it is
<br>easy to decide which process to kill so that the OS can continue. In the
<br>BEAM, however, it is the VM as a whole which has run out of memory not a
<br>specific, it is. therefore, much more difficult to work out which process is
<br>the culprit and to decide what to do. For example it might be that the
<br>process which causes the OOM is not the actual problem process, it might
<br>just the last straw. Or the actual cause may that the whole app might be
<br>generating large binaries too quickly. Or it might be that the whole app is
<br>spawning to many processes without any one process being the cause. Or ...
<br>In all these cases killing the process which triggered the OOM would be the
<br>Wrong Thing. We found that it was difficult to work out a reasonable
<br>strategy to handle the actual cause so we decided not handle it.
<br><br>"Don't catch an error which you can't handle" as the bard put it.
<br><br>Robert
<br><br>2009/11/9 Max Lapshin <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26262631&i=0" target="_top" rel="nofollow">max.lapshin@...</a>>
<br><br>> Yes, there are techniques to write watchdogs, but my question was: is
<br>> it possible to prevent Erlang VM from crash?
<br>>
<br>> ________________________________________________________________
<br>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> erlang-questions (at) erlang.org
<br>>
<br>>
<br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26262318Re: Why Beam.smp crashes when memory is over?2009-11-08T23:37:32Z2009-11-08T23:37:32ZMax Lapshin-2Yes, there are techniques to write watchdogs, but my question was: is
<br>it possible to prevent Erlang VM from crash?
<br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26262244Re: Why Beam.smp crashes when memory is over?2009-11-08T23:31:24Z2009-11-08T23:31:24ZZoltan Lajos KisWhy don't you create a process, which regularly checks on the memory usage
<br>of those processes, and kills them when they begin to eat up too much
<br>memory?
<br><br>Zoltan.
<br><div class='shrinkable-quote'><br>> Look, I have a program, that reads MPEG TS stream from network. There
<br>> is some bug in my code
<br>> and my ts_lander crashes beam. How is it possible to store state of
<br>> external HTTP stream??
<br>>
<br>> If Erlang VM wouldn't crash, but only kill the leaking process, I
<br>> could buffer stream in other process and feed
<br>> new mpeg ts lander after restart. But it is impossible =(
<br>>
<br>> ________________________________________________________________
<br>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> erlang-questions (at) erlang.org
<br>>
<br>>
</div><br><br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26262143accessing ets from the process on other node2009-11-08T23:16:39Z2009-11-08T23:16:39Zsapan shahHI,<br><br>I use digraph module to create graphs and apply some functions on it. The digraph is internally stored as ets tuples.<br>I am using mapreduce architecture to speed up the computations on the digraph (using plists.erl from Stephen Marsh).<br>
So, When the graph is passed to a process on some other node on network, it exits with error. (may be because it is not able to access the ets tuples)<br clear="all">Any solution???<br><br>Below is a small test example to generate the error<br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>-module(pliststest).<br>-compile(export_all).<br><br>malt() -><br> [100, {nodes, [{'sap@laptop1', 3}, {'sap@laptop2', 3}]}].<br><br>test(G, List) -><br> Fun = fun(X) -> io:format("TROUBLE: ~p~n",[digraph:vertices(G)]), lists:foldr(fun(_Y,ACC) -> ACC+1 end, 0, X) end,<br>
Fuse = fun(X, Y) -> X+Y end,<br> plists:runmany(Fun, Fuse, List, malt()).<br>%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br><br>Also, PFA the plists.erl (from Stephen Marsh).<br><br>-- <br>Regards,<br>Sapan Shah<br>
<br /><tt>[plists.erl]</tt><br /><hr align="left" width="300" /><tt>% @author Stephen Marsh
<br>% @copyright 2007 Stephen Marsh freeyourmind ++ [$@|gmail.com]
<br>% @doc plists is a drop-in replacement for module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>,
<br>% making most list operations parallel. It can operate on each element in
<br>% parallel, for IO-bound operations, on sublists in parallel, for
<br>% taking advantage of multi-core machines with CPU-bound operations, and
<br>% across erlang nodes, for parallizing inside a cluster. It handles
<br>% errors and node failures. It can be configured, tuned, and tweaked to
<br>% get optimal performance while minimizing overhead.
<br>%
<br>% Almost all the functions are
<br>% identical to equivalent functions in lists, returning exactly the same
<br>% result, and having both a form with an identical syntax that operates on
<br>% each element in parallel and a form which takes an optional "malt",
<br>% a specification for how to parallize the operation.
<br>%
<br>% fold is the one exception, parallel fold is different from linear fold.
<br>% This module also include a simple mapreduce implementation, and the
<br>% function runmany. All the other functions are implemented with runmany,
<br>% which is as a generalization of parallel list operations.
<br>%
<br>% == Malts ==
<br>% A malt specifies how to break a list into sublists, and can optionally
<br>% specify a timeout, which nodes to run on, and how many processes to start
<br>% per node.
<br>%
<br>% Malt = MaltComponent | [MaltComponent]<br/>
<br>% MaltComponent = SubListSize::integer() | {processes, integer()} |
<br>% {timeout, Milliseconds::integer()} | {nodes, [NodeSpec]}<br/>
<br>% NodeSpec = Node::atom() | {Node::atom(), NumProcesses::integer()}
<br>%
<br>% An integer can be given to specify the exact size for
<br>% sublists. 1 is a good choice for IO-bound operations and when
<br>% the operation on each list element is expensive. Larger numbers
<br>% minimize overhead and are faster for cheap operations.
<br>%
<br>% If the integer is omitted, and
<br>% you have specified a {processes, X}, the list is
<br>% split into X sublists. This is only
<br>% useful when the time to process each element is close to identical and you
<br>% know exactly how many lines of execution are available to you.
<br>%
<br>% If neither of the above applies, the sublist size defaults to 1.
<br>%
<br>% You can use {processes, X} to have the list processed
<br>% by X processes on the local machine. A good choice for X is the number of
<br>% lines of execution (cores) the machine provides.
<br>%
<br>% {timeout, Milliseconds} specifies a timeout. This is a timeout for the entire
<br>% operation, both operating on the sublists and combining the results.
<br>% exit(timeout) is evaluated if the timeout is exceeded.
<br>%
<br>% {nodes, NodeList} specifies that the operation should be done across nodes.
<br>% Every element of NodeList is of the form {NodeName, NumProcesses} or
<br>% NodeName, which means the same as {NodeName, 1}. plists runs
<br>% NumProcesses process on NodeName concurrently. A good choice for
<br>% NumProcesses is the number of lines of execution (cores) a machine provides
<br>% plus one. This ensures the machine is completely busy even when
<br>% fetching a new sublist. plists is able to recover if a node goes down.
<br>% If all nodes go down, exit(allnodescrashed) is evaluated.
<br>%
<br>% Any of the above may be used as a malt, or may be combined into a list.
<br>% {nodes, NodeList} and {processes, X} may not be combined.
<br>%
<br>% === Examples ===
<br>% % start a process for each element (1-element sublists)<br/>
<br>% 1
<br>%
<br>% % start a process for each ten elements (10-element sublists)<br/>
<br>% 10
<br>%
<br>% % split the list into two sublists and process in two processes<br/>
<br>% {processes, 2}
<br>%
<br>% % split the list into 10-element sublists and process in two processes<br/>
<br>% [10, {processes, 2}]
<br>%
<br>% % timeout after one second. Assumes that a process should be started<br/>
<br>% % for each element.<br/>
<br>% {timeout, 1000}
<br>%
<br>% % Runs 3 processes at a time on apple@desktop,
<br>% and 2 on orange@laptop<br/>
<br>% % This is the best way to utilize all the CPU-power of a dual-core<br/>
<br>% % desktop and a single-core laptop. Assumes that the list should be<br/>
<br>% % split into 1-element sublists.<br/>
<br>% {nodes, [{apple@desktop, 3}, {orange@laptop, 2}]}
<br>%
<br>% % Gives apple and orange three seconds to process the list as<br/>
<br>% % 100-element sublists.<br/>
<br>% [100, {timeout, 3000}, {nodes, [{apple@desktop, 3}, {orange@laptop, 2}]}]
<br>%
<br>% === Aside: Why Malt? ===
<br>% I needed a word for this concept, so maybe my subconsciousness gave me one by
<br>% making me misspell multiply. Maybe it is an acronym for Malt is A List
<br>% Tearing Specification. Maybe it is a beer metaphor, suggesting that code
<br>% only runs in parallel if bribed with spirits. It's jargon, learn it
<br>% or you can't be part of the in-group.
<br>%
<br>% == Messages and Errors ==
<br>% plists assures that no extraneous messages are left in or will later
<br>% enter the message queue. This is guaranteed even in the event of an error.
<br>%
<br>% Errors in spawned processes are caught and propagated to the calling
<br>% process. If you invoke
<br>%
<br>% plists:map(fun (X) -> 1/X end, [1, 2, 3, 0]).
<br>%
<br>% you get a badarith error, exactly like when you use lists:map.
<br>%
<br>% plists uses monitors to watch the processes it spawns. It is not a good idea
<br>% to invoke plists when you are already monitoring processes. If one of them
<br>% does a non-normal exit, plists receives the 'DOWN' message believing it to be
<br>% from one of its own processes. The error propagation system goes into
<br>% effect, which results in the error occuring in the calling process.
<br>%
<br>% == License ==
<br>% The MIT License
<br>%
<br>% Copyright (c) 2007 Stephen Marsh
<br>%
<br>% Permission is hereby granted, free of charge, to any person obtaining a copy
<br>% of this software and associated documentation files (the "Software"), to deal
<br>% in the Software without restriction, including without limitation the rights
<br>% to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
<br>% copies of the Software, and to permit persons to whom the Software is
<br>% furnished to do so, subject to the following conditions:
<br>%
<br>% The above copyright notice and this permission notice shall be included in
<br>% all copies or substantial portions of the Software.
<br>%
<br>% THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
<br>% IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
<br>% FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
<br>% AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
<br>% LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
<br>% OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
<br>% THE SOFTWARE.
<br><br><br>-module(plists).
<br>-export([all/2, all/3, any/2, any/3, filter/2, filter/3,
<br>fold/3, fold/4, fold/5, foreach/2, foreach/3, map/2, map/3,
<br>partition/2, partition/3, sort/1, sort/2, sort/3,
<br>usort/1, usort/2, usort/3, mapreduce/2, mapreduce/3,
<br>runmany/3, runmany/4]).
<br><br>% Everything here is defined in terms of runmany.
<br>% The following methods are convient interfaces to runmany.
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List) -> bool()
<br>all(Fun, List) ->
<br> all(Fun, List, 1).
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List, Malt) -> bool()
<br>all(Fun, List, Malt) ->
<br> try runmany(fun (L) ->
<br> B = lists:all(Fun, L),
<br> if B ->
<br> nil;
<br> true ->
<br> exit(notall)
<br> end
<br> end,
<br> fun (_A1, _A2) ->
<br> nil
<br> end,
<br> List, Malt) of
<br> _ ->
<br> true
<br> catch exit:notall ->
<br> false
<br> end.
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List) -> bool()
<br>any(Fun, List) ->
<br> any(Fun, List, 1).
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List, Malt) -> bool()
<br>any(Fun, List, Malt) ->
<br> try runmany(fun (L) ->
<br> B = lists:any(Fun, L),
<br> if B ->
<br> exit(any);
<br> true ->
<br> nil
<br> end
<br> end,
<br> fun (_A1, _A2) ->
<br> nil
<br> end,
<br> List, Malt) of
<br> _ ->
<br> false
<br> catch exit:any ->
<br> true
<br> end.
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List) -> list()
<br>filter(Fun, List) ->
<br> filter(Fun, List, 1).
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List, Malt) -> list()
<br>filter(Fun, List, Malt) ->
<br> runmany(fun (L) ->
<br> lists:filter(Fun, L)
<br> end,
<br> fun (A1, A2) ->
<br> A1 ++ A2
<br> end,
<br> List, Malt).
<br><br>% Note that with parallel fold there is not foldl and foldr,
<br>% instead just one fold that can fuse Accumlators.
<br><br>% @doc Like below, but assumes 1 as the Malt. This function is almost useless,
<br>% and is intended only to aid converting code from using lists to plists.
<br>% @spec (Fun, InitAcc, List) -> term()
<br>fold(Fun, InitAcc, List) ->
<br> fold(Fun, Fun, InitAcc, List, 1).
<br><br>% @doc Like below, but uses the Fun as the Fuse by default.
<br>% @spec (Fun, InitAcc, List, Malt) -> term()
<br>fold(Fun, InitAcc, List, Malt) ->
<br> fold(Fun, Fun, InitAcc, List, Malt).
<br><br>% @doc fold is more complex when made parallel. There is no foldl and foldr,
<br>% accumulators aren't passed in any defined order.
<br>% The list is split into sublists which are folded together. Fun is
<br>% identical to the function passed to lists:fold[lr], it takes
<br>% (an element, and the accumulator) and returns -> a new accumulator.
<br>% It is used for the initial stage of folding sublists. Fuse fuses together
<br>% the results, it takes (Results1, Result2) and returns -> a new result.
<br>% By default sublists are fused left to right, each result of a fuse being
<br>% fed into the first element of the next fuse. The result of the last fuse
<br>% is the result.
<br>%
<br>% Fusing may also run in parallel using a recursive algorithm,
<br>% by specifying the fuse as {recursive, Fuse}. See
<br>% the discussion in {@link runmany/4}.
<br>%
<br>% Malt is the malt for the initial folding of sublists, and for the
<br>% possible recursive fuse.
<br>% @spec (Fun, Fuse, InitAcc, List, Malt) -> term()
<br>fold(Fun, Fuse, InitAcc, List, Malt) ->
<br> Fun2 = fun (L) -> lists:foldl(Fun, InitAcc, L) end,
<br> runmany(Fun2, Fuse, List, Malt).
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List) -> void()
<br>foreach(Fun, List) ->
<br> foreach(Fun, List, 1).
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List, Malt) -> void()
<br>foreach(Fun, List, Malt) ->
<br> runmany(fun (L) ->
<br> lists:foreach(Fun, L)
<br> end,
<br> fun (_A1, _A2) ->
<br> ok
<br> end,
<br> List, Malt).
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List) -> list()
<br>map(Fun, List) ->
<br> map(Fun, List, 1).
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List, Malt) -> list()
<br>map(Fun, List, Malt) ->
<br> runmany(fun (L) ->
<br> lists:map(Fun, L)
<br> end,
<br> fun (A1, A2) ->
<br> A1 ++ A2
<br> end,
<br> List, Malt).
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List) -> {list(), list()}
<br>partition(Fun, List) ->
<br> partition(Fun, List, 1).
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List, Malt) -> {list(), list()}
<br>partition(Fun, List, Malt) ->
<br> runmany(fun (L) ->
<br> lists:partition(Fun, L)
<br> end,
<br> fun ({True1, False1}, {True2, False2}) ->
<br> {True1 ++ True2, False1 ++ False2}
<br> end,
<br> List, Malt).
<br><br>% SORTMALT needs to be tuned
<br>-define(SORTMALT, 100).
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (List) -> list()
<br>sort(List) ->
<br> sort(fun (A, B) ->
<br> A =< B
<br> end,
<br> List).
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List) -> list()
<br>sort(Fun, List) ->
<br> sort(Fun, List, ?SORTMALT).
<br><br>% @doc This version lets you specify your own malt for sort.
<br>%
<br>% sort splits the list into sublists and sorts them, and it merges the
<br>% sorted lists together. These are done in parallel. Each sublist is
<br>% sorted in a seperate process, and each merging of results is done in a
<br>% seperate process. Malt defaults to 100, causing the list to be split into
<br>% 100-element sublists.
<br>% @spec (Fun, List, Malt) -> list()
<br>sort(Fun, List, Malt) ->
<br> Fun2 = fun (L) ->
<br> lists:sort(Fun, L)
<br> end,
<br> Fuse = fun (A1, A2) ->
<br> lists:merge(Fun, A1, A2)
<br> end,
<br> runmany(Fun2, {recursive, Fuse}, List, Malt).
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (List) -> list()
<br>usort(List) ->
<br> usort(fun (A, B) ->
<br> A =< B
<br> end,
<br> List).
<br><br>% @doc Same semantics as in module
<br>% <a href="http://www.erlang.org/doc/man/lists.html">lists</a>.
<br>% @spec (Fun, List) -> list()
<br>usort(Fun, List) ->
<br> usort(Fun, List, ?SORTMALT).
<br><br>% @doc This version lets you specify your own malt for usort.
<br>%
<br>% usort splits the list into sublists and sorts them, and it merges the
<br>% sorted lists together. These are done in parallel. Each sublist is
<br>% sorted in a seperate process, and each merging of results is done in a
<br>% seperate process. Malt defaults to 100, causing the list to be split into
<br>% 100-element sublists.
<br>%
<br>% usort removes duplicate elments while it sorts.
<br>% @spec (Fun, List, Malt) -> list()
<br>usort(Fun, List, Malt) ->
<br> Fun2 = fun (L) ->
<br> lists:usort(Fun, L)
<br> end,
<br> Fuse = fun (A1, A2) ->
<br> lists:umerge(Fun, A1, A2)
<br> end,
<br> runmany(Fun2, {recursive, Fuse}, List, Malt).
<br><br>% @doc Like below, assumes default MapMalt of 1.
<br>% @spec (MapFunc, List) -> Dict
<br>% MapFunc = (term()) -> DeepListOfKeyValuePairs
<br>% DeepListOfKeyValuePairs = [DeepListOfKeyValuePairs] | {Key, Value}
<br>mapreduce(MapFunc, List) ->
<br> mapreduce(MapFunc, List, 1).
<br><br>% @doc This is a very basic mapreduce. You won't write a Google-rivaling
<br>% search engine with it. It has no equivalent in lists. Each
<br>% element in the list is run through the MapFunc, which produces either
<br>% a {Key, Value} pair, or a lists of key value pairs, or a list of lists of
<br>% key value pairs...etc. A reducer process runs in parallel with the mapping
<br>% processes, collecting the key value pairs into a
<br>% <a href="http://www.erlang.org/doc/man/dict.html">dict</a>. This dict
<br>% is returned as the result.
<br>%
<br>% MapMalt is the malt for the mapping operation, with a default value of 1,
<br>% meaning each element of the list is mapped by a seperate process.
<br>%
<br>% mapreduce requires OTP R11B, or it may leave monitoring messages in the
<br>% message queue.
<br>% @spec (MapFunc, List, MapMalt) -> Dict
<br>% MapFunc = (term()) -> DeepListOfKeyValuePairs
<br>% DeepListOfKeyValuePairs = [DeepListOfKeyValuePairs] | {Key, Value}
<br>mapreduce(MapFunc, List, MapMalt) ->
<br> Parent = self(),
<br> {Reducer, ReducerRef} =
<br> erlang:spawn_monitor(fun () ->
<br> reducer(Parent, dict:new(), 0)
<br> end),
<br> MapFunc2 = fun (L) ->
<br> Reducer ! lists:map(MapFunc, L),
<br> 1
<br> end,
<br> SentMessages = try runmany(MapFunc2, fun (A, B) -> A+B end, List, MapMalt)
<br> catch
<br> exit:Reason ->
<br> erlang:demonitor(ReducerRef, [flush]),
<br> Reducer ! die,
<br> exit(Reason)
<br> end,
<br> Reducer ! {done, SentMessages},
<br> Results = receive
<br> {Reducer, Results2} ->
<br> Results2;
<br> {'DOWN', _, _, Reducer, Reason2} ->
<br> exit(Reason2)
<br> end,
<br> receive
<br> {'DOWN', _, _, Reducer, normal} ->
<br> nil
<br> end,
<br> Results.
<br><br>reducer(Parent, Dict, NumReceived) ->
<br> receive
<br> die ->
<br> nil;
<br> {done, NumReceived} ->
<br> Parent ! {self (), Dict};
<br> Keys ->
<br> reducer(Parent, add_keys(Dict, Keys), NumReceived + 1)
<br> end.
<br><br>add_keys(Dict, [{Key, Value}|Keys]) ->
<br> case dict:is_key(Key, Dict) of
<br> true ->
<br> add_keys(dict:append(Key, Value, Dict), Keys);
<br> false ->
<br> add_keys(dict:store(Key, [Value], Dict), Keys)
<br> end;
<br>add_keys(Dict, [List|Keys]) when is_list(List) ->
<br> add_keys(add_keys(Dict, List), Keys);
<br>add_keys(Dict, []) ->
<br> Dict.
<br><br>% @doc Like below, but assumes a Malt of 1,
<br>% meaning each element of the list is processed by a seperate process.
<br>% @spec (Fun, Fuse, List) -> term()
<br>runmany(Fun, Fuse, List) ->
<br> runmany(Fun, Fuse, List, 1).
<br><br>% Begin internal stuff (though runmany/4 is exported).
<br><br>% @doc All of the other functions are implemented with runmany. runmany
<br>% takes a List, splits it into sublists, and starts processes to operate on
<br>% each sublist, all done according to Malt. Each process passes its sublist
<br>% into Fun and sends the result back.
<br>%
<br>% The results are then fused together to get the final result. There are two
<br>% ways this can operate, lineraly and recursively. If Fuse is a function,
<br>% a fuse is done linearly left-to-right on the sublists, the results
<br>% of processing the first and second sublists being passed to Fuse, then
<br>% the result of that fuse and processing the third sublits, and so on.
<br>% This method preserves the original order of the lists elements.
<br>%
<br>% To do a recursive fuse, pass Fuse as {recursive, FuseFunc}.
<br>% The recursive fuse makes no guarantee about the order the results of
<br>% sublists, or the results of fuses are passed to FuseFunc. It
<br>% continues fusing pairs of results until it is down to one.
<br>%
<br>% Recursive fuse is down in parallel with processing the sublists, and a
<br>% process is spawned to fuse each pair of results. It is a parallized
<br>% algorithm. Linear fuse is done after all results of processing sublists
<br>% have been collected, and can only run in a single process.
<br>%
<br>% Even if you pass {recursive, FuseFunc}, a recursive fuse is only done if
<br>% the malt contains {nodes, NodeList} or {processes, X}. If this is not the
<br>% case, a linear fuse is done.
<br>% @spec (Fun, Fuse, List, Malt) -> term()
<br>% Fun = (list()) -> term()
<br>% Fuse = FuseFunc | {recursive, FuseFunc}
<br>% FuseFunc = (term(), term()) -> term()
<br>runmany(Fun, Fuse, List, Malt) when is_list(Malt) ->
<br> % If the sublist size wasn't specified, assume a default.
<br> case lists:any(fun (X) when is_integer(X) ->
<br> true;
<br> (_) ->
<br> false
<br> end, Malt) of
<br> true ->
<br> runmany(Fun, Fuse, List, local, Malt);
<br> false ->
<br> case lists:foldl(fun({processes, X}, _A) ->
<br> X;
<br> (_, A) ->
<br> A
<br> end, no_processes, Malt) of
<br> no_processes ->
<br> runmany(Fun, Fuse, List, local, [1|Malt]);
<br> X ->
<br> % split list into X sublists.
<br> L = length(List),
<br> case L rem X of
<br> 0 ->
<br> runmany(Fun, Fuse, List, local, [L div X|Malt]);
<br> _ ->
<br> runmany(Fun, Fuse, List, local, [L div X + 1|Malt])
<br> end
<br> end
<br> end;
<br>runmany(Fun, Fuse, List, Malt) ->
<br> runmany(Fun, Fuse, List, [Malt]).
<br><br>runmany(Fun, Fuse, List, Nodes, [MaltTerm|Malt]) when is_integer(MaltTerm) ->
<br> runmany(Fun, Fuse, splitmany(List, MaltTerm), Nodes, Malt);
<br>% run X process on local machine
<br>runmany(Fun, Fuse, List, local, [{processes, X}|Malt]) ->
<br> Nodes = lists:duplicate(X, node()),
<br> runmany(Fun, Fuse, List, Nodes, Malt);
<br>runmany(Fun, Fuse, List, Nodes, [{timeout, X}|Malt]) ->
<br> Parent = self(),
<br> Timer = spawn(fun () ->
<br> receive
<br> stoptimer ->
<br> Parent ! {timerstopped, self()}
<br> after X ->
<br> Parent ! {timerrang, self()},
<br> receive
<br> stoptimer ->
<br> Parent ! {timerstopped, self()}
<br> end
<br> end
<br> end),
<br> Ans = try runmany(Fun, Fuse, List, Nodes, Malt)
<br> catch
<br> % we really just want the after block, the syntax
<br> % makes this catch necessary.
<br> willneverhappen ->
<br> nil
<br> after
<br> Timer ! stoptimer,
<br> cleanup_timer(Timer)
<br> end,
<br> Ans;
<br>runmany(Fun, Fuse, List, local, [{nodes, NodeList}|Malt]) ->
<br> Nodes = lists:foldl(fun ({Node, X}, A) ->
<br> lists:reverse(lists:duplicate(X, Node), A);
<br> (Node, A) ->
<br> [Node|A]
<br> end,
<br> [], NodeList),
<br> runmany(Fun, Fuse, List, Nodes, Malt);
<br>% local recursive fuse, for when we weren't invoked with {processes, X}
<br>% or {nodes, NodeList}. Degenerates recursive fuse into linear fuse.
<br>runmany(Fun, {recursive, Fuse}, List, local, []) ->
<br> runmany(Fun, Fuse, List, local, []);
<br>runmany(Fun, Fuse, List, local, []) ->
<br> local_runmany(Fun, Fuse, List);
<br>runmany(Fun, Fuse, List, Nodes, []) ->
<br> cluster_runmany(Fun, Fuse, List, Nodes).
<br><br>cleanup_timer(Timer) ->
<br> receive
<br> {timerrang, Timer} ->
<br> cleanup_timer(Timer);
<br> {timerstopped, Timer} ->
<br> nil
<br> end.
<br><br>% local runmany, for when we weren't invoked with {processes, X}
<br>% or {nodes, NodeList}. Every sublist is processed in parallel.
<br>local_runmany(Fun, Fuse, List) ->
<br> Parent = self (),
<br> Pids = lists:map(fun (L) ->
<br> F = fun () ->
<br> Parent !
<br> {self (), Fun(L)}
<br> end,
<br> {Pid, _} = erlang:spawn_monitor(F),
<br> Pid
<br> end,
<br> List),
<br> Answers = try lists:map(fun receivefrom/1, Pids)
<br> catch throw:Message ->
<br> {BadPid, Reason} = Message,
<br> handle_error(BadPid, Reason, Pids)
<br> end,
<br> lists:foreach(fun (Pid) ->
<br> normal_cleanup(Pid)
<br> end, Pids),
<br> fuse(Fuse, Answers).
<br><br>receivefrom(Pid) ->
<br> receive
<br> {Pid, R} ->
<br> R;
<br> {'DOWN', _, _, BadPid, Reason} when Reason =/= normal ->
<br> throw({BadPid, Reason});
<br> {timerrang, _} ->
<br> throw({nil, timeout})
<br> end.
<br><br>% Convert List into [{Number, Sublist}]
<br>cluster_runmany(Fun, Fuse, List, Nodes) ->
<br> {List2, _} = lists:foldl(fun (X, {L, Count}) ->
<br> {[{Count, X}|L], Count+1}
<br> end,
<br> {[], 0}, List),
<br> cluster_runmany(Fun, Fuse, List2, Nodes, [], []).
<br><br>% Add a pair of results into the TaskList as a fusing task
<br>cluster_runmany(Fun, {recursive, Fuse}, [], Nodes, Running,
<br> [{_, R1}, {_, R2}|Results]) ->
<br> cluster_runmany(Fun, {recursive, Fuse}, [{fuse, R1, R2}], Nodes,
<br> Running, Results);
<br>% recursive fuse done, return result
<br>cluster_runmany(_, {recursive, _Fuse}, [], _Nodes, [], [{_, Result}]) ->
<br> Result;
<br>% edge case where we are asked to do nothing
<br>cluster_runmany(_, {recursive, _Fuse}, [], _Nodes, [], []) ->
<br> [];
<br>% We're done, now we just have to [linear] fuse the results
<br>cluster_runmany(_, Fuse, [], _Nodes, [], Results) ->
<br> fuse(Fuse, lists:map(fun ({_, R}) -> R end,
<br> lists:sort(fun ({A, _}, {B, _}) ->
<br> A =< B
<br> end,
<br> lists:reverse(Results))));
<br>% We have a ready node and a sublist or fuse to be processed, so we start
<br>% a new process
<br>cluster_runmany(Fun, Fuse, [Task|TaskList], [N|Nodes], Running, Results) ->
<br> Parent = self(),
<br> case Task of
<br> {Num, L2} ->
<br> Fun2 = fun () ->
<br> Parent ! {self(), Num, Fun(L2)}
<br> end;
<br> {fuse, R1, R2} ->
<br> {recursive, FuseFunc} = Fuse,
<br> Fun2 = fun () ->
<br> Parent ! {self(), junknum, FuseFunc(R1, R2)}
<br> end
<br> end,
<br> Pid = spawn(N, Fun2),
<br> erlang:monitor(process, Pid),
<br> cluster_runmany(Fun, Fuse, TaskList, Nodes, [{Pid, N, Task}|Running], Results);
<br>% We can't start a new process, but can watch over already running ones
<br>cluster_runmany(Fun, Fuse, TaskList, Nodes, Running, Results) when length(Running) > 0 ->
<br> receive
<br> {Pid, Num, Result} ->
<br> % throw out the normal exit message
<br> receive {'DOWN', _, _, Pid, normal} ->
<br> nil;
<br> {'DOWN', _, _, Pid, noproc}->
<br> nil
<br> end,
<br> {Running2, FinishedNode, _} = delete_running(Pid, Running, []),
<br> cluster_runmany(Fun, Fuse, TaskList,
<br> [FinishedNode|Nodes], Running2, [{Num, Result}|Results]);
<br> {timerrang, _} ->
<br> RunningPids = lists:map(fun ({Pid, _, _}) ->
<br> Pid
<br> end,
<br> Running),
<br> handle_error(nil, timeout, RunningPids);
<br> % node failure
<br> {'DOWN', _, _, Pid, noconnection} ->
<br> {Running2, _DeadNode, Task} = delete_running(Pid, Running, []),
<br> cluster_runmany(Fun, Fuse, [Task|TaskList], Nodes,
<br> Running2, Results);
<br> {'DOWN', _, _, BadPid, Reason} when Reason =/= normal ->
<br> RunningPids = lists:map(fun ({Pid, _, _}) ->
<br> Pid
<br> end,
<br> Running),
<br> handle_error(BadPid, Reason, RunningPids)
<br> end;
<br>% We have data, but no nodes either available or occupied
<br>cluster_runmany(_, _, [_Non|_Empty], []=_Nodes, []=_Running, _) ->
<br> exit(allnodescrashed).
<br><br>delete_running(Pid, [{Pid, Node, List}|Running], Acc) ->
<br> {Running ++ Acc, Node, List};
<br>delete_running(Pid, [R|Running], Acc) ->
<br> delete_running(Pid, Running, [R|Acc]).
<br><br>handle_error(BadPid, Reason, Pids) ->
<br> lists:foreach(fun (Pid) ->
<br> exit(Pid, siblingdied)
<br> end, Pids),
<br> lists:foreach(fun (Pid) ->
<br> error_cleanup(Pid, BadPid)
<br> end, Pids),
<br> exit(Reason).
<br><br>error_cleanup(BadPid, BadPid) ->
<br> ok;
<br>error_cleanup(Pid, BadPid) ->
<br> receive
<br> {Pid, _} ->
<br> error_cleanup(Pid, BadPid);
<br> {Pid, _, _} ->
<br> error_cleanup(Pid, BadPid);
<br> {'DOWN', _, _, Pid, _Reason} ->
<br> ok
<br> end.
<br><br>normal_cleanup(Pid) ->
<br> receive
<br> {'DOWN', _, _, Pid, _Reason} ->
<br> ok
<br> end.
<br><br>fuse(Fuse, [R1|Results]) ->
<br> fuse(Fuse, Results, R1);
<br>fuse(_, []) ->
<br> [].
<br><br>fuse(Fuse, [R2|Results], R1) ->
<br> fuse(Fuse, Results, Fuse(R1, R2));
<br>fuse(_, [], R) ->
<br> R.
<br><br>% Splits a list into a list of sublists, each of size Size,
<br>% except for the last element which is less if the original list
<br>% could not be evenly divided into Size-sized lists.
<br>splitmany(List, Size) ->
<br> splitmany(List, [], Size).
<br><br>splitmany([], Acc, _) ->
<br> lists:reverse(Acc);
<br>splitmany(List, Acc, Size) ->
<br> {Top, NList} = split(Size, List),
<br> splitmany(NList, [Top|Acc], Size).
<br><br>% Like lists:split, except it splits a list smaller than its first
<br>% parameter
<br>split(Size, List) ->
<br> split(Size, List, []).
<br><br>split(0, List, Acc) ->
<br> {lists:reverse(Acc), List};
<br>split(Size, [H|List], Acc) ->
<br> split(Size - 1, List, [H|Acc]);
<br>split(_, [], Acc) ->
<br> {lists:reverse(Acc), []}.
<br></tt><hr align="left" width="300" /><br /><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org<p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26261940Re: Why Beam.smp crashes when memory is over?2009-11-08T22:48:11Z2009-11-08T22:48:11ZAndrew Thompson-15On Sun, Nov 08, 2009 at 10:43:25PM -0700, Tony Arcieri wrote:
<div class='shrinkable-quote'><br>> I'm not really sure if we're speaking about the same thing then. Erlang is
<br>> best utilized as a stateless platform, with a "fail early fail often"
<br>> philosophy, and crashing the entire interpreter in the event of some
<br>> underlying problem with the OS fits that entirely. You seem to be seeking a
<br>> platform that lets you program defensively, handling errors rather than
<br>> crashing and restoring with a clean state. In that regard, Erlang may not
<br>> be the best language for you to work with, but then again the problem of
<br>> successfully recovering from an out-of-resource condition is an extremely
<br>> difficult one to handle well without scrapping it all and trying to restart
<br>> with a clean state.
<br>>
</div><br>I agree that Erlang is best used when your application is as stateless
<br>as possible, or has clear transfer of responsibility (HTTP server, SMTP
<br>server, etc), but sometimes Erlang still has a lot of benefits even if
<br>you're lugging some state around, and that's when the current OOM
<br>behaviour is a little annoying. I don't say that the current behaviour
<br>is not the best in a lot of situations, but it'd be nice to have an
<br>alternative in cases where nuking the VM and having to recover or simply
<br>discard a lot of state is unpleasant.
<br><br>In an ideal world I'd like to see an *optional* OOM behaviour similar to
<br>the following:
<br><br>On startup, the VM pre-allocates enough memory to be able to attempt OOM
<br>recovery without additional malloc calls. If and when an OOM condition
<br>is detected, freeze the VM, iterate the process list and simply
<br>exit(FatPid, oom) the process with the most memory allocated (obviously
<br>this strategy is full of holes, epecially with regard to off-heap
<br>binaries), but *sometimes* it's better than just blowing away the whole
<br>thing. Then, if the malloc fails again, revert to the current behaviour.
<br><br>You could, of course try to be smarter about it (look at the process
<br>calling malloc, look at the reference counted binaries, try to avoid
<br>killing SASL or system processes), but the point is that sometimes the
<br>obvious offender is the only one you need to kill to restore a system's
<br>functionality.
<br><br>Of course, I'm no expert on this, these are merely musings from having
<br>(entirely due to my own fault) triggered plenty of OOM VM kills that
<br>would have been as easily solved if a single process had been killed.
<br><br>Andrew
<br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26261578Re: Why Beam.smp crashes when memory is over?2009-11-08T21:43:25Z2009-11-08T21:43:25ZTony ArcieriOn Sun, Nov 8, 2009 at 9:56 PM, Max Lapshin <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26261578&i=0" target="_top" rel="nofollow">max.lapshin@...</a>> wrote:
<br><br>> I understand your arguments. Yet, It would be great to have
<br>> instrument, that tells supervisor, that child
<br>> is eating too much memory. Look, if we are speaking about reliable
<br>> platform, than it is better to have some
<br>> predictable behaviour.
<br>>
<br><br>I'm not really sure if we're speaking about the same thing then. Erlang is
<br>best utilized as a stateless platform, with a "fail early fail often"
<br>philosophy, and crashing the entire interpreter in the event of some
<br>underlying problem with the OS fits that entirely. You seem to be seeking a
<br>platform that lets you program defensively, handling errors rather than
<br>crashing and restoring with a clean state. In that regard, Erlang may not
<br>be the best language for you to work with, but then again the problem of
<br>successfully recovering from an out-of-resource condition is an extremely
<br>difficult one to handle well without scrapping it all and trying to restart
<br>with a clean state.
<br><br>I'm not really sure what alternative behavior you're proposing, either?
<br><br>--
<br>Tony Arcieri
<br>Medioh/Nagravision
<br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26261387Re: Why Beam.smp crashes when memory is over?2009-11-08T21:08:58Z2009-11-08T21:08:58ZMax Lapshin-2Look, I have a program, that reads MPEG TS stream from network. There
<br>is some bug in my code
<br>and my ts_lander crashes beam. How is it possible to store state of
<br>external HTTP stream??
<br><br>If Erlang VM wouldn't crash, but only kill the leaking process, I
<br>could buffer stream in other process and feed
<br>new mpeg ts lander after restart. But it is impossible =(
<br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26261326Re: Why Beam.smp crashes when memory is over?2009-11-08T20:56:38Z2009-11-08T20:56:38ZMax Lapshin-2I understand your arguments. Yet, It would be great to have
<br>instrument, that tells supervisor, that child
<br>is eating too much memory. Look, if we are speaking about reliable
<br>platform, than it is better to have some
<br>predictable behaviour.
<br><br>Perhaps some startup options, like memory threshold for OOM killer?
<br>Supervisor will never grow up to 20 GB,
<br>but our rabbitmq production instance error_logger used to grow up to 20GB+
<br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26259262Re: Dialyzer problem on OSX 10.6 (Erlang R13B02-1)2009-11-08T15:35:12Z2009-11-08T15:35:12ZTamas Nagy-3Hi Kostis,
<br><br>This is what all I get:
<br>$ dialyzer --build_plt -r /usr/local/Cellar/erlang/R13B02-1/lib/erlang/
<br>lib/erts-5.7.3/ebin/ /usr/local/Cellar/erlang/R13B02-1/lib/erlang/lib/
<br>kernel-2.13.3/ebin/ /usr/local/Cellar/erlang/R13B02-1/lib/erlang/lib/
<br>stdlib-1.16.3/ebin/
<br> Creating PLT /Users/lestat/.dialyzer_plt ...
<br><br>I will try to rebuild erlang with disabling hipe to see if that helps.
<br><br>I am happy to run some tests/traces you would like to see the results
<br>of to help.
<br><br>Regards,
<br> Tamas
<br><br>Tamas Nagy
<br>Erlang Training & Consulting
<br><a href="http://www.erlang-consulting.com" target="_top" rel="nofollow">http://www.erlang-consulting.com</a><br><br>On 8 Nov 2009, at 19:52, Kostis Sagonas wrote:
<br><div class='shrinkable-quote'><br>> Tamas Nagy wrote:
<br>>> Hi,
<br>>> Am I the only one who cannot build a plt under OSX 10.6 in R13B02-1?
<br>>> No matter what applications I specify for the initial plt it just
<br>>> hangs eating processing power but not producing anything.
<br>>> Erlang was installed with homebrew, but that shouldn't matter. Or
<br>>> maybe it does. (beam is compiled for 64 bit)
<br>>> Here is the command I used:
<br>>> dialyzer --build_plt -r /usr/local/Cellar/erlang/R13B02-1/lib/
<br>>> erlang/lib/erts-5.7.3/ebin/ /usr/local/Cellar/erlang/R13B02-1/lib/
<br>>> erlang/lib/kernel-2.13.3/ebin/ /usr/local/Cellar/erlang/R13B02-1/
<br>>> lib/erlang/lib/stdlib-1.16.3/ebin/
<br>>> I gave waiting for it to finish after ~1 hour (on an old type white
<br>>> MacBook (MacBook2,1) with 2GB of RAM).
<br>>
<br>> None of the dialyzer developers has access to such a machine to see
<br>> what's happening. Perhaps somebody else in that list might want to
<br>> report whether this is reproducible or not.
<br>>
<br>> However, perhaps the following might help: when building the PLT, do
<br>> you see a message:
<br>>
<br>> Compiling some key modules to native code...
<br>>
<br>> If so, can you disable hipe (64-bit Mac is not a platform we have
<br>> ever tested due to no access to such a machine) to see whether this
<br>> may be the culprit?
<br>>
<br>> Kostis
</div><br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26259181Re: Why Beam.smp crashes when memory is over?2009-11-08T15:25:17Z2009-11-08T15:25:17ZTony ArcieriOn Sun, Nov 8, 2009 at 1:57 PM, Jayson Vantuyl <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26259181&i=0" target="_top" rel="nofollow">kagato@...</a>> wrote:
<br><br>> In general, I believe the heartbeat functionality (i.e. erl -heart) is
<br>> supposed to handle this by letting it crash and restarting the whole
<br>> application.
<br><br><br>Yes, the design philosophy of Erlang is that your application should be
<br>stateless aside from things residing in dets tables or other external,
<br>stateful resources.
<br><br>Following the fail early, fail often approach, Erlang recovers from an out
<br>of memory condition by crashing the entire Erlang intrepreter, restarting it
<br>via heartbeat, and reloading your application.
<br><br>--
<br>Tony Arcieri
<br>Medioh/Nagravision
<br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26259029some questions about beepbeep2009-11-08T14:04:34Z2009-11-08T14:04:34ZPablo PlattHi,
<br><br>I'm trying to understand beepbeep web framework and ewgi.
<br>I like the ideas behind it.
<br><br>Is there a mailing list because I couldn't find one.
<br>Does anyone using it and can comment about it?
<br><br>Does it support arbitrary url dispatching like django and webmachine?
<br>I only saw examples like login_controller that handles /login/some/path
<br>I want to dispatch /admin/pages/15 to the admin_pages_controller.
<br>And sometimes you need to dispatch /user/15/album/12 to album_controller(user, album)
<br><br>Is there a form validation module?
<br><br>Thanks
<br><br><br> <p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26258507Re: Exceptions and gen_server2009-11-08T14:01:58Z2009-11-08T14:01:58ZAnton KrasovskyThanks for clarifying that, especially throw vs error part!
<br><br><br>Regards,
<br>Anton
<br><br>On Sun, Nov 8, 2009 at 6:47 PM, Robert Virding <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26258507&i=0" target="_top" rel="nofollow">rvirding@...</a>> wrote:
<div class='shrinkable-quote'><br>> There have been a number of different suggestions as to how you could
<br>> program this. I think the main thing for you to decide is *HOW* you want to
<br>> see the error, then it is relatively easy for you to program it. For
<br>> example:
<br>>
<br>> - First off: do you want the error to crash the server or not? If so is
<br>> links and exit signals a good enough way for you to see the error?
<br>>
<br>> - If the server is to live, how should your app see the error? Two valid
<br>> suggestions which have been given here are:
<br>>
<br>> 1. Signal the error using exit/2. This "simulates" the exit signals as if
<br>> the server had crashed.
<br>> 2. Return an error value which either: returns an error value to the caller;
<br>> or generates an error using erlang:error/1. Which is best for you depends on
<br>> your app.
<br>>
<br>> All these solutions are easy to program so the main problem is for you to
<br>> decide which is best for your app.
<br>>
<br>> One thing I would *NOT* do is use throw/1 to signal error! Throw is for
<br>> non-local returns not errors. Using it for errors will just lead to
<br>> confusion and maybe errors if other parts of app assume it is a non-local
<br>> return. As try can differentiate between throw, exit and error it is
<br>> important that they are used for the right thing.
<br>>
<br>> Robert
<br>>
<br>> P.S. I would LOVE to change catch now so it only catches throws, but I don't
<br>> think it would be possible. :-(
<br>>
<br>> 2009/11/6 Anton Krasovsky <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26258507&i=1" target="_top" rel="nofollow">anton.krasovsky@...</a>>
<br>>>
<br>>> What would be the standard way of returning/throwing an exception in a
<br>>> result to a gen_server:call?
<br>>>
<br>>> Would something like that be reasonable:
<br>>>
<br>>> a() ->
<br>>> case gen_server:call(..., a) of
<br>>> {ok, Value} -> Value;
<br>>> {error, Error} -> throw({error, Error})
<br>>> end
<br>>>
<br>>> and
<br>>>
<br>>> handle_call(a, ...) ->
<br>>> case ... of
<br>>> {ok, Value} -> {reply, {ok, Value} , State};
<br>>> {error, Error} -> {reply, {error, Error} , State}
<br>>> end
<br>>>
<br>>> Regards,
<br>>> Anton
<br>>>
<br>>> ________________________________________________________________
<br>>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>>> erlang-questions (at) erlang.org
<br>>>
<br>>
<br>>
</div><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26258155Re: Why Beam.smp crashes when memory is over?2009-11-08T13:26:43Z2009-11-08T13:26:43ZJayson Vantuyl-2Erlang needs to allocate memory in any number of situations. For
<br>example, assume that Erlang tried to tell your code. Should it
<br>generate a message? Should it call a function? Should it create an
<br>exception? Guess what all of these have in common? They allocate
<br>memory (which isn't there).
<br><br>You can try to work around this. You can have reserved memory just
<br>for this. However, there's still no clue on where it should happen.
<br>There is not a very good chance that this error will happen in the
<br>process that has all of the memory allocated. If you take the Linux
<br>OOM approach, you would have to scan all of the processes, weigh them,
<br>mix in some randomness, and message it. There's no memory for it do
<br>think much about the problem. Even if you killed it, that would just
<br>trigger the supervisor to restart it, even though we may not have
<br>actually stopped the memory leak.
<br><br>Worse, this means that an "out of memory error" can happen anywhere
<br>and must be handled everywhere, even the supervisors. Patching the
<br>supervisors to reliably handle this would be insane. Suddenly,
<br>reliability under load becomes impossible to guarantee.
<br><br>Even if you emulate Linux and provide an OOM-killer (i.e. kill
<br>processes based on randomness + heuristics to detect runaway
<br>processes), you introduce tons of random behavior into the VM, when a
<br>VM restart would be recognizable, loggable, and generally easier to
<br>debug.
<br><br>Exposing those errors creates an ugly situation. This extra error
<br>handling would cause an explosion of corner cases, decreases in
<br>reliability, and volumes of code (i.e. where bugs live). Inside of
<br>Erlang, the philosophy is to use supervisors and writing daemons to be
<br>able to recover from a restart. Heartbeat gives the same behavior for
<br>the entire VM. It's a philosophical design choice to try to handle
<br>critical faults rather than mask critical faults. It's really better
<br>than trying to handle this.
<br><br>It's seems obvious that there should be a better way to handle OOM,
<br>but it's is all devilishly difficult to do in any meaningfully
<br>portable (or useful) way.
<br><br>On Nov 8, 2009, at 1:02 PM, Max Lapshin wrote:
<br><div class='shrinkable-quote'><br>> On Sun, Nov 8, 2009 at 11:57 PM, Jayson Vantuyl <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26258155&i=0" target="_top" rel="nofollow">kagato@...</a>>
<br>> wrote:
<br>>> From within Erlang, I don't believe so.
<br>>
<br>> And what are the problems? OS never crashes when memory is over, OOM
<br>> killer does the job well.
<br>> Why should die Erlang VM?
<br>>
<br>> ________________________________________________________________
<br>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> erlang-questions (at) erlang.org
<br>>
</div><br>--
<br>Jayson Vantuyl
<br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26258155&i=1" target="_top" rel="nofollow">kagato@...</a>
<br><br><br><br><br><br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26257949Re: Why Beam.smp crashes when memory is over?2009-11-08T13:02:31Z2009-11-08T13:02:31ZMax Lapshin-2On Sun, Nov 8, 2009 at 11:57 PM, Jayson Vantuyl <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26257949&i=0" target="_top" rel="nofollow">kagato@...</a>> wrote:
<br>> From within Erlang, I don't believe so.
<br><br>And what are the problems? OS never crashes when memory is over, OOM
<br>killer does the job well.
<br>Why should die Erlang VM?
<br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26257923Re: Why Beam.smp crashes when memory is over?2009-11-08T12:57:45Z2009-11-08T12:57:45ZJayson Vantuyl-2 From within Erlang, I don't believe so.
<br><br> From outside, maybe, with the right debugger.
<br><br>In general, I believe the heartbeat functionality (i.e. erl -heart) is
<br>supposed to handle this by letting it crash and restarting the whole
<br>application.
<br><br>On Nov 8, 2009, at 12:56 PM, Max Lapshin wrote:
<br><div class='shrinkable-quote'><br>> Why does Erlang crashes, when physical memory is over?
<br>>
<br>> Error is:
<br>>
<br>> beam.smp(5454,0xb01aa000) malloc: *** mmap(size=41943040) failed
<br>> (error code=12)
<br>>
<br>> *** error: can't allocate region
<br>>
<br>> *** set a breakpoint in
<br>> malloc_error_break to debug
<br>>
<br>>
<br>> Isn't it possible to intercept this error and stop erlang process,
<br>> that requests memory?
<br>>
<br>> ________________________________________________________________
<br>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> erlang-questions (at) erlang.org
</div><br><br>--
<br>Jayson Vantuyl
<br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26257923&i=0" target="_top" rel="nofollow">kagato@...</a>
<br><br><br><br><br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26257892Why Beam.smp crashes when memory is over?2009-11-08T12:56:05Z2009-11-08T12:56:05ZMax Lapshin-2Why does Erlang crashes, when physical memory is over?
<br><br>Error is:
<br><br>beam.smp(5454,0xb01aa000) malloc: *** mmap(size=41943040) failed (error code=12)
<br><br> *** error: can't allocate region
<br><br> *** set a breakpoint in
<br>malloc_error_break to debug
<br><br><br>Isn't it possible to intercept this error and stop erlang process,
<br>that requests memory?
<br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26257285Re: Dialyzer problem on OSX 10.6 (Erlang R13B02-1)2009-11-08T11:52:12Z2009-11-08T11:52:12ZKostis Sagonas-2Tamas Nagy wrote:
<div class='shrinkable-quote'><br>> Hi,
<br>>
<br>> Am I the only one who cannot build a plt under OSX 10.6 in R13B02-1?
<br>>
<br>> No matter what applications I specify for the initial plt it just hangs
<br>> eating processing power but not producing anything.
<br>> Erlang was installed with homebrew, but that shouldn't matter. Or maybe
<br>> it does. (beam is compiled for 64 bit)
<br>>
<br>> Here is the command I used:
<br>> dialyzer --build_plt -r
<br>> /usr/local/Cellar/erlang/R13B02-1/lib/erlang/lib/erts-5.7.3/ebin/
<br>> /usr/local/Cellar/erlang/R13B02-1/lib/erlang/lib/kernel-2.13.3/ebin/
<br>> /usr/local/Cellar/erlang/R13B02-1/lib/erlang/lib/stdlib-1.16.3/ebin/
<br>>
<br>> I gave waiting for it to finish after ~1 hour (on an old type white
<br>> MacBook (MacBook2,1) with 2GB of RAM).
</div><br>None of the dialyzer developers has access to such a machine to see
<br>what's happening. Perhaps somebody else in that list might want to
<br>report whether this is reproducible or not.
<br><br>However, perhaps the following might help: when building the PLT, do you
<br>see a message:
<br><br> Compiling some key modules to native code...
<br><br>If so, can you disable hipe (64-bit Mac is not a platform we have ever
<br>tested due to no access to such a machine) to see whether this may be
<br>the culprit?
<br><br>Kostis
<br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26256992Dialyzer problem on OSX 10.6 (Erlang R13B02-1)2009-11-08T11:23:36Z2009-11-08T11:23:36ZTamas Nagy-3Hi,
<br><br>Am I the only one who cannot build a plt under OSX 10.6 in R13B02-1?
<br><br>No matter what applications I specify for the initial plt it just
<br>hangs eating processing power but not producing anything.
<br>Erlang was installed with homebrew, but that shouldn't matter. Or
<br>maybe it does. (beam is compiled for 64 bit)
<br><br>Here is the command I used:
<br>dialyzer --build_plt -r /usr/local/Cellar/erlang/R13B02-1/lib/erlang/
<br>lib/erts-5.7.3/ebin/ /usr/local/Cellar/erlang/R13B02-1/lib/erlang/lib/
<br>kernel-2.13.3/ebin/ /usr/local/Cellar/erlang/R13B02-1/lib/erlang/lib/
<br>stdlib-1.16.3/ebin/
<br><br>I gave waiting for it to finish after ~1 hour (on an old type white
<br>MacBook (MacBook2,1) with 2GB of RAM).
<br><br>The info I get from the dialyzer seems to suggest that it still
<br>assumes that erts version is 5.7 can the problem be around there?
<br><br> Checking whether the PLT /Users/lestat/.dialyzer_plt is up-to-
<br>date...dialyzer: Could not find the PLT: /Users/lestat/.dialyzer_plt
<br>Use the options:
<br> --build_plt to build a new one; or
<br> --add_to_plt to add to an existing PLT
<br>For example (in an installed Erlang/OTP system):
<br> dialyzer --build_plt -r $ERL_TOP/lib/erts-5.7/ebin\
<br> $ERL_TOP/lib/kernel/ebin\
<br> $ERL_TOP/lib/stdlib/ebin\
<br> $ERL_TOP/lib/mnesia/ebin\
<br> $ERL_TOP/lib/crypto/ebin
<br>Note that building a PLT such as the above may take 20 mins or so
<br><br>If I try to use a plt generated on another machine (on Ubuntu there
<br>does not seem to be a problem) it seems to work.
<br><br>P.S.: I've tried rebuilding erlang with and without disable-debug flag
<br>as well to no avail.
<br><br>Tamas Nagy
<br>Erlang Training & Consulting
<br><a href="http://www.erlang-consulting.com" target="_top" rel="nofollow">http://www.erlang-consulting.com</a><br><br><br>________________________________________________________________
<br>erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>erlang-questions (at) erlang.org
<br><br><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>tag:old.nabble.com,2006:post-26256657Re: Exceptions and gen_server2009-11-08T10:47:29Z2009-11-08T10:47:29ZRobert VirdingThere have been a number of different suggestions as to how you could
<br>program this. I think the main thing for you to decide is *HOW* you want to
<br>see the error, then it is relatively easy for you to program it. For
<br>example:
<br><br>- First off: do you want the error to crash the server or not? If so is
<br>links and exit signals a good enough way for you to see the error?
<br><br>- If the server is to live, how should your app see the error? Two valid
<br>suggestions which have been given here are:
<br><br>1. Signal the error using exit/2. This "simulates" the exit signals as if
<br>the server had crashed.
<br>2. Return an error value which either: returns an error value to the caller;
<br>or generates an error using erlang:error/1. Which is best for you depends on
<br>your app.
<br><br>All these solutions are easy to program so the main problem is for you to
<br>decide which is best for your app.
<br><br>One thing I would *NOT* do is use throw/1 to signal error! Throw is for
<br>non-local returns not errors. Using it for errors will just lead to
<br>confusion and maybe errors if other parts of app assume it is a non-local
<br>return. As try can differentiate between throw, exit and error it is
<br>important that they are used for the right thing.
<br><br>Robert
<br><br>P.S. I would LOVE to change catch now so it only catches throws, but I don't
<br>think it would be possible. :-(
<br><br>2009/11/6 Anton Krasovsky <<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26256657&i=0" target="_top" rel="nofollow">anton.krasovsky@...</a>>
<br><div class='shrinkable-quote'><br>> What would be the standard way of returning/throwing an exception in a
<br>> result to a gen_server:call?
<br>>
<br>> Would something like that be reasonable:
<br>>
<br>> a() ->
<br>> case gen_server:call(..., a) of
<br>> {ok, Value} -> Value;
<br>> {error, Error} -> throw({error, Error})
<br>> end
<br>>
<br>> and
<br>>
<br>> handle_call(a, ...) ->
<br>> case ... of
<br>> {ok, Value} -> {reply, {ok, Value} , State};
<br>> {error, Error} -> {reply, {error, Error} , State}
<br>> end
<br>>
<br>> Regards,
<br>> Anton
<br>>
<br>> ________________________________________________________________
<br>> erlang-questions mailing list. See <a href="http://www.erlang.org/faq.html" target="_top" rel="nofollow">http://www.erlang.org/faq.html</a><br>> erlang-questions (at) erlang.org
<br>>
<br>>
<br></div><p>From forum: <a href="http://old.nabble.com/Erlang-Questions-f14096.html" embed="fixTarget[14096]" target="_top" >Erlang Questions</a></p>