Dynamically Allocating Memory in C

[Q Beukes]

Hey,

I am using the HiTech compiler, since I know this might be compiler dependent.

I was hoping for some advice/corrections on dynamically allocating memory.

My understanding is this:
1. The device has X bytes of memory.
2. I can assign the address of memory I wish to use to a pointer, and
work from there.
3. I need to somehow make sure I don't use memory which is already
assigned by the compiler.
4. I _assume_ the compiler will use the lower memory ranges, so if I
know how much memory the compiler has assigned, I can use any other
above that address? I can determine this from the following output:
Memory Summary:
Program space        used   8DEh (  2270) of  8000h bytes   (  6.9%)
Data space           used    A9h (   169) of   600h bytes   ( 11.0%)
EEPROM space         used     0h (     0) of   100h bytes   (  0.0%)
External data memory None available
ID Location space    used     0h (     0) of     8h nibbles (  0.0%)
Configuration bits   used     7h (     7) of     7h words   (100.0%)

>From here I figured, I can choose an address range based on the following data:
1. I have a PIC 18F2520.
2. It has 1536 bytes of SRAM (600h), as seen in row 2 of the summary above.
3. The compiler assigned 169 bytes (A9h) of memory.

Then chosen as follows: Any addresses above A9, and to accommodate
future changes, use any above 200, and carefully monitor my compiles?

Alternatively. I had a quick glance over the compiler's manual, and
noticed memory areas called "psects", which can be user defined with
#pragma psect, but I don't see how I can use this for the described
purpose.

Any/all advice would be appreciated.

Quintin Beukes
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[sergio masci-2]

On Tue, 6 Oct 2009, Quintin Beukes wrote:

> Any/all advice would be appreciated.

Are you aware of the standard C library functions "alloc", "malloc" and
"free"? Are these available for the HiTech compiler?

Regards
Sergio Masci
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Isaac Marino Bavaresco]

Quintin Beukes escreveu:

> Hey,
>
> I am using the HiTech compiler, since I know this might be compiler dependent.
>
> I was hoping for some advice/corrections on dynamically allocating memory.
>
> My understanding is this:
> 1. The device has X bytes of memory.
> 2. I can assign the address of memory I wish to use to a pointer, and
> work from there.
> 3. I need to somehow make sure I don't use memory which is already
> assigned by the compiler.
> 4. I _assume_ the compiler will use the lower memory ranges, so if I
> know how much memory the compiler has assigned, I can use any other
> above that address? I can determine this from the following output:
> Memory Summary:
> Program space        used   8DEh (  2270) of  8000h bytes   (  6.9%)
> Data space           used    A9h (   169) of   600h bytes   ( 11.0%)
> EEPROM space         used     0h (     0) of   100h bytes   (  0.0%)
> External data memory None available
> ID Location space    used     0h (     0) of     8h nibbles (  0.0%)
> Configuration bits   used     7h (     7) of     7h words   (100.0%)
>
> >From here I figured, I can choose an address range based on the following data:
> 1. I have a PIC 18F2520.
> 2. It has 1536 bytes of SRAM (600h), as seen in row 2 of the summary above.
> 3. The compiler assigned 169 bytes (A9h) of memory.
>
> Then chosen as follows: Any addresses above A9, and to accommodate
> future changes, use any above 200, and carefully monitor my compiles?
>
> Alternatively. I had a quick glance over the compiler's manual, and
> noticed memory areas called "psects", which can be user defined with
> #pragma psect, but I don't see how I can use this for the described
> purpose.
>
> Any/all advice would be appreciated.
>
> Quintin Beukes
>  

Greetings,

Take a look at my page:
<http://www.piclist.com/techref/member/IMB-yahoo-J86/heap-mgmt.htm>
Directly related to your needs, probably off-the-shelf.

See other PIC-related stuff also:
<http://www.piclist.com/techref/member/IMB-yahoo-J86/index.htm>

Best regards,

Isaac

__________________________________________________
Faça ligações para outros computadores com o novo Yahoo! Messenger
http://br.beta.messenger.yahoo.com/ 
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Olin Lathrop]

Quintin Beukes wrote:
> I was hoping for some advice/corrections on dynamically allocating
> memory.

Some compilers come with heap and dynamic memory management routines.  Use
them if they are there, or roll your own if you want or need to, but don't
just go grabbing memory you think isn't in use.

Why do you need to dynamically allocate memory anyway?  This is rather
unusual in a small embedded system, and likely indicates big system thinking
on a small system.  I've only done on the fly real persistant memory
allocation on a PIC once that I can remember, and in that case the memory
being allocated was actually out on a ENC28J60 ethernet MAC/PHY with the PIC
keeping track of the allocated regions.  I have sometimes needed a temporary
buffer and the like inside a single routine on a 18F.  In those cases I have
so far just temporarily appended the buffer to the data stack.

What problem are you really trying to solve?

> 4. I _assume_ the compiler will use ...

This sounds like really bad design.  Don't assume, know.  But even if you
did know for sure, that only applies to this rev of this compiler.  Guessing
where unused memory is without using the proper mechanisms to allocate it is
heading for trouble.  For example, how do you know the data stack isn't
allowed to grow into the left over memory?

> Then chosen as follows: Any addresses above A9, and to accommodate
> future changes, use any above 200, and carefully monitor my compiles?

Yuck!  That is really bad design.


********************************************************************
Embed Inc, Littleton Massachusetts, http://www.embedinc.com/products
[phone].  Gold level PIC consultants since 2000.
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Q Beukes]

Yes, Unfortunately not. I am having a very hard time without it.

Quintin Beukes



On Tue, Oct 6, 2009 at 3:42 PM, sergio masci <smplx@...> wrote:

>
>
> On Tue, 6 Oct 2009, Quintin Beukes wrote:
>
>> Any/all advice would be appreciated.
>
> Are you aware of the standard C library functions "alloc", "malloc" and
> "free"? Are these available for the HiTech compiler?
>
> Regards
> Sergio Masci
> --
> http://www.piclist.com PIC/SX FAQ & list archive
> View/change your membership options at
> http://mailman.mit.edu/mailman/listinfo/piclist
>

--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Q Beukes]

That's awesome. Thanks alot.

Quintin Beukes



On Tue, Oct 6, 2009 at 1:19 PM, Isaac Marino Bavaresco
<isaacbavaresco@...> wrote:

> Quintin Beukes escreveu:
>> Hey,
>>
>> I am using the HiTech compiler, since I know this might be compiler dependent.
>>
>> I was hoping for some advice/corrections on dynamically allocating memory.
>>
>> My understanding is this:
>> 1. The device has X bytes of memory.
>> 2. I can assign the address of memory I wish to use to a pointer, and
>> work from there.
>> 3. I need to somehow make sure I don't use memory which is already
>> assigned by the compiler.
>> 4. I _assume_ the compiler will use the lower memory ranges, so if I
>> know how much memory the compiler has assigned, I can use any other
>> above that address? I can determine this from the following output:
>> Memory Summary:
>> Program space        used   8DEh (  2270) of  8000h bytes   (  6.9%)
>> Data space           used    A9h (   169) of   600h bytes   ( 11.0%)
>> EEPROM space         used     0h (     0) of   100h bytes   (  0.0%)
>> External data memory None available
>> ID Location space    used     0h (     0) of     8h nibbles (  0.0%)
>> Configuration bits   used     7h (     7) of     7h words   (100.0%)
>>
>> >From here I figured, I can choose an address range based on the following data:
>> 1. I have a PIC 18F2520.
>> 2. It has 1536 bytes of SRAM (600h), as seen in row 2 of the summary above.
>> 3. The compiler assigned 169 bytes (A9h) of memory.
>>
>> Then chosen as follows: Any addresses above A9, and to accommodate
>> future changes, use any above 200, and carefully monitor my compiles?
>>
>> Alternatively. I had a quick glance over the compiler's manual, and
>> noticed memory areas called "psects", which can be user defined with
>> #pragma psect, but I don't see how I can use this for the described
>> purpose.
>>
>> Any/all advice would be appreciated.
>>
>> Quintin Beukes
>>
>
> Greetings,
>
> Take a look at my page:
> <http://www.piclist.com/techref/member/IMB-yahoo-J86/heap-mgmt.htm>
> Directly related to your needs, probably off-the-shelf.
>
> See other PIC-related stuff also:
> <http://www.piclist.com/techref/member/IMB-yahoo-J86/index.htm>
>
> Best regards,
>
> Isaac
>
> __________________________________________________
> Faça ligações para outros computadores com o novo Yahoo! Messenger
> http://br.beta.messenger.yahoo.com/
> --
> http://www.piclist.com PIC/SX FAQ & list archive
> View/change your membership options at
> http://mailman.mit.edu/mailman/listinfo/piclist
>

--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Isaac Marino Bavaresco]

Olin Lathrop escreveu:

> Quintin Beukes wrote:
>  
>> I was hoping for some advice/corrections on dynamically allocating
>> memory.
>>    
>
> Some compilers come with heap and dynamic memory management routines.  Use
> them if they are there, or roll your own if you want or need to, but don't
> just go grabbing memory you think isn't in use.
>
> Why do you need to dynamically allocate memory anyway?  This is rather
> unusual in a small embedded system, and likely indicates big system thinking
> on a small system.  I've only done on the fly real persistant memory
> allocation on a PIC once that I can remember, and in that case the memory
> being allocated was actually out on a ENC28J60 ethernet MAC/PHY with the PIC
> keeping track of the allocated regions.  I have sometimes needed a temporary
> buffer and the like inside a single routine on a 18F.  In those cases I have
> so far just temporarily appended the buffer to the data stack.
>
> What problem are you really trying to solve?
>
>  
>> 4. I _assume_ the compiler will use ...
>>    
>
> This sounds like really bad design.  Don't assume, know.  But even if you
> did know for sure, that only applies to this rev of this compiler.  Guessing
> where unused memory is without using the proper mechanisms to allocate it is
> heading for trouble.  For example, how do you know the data stack isn't
> allowed to grow into the left over memory?
>
>  
>> Then chosen as follows: Any addresses above A9, and to accommodate
>> future changes, use any above 200, and carefully monitor my compiles?
>>    
>
> Yuck!  That is really bad design.

Indeed, your heap must be declared as a normal variable (big array of
char) so the compiler and linker locate it properly. If you just use an
address range you *think* is not in use then next time you link it, it
may be on top of some other variable.

Then your allocator "cuts" chunks of it and gives to your application
(return the chunk address).

See the sink I posted in my previous reply, it is exactly what you need,
and very well tested. If you need help to get started just send me a
message.

Best regards,

Isaac

__________________________________________________
Faça ligações para outros computadores com o novo Yahoo! Messenger
http://br.beta.messenger.yahoo.com/ 
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Tamas Rudnai]

On Tue, Oct 6, 2009 at 3:24 PM, Isaac Marino Bavaresco <
isaacbavaresco@...> wrote:

> Indeed, your heap must be declared as a normal variable (big array of
> char) so the compiler and linker locate it properly. If you just use an
> address range you *think* is not in use then next time you link it, it
> may be on top of some other variable.
>

I think it is more common to declare a chunk of memory in linker script so
that area will not be used by the linker therefore free to use. With a
variable it could happen that the compiler thinks that it was not used and
then the optimization chops it off...

BTW: I think what Olin is trying to express is that a dynamic memory
allocator needs an overhead plus some trouble with the memory fragmentation
so overall it is not the best choice for a small device.

Tamas

>
> Then your allocator "cuts" chunks of it and gives to your application
> (return the chunk address).
>
> See the sink I posted in my previous reply, it is exactly what you need,
> and very well tested. If you need help to get started just send me a
> message.
>
> Best regards,
>
> Isaac
>
> __________________________________________________
> Faça ligações para outros computadores com o novo Yahoo! Messenger
> http://br.beta.messenger.yahoo.com/
> --
> http://www.piclist.com PIC/SX FAQ & list archive
> View/change your membership options at
> http://mailman.mit.edu/mailman/listinfo/piclist
>

--
http://www.mcuhobby.com
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[William "Chops" Westfield]

On Oct 6, 2009, at 7:24 AM, Isaac Marino Bavaresco wrote:

> Indeed, your heap must be declared as a normal variable (big array of
> char) so the compiler and linker locate it properly.

This is the easy way to do things, but of course it won't correct for  
suddenly switching to a different PIC with more RAM.  Compiler/linker  
combinations are usually pretty good about defining some magic symbols  
(_edata for gcc, for instance) that indicate the end of the pre-
allocated data space.  If you combine that with detecting or "knowing"  
where the stack is, and how much memory actually exists, you can get  
away without having the array...

BillW
 
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Q Beukes]

That's actually a very good technique. I'm actually embarrassed for my
e-mail now, as I can't believe I didn't think of that :/

Either way, to answer the questions.

There is no real reason for this. I understand the constraints of an
embedded device fairly well. I was mostly wondering out of curiosity.
It was inspired by a redevelopment of a program, where I constantly
run into situations where the first impulse is a malloc(), then a stop
- think and redevelop a bunch of code.

I'm busy making a base framework for all our applications while
redeveloping one specific one, and the common behavior in them is
difficult to define without dynamic data if you want a simple
structure. I've had to make a few sacrifices so far, though I must say
that without malloc things have been going pretty well. In the most
severe cases I got away with using static (where I have declarations
inside a function).

Quintin Beukes



On Tue, Oct 6, 2009 at 4:24 PM, Isaac Marino Bavaresco
<isaacbavaresco@...> wrote:

> Olin Lathrop escreveu:
>> Quintin Beukes wrote:
>>
>>> I was hoping for some advice/corrections on dynamically allocating
>>> memory.
>>>
>>
>> Some compilers come with heap and dynamic memory management routines.  Use
>> them if they are there, or roll your own if you want or need to, but don't
>> just go grabbing memory you think isn't in use.
>>
>> Why do you need to dynamically allocate memory anyway?  This is rather
>> unusual in a small embedded system, and likely indicates big system thinking
>> on a small system.  I've only done on the fly real persistant memory
>> allocation on a PIC once that I can remember, and in that case the memory
>> being allocated was actually out on a ENC28J60 ethernet MAC/PHY with the PIC
>> keeping track of the allocated regions.  I have sometimes needed a temporary
>> buffer and the like inside a single routine on a 18F.  In those cases I have
>> so far just temporarily appended the buffer to the data stack.
>>
>> What problem are you really trying to solve?
>>
>>
>>> 4. I _assume_ the compiler will use ...
>>>
>>
>> This sounds like really bad design.  Don't assume, know.  But even if you
>> did know for sure, that only applies to this rev of this compiler.  Guessing
>> where unused memory is without using the proper mechanisms to allocate it is
>> heading for trouble.  For example, how do you know the data stack isn't
>> allowed to grow into the left over memory?
>>
>>
>>> Then chosen as follows: Any addresses above A9, and to accommodate
>>> future changes, use any above 200, and carefully monitor my compiles?
>>>
>>
>> Yuck!  That is really bad design.
>
> Indeed, your heap must be declared as a normal variable (big array of
> char) so the compiler and linker locate it properly. If you just use an
> address range you *think* is not in use then next time you link it, it
> may be on top of some other variable.
>
> Then your allocator "cuts" chunks of it and gives to your application
> (return the chunk address).
>
> See the sink I posted in my previous reply, it is exactly what you need,
> and very well tested. If you need help to get started just send me a
> message.
>
> Best regards,
>
> Isaac
>
> __________________________________________________
> Faça ligações para outros computadores com o novo Yahoo! Messenger
> http://br.beta.messenger.yahoo.com/
> --
> http://www.piclist.com PIC/SX FAQ & list archive
> View/change your membership options at
> http://mailman.mit.edu/mailman/listinfo/piclist
>

--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Olin Lathrop]

Tamas Rudnai wrote:
> BTW: I think what Olin is trying to express is that a dynamic memory
> allocator needs an overhead plus some trouble with the memory
> fragmentation so overall it is not the best choice for a small device.

Yes, and usually small embedded systems run fixed problems where there isn't
a need to dynamically allocate memory.  Asking for dynamic memory allocation
is a warning that the system wasn't architected with the small system
mindset.  There's usually a way around dynamic memory if you think about the
problem differently.

Another drawback of dynamic memory on a small system is how to guarantee you
won't run out.  If you don't know how much memory you need up front, then
how do you know you don't need more than there is available under the right
set of input conditions.  And if you do use malloc or the like, what are you
going to do when it fails?  Punting back to the operating system with a
non-zero exit status doesn't work here.


********************************************************************
Embed Inc, Littleton Massachusetts, http://www.embedinc.com/products
[phone].  Gold level PIC consultants since 2000.
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Isaac Marino Bavaresco]

Tamas Rudnai escreveu:

> On Tue, Oct 6, 2009 at 3:24 PM, Isaac Marino Bavaresco <
> isaacbavaresco@...> wrote:
>
>  
>> Indeed, your heap must be declared as a normal variable (big array of
>> char) so the compiler and linker locate it properly. If you just use an
>> address range you *think* is not in use then next time you link it, it
>> may be on top of some other variable.
>>
>>    
>
> I think it is more common to declare a chunk of memory in linker script so
> that area will not be used by the linker therefore free to use. With a
> variable it could happen that the compiler thinks that it was not used and
> then the optimization chops it off...
>
>  

Yes, you will need to change the linker-script to define an area big
enough for your heap, but you must declare also a variable so you can
take its address, initialize it, etc.
There are other approaches, usually needing knowledge of the compiler´s
internals (automatically defined symbols, etc). When the compiler
manufacturers provide heap management they make the compiler help/know
about it. Without this "privileged" knowledge the simplest way is to
declare it as a a big array, and it is more portable too.

> BTW: I think what Olin is trying to express is that a dynamic memory
> allocator needs an overhead plus some trouble with the memory fragmentation
> so overall it is not the best choice for a small device.
>
> Tamas

I use FreeRTOS with PIC18 and dsPICs and it doesn't work without dynamic
allocation.


Best regards,

Isaac

__________________________________________________
Faça ligações para outros computadores com o novo Yahoo! Messenger
http://br.beta.messenger.yahoo.com/ 
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Q Beukes]

It's very true what you say. I think I did mention it, but rethinking
all my problems where I felt like using malloc ended up in an
alternative, though less elegant (another word for "big system"
perhaps? ;>).

Though so far I've not been able to come up with a solution for a
specific problem where the mode of the device can change and so the
data it stores changes. Though both structures sometimes need memory,
depending on it's status. We basically solved it by overlaying the
same structure over a memory block and giving it an ID to know what to
cast to, though 2 dynamically sized linked lists would have been the
ideal solution here.

We have an SD card and allocate a large array to store a certain
amount of blocks, flushing them as they are written to make space in
the array, like when we need more of a certain structure, which is
itself slow both times, as you don't only replace items (due to their
very different sizes), when you clean old items you have to shift the
array data. If you had a more dynamically sized linked list, we could
gain a lot of performance if these structures could be more dynamic,
and only flush to SD card "cold" data when more memory is needed.

I'm sure many solutions could be thought up for these, as have been my
experience so far when I think about them "embedded-ly" a bit more,
but given the way the systems work a lot could be gained in both there
cases with semi-dynamically sized structures, by which I mean they
share memory, but can allocate/free like with malloc() and free().
Almost like a smart union, and together with the SD card you keep in
memory the "hot" data.The SD card is very slow, and network + RF
transfers as well. With a bit of multitasking and a some "dynamic
caches" (perverted the meaning a bit - sorry) the system could work
much more optimally. We for instance have a problem that where the RF
transfers happen they need to happen very fast, and the restructuring
of the array to make space, and a lot of flushing to/from SD card in
between wastes a lot of time. Same goes for reading from the SD card
when sending back over the network. If this happens with the "other
than to be transfered" structure mostly in memory because of a
previous "exchange which overwrote it's data", it's almost purely
reading from the SD card. Reading into dynamically allocated areas and
freeing as you go could help, especially when the hot data is
available and as you start writing to the network port you can start
reading from the SD card, using what is available in RAM while you
wait for new data.

But like discussed, there is a lot involved, and I understand the
implications/constraints. I haven't gotten around to analyzing the
system yet (only a few thoughts/looks here/there and an RF data
structure optimization once), but from initial impressions this might
be a possibly justified use of dynamic allocation.

Quintin Beukes



On Tue, Oct 6, 2009 at 5:39 PM, Olin Lathrop <olin_piclist@...> wrote:

> Tamas Rudnai wrote:
>> BTW: I think what Olin is trying to express is that a dynamic memory
>> allocator needs an overhead plus some trouble with the memory
>> fragmentation so overall it is not the best choice for a small device.
>
> Yes, and usually small embedded systems run fixed problems where there isn't
> a need to dynamically allocate memory.  Asking for dynamic memory allocation
> is a warning that the system wasn't architected with the small system
> mindset.  There's usually a way around dynamic memory if you think about the
> problem differently.
>
> Another drawback of dynamic memory on a small system is how to guarantee you
> won't run out.  If you don't know how much memory you need up front, then
> how do you know you don't need more than there is available under the right
> set of input conditions.  And if you do use malloc or the like, what are you
> going to do when it fails?  Punting back to the operating system with a
> non-zero exit status doesn't work here.
>
>
> ********************************************************************
> Embed Inc, Littleton Massachusetts, http://www.embedinc.com/products
> [phone].  Gold level PIC consultants since 2000.
> --
> http://www.piclist.com PIC/SX FAQ & list archive
> View/change your membership options at
> http://mailman.mit.edu/mailman/listinfo/piclist
>

--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Bill Knight]

One of the problems with malloc & free is memory fragmentation due to
the variable size of the blocks that are allocated and released.
Another approach that is often used in embedded systems is a memory pool
of fixed sized blocks where the size of the blocks in a given pool is
large enough to hold all of a specific item item.  If an item is
somewhat smaller than a block, it just doesn't use all of it.  The
result is still much safer.  If you need both big and small blocks then
have the small block manager get a big block from the big block
allocator and divide it into small fixed size blocks.  If it needs more
small blocks, have it get another big block and divide it.

I'm not sure if this will fit your purpose.  Just tossing it out for
discussion.

Regards
-Bill Knight
R O SoftWare

Quintin Beukes wrote:

> It's very true what you say. I think I did mention it, but rethinking
> all my problems where I felt like using malloc ended up in an
> alternative, though less elegant (another word for "big system"
> perhaps? ;>).
>
> Though so far I've not been able to come up with a solution for a
> specific problem where the mode of the device can change and so the
> data it stores changes. Though both structures sometimes need memory,
> depending on it's status. We basically solved it by overlaying the
> same structure over a memory block and giving it an ID to know what to
> cast to, though 2 dynamically sized linked lists would have been the
> ideal solution here.
>
> We have an SD card and allocate a large array to store a certain
> amount of blocks, flushing them as they are written to make space in
> the array, like when we need more of a certain structure, which is
> itself slow both times, as you don't only replace items (due to their
> very different sizes), when you clean old items you have to shift the
> array data. If you had a more dynamically sized linked list, we could
> gain a lot of performance if these structures could be more dynamic,
> and only flush to SD card "cold" data when more memory is needed.
>
> I'm sure many solutions could be thought up for these, as have been my
> experience so far when I think about them "embedded-ly" a bit more,
> but given the way the systems work a lot could be gained in both there
> cases with semi-dynamically sized structures, by which I mean they
> share memory, but can allocate/free like with malloc() and free().
> Almost like a smart union, and together with the SD card you keep in
> memory the "hot" data.The SD card is very slow, and network + RF
> transfers as well. With a bit of multitasking and a some "dynamic
> caches" (perverted the meaning a bit - sorry) the system could work
> much more optimally. We for instance have a problem that where the RF
> transfers happen they need to happen very fast, and the restructuring
> of the array to make space, and a lot of flushing to/from SD card in
> between wastes a lot of time. Same goes for reading from the SD card
> when sending back over the network. If this happens with the "other
> than to be transfered" structure mostly in memory because of a
> previous "exchange which overwrote it's data", it's almost purely
> reading from the SD card. Reading into dynamically allocated areas and
> freeing as you go could help, especially when the hot data is
> available and as you start writing to the network port you can start
> reading from the SD card, using what is available in RAM while you
> wait for new data.
>
> But like discussed, there is a lot involved, and I understand the
> implications/constraints. I haven't gotten around to analyzing the
> system yet (only a few thoughts/looks here/there and an RF data
> structure optimization once), but from initial impressions this might
> be a possibly justified use of dynamic allocation.
>
> Quintin Beukes
>
>
>
> On Tue, Oct 6, 2009 at 5:39 PM, Olin Lathrop <olin_piclist@...> wrote:
>> Tamas Rudnai wrote:
>>> BTW: I think what Olin is trying to express is that a dynamic memory
>>> allocator needs an overhead plus some trouble with the memory
>>> fragmentation so overall it is not the best choice for a small device.
>> Yes, and usually small embedded systems run fixed problems where there isn't
>> a need to dynamically allocate memory.  Asking for dynamic memory allocation
>> is a warning that the system wasn't architected with the small system
>> mindset.  There's usually a way around dynamic memory if you think about the
>> problem differently.
>>
>> Another drawback of dynamic memory on a small system is how to guarantee you
>> won't run out.  If you don't know how much memory you need up front, then
>> how do you know you don't need more than there is available under the right
>> set of input conditions.  And if you do use malloc or the like, what are you
>> going to do when it fails?  Punting back to the operating system with a
>> non-zero exit status doesn't work here.
>>
>>
>> ********************************************************************
>> Embed Inc, Littleton Massachusetts, http://www.embedinc.com/products
>> [phone].  Gold level PIC consultants since 2000.
>> --
>> http://www.piclist.com PIC/SX FAQ & list archive
>> View/change your membership options at
>> http://mailman.mit.edu/mailman/listinfo/piclist
>>
>

--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Gerhard Fiedler]

Bill Knight wrote:

> Quintin Beukes wrote:
>> On Tue, Oct 6, 2009 at 5:39 PM, Olin Lathrop <olin_piclist@...> wrote:
>>> Another drawback of dynamic memory on a small system is how to
>>> guarantee you won't run out.  If you don't know how much memory you
>>> need up front, then how do you know you don't need more than there
>>> is available under the right set of input conditions.  And if you
>>> do use malloc or the like, what are you going to do when it fails?
>>> Punting back to the operating system with a non-zero exit status
>>> doesn't work here.
>>
>> Though so far I've not been able to come up with a solution for a
>> specific problem where the mode of the device can change and so the
>> data it stores changes. Though both structures sometimes need
>> memory, depending on it's status. We basically solved it by
>> overlaying the same structure over a memory block and giving it an
>> ID to know what to cast to, though 2 dynamically sized linked lists
>> would have been the ideal solution here.
>
> One of the problems with malloc & free is memory fragmentation due to
> the variable size of the blocks that are allocated and released.
> Another approach that is often used in embedded systems is a memory
> pool of fixed sized blocks where the size of the blocks in a given
> pool is large enough to hold all of a specific item item.  If an item
> is somewhat smaller than a block, it just doesn't use all of it.  The
> result is still much safer.  If you need both big and small blocks
> then have the small block manager get a big block from the big block
> allocator and divide it into small fixed size blocks.  If it needs
> more small blocks, have it get another big block and divide it.

Another approach could be a union of two arrays with different member
types. No list => no dynamic allocation needed. Array => can be used as
circular buffer or stack. Overall union (as opposed to individual union)
=> all the available space is used (in case the types have sufficiently
different sizes that this makes a difference).

One of the main issues here is that dynamic memory is not predictable.
Usually you need to make sure that in the worst case scenario (per spec)
the available memory is enough (queue length, etc.) With dynamic memory,
you don't even know before-hand how much usable memory you have, due to
fragmentation. Only a 'designed' memory solution can give you that. With
this in mind, malloc et al is generally out of the picture anyway.

Gerhard
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Tamas Rudnai]

On Wed, Oct 7, 2009 at 12:07 PM, Gerhard Fiedler <lists@...
> wrote:

> One of the main issues here is that dynamic memory is not predictable.
> Usually you need to make sure that in the worst case scenario (per spec)
> the available memory is enough (queue length, etc.) With dynamic memory,
> you don't even know before-hand how much usable memory you have, due to
> fragmentation. Only a 'designed' memory solution can give you that. With
> this in mind, malloc et al is generally out of the picture anyway.
>

In many cases a "double-pointer" (or in more fashionable naming a memory
handler) should be used for dynamic memories instead of pure pointers, so
that when the memory is fragmented you can do a garbage collection. The only
problem is when are you going to start the garbage collector so that it will
not affect the device functionality -- in many systems you can initiate it
manually when you know you have enough time to finish, while some other
system the os / memory manager do that automatically for you (like .NET
brrr, or Java or Mac 68k, and even Apple II with the Apple Basic had this
kind of approach).

Again, not sure if you want to get into trouble like this on an MCU.

Tamas



>
> Gerhard
> --
> http://www.piclist.com PIC/SX FAQ & list archive
> View/change your membership options at
> http://mailman.mit.edu/mailman/listinfo/piclist
>



--
http://www.mcuhobby.com
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Isaac Marino Bavaresco]

Gerhard Fiedler escreveu:

> Bill Knight wrote:
>
>  
>> Quintin Beukes wrote:
>>    
>>> On Tue, Oct 6, 2009 at 5:39 PM, Olin Lathrop <olin_piclist@...> wrote:
>>>      
>>>> Another drawback of dynamic memory on a small system is how to
>>>> guarantee you won't run out.  If you don't know how much memory you
>>>> need up front, then how do you know you don't need more than there
>>>> is available under the right set of input conditions.  And if you
>>>> do use malloc or the like, what are you going to do when it fails?
>>>> Punting back to the operating system with a non-zero exit status
>>>> doesn't work here.
>>>>        
>>> Though so far I've not been able to come up with a solution for a
>>> specific problem where the mode of the device can change and so the
>>> data it stores changes. Though both structures sometimes need
>>> memory, depending on it's status. We basically solved it by
>>> overlaying the same structure over a memory block and giving it an
>>> ID to know what to cast to, though 2 dynamically sized linked lists
>>> would have been the ideal solution here.
>>>      
>> One of the problems with malloc & free is memory fragmentation due to
>> the variable size of the blocks that are allocated and released.
>> Another approach that is often used in embedded systems is a memory
>> pool of fixed sized blocks where the size of the blocks in a given
>> pool is large enough to hold all of a specific item item.  If an item
>> is somewhat smaller than a block, it just doesn't use all of it.  The
>> result is still much safer.  If you need both big and small blocks
>> then have the small block manager get a big block from the big block
>> allocator and divide it into small fixed size blocks.  If it needs
>> more small blocks, have it get another big block and divide it.
>>    
>
> Another approach could be a union of two arrays with different member
> types. No list => no dynamic allocation needed. Array => can be used as
> circular buffer or stack. Overall union (as opposed to individual union)
> => all the available space is used (in case the types have sufficiently
> different sizes that this makes a difference).
>
> One of the main issues here is that dynamic memory is not predictable.
> Usually you need to make sure that in the worst case scenario (per spec)
> the available memory is enough (queue length, etc.) With dynamic memory,
> you don't even know before-hand how much usable memory you have, due to
> fragmentation. Only a 'designed' memory solution can give you that. With
> this in mind, malloc et al is generally out of the picture anyway.
>
> Gerhard
>  

Not always true. I have a system where there are a few types of dynamic
data, and all of them have fixed sizes.
What changes is how many of each type is needed at any time. And
sometimes when one type is in use, by design, other types are not needed.

Another example, to avoid priority inversion I have a queue of messages
(real text messages, for LCD). The message length may vary from a couple
up to 32 bytes, but I always allocate in multiples of 16 bytes.

Best regards,

Isaac

__________________________________________________
Faça ligações para outros computadores com o novo Yahoo! Messenger
http://br.beta.messenger.yahoo.com/ 
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Harold Hallikainen-2]

I've done some applications where I need to allocate buffers, the number
of which depend on user configuration of the system. I know I have so much
RAM available, but do not know how many buffers will be needed. I'd like
to make the buffers as large as possible. So, I allocate them during
initialization at run time and never discard them. Seems pretty safe...

Harold



--
FCC Rules Updated Daily at http://www.hallikainen.com - Advertising
opportunities available!
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Vitaliy-14]

Olin Lathrop wrote:
> Why do you need to dynamically allocate memory anyway?  This is rather
> unusual in a small embedded system, and likely indicates big system
> thinking
> on a small system.

FWIW, we use dynamic memory allocation in all of our current projects (on
24H and 33F). In fact, some of the things we use it for, would be impossible
to implement in any other way.


> Another drawback of dynamic memory on a small system is how to guarantee
> you
> won't run out.  If you don't know how much memory you need up front, then
> how do you know you don't need more than there is available under the
> right
> set of input conditions.  And if you do use malloc or the like, what are
> you
> going to do when it fails?  Punting back to the operating system with a
> non-zero exit status doesn't work here.

In our case, we just tell the user that there's no more RAM (because if it
happens, they're the ones who added too many things to begin with).

When there's no user input, there's no risk of running out of RAM. Memory
gets allocated during initialization and doesn't change.

But, yeah -- stay away from malloc() if you can help it. :)

Vitaliy

--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Olin Lathrop]

Vitaliy wrote:
> FWIW, we use dynamic memory allocation in all of our current projects
> (on 24H and 33F).
>
> ...
>
> Memory gets allocated during initialization and doesn't change.

These two statements seem to be at odds with each other.


********************************************************************
Embed Inc, Littleton Massachusetts, http://www.embedinc.com/products
[phone].  Gold level PIC consultants since 2000.
--
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist