[rvm-core] Renaming/aligning methods in org.mmtk.vm.Memory with Harmony?

[Ian Rogers (nabble)]

Hi,

I'm interested in the interface to underlying OS operations used by MMTk:

dzmmap - demand zero mmap
mprotect - remove read/write protections to a region of memory
munprotect - allow read/write access to a region of memory

the equivalent Harmony interfaces are:

reserve - reserve a region of virtual memory for use
commit - allow access in a region previously reserved
uncommit - remove access to a region

for completeness there is unreserve (but MMTk has no munmap).

MMTk's dzmmap is equivalent to a reserve & commit, mprotect to uncommit, munprotect to commit. The Harmony interface more directly maps to that of Windows. It would make sense to me to push the Harmony naming convention through the RVM interface into MMTk's. I will provide a patch if other people agree.

Ian

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

[Ian Rogers (nabble)]

2009/3/1 Ian Rogers <ian.rogers@...>

Hi,

I'm interested in the interface to underlying OS operations used by MMTk:

dzmmap - demand zero mmap
mprotect - remove read/write protections to a region of memory
munprotect - allow read/write access to a region of memory

the equivalent Harmony interfaces are:

reserve - reserve a region of virtual memory for use
commit - allow access in a region previously reserved
uncommit - remove access to a region

for uncommit read decommit
 

for completeness there is unreserve (but MMTk has no munmap).

MMTk's dzmmap is equivalent to a reserve & commit, mprotect to uncommit, munprotect to commit. The Harmony interface more directly maps to that of Windows. It would make sense to me to push the Harmony naming convention through the RVM interface into MMTk's. I will provide a patch if other people agree.

Ian

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

[Eliot Moss]

Ting Yang's work further suggested the following:

- If running benchmarks more or less standalone, with a known amount of
demand for real memory, etc., it does not make much sense to munmap,
given the overhead of remapping later.

- If running in real life situations under varying load, and using
something like Ting's heap size adjustment, you get better performance
overall if you restrict your usage to the real memory available,
which may sometimes decrease. When this happens, it may be a good idea
to unmap those pages you don't care about, so the OS won't be as
inclined to page out the ones you *do* care about.

In sum, unmapping does not speed up *benchmarks*, so it was never
really explored much and supported. But *real life* probably wants
it there, and algorithms that will actually invoke it. However,
unmapping after each gc remains a bad idea -- one should unmap
when one is reducing the demand for real memory ....

Bottom line: MMTk *should* support it, even if it does not do so
now.

Contrary opinions from down under?

Best wishes -- Eliot

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

[Daniel Frampton]

I think this is just a question of priorities. Any change like this we
make obviously introduces the possibility of performance and stability
problems. As Eliot says, I think it makes sense to tie freeing virtual
memory to the heap shrinking logic. However, heap growth/shrinking in
general is not something that has got a huge amount of attention in
MMTk. When comparing individual collectors we usually take it out of
the picture entirely by evaluating across a range of fixed heap sizes.

Of course for research in other areas this is not the case, and
contributions of any improvements to our heap resizing logic could
both improve the default behavior of Jikes RVM, as well as assist
future research.

On the separate question of changing the names for the virtual memory
functions, I really don't think it is a good idea. The operations we
have are fairly simple and I would argue quite obvious from the names
(or at least javadoc).

I think you have also misunderstood what the protection methods do
anyway. We want to be able to actually control memory protection
(read/write/execute) to both support collectors and help debugging.
Currently we support protecting and then unprotecting memory. This is
different to decommitting memory, which if I understand correctly
means the physical storage is thrown away.

Under windows... I think we would want:

dzmmap -> VirtualAlloc (reserve + commit)
mprotect -> VirtualProtect
munprotect -> VirtualProtect
munmap -> VirtualFree (decommit + unreserve)

Under harmony, I don't know what the equivalent calls are. Maybe
harmony does not provide the same level of access to memory
protection.

Cheers,
Daniel.

On Mon, Mar 2, 2009 at 9:27 AM, Eliot Moss <moss@...> wrote:

> Ting Yang's work further suggested the following:
>
> - If running benchmarks more or less standalone, with a known amount of
> demand for real memory, etc., it does not make much sense to munmap,
> given the overhead of remapping later.
>
> - If running in real life situations under varying load, and using
> something like Ting's heap size adjustment, you get better performance
> overall if you restrict your usage to the real memory available,
> which may sometimes decrease. When this happens, it may be a good idea
> to unmap those pages you don't care about, so the OS won't be as
> inclined to page out the ones you *do* care about.
>
> In sum, unmapping does not speed up *benchmarks*, so it was never
> really explored much and supported. But *real life* probably wants
> it there, and algorithms that will actually invoke it. However,
> unmapping after each gc remains a bad idea -- one should unmap
> when one is reducing the demand for real memory ....
>
> Bottom line: MMTk *should* support it, even if it does not do so
> now.
>
> Contrary opinions from down under?
>
> Best wishes -- Eliot
>
> ------------------------------------------------------------------------------
> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
> -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
> -Strategies to boost innovation and cut costs with open source participation
> -Receive a $600 discount off the registration fee with the source code: SFAD
> http://p.sf.net/sfu/XcvMzF8H
> _______________________________________________
> Jikesrvm-core mailing list
> Jikesrvm-core@...
> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core
>

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

[Ian Rogers (nabble)]

2009/3/2 Daniel Frampton <zyridium@...>

I think this is just a question of priorities. Any change like this we
make obviously introduces the possibility of performance and stability
problems. As Eliot says, I think it makes sense to tie freeing virtual
memory to the heap shrinking logic. However, heap growth/shrinking in
general is not something that has got a huge amount of attention in
MMTk. When comparing individual collectors we usually take it out of
the picture entirely by evaluating across a range of fixed heap sizes.

Of course for research in other areas this is not the case, and
contributions of any improvements to our heap resizing logic could
both improve the default behavior of Jikes RVM, as well as assist
future research.

On the separate question of changing the names for the virtual memory
functions, I really don't think it is a good idea. The operations we
have are fairly simple and I would argue quite obvious from the names
(or at least javadoc).

I think you have also misunderstood what the protection methods do
anyway. We want to be able to actually control memory protection
(read/write/execute) to both support collectors and help debugging.
Currently we support protecting and then unprotecting memory. This is
different to decommitting memory, which if I understand correctly
means the physical storage is thrown away.

Under windows... I think we would want:

dzmmap -> VirtualAlloc (reserve + commit)
mprotect -> VirtualProtect
munprotect -> VirtualProtect
munmap -> VirtualFree (decommit + unreserve)

Under harmony, I don't know what the equivalent calls are. Maybe
harmony does not provide the same level of access to memory
protection.

My understanding is Harmony's names are designed to align with Windows, so you commit or decommit memory and specify protection at the same time. When you reserve memory you can specify protection and you can also specify that memory should be committed when it is reserved. If you wish to change the protection on committed memory you just commit it again with different protection.

I think the problems with your proposal are that:
- there is no way to reserve but not commit memory, which would be a good way to stop the problem we get when shared libraries are loaded into memory that MMTk thinks it controls.
- it makes us have to write the OS interface and breaks the advantage of the Harmony portability library (portlib).

Ian

Cheers,
Daniel.

On Mon, Mar 2, 2009 at 9:27 AM, Eliot Moss <moss@...> wrote:
> Ting Yang's work further suggested the following:
>
> - If running benchmarks more or less standalone, with a known amount of
> demand for real memory, etc., it does not make much sense to munmap,
> given the overhead of remapping later.
>
> - If running in real life situations under varying load, and using
> something like Ting's heap size adjustment, you get better performance
> overall if you restrict your usage to the real memory available,
> which may sometimes decrease. When this happens, it may be a good idea
> to unmap those pages you don't care about, so the OS won't be as
> inclined to page out the ones you *do* care about.
>
> In sum, unmapping does not speed up *benchmarks*, so it was never
> really explored much and supported. But *real life* probably wants
> it there, and algorithms that will actually invoke it. However,
> unmapping after each gc remains a bad idea -- one should unmap
> when one is reducing the demand for real memory ....
>
> Bottom line: MMTk *should* support it, even if it does not do so
> now.
>
> Contrary opinions from down under?
>
> Best wishes -- Eliot
>
> ------------------------------------------------------------------------------
> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
> -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
> -Strategies to boost innovation and cut costs with open source participation
> -Receive a $600 discount off the registration fee with the source code: SFAD
> http://p.sf.net/sfu/XcvMzF8H
> _______________________________________________
> Jikesrvm-core mailing list
> Jikesrvm-core@...
> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core
>

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

[Daniel Frampton]

My windows comment was about logically what methods could be called,
not a suggestion to code to it directly.

it seems to me that it is trivial to implement the primitives we
currently require in terms of AIX, linux, harmony, or windows
directly.

Cheers,
Daniel.

On Mon, Mar 2, 2009 at 11:22 AM, Ian Rogers <ian.rogers@...> wrote:

> 2009/3/2 Daniel Frampton <zyridium@...>
>>
>> I think this is just a question of priorities. Any change like this we
>> make obviously introduces the possibility of performance and stability
>> problems. As Eliot says, I think it makes sense to tie freeing virtual
>> memory to the heap shrinking logic. However, heap growth/shrinking in
>> general is not something that has got a huge amount of attention in
>> MMTk. When comparing individual collectors we usually take it out of
>> the picture entirely by evaluating across a range of fixed heap sizes.
>>
>> Of course for research in other areas this is not the case, and
>> contributions of any improvements to our heap resizing logic could
>> both improve the default behavior of Jikes RVM, as well as assist
>> future research.
>>
>> On the separate question of changing the names for the virtual memory
>> functions, I really don't think it is a good idea. The operations we
>> have are fairly simple and I would argue quite obvious from the names
>> (or at least javadoc).
>>
>> I think you have also misunderstood what the protection methods do
>> anyway. We want to be able to actually control memory protection
>> (read/write/execute) to both support collectors and help debugging.
>> Currently we support protecting and then unprotecting memory. This is
>> different to decommitting memory, which if I understand correctly
>> means the physical storage is thrown away.
>>
>> Under windows... I think we would want:
>>
>> dzmmap -> VirtualAlloc (reserve + commit)
>> mprotect -> VirtualProtect
>> munprotect -> VirtualProtect
>> munmap -> VirtualFree (decommit + unreserve)
>>
>> Under harmony, I don't know what the equivalent calls are. Maybe
>> harmony does not provide the same level of access to memory
>> protection.
>
> My understanding is Harmony's names are designed to align with Windows, so
> you commit or decommit memory and specify protection at the same time. When
> you reserve memory you can specify protection and you can also specify that
> memory should be committed when it is reserved. If you wish to change the
> protection on committed memory you just commit it again with different
> protection.
>
> I think the problems with your proposal are that:
> - there is no way to reserve but not commit memory, which would be a good
> way to stop the problem we get when shared libraries are loaded into memory
> that MMTk thinks it controls.
> - it makes us have to write the OS interface and breaks the advantage of the
> Harmony portability library (portlib).
>
> Ian
>
>> Cheers,
>> Daniel.
>>
>> On Mon, Mar 2, 2009 at 9:27 AM, Eliot Moss <moss@...> wrote:
>> > Ting Yang's work further suggested the following:
>> >
>> > - If running benchmarks more or less standalone, with a known amount of
>> > demand for real memory, etc., it does not make much sense to munmap,
>> > given the overhead of remapping later.
>> >
>> > - If running in real life situations under varying load, and using
>> > something like Ting's heap size adjustment, you get better performance
>> > overall if you restrict your usage to the real memory available,
>> > which may sometimes decrease. When this happens, it may be a good idea
>> > to unmap those pages you don't care about, so the OS won't be as
>> > inclined to page out the ones you *do* care about.
>> >
>> > In sum, unmapping does not speed up *benchmarks*, so it was never
>> > really explored much and supported. But *real life* probably wants
>> > it there, and algorithms that will actually invoke it. However,
>> > unmapping after each gc remains a bad idea -- one should unmap
>> > when one is reducing the demand for real memory ....
>> >
>> > Bottom line: MMTk *should* support it, even if it does not do so
>> > now.
>> >
>> > Contrary opinions from down under?
>> >
>> > Best wishes -- Eliot
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > Open Source Business Conference (OSBC), March 24-25, 2009, San
>> > Francisco, CA
>> > -OSBC tackles the biggest issue in open source: Open Sourcing the
>> > Enterprise
>> > -Strategies to boost innovation and cut costs with open source
>> > participation
>> > -Receive a $600 discount off the registration fee with the source code:
>> > SFAD
>> > http://p.sf.net/sfu/XcvMzF8H
>> > _______________________________________________
>> > Jikesrvm-core mailing list
>> > Jikesrvm-core@...
>> > https://lists.sourceforge.net/lists/listinfo/jikesrvm-core
>> >
>>
>>
>> ------------------------------------------------------------------------------
>> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco,
>> CA
>> -OSBC tackles the biggest issue in open source: Open Sourcing the
>> Enterprise
>> -Strategies to boost innovation and cut costs with open source
>> participation
>> -Receive a $600 discount off the registration fee with the source code:
>> SFAD
>> http://p.sf.net/sfu/XcvMzF8H
>> _______________________________________________
>> Jikesrvm-core mailing list
>> Jikesrvm-core@...
>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core
>
>
> ------------------------------------------------------------------------------
> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
> -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
> -Strategies to boost innovation and cut costs with open source participation
> -Receive a $600 discount off the registration fee with the source code: SFAD
> http://p.sf.net/sfu/XcvMzF8H
> _______________________________________________
> Jikesrvm-core mailing list
> Jikesrvm-core@...
> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core
>
>

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

[Tony Hosking]

Hi Ian,

This all seems like window-dressing (pardon the pun) to me.  What difference does it make what the operations are called?  Every OS will require some sort of adaptation layer to meet the needs of MMTk.  To my Unix-blinkered mind the current naming scheme seems intuitive.  Or perhaps you are trying to say that there is a deeper problem of mapping the MMTk abstractions to the underlying OS (in other words, I may be missing the point).

Antony Hosking | Associate Professor | Computer Science | Purdue University

305 N. University Street | West Lafayette | IN 47907 | USA

Office [phone] | Mobile [phone]

On 2 Mar 2009, at 08:17, Ian Rogers wrote:

Hi,

I'm interested in the interface to underlying OS operations used by MMTk:

dzmmap - demand zero mmap
mprotect - remove read/write protections to a region of memory
munprotect - allow read/write access to a region of memory

the equivalent Harmony interfaces are:

reserve - reserve a region of virtual memory for use
commit - allow access in a region previously reserved
uncommit - remove access to a region

for completeness there is unreserve (but MMTk has no munmap).

MMTk's dzmmap is equivalent to a reserve & commit, mprotect to uncommit, munprotect to commit. The Harmony interface more directly maps to that of Windows. It would make sense to me to push the Harmony naming convention through the RVM interface into MMTk's. I will provide a patch if other people agree.

Ian
------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

[Robin Garner]

My feeling is that the names used in the MMTk interface should most
closely align with the effect that MMTk expects, rather than the
implementation on any given platform.  I'm not opposed to changing the
operation names, but I don't see that reserve/commit/uncommit expresses
the intended effects any better than dzmmap/mprotect/munprotect.

As you say, there is a simple 1:n mapping between the MMTk interface and
the Harmony abstractions, so I don't see a good reason to change MMTk at
this stage.

Regards,
Robin

Ian Rogers wrote:

> Hi,
>
> I'm interested in the interface to underlying OS operations used by MMTk:
>
> dzmmap - demand zero mmap
> mprotect - remove read/write protections to a region of memory
> munprotect - allow read/write access to a region of memory
>
> the equivalent Harmony interfaces are:
>
> reserve - reserve a region of virtual memory for use
> commit - allow access in a region previously reserved
> uncommit - remove access to a region
>
> for completeness there is unreserve (but MMTk has no munmap).
>
> MMTk's dzmmap is equivalent to a reserve & commit, mprotect to
> uncommit, munprotect to commit. The Harmony interface more directly
> maps to that of Windows. It would make sense to me to push the Harmony
> naming convention through the RVM interface into MMTk's. I will
> provide a patch if other people agree.
>
> Ian
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
> -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
> -Strategies to boost innovation and cut costs with open source participation
> -Receive a $600 discount off the registration fee with the source code: SFAD
> http://p.sf.net/sfu/XcvMzF8H
> ------------------------------------------------------------------------
>
> _______________________________________________
> Jikesrvm-core mailing list
> Jikesrvm-core@...
> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core
>  

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

[Filip Pizlo-2]

I agree with Daniel.

I would advise against changing all of our internal VM APIs to follow  
Harmony conventions.  If our APIs aren't portable, then we should  
change them, but as far as I can tell, they already are portable, so  
we can leave them as-is.

More crucially, changing our already proven internal VM APIs, with  
names and semantics we (and our users!) already understand, to use  
names and/or semantics of a foreign library seems like a sure way of  
generating bugs in the long run.

Bottom line: we can support Harmony, on Linux, AIX, and Windows  
without having to move to Harmony conventions.  I've yet to see an  
argument to the contrary.

-F



On Mar 1, 2009, at 8:22 PM, Daniel Frampton wrote:

> My windows comment was about logically what methods could be called,
> not a suggestion to code to it directly.
>
> it seems to me that it is trivial to implement the primitives we
> currently require in terms of AIX, linux, harmony, or windows
> directly.
>
> Cheers,
> Daniel.
>
> On Mon, Mar 2, 2009 at 11:22 AM, Ian Rogers <ian.rogers@...
> > wrote:
>> 2009/3/2 Daniel Frampton <zyridium@...>
>>>
>>> I think this is just a question of priorities. Any change like  
>>> this we
>>> make obviously introduces the possibility of performance and  
>>> stability
>>> problems. As Eliot says, I think it makes sense to tie freeing  
>>> virtual
>>> memory to the heap shrinking logic. However, heap growth/shrinking  
>>> in
>>> general is not something that has got a huge amount of attention in
>>> MMTk. When comparing individual collectors we usually take it out of
>>> the picture entirely by evaluating across a range of fixed heap  
>>> sizes.
>>>
>>> Of course for research in other areas this is not the case, and
>>> contributions of any improvements to our heap resizing logic could
>>> both improve the default behavior of Jikes RVM, as well as assist
>>> future research.
>>>
>>> On the separate question of changing the names for the virtual  
>>> memory
>>> functions, I really don't think it is a good idea. The operations we
>>> have are fairly simple and I would argue quite obvious from the  
>>> names
>>> (or at least javadoc).
>>>
>>> I think you have also misunderstood what the protection methods do
>>> anyway. We want to be able to actually control memory protection
>>> (read/write/execute) to both support collectors and help debugging.
>>> Currently we support protecting and then unprotecting memory. This  
>>> is
>>> different to decommitting memory, which if I understand correctly
>>> means the physical storage is thrown away.
>>>
>>> Under windows... I think we would want:
>>>
>>> dzmmap -> VirtualAlloc (reserve + commit)
>>> mprotect -> VirtualProtect
>>> munprotect -> VirtualProtect
>>> munmap -> VirtualFree (decommit + unreserve)
>>>
>>> Under harmony, I don't know what the equivalent calls are. Maybe
>>> harmony does not provide the same level of access to memory
>>> protection.
>>
>> My understanding is Harmony's names are designed to align with  
>> Windows, so
>> you commit or decommit memory and specify protection at the same  
>> time. When
>> you reserve memory you can specify protection and you can also  
>> specify that
>> memory should be committed when it is reserved. If you wish to  
>> change the
>> protection on committed memory you just commit it again with  
>> different
>> protection.
>>
>> I think the problems with your proposal are that:
>> - there is no way to reserve but not commit memory, which would be  
>> a good
>> way to stop the problem we get when shared libraries are loaded  
>> into memory
>> that MMTk thinks it controls.
>> - it makes us have to write the OS interface and breaks the  
>> advantage of the
>> Harmony portability library (portlib).
>>
>> Ian
>>
>>> Cheers,
>>> Daniel.
>>>
>>> On Mon, Mar 2, 2009 at 9:27 AM, Eliot Moss <moss@...>  
>>> wrote:
>>>> Ting Yang's work further suggested the following:
>>>>
>>>> - If running benchmarks more or less standalone, with a known  
>>>> amount of
>>>> demand for real memory, etc., it does not make much sense to  
>>>> munmap,
>>>> given the overhead of remapping later.
>>>>
>>>> - If running in real life situations under varying load, and using
>>>> something like Ting's heap size adjustment, you get better  
>>>> performance
>>>> overall if you restrict your usage to the real memory available,
>>>> which may sometimes decrease. When this happens, it may be a good  
>>>> idea
>>>> to unmap those pages you don't care about, so the OS won't be as
>>>> inclined to page out the ones you *do* care about.
>>>>
>>>> In sum, unmapping does not speed up *benchmarks*, so it was never
>>>> really explored much and supported. But *real life* probably wants
>>>> it there, and algorithms that will actually invoke it. However,
>>>> unmapping after each gc remains a bad idea -- one should unmap
>>>> when one is reducing the demand for real memory ....
>>>>
>>>> Bottom line: MMTk *should* support it, even if it does not do so
>>>> now.
>>>>
>>>> Contrary opinions from down under?
>>>>
>>>> Best wishes -- Eliot
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Open Source Business Conference (OSBC), March 24-25, 2009, San
>>>> Francisco, CA
>>>> -OSBC tackles the biggest issue in open source: Open Sourcing the
>>>> Enterprise
>>>> -Strategies to boost innovation and cut costs with open source
>>>> participation
>>>> -Receive a $600 discount off the registration fee with the source  
>>>> code:
>>>> SFAD
>>>> http://p.sf.net/sfu/XcvMzF8H
>>>> _______________________________________________
>>>> Jikesrvm-core mailing list
>>>> Jikesrvm-core@...
>>>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core
>>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Open Source Business Conference (OSBC), March 24-25, 2009, San  
>>> Francisco,
>>> CA
>>> -OSBC tackles the biggest issue in open source: Open Sourcing the
>>> Enterprise
>>> -Strategies to boost innovation and cut costs with open source
>>> participation
>>> -Receive a $600 discount off the registration fee with the source  
>>> code:
>>> SFAD
>>> http://p.sf.net/sfu/XcvMzF8H
>>> _______________________________________________
>>> Jikesrvm-core mailing list
>>> Jikesrvm-core@...
>>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core
>>
>>
>> ------------------------------------------------------------------------------
>> Open Source Business Conference (OSBC), March 24-25, 2009, San  
>> Francisco, CA
>> -OSBC tackles the biggest issue in open source: Open Sourcing the  
>> Enterprise
>> -Strategies to boost innovation and cut costs with open source  
>> participation
>> -Receive a $600 discount off the registration fee with the source  
>> code: SFAD
>> http://p.sf.net/sfu/XcvMzF8H
>> _______________________________________________
>> Jikesrvm-core mailing list
>> Jikesrvm-core@...
>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core
>>
>>
>
> ------------------------------------------------------------------------------
> Open Source Business Conference (OSBC), March 24-25, 2009, San  
> Francisco, CA
> -OSBC tackles the biggest issue in open source: Open Sourcing the  
> Enterprise
> -Strategies to boost innovation and cut costs with open source  
> participation
> -Receive a $600 discount off the registration fee with the source  
> code: SFAD
> http://p.sf.net/sfu/XcvMzF8H
> _______________________________________________
> Jikesrvm-core mailing list
> Jikesrvm-core@...
> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

[Ian Rogers (nabble)]

2009/3/2 Tony Hosking <hosking@...>

Hi Ian,

This all seems like window-dressing (pardon the pun) to me.  What difference does it make what the operations are called?  Every OS will require some sort of adaptation layer to meet the needs of MMTk.  To my Unix-blinkered mind the current naming scheme seems intuitive.  Or perhaps you are trying to say that there is a deeper problem of mapping the MMTk abstractions to the underlying OS (in other words, I may be missing the point).

Antony Hosking | Associate Professor | Computer Science | Purdue University

305 N. University Street | West Lafayette | IN 47907 | USA

Office [phone] | Mobile [phone]

Where Harmony goes beyond our current abstractions is that it is much more rigorous. For example, memory pools are used to manage malloc/free. By following their portability library we get the ability to be a 1st class VM wrt integrating with applet viewers, launchers, etc. Currently Jikes RVM doesn't look like a regular Java VM, implemented through a jvm.dll, etc. This is all logged through the trackers and is a long standing problem.

The reserve, commit, decommit, free strategy is fundamentally different to the current mmap, mprotect, munmap strategy. With the mmap strategy we don't have the ability to mark memory as being reserved but not yet in use. Without being able to reserve memory we are not a good citizen with the OS and we frequently get libraries and the like mapped into the MMTk managed address space. The consequence of this is that MMTk can only support heaps smaller than the address space and Jikes RVM breaks when a Linux vendor changes the memory layout (we had a spate of these during the 2.4 releases).

Whilst it is possible to emulate the Harmony behaviour partially on the Linux calls (the notion of commit/decommit is particular to Windows and decommit is a nop on Linux) imo you end up with an ugly API, especially when you need to put it on top of Windows. Harmony have solved this problem already with their portability library and this is being used as the basis for IBM J9 (via IBM Hursley) and Intel DRLVM development work (which both include concurrent garbage collectors, etc.).

By effectively having system calls as our own portability library we are always having to do leg work. We had to work constantly, me in particular (and Steven Augurt before me), doing leg work to keep Classpath working with the RVM as we were a special case (see syswrap and portable native sync in particular). If we can be a special case as little as possible it is inevitable that the development effort will be less.

Ian
 

On 2 Mar 2009, at 08:17, Ian Rogers wrote:

Hi,

I'm interested in the interface to underlying OS operations used by MMTk:

dzmmap - demand zero mmap
mprotect - remove read/write protections to a region of memory
munprotect - allow read/write access to a region of memory

the equivalent Harmony interfaces are:

reserve - reserve a region of virtual memory for use
commit - allow access in a region previously reserved
uncommit - remove access to a region

for completeness there is unreserve (but MMTk has no munmap).

MMTk's dzmmap is equivalent to a reserve & commit, mprotect to uncommit, munprotect to commit. The Harmony interface more directly maps to that of Windows. It would make sense to me to push the Harmony naming convention through the RVM interface into MMTk's. I will provide a patch if other people agree.

Ian

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

[Steve Blackburn]

Ian,

The org.mmtk.vm.* API is (by definition) an abstraction that defines the MMTk *requirements*, which are explicitly intended to be OS and VM neutral and express nothing other than the current requirements of MMTk.  The API should have nothing to do with Harmony.

When MMTk changes its requirements, then we can change the API that defines those requirements.  I can foresee this happening (of course), but it is certainly not on our critical todo list right now, so at this time I see no call for changing those particular APIs.

If there is something you'd like changed in MMTk, by all means create a JIRA.  This allows the issue to be noted and suitably prioritized.

--Steve

On 02/03/2009, at 7:30 PM, Ian Rogers wrote:

2009/3/2 Tony Hosking <hosking@...>

Hi Ian,

This all seems like window-dressing (pardon the pun) to me.  What difference does it make what the operations are called?  Every OS will require some sort of adaptation layer to meet the needs of MMTk.  To my Unix-blinkered mind the current naming scheme seems intuitive.  Or perhaps you are trying to say that there is a deeper problem of mapping the MMTk abstractions to the underlying OS (in other words, I may be missing the point).

Antony Hosking | Associate Professor | Computer Science | Purdue University

305 N. University Street | West Lafayette | IN 47907 | USA

Office [phone] | Mobile [phone]

Where Harmony goes beyond our current abstractions is that it is much more rigorous. For example, memory pools are used to manage malloc/free. By following their portability library we get the ability to be a 1st class VM wrt integrating with applet viewers, launchers, etc. Currently Jikes RVM doesn't look like a regular Java VM, implemented through a jvm.dll, etc. This is all logged through the trackers and is a long standing problem.

The reserve, commit, decommit, free strategy is fundamentally different to the current mmap, mprotect, munmap strategy. With the mmap strategy we don't have the ability to mark memory as being reserved but not yet in use. Without being able to reserve memory we are not a good citizen with the OS and we frequently get libraries and the like mapped into the MMTk managed address space. The consequence of this is that MMTk can only support heaps smaller than the address space and Jikes RVM breaks when a Linux vendor changes the memory layout (we had a spate of these during the 2.4 releases).

Whilst it is possible to emulate the Harmony behaviour partially on the Linux calls (the notion of commit/decommit is particular to Windows and decommit is a nop on Linux) imo you end up with an ugly API, especially when you need to put it on top of Windows. Harmony have solved this problem already with their portability library and this is being used as the basis for IBM J9 (via IBM Hursley) and Intel DRLVM development work (which both include concurrent garbage collectors, etc.).

By effectively having system calls as our own portability library we are always having to do leg work. We had to work constantly, me in particular (and Steven Augurt before me), doing leg work to keep Classpath working with the RVM as we were a special case (see syswrap and portable native sync in particular). If we can be a special case as little as possible it is inevitable that the development effort will be less.

Ian
 

On 2 Mar 2009, at 08:17, Ian Rogers wrote:

Hi,

I'm interested in the interface to underlying OS operations used by MMTk:

dzmmap - demand zero mmap
mprotect - remove read/write protections to a region of memory
munprotect - allow read/write access to a region of memory

the equivalent Harmony interfaces are:

reserve - reserve a region of virtual memory for use
commit - allow access in a region previously reserved
uncommit - remove access to a region

for completeness there is unreserve (but MMTk has no munmap).

MMTk's dzmmap is equivalent to a reserve & commit, mprotect to uncommit, munprotect to commit. The Harmony interface more directly maps to that of Windows. It would make sense to me to push the Harmony naming convention through the RVM interface into MMTk's. I will provide a patch if other people agree.

Ian

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

[Ian Rogers (nabble)]

2009/3/2 Steve Blackburn <Steve.Blackburn@...>

Ian,

The org.mmtk.vm.* API is (by definition) an abstraction that defines the MMTk *requirements*, which are explicitly intended to be OS and VM neutral and express nothing other than the current requirements of MMTk.  The API should have nothing to do with Harmony.

When MMTk changes its requirements, then we can change the API that defines those requirements.  I can foresee this happening (of course), but it is certainly not on our critical todo list right now, so at this time I see no call for changing those particular APIs.

If there is something you'd like changed in MMTk, by all means create a JIRA.  This allows the issue to be noted and suitably prioritized.

--Steve

Moved to an MMTk bug in RVM-799:
http://jira.codehaus.org/browse/RVM-799

Ian
 

On 02/03/2009, at 7:30 PM, Ian Rogers wrote:

2009/3/2 Tony Hosking <hosking@...>

Hi Ian,

This all seems like window-dressing (pardon the pun) to me.  What difference does it make what the operations are called?  Every OS will require some sort of adaptation layer to meet the needs of MMTk.  To my Unix-blinkered mind the current naming scheme seems intuitive.  Or perhaps you are trying to say that there is a deeper problem of mapping the MMTk abstractions to the underlying OS (in other words, I may be missing the point).

Antony Hosking | Associate Professor | Computer Science | Purdue University

305 N. University Street | West Lafayette | IN 47907 | USA

Office [phone] | Mobile [phone]

Where Harmony goes beyond our current abstractions is that it is much more rigorous. For example, memory pools are used to manage malloc/free. By following their portability library we get the ability to be a 1st class VM wrt integrating with applet viewers, launchers, etc. Currently Jikes RVM doesn't look like a regular Java VM, implemented through a jvm.dll, etc. This is all logged through the trackers and is a long standing problem.

The reserve, commit, decommit, free strategy is fundamentally different to the current mmap, mprotect, munmap strategy. With the mmap strategy we don't have the ability to mark memory as being reserved but not yet in use. Without being able to reserve memory we are not a good citizen with the OS and we frequently get libraries and the like mapped into the MMTk managed address space. The consequence of this is that MMTk can only support heaps smaller than the address space and Jikes RVM breaks when a Linux vendor changes the memory layout (we had a spate of these during the 2.4 releases).

Whilst it is possible to emulate the Harmony behaviour partially on the Linux calls (the notion of commit/decommit is particular to Windows and decommit is a nop on Linux) imo you end up with an ugly API, especially when you need to put it on top of Windows. Harmony have solved this problem already with their portability library and this is being used as the basis for IBM J9 (via IBM Hursley) and Intel DRLVM development work (which both include concurrent garbage collectors, etc.).

By effectively having system calls as our own portability library we are always having to do leg work. We had to work constantly, me in particular (and Steven Augurt before me), doing leg work to keep Classpath working with the RVM as we were a special case (see syswrap and portable native sync in particular). If we can be a special case as little as possible it is inevitable that the development effort will be less.

Ian
 

On 2 Mar 2009, at 08:17, Ian Rogers wrote:

Hi,

I'm interested in the interface to underlying OS operations used by MMTk:

dzmmap - demand zero mmap
mprotect - remove read/write protections to a region of memory
munprotect - allow read/write access to a region of memory

the equivalent Harmony interfaces are:

reserve - reserve a region of virtual memory for use
commit - allow access in a region previously reserved
uncommit - remove access to a region

for completeness there is unreserve (but MMTk has no munmap).

MMTk's dzmmap is equivalent to a reserve & commit, mprotect to uncommit, munprotect to commit. The Harmony interface more directly maps to that of Windows. It would make sense to me to push the Harmony naming convention through the RVM interface into MMTk's. I will provide a patch if other people agree.

Ian

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core