[RFC PATCH 0/4] (Take 2): transcendent memory ("tmem") for Linux

[Dan Magenheimer-2]

> > That 63GB requires no page structs or other data structures in the
> > guest.  And in the current (external) implementation, the size
> > of each pool is constantly changing, sometimes dramatically so
> > the guest would have to be prepared to handle this.  I also wonder
> > if this would make shared-tmem-pools more difficult.  
>
> Having no struct pages is also a downside; for example this
> guest cannot
> have more than 1GB of anonymous memory without swapping like mad.  
> Swapping to tmem is fast but still a lot slower than having
> the memory
> available.

Yes, true.  Tmem offers little additional advantage for workloads
that have a huge variation in working set size that is primarily
anonymous memory.  That larger scale "memory shaping" is left to
ballooning and hotplug.

> tmem makes life a lot easier to the hypervisor and to the guest, but
> also gives up a lot of flexibility.  There's a difference
> between memory
> and a very fast synchronous backing store.

I don't see that it gives up that flexibility.  System adminstrators
are still free to size their guests properly.  Tmem's contribution
is in environments that are highly dynamic, where the only
alternative is really sizing memory maximally (and thus wasting
it for the vast majority of time in which the working set is smaller).

> > I can see how it might be useful for KVM though.  Once the
> > core API and all the hooks are in place, a KVM implementation of
> > tmem could attempt something like this.
> >    
>
> My worry is that tmem for kvm leaves a lot of niftiness on the table,
> since it was designed for a hypervisor with much simpler memory
> management.  kvm can already use spare memory for backing guest swap,
> and can already convert unused guest memory to free memory
> (by swapping
> it).  tmem doesn't really integrate well with these capabilities.

I'm certainly open to identifying compromises and layer modifications
that help meet the needs of both Xen and KVM (and others).  For
example, if we can determine that the basic hook placement for
precache/preswap (or even just precache for KVM) can be built
on different underlying layers, that would be great!

Dan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Avi Kivity-4]

On 07/12/2009 07:20 PM, Dan Magenheimer wrote:

>>> that information; but tmem is trying to go a step further by making
>>> the cooperation between the OS and hypervisor more explicit
>>> and directly beneficial to the OS.
>>>        
>> KVM definitely falls into the camp of trying to minimize
>> modification to the guest.
>>      
>
> No argument there.  Well, maybe one :-) Yes, but KVM
> also heavily encourages unmodified guests.  Tmem is
> philosophically in favor of finding a balance between
> things that work well with no changes to any OS (and
> thus work just fine regardless of whether the OS is
> running in a virtual environment or not), and things
> that could work better if the OS is knowledgable that
> it is running in a virtual environment.
>    

CMM2 and tmem are not any different in this regard; both require OS
modification, and both make information available to the hypervisor.  In
fact CMM2 is much more intrusive (but on the other hand provides much
more information).

> For those that believe virtualization is a flash-in-
> the-pan, no modifications to the OS is the right answer.
> For those that believe it will be pervasive in the
> future, finding the right balance is a critical step
> in operating system evolution.
>    

You're arguing for CMM2 here IMO.

> Is it the tmem API or the precache/preswap API layered on
> top of it that is problematic?  Both currently assume copying
> but perhaps the precache/preswap API could, with minor
> modifications, meet KVM's needs better?
>
>    

My take on this is that precache (predecache?) / preswap can be
implemented even without tmem by using write-through backing for the
virtual disk.  For swap this is actually slight;y more efficient than
tmem preswap, for preuncache slightly less efficient (since there will
be some double caching).  So I'm more interested in other use cases of
tmem/CMM2.Well, first, tmem's very name means memory that is "beyond the
> range of normal perception".  This is certainly not the first class
> of memory in use in data centers that can't be accounted at
> process granularity.  I'm thinking disk array caches as the
> primary example.  Also lots of tools that work great in a
> non-virtualized OS are worthless or misleading in a virtual
> environment.
>
>    

Right, the transient uses of tmem when applied to disk objects
(swap/pagecache) are very similar to disk caches.  Which is why you can
get a very similar effect when caching your virtual disks; this can be
done without any guest modification.


--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Avi Kivity-4]

On 07/12/2009 07:28 PM, Dan Magenheimer wrote:

>> Having no struct pages is also a downside; for example this
>> guest cannot
>> have more than 1GB of anonymous memory without swapping like mad.
>> Swapping to tmem is fast but still a lot slower than having
>> the memory
>> available.
>>      
>
> Yes, true.  Tmem offers little additional advantage for workloads
> that have a huge variation in working set size that is primarily
> anonymous memory.  That larger scale "memory shaping" is left to
> ballooning and hotplug.
>    

And this is where the policy problems erupt.  When do you balloon in
favor of tmem?  which guest do you balloon? do you leave it to the
administrator? there's the host's administrator and the guests'
administrators.

CMM2 solves this neatly by providing information to the host.  The host
can pick the least recently used page (or a better algorithm) and evict
it using information from the guest, either dropping it or swapping it.  
It also provides information back to the guest when it references an
evicted page: either the guest needs to recreate the page or it just
needs to wait.

>> tmem makes life a lot easier to the hypervisor and to the guest, but
>> also gives up a lot of flexibility.  There's a difference
>> between memory
>> and a very fast synchronous backing store.
>>      
>
> I don't see that it gives up that flexibility.  System adminstrators
> are still free to size their guests properly.  Tmem's contribution
> is in environments that are highly dynamic, where the only
> alternative is really sizing memory maximally (and thus wasting
> it for the vast majority of time in which the working set is smaller).
>    

I meant that once a page is converted to tmem, there's a limited amount
of things you can do with it compared to normal memory.  For example
tmem won't help with a dcache intensive workload.

> I'm certainly open to identifying compromises and layer modifications
> that help meet the needs of both Xen and KVM (and others).  For
> example, if we can determine that the basic hook placement for
> precache/preswap (or even just precache for KVM) can be built
> on different underlying layers, that would be great!
>    

I'm not sure preswap/precache by itself justifies tmem since it can be
emulated by backing the disk with a cached file.  What I'm missing in
tmem is the ability for the hypervisor to take a global view on memory;
instead it's forced to look at memory and tmem separately.  That's fine
for Xen since it can't really make any decisions on normal memory
(lacking swap); on the other hand kvm doesn't map well to tmem since
"free memory" is already used by the host pagecache.

--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Anthony Liguori-2]

Avi Kivity wrote:
>
> In fact CMM2 is much more intrusive (but on the other hand provides
> much more information).
I don't think this will remain true long term.  CMM2 touches a lot of
core mm code and certainly qualifies as intrusive.  However the result
is that the VMM has a tremendous amount of insight into how the guest is
using it's memory and can implement all sorts of fancy policy for
reclaim.  Since the reclaim policy can evolve without any additional
assistance from the guest, the guest doesn't have to change as policy
evolves.

Since tmem requires that reclaim policy is implemented within the guest,
I think in the long term, tmem will have to touch a broad number of
places within Linux.  Beside the core mm, the first round of patches
already touch filesystems (just ext3 to start out with).  To truly be
effective, tmem would have to be a first class kernel citizen and I
suspect a lot of code would have to be aware of it.

So while CMM2 does a lot of code no one wants to touch, I think in the
long term it would remain relatively well contained compared to tmem
which will steadily increase in complexity within the guest.

Regards,

Anthony Liguori
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Dan Magenheimer-2]

> CMM2 and tmem are not any different in this regard; both require OS
> modification, and both make information available to the
> hypervisor.  In
> fact CMM2 is much more intrusive (but on the other hand provides much
> more information).
>
> > For those that believe it will be pervasive in the
> > future, finding the right balance is a critical step
> > in operating system evolution.
>
> You're arguing for CMM2 here IMO.

I'm arguing that both are a good thing and a step in
the right direction.  In some ways, tmem is a bigger
step and in some ways CMM2 is a bigger step.

> My take on this is that precache (predecache?) / preswap can be
> implemented even without tmem by using write-through backing for the
> virtual disk.  For swap this is actually slight;y more efficient than
> tmem preswap, for preuncache slightly less efficient (since
> there will
> be some double caching).  So I'm more interested in other use
> cases of tmem/CMM2.
>
> Right, the transient uses of tmem when applied to disk objects
> (swap/pagecache) are very similar to disk caches.  Which is
> why you can
> get a very similar effect when caching your virtual disks;
> this can be
> done without any guest modification.

Write-through backing and virtual disk cacheing offer a
similar effect, but it is far from the same.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Avi Kivity-4]

On 07/12/2009 11:39 PM, Dan Magenheimer wrote:

>> Right, the transient uses of tmem when applied to disk objects
>> (swap/pagecache) are very similar to disk caches.  Which is
>> why you can
>> get a very similar effect when caching your virtual disks;
>> this can be
>> done without any guest modification.
>>      
>
> Write-through backing and virtual disk cacheing offer a
> similar effect, but it is far from the same.
>    

Can you explain how it differs for the swap case?  Maybe I don't
understand how tmem preswap works.

--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Dan Magenheimer-2]

> > anonymous memory.  That larger scale "memory shaping" is left to
> > ballooning and hotplug.
>
> And this is where the policy problems erupt.  When do you balloon in
> favor of tmem?  which guest do you balloon? do you leave it to the
> administrator? there's the host's administrator and the guests'
> administrators.
> :
> CMM2 solves this neatly by providing information to the host.

As with CMM2, ballooning is for larger scale memory shaping.
Tmem provides a safety valve if the shaping is too aggressive
(and thus encourages more aggressive ballooning).  So they
are complementary.  Tmem also provides plenty of information
to the host that can be used to fine tune ballooning policy
if desired (and this can be done in userland and/or management
tools).
 

> > I don't see that it gives up that flexibility.  System adminstrators
> > are still free to size their guests properly.  Tmem's contribution
> > is in environments that are highly dynamic, where the only
> > alternative is really sizing memory maximally (and thus wasting
> > it for the vast majority of time in which the working set
> is smaller).
>
> I meant that once a page is converted to tmem, there's a
> limited amount
> of things you can do with it compared to normal memory.  For example
> tmem won't help with a dcache intensive workload.

Yes that's true.  But that's part of the point of tmem.  Tmem
isn't just providing benefits to a single guest.  It's
providing "memory liquidity" (Jeremy's term, but I like it)
which benefits the collective of guests on a machine and
across the data center.  For KVM+CMM2, I suppose this might be
less valuable because of the more incestuous relationship
between the host and guests.

> > I'm certainly open to identifying compromises and layer
> modifications
> > that help meet the needs of both Xen and KVM (and others).  For
> > example, if we can determine that the basic hook placement for
> > precache/preswap (or even just precache for KVM) can be built
> > on different underlying layers, that would be great!
>
> I'm not sure preswap/precache by itself justifies tmem since
> it can be
> emulated by backing the disk with a cached file.

I don't see that it can... though perhaps it can in the KVM
world.

> What I'm missing in
> tmem is the ability for the hypervisor to take a global view
> on memory;
> instead it's forced to look at memory and tmem separately.  

Again, I guess I see this as one of the key values of tmem.
Memory *does* have different attributes and calling out the
differences in some cases allows more flexibility to the
whole collective of guests with very little impact to any
one guest.

P.S.  I have to mostly disconnect from this discussion for
a few days except for short replies.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Dan Magenheimer-2]

> >> Right, the transient uses of tmem when applied to disk objects
> >> (swap/pagecache) are very similar to disk caches.  Which is
> >> why you can
> >> get a very similar effect when caching your virtual disks;
> >> this can be
> >> done without any guest modification.
> >
> > Write-through backing and virtual disk cacheing offer a
> > similar effect, but it is far from the same.
>
> Can you explain how it differs for the swap case?  Maybe I don't
> understand how tmem preswap works.

The key differences I see are the "please may I store something"
API and the fact that the reply (yes or no) can vary across time
depending on the state of the collective of guests.  Virtual
disk cacheing requires the host to always say yes and always
deliver persistence.  I can see that this is less of a concern
for KVM because the host can swap... though doesn't this hide
information from the guest and potentially have split-brain
swapping issues?

(thanks for the great discussion so far... going offline mostly now
for a few days)

Dan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Avi Kivity-4]

On 07/13/2009 12:08 AM, Dan Magenheimer wrote:
>> Can you explain how it differs for the swap case?  Maybe I don't
>> understand how tmem preswap works.
>>      
>
> The key differences I see are the "please may I store something"
> API and the fact that the reply (yes or no) can vary across time
> depending on the state of the collective of guests.  Virtual
> disk cacheing requires the host to always say yes and always
> deliver persistence.

We need to compare tmem+swap to swap+cache, not just tmem to cache.  
Here's how I see it:

tmem+swap swapout:
   - guest copies page to tmem (may fail)
   - guest writes page to disk

cached drive swapout:
   - guest writes page to disk
   - host copies page to cache

tmem+swap swapin:
   - guest reads page from tmem (may fail)
   - on tmem failure, guest reads swap from disk
   - guest drops tmem page

cached drive swapin:
   - guest reads page from disk
   - host may satisfy read from cache

tmem+swap ageing:
   - host may drop tmem page at any time

cached drive ageing:
   - host may drop cached page at any time

So they're pretty similar.  The main difference is that tmem can drop
the page on swapin.  It could be made to work with swap by supporting
the TRIM command.

> I can see that this is less of a concern
> for KVM because the host can swap... though doesn't this hide
> information from the guest and potentially have split-brain
> swapping issues?
>    

Double swap is bad for performance, yes.  CMM2 addresses it nicely.  
tmem doesn't address it at all - it assumes you have excess memory.

> (thanks for the great discussion so far... going offline mostly now
> for a few days)
>    

I'm going offline too so it cancels out.

--
error compiling committee.c: too many arguments to function

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Chris Mason-4]

On Sun, Jul 12, 2009 at 02:34:25PM -0500, Anthony Liguori wrote:

> Avi Kivity wrote:
>>
>> In fact CMM2 is much more intrusive (but on the other hand provides  
>> much more information).
> I don't think this will remain true long term.  CMM2 touches a lot of  
> core mm code and certainly qualifies as intrusive.  However the result  
> is that the VMM has a tremendous amount of insight into how the guest is  
> using it's memory and can implement all sorts of fancy policy for  
> reclaim.  Since the reclaim policy can evolve without any additional  
> assistance from the guest, the guest doesn't have to change as policy  
> evolves.
>
> Since tmem requires that reclaim policy is implemented within the guest,  
> I think in the long term, tmem will have to touch a broad number of  
> places within Linux.  Beside the core mm, the first round of patches  
> already touch filesystems (just ext3 to start out with).  To truly be  
> effective, tmem would have to be a first class kernel citizen and I  
> suspect a lot of code would have to be aware of it.

This depends on the extent to which tmem is integrated into the VM.  For
filesystem usage, the hooks are relatively simple because we already
have a lot of code sharing in this area.  Basically tmem is concerned
with when we free a clean page and when the contents of a particular
offset in the file are no longer valid.

The nice part about tmem is that any time a given corner case gets
tricky, you can just invalidate that offset in tmem and move on.

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Anthony Liguori-2]

Chris Mason wrote:
> This depends on the extent to which tmem is integrated into the VM.  For
> filesystem usage, the hooks are relatively simple because we already
> have a lot of code sharing in this area.  Basically tmem is concerned
> with when we free a clean page and when the contents of a particular
> offset in the file are no longer valid.
>  

But filesystem usage is perhaps the least interesting part of tmem.

The VMM already knows which pages in the guest are the result of disk IO
(it's the one that put it there, afterall).  It also knows when those
pages have been invalidated (or it can tell based on write-faulting).

The VMM also knows when the disk IO has been rerequested by tracking
previous requests.  It can keep the old IO requests cached in memory and
use that to satisfy re-reads as long as the memory isn't needed for
something else.  Basically, we have tmem today with kvm and we use it by
default by using the host page cache to do I/O caching (via
cache=writethrough).

The difference between our "tmem" is that instead of providing an
interface where the guest explicitly says, "I'm throwing away this
memory, I may need it later", and then asking again for it, the guest
throws away the page and then we can later satisfy the disk I/O request
that results from re-requesting the page instantaneously.

This transparent approach is far superior too because it enables
transparent sharing across multiple guests.  This works well for CoW
images and would work really well if we had a file system capable of
block-level deduplification... :-)

Regards,

Anthony Liguori
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Chris Mason-4]

On Mon, Jul 13, 2009 at 03:38:45PM -0500, Anthony Liguori wrote:

> Chris Mason wrote:
>> This depends on the extent to which tmem is integrated into the VM.  For
>> filesystem usage, the hooks are relatively simple because we already
>> have a lot of code sharing in this area.  Basically tmem is concerned
>> with when we free a clean page and when the contents of a particular
>> offset in the file are no longer valid.
>>  
>
> But filesystem usage is perhaps the least interesting part of tmem.
>
> The VMM already knows which pages in the guest are the result of disk IO  
> (it's the one that put it there, afterall).  It also knows when those  
> pages have been invalidated (or it can tell based on write-faulting).
>
> The VMM also knows when the disk IO has been rerequested by tracking  
> previous requests.  It can keep the old IO requests cached in memory and  
> use that to satisfy re-reads as long as the memory isn't needed for  
> something else.  Basically, we have tmem today with kvm and we use it by  
> default by using the host page cache to do I/O caching (via  
> cache=writethrough).

I'll definitely grant that caching with writethough adds more caching,
but it does need trim support before it is similar to tmem.  The caching
is transparent to the guest, but it is also transparent to qemu, and so
it is harder to manage and size (or even get a stat for how big it
currently is).

>
> The difference between our "tmem" is that instead of providing an  
> interface where the guest explicitly says, "I'm throwing away this  
> memory, I may need it later", and then asking again for it, the guest  
> throws away the page and then we can later satisfy the disk I/O request  
> that results from re-requesting the page instantaneously.
>
> This transparent approach is far superior too because it enables  
> transparent sharing across multiple guests.  This works well for CoW  
> images and would work really well if we had a file system capable of  
> block-level deduplification... :-)

Grin, I'm afraid that even if someone were to jump in and write the
perfect cow based filesystem and then find a willing contributor to code
up a dedup implementation, each cow image would be a different file
and so it would have its own address space.

Dedup and COW are an easy way to have hints about which pages are
supposed to be have the same contents, but they would have to go with
some other duplicate page sharing scheme.

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Anthony Liguori-2]

Chris Mason wrote:
> On Mon, Jul 13, 2009 at 03:38:45PM -0500, Anthony Liguori wrote:
>  
> I'll definitely grant that caching with writethough adds more caching,
> but it does need trim support before it is similar to tmem.

I think trim is somewhat orthogonal but even if you do need it, the nice
thing about implementing ATA trim support verses a paravirtualization is
that it works with a wide variety of guests.

 From the perspective of the VMM, it seems like a good thing.

>   The caching
> is transparent to the guest, but it is also transparent to qemu, and so
> it is harder to manage and size (or even get a stat for how big it
> currently is).
>  

That's certainly a fixable problem though.  We could expose statistics
to userspace and then further expose those to guests.  I think the first
question to answer though is what you would use those statistics for.

>> The difference between our "tmem" is that instead of providing an  
>> interface where the guest explicitly says, "I'm throwing away this  
>> memory, I may need it later", and then asking again for it, the guest  
>> throws away the page and then we can later satisfy the disk I/O request  
>> that results from re-requesting the page instantaneously.
>>
>> This transparent approach is far superior too because it enables  
>> transparent sharing across multiple guests.  This works well for CoW  
>> images and would work really well if we had a file system capable of  
>> block-level deduplification... :-)
>>    
>
> Grin, I'm afraid that even if someone were to jump in and write the
> perfect cow based filesystem and then find a willing contributor to code
> up a dedup implementation, each cow image would be a different file
> and so it would have its own address space.
>
> Dedup and COW are an easy way to have hints about which pages are
> supposed to be have the same contents, but they would have to go with
> some other duplicate page sharing scheme.
>  

Yes.  We have the information we need to dedup this memory though.  We
just need a way to track non-dirty pages that result from DMA, map the
host page cache directly into the guest, and then CoW when the guest
tries to dirty that memory.

We don't quite have the right infrastructure in Linux yet to do this
effectively, but this is entirely an issue with the host.  The guest
doesn't need any changes here.

Regards,

Anthony Liguori
> -chris
>
>  

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Avi Kivity-4]

On 07/14/2009 12:17 AM, Anthony Liguori wrote:

> Chris Mason wrote:
>> On Mon, Jul 13, 2009 at 03:38:45PM -0500, Anthony Liguori wrote:
>>   I'll definitely grant that caching with writethough adds more caching,
>> but it does need trim support before it is similar to tmem.
>
> I think trim is somewhat orthogonal but even if you do need it, the
> nice thing about implementing ATA trim support verses a
> paravirtualization is that it works with a wide variety of guests.
>
> From the perspective of the VMM, it seems like a good thing.

trim is also lovely in that images will no longer grow monotonously even
though guest disk usage is constant or is even reduced.

--
error compiling committee.c: too many arguments to function

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/