MIPS: bus_dma(9) and cache problems

[Oleksandr Tymoshenko-4]

This problem haunts for a couple of days and I can't find a
nice and clean solution so this email is actually a cry for help.


The problem:
     There is a buffer loaded by bus_dmamap_load for use as a
DMA buffer. Right before this buffer resides block of vital
data structure. Consider following scenario:

     1. code modifies data in block and this modification ends up in
        cache and is not written back to memory
     2. right after this code calls bus_dmamap_sync for this buffer
        and as a result cache invalidation is performed
     3. Cache function operates on cache line size-aligned addresses
        and the block in question happens to share the same cache line
        with the buffer. So modification made at step (1) is lost.

If busdma code controls allocation (bus_dmamem_alloc) this situation
can be avoided by forcing pointer alignment. But when address come
from the "outer space" via bus_dmamap_load* there is not much to do.

There are two solutions I've figured so far:
   - Create bounce page for not properly aligned memory. Which would
       reduce performance a lot.
   - Remap buffer's page(s) as uncached. I haven't succeeded with this
      one yet and not sure it's always possible.

May be someone can suggest better way to clean this mess?

Thanks!

--
gonzo

_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Jason Harmening]

>     1. code modifies data in block and this modification ends up in
>        cache and is not written back to memory
>     2. right after this code calls bus_dmamap_sync for this buffer
>        and as a result cache invalidation is performed
>     3. Cache function operates on cache line size-aligned addresses
>        and the block in question happens to share the same cache line
>        with the buffer. So modification made at step (1) is lost.

What sync operation are you doing?  At least for PREREAD or PREWRITE,
I'd expect any dirty cache lines to be flushed to RAM.  If this isn't
happening, then you may want to submit a bug report.

BTW, if you haven't already found it the MIPS sync code for 9-CURRENT is here:

http://fxr.watson.org/fxr/source/mips/mips/busdma_machdep.c#L760
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Ryan Stone-2]

> What sync operation are you doing?  At least for PREREAD or PREWRITE,
> I'd expect any dirty cache lines to be flushed to RAM.  If this isn't
> happening, then you may want to submit a bug report.

For a PREREAD, I don't believe that it's correct to flush a dirty
cache line to RAM.  That would overwrite whatever had been DMA'ed into
that cache line.

What about the following procedure for a PREREAD for a non-cache
aligned buffer I'll call dma_buf

1) read the entire cache line into a buffer, buf1
2) issue the invalidate
3) copy the portion of buf1 that preceeds dma_buf back to that address

One problem I can see immediately is that there is a race here: if
something tries to access the memory preceeding dma_buf after the
invalidate is issued but before the copy completes they will see
inconsistent data.  Maybe somebody else can think of a way around
that.

Ryan Stone
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Jason Harmening]

On Mon, Nov 2, 2009 at 12:24 PM, Ryan Stone <rysto32@...> wrote:
>> What sync operation are you doing?  At least for PREREAD or PREWRITE,
>> I'd expect any dirty cache lines to be flushed to RAM.  If this isn't
>> happening, then you may want to submit a bug report.
>
> For a PREREAD, I don't believe that it's correct to flush a dirty
> cache line to RAM.  That would overwrite whatever had been DMA'ed into
> that cache line.
>

I don't think that should matter--if you're issuing a PREREAD,
*before* the start of a DMA transfer, the CPU either doesn't care
about what's currently in the part of the line that is to be DMA'ed
into (because it's about to be overwritten by the device anyway), or
it's finished accessing what's in there from a previous DMA operation
(in which case you'd expect it to either be present in the cache or
already flushed out by something else anyway).

But the basic idea is that the CPU shouldn't access the cache line
from the time the PRE-whatever operation is issued to the time the
POST-whatever operation is issued, so if you have multiple threads
(anywhere on the system) accessing that line, you could be screwed.

Heh, I just noticed the copyright at the top of the MIPS busdma
implementation--apparently you ARE familiar w/ the MIPS sync
implementation :)
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."