OS X 10.6 (Snow Leopard) HFS+ File Compression

[Tony-192]

Are there any patches (or planned updates) to rsync v3.0.6 to handle  
the HFS+ File Compression that Apple introduced with Snow Leopard?
 
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Matt McCutchen-7]

On Tue, 2009-10-27 at 19:31 -0400, Tony wrote:
> Are there any patches (or planned updates) to rsync v3.0.6 to handle  
> the HFS+ File Compression that Apple introduced with Snow Leopard?

What kind of special treatment from rsync were you expecting?  I read
http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars/3 , and
as far as I can tell, the compression is handled entirely by the
filesystem with no intervention from applications needed.

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Matt McCutchen-7]

On Tue, 2009-10-27 at 23:38 -0400, Tony wrote:

> When rsync 3.0.6 copies files with HFS+ File Compression, the new  
> extended attribute decmpfs is not preserved, and the UF_COMPRESSED  
> flag is not set on the destination and the destination file is not  
> compressed.
>
> I examined the destination file as described in ars technica  (with ls  
> and xattr from a 10.5 Leopard boot), and the compressed data is moved  
> from the resource fork to the data fork, and the extended attributes  
> '@' are removed from the file.
>
> As far as I know, only ditto in 10.6 can handle HFS+ File  
> Compression.  (I even tested a 'clone' with disk utility (file copy,  
> not block), and it also failed (block copy, of course works).

Rsync is just reading and writing files via the filesystem API; it has
no access to any of the flags or xattrs used to implement the
compression.

I guess the filesystem doesn't compress new files by default.  If it had
an API to request compression, rsync could use that API when writing the
destination files.  Unfortunately, the API ditto is using appears to be
private to Apple.  See the post from brkirch beginning "The first thing
that I tried to do" on this page:

http://www.macosxhints.com/article.php?story=20090902223042255

So anyone interested in making rsync compress the destination files
would probably have to copy the relevant code from afsctool.  This could
be shared as a patch; I feel quite sure it would not be adopted in the
main version of rsync.

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Mike Bombich]

HFS compression can be preserved as long as the relevant xattr(s) and flags on those files are preserved.  A compressed file has the compressed data in a hidden xattr (com.apple.decmpfs if < 4Kb, com.apple.ResourceFork if more), and has the UF_COMPRESSED flag set (decimal 40).  When rsync encounters a file like this, it should ignore the data fork of the file, which will appear to contain normal, uncompressed data.  It should also pass a special flag to the xattr calls to expose the decmpfs xattrs.

I've already implemented this in rsync (3.0.6), I just hadn't taken the time to craft the HFS-compression-specific changes into a patch.  I did that this evening and attached it below.  These are changes against the 3.0.6 base, plus the crtimes, fileflags, and backup-dir-dels patches.  It should work, at minimum, against the 3.0.6 base plus the fileflags patch (that patch is required).

Let me know if it doesn't work for you, it's entirely possible that I overlooked something in the extraction.

Mike

On Oct 27, 2009, at 11:08 PM, Matt McCutchen wrote:

On Tue, 2009-10-27 at 23:38 -0400, Tony wrote:

When rsync 3.0.6 copies files with HFS+ File Compression, the new  

extended attribute decmpfs is not preserved, and the UF_COMPRESSED  

flag is not set on the destination and the destination file is not  

compressed.

I examined the destination file as described in ars technica  (with ls  

and xattr from a 10.5 Leopard boot), and the compressed data is moved  

from the resource fork to the data fork, and the extended attributes  

'@' are removed from the file.

As far as I know, only ditto in 10.6 can handle HFS+ File  

Compression.  (I even tested a 'clone' with disk utility (file copy,  

not block), and it also failed (block copy, of course works).

Rsync is just reading and writing files via the filesystem API; it has
no access to any of the flags or xattrs used to implement the
compression.

I guess the filesystem doesn't compress new files by default.  If it had
an API to request compression, rsync could use that API when writing the
destination files.  Unfortunately, the API ditto is using appears to be
private to Apple.  See the post from brkirch beginning "The first thing
that I tried to do" on this page:

http://www.macosxhints.com/article.php?story=20090902223042255

So anyone interested in making rsync compress the destination files
would probably have to copy the relevant code from afsctool.  This could
be shared as a patch; I feel quite sure it would not be adopted in the
main version of rsync.

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Tony-192]

When rsync 3.0.6 copies files with HFS+ File Compression, the new  
extended attribute decmpfs is not preserved, and the UF_COMPRESSED  
flag is not set on the destination and the destination file is not  
compressed.

I examined the destination file as described in ars technica  (with ls  
and xattr from a 10.5 Leopard boot), and the compressed data is moved  
from the resource fork to the data fork, and the extended attributes  
'@' are removed from the file.

As far as I know, only ditto in 10.6 can handle HFS+ File  
Compression.  (I even tested a 'clone' with disk utility (file copy,  
not block), and it also failed (block copy, of course works).

On Oct 27, 2009, at 7:39 PM, Matt McCutchen wrote:

>
> What kind of special treatment from rsync were you expecting?  I read
> http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars/3 , and
> as far as I can tell, the compression is handled entirely by the
> filesystem with no intervention from applications needed.
>
> --
> Matt
>
> On Tue, 2009-10-27 at 19:31 -0400, Tony wrote:
>> Are there any patches (or planned updates) to rsync v3.0.6 to handle
>> the HFS+ File Compression that Apple introduced with Snow Leopard?

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Tony-192]

Thanks.  The C code that brkirch provides takes care of a lot of the  
work, so hopefully someone will be able to provide a patch (Its been  
over 15 years since I did any C programing, so unfortunately I won't  
be able to contribute)

On Oct 28, 2009, at 12:08 AM, Matt McCutchen wrote:

> Rsync is just reading and writing files via the filesystem API; it has
> no access to any of the flags or xattrs used to implement the
> compression.
>
> I guess the filesystem doesn't compress new files by default.  If it  
> had
> an API to request compression, rsync could use that API when writing  
> the
> destination files.  Unfortunately, the API ditto is using appears to  
> be
> private to Apple.  See the post from brkirch beginning "The first  
> thing
> that I tried to do" on this page:
>
> http://www.macosxhints.com/article.php?story=20090902223042255
>
> So anyone interested in making rsync compress the destination files
> would probably have to copy the relevant code from afsctool.  This  
> could
> be shared as a patch; I feel quite sure it would not be adopted in the
> main version of rsync.
>
>

> On Tue, 2009-10-27 at 23:38 -0400, Tony wrote:
>> When rsync 3.0.6 copies files with HFS+ File Compression, the new
>> extended attribute decmpfs is not preserved, and the UF_COMPRESSED
>> flag is not set on the destination and the destination file is not
>> compressed.
>>
>> I examined the destination file as described in ars technica  (with  
>> ls
>> and xattr from a 10.5 Leopard boot), and the compressed data is moved
>> from the resource fork to the data fork, and the extended attributes
>> '@' are removed from the file.
>>
>> As far as I know, only ditto in 10.6 can handle HFS+ File
>> Compression.  (I even tested a 'clone' with disk utility (file copy,
>> not block), and it also failed (block copy, of course works).
>
> Rsync is just reading and writing files via the filesystem API; it has
> no access to any of the flags or xattrs used to implement the
> compression.
>
> I guess the filesystem doesn't compress new files by default.  If it  
> had
> an API to request compression, rsync could use that API when writing  
> the
> destination files.  Unfortunately, the API ditto is using appears to  
> be
> private to Apple.  See the post from brkirch beginning "The first  
> thing
> that I tried to do" on this page:
>
> http://www.macosxhints.com/article.php?story=20090902223042255
>
> So anyone interested in making rsync compress the destination files
> would probably have to copy the relevant code from afsctool.  This  
> could
> be shared as a patch; I feel quite sure it would not be adopted in the
> main version of rsync.
>
> --
> Matt
>

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Tony-192]

Mike, thanks for the patch.  Will this patch be maintained in rsync-patches-3.0.6.tar.gz ?

On Oct 28, 2009, at 1:20 AM, Mike Bombich wrote:

HFS compression can be preserved as long as the relevant xattr(s) and flags on those files are preserved.  A compressed file has the compressed data in a hidden xattr (com.apple.decmpfs if < 4Kb, com.apple.ResourceFork if more), and has the UF_COMPRESSED flag set (decimal 40).  When rsync encounters a file like this, it should ignore the data fork of the file, which will appear to contain normal, uncompressed data.  It should also pass a special flag to the xattr calls to expose the decmpfs xattrs.

I've already implemented this in rsync (3.0.6), I just hadn't taken the time to craft the HFS-compression-specific changes into a patch.  I did that this evening and attached it below.  These are changes against the 3.0.6 base, plus the crtimes, fileflags, and backup-dir-dels patches.  It should work, at minimum, against the 3.0.6 base plus the fileflags patch (that patch is required).

Let me know if it doesn't work for you, it's entirely possible that I overlooked something in the extraction.

Mike

<rsync_3.0.6-hfs-compression_20091027.diff>

On Oct 27, 2009, at 11:08 PM, Matt McCutchen wrote:

On Tue, 2009-10-27 at 23:38 -0400, Tony wrote:

When rsync 3.0.6 copies files with HFS+ File Compression, the new  

extended attribute decmpfs is not preserved, and the UF_COMPRESSED  

flag is not set on the destination and the destination file is not  

compressed.

I examined the destination file as described in ars technica  (with ls  

and xattr from a 10.5 Leopard boot), and the compressed data is moved  

from the resource fork to the data fork, and the extended attributes  

'@' are removed from the file.

As far as I know, only ditto in 10.6 can handle HFS+ File  

Compression.  (I even tested a 'clone' with disk utility (file copy,  

not block), and it also failed (block copy, of course works).

Rsync is just reading and writing files via the filesystem API; it has
no access to any of the flags or xattrs used to implement the
compression.

I guess the filesystem doesn't compress new files by default.  If it had
an API to request compression, rsync could use that API when writing the
destination files.  Unfortunately, the API ditto is using appears to be
private to Apple.  See the post from brkirch beginning "The first thing
that I tried to do" on this page:

http://www.macosxhints.com/article.php?story=20090902223042255

So anyone interested in making rsync compress the destination files
would probably have to copy the relevant code from afsctool.  This could
be shared as a patch; I feel quite sure it would not be adopted in the
main version of rsync.

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Mike Bombich]

I wouldn't blame the rsync team for not wanting to maintain it, it's a pretty narrow-scope patch affecting only one OS.  I'm pretty motivated to keep it up, though, so I'll repost my patches to this list when I update them.  I'll probably get it updated to 3.1.0 in the next month or so.

Mike

On Nov 1, 2009, at 7:57 PM, Tony wrote:

Mike, thanks for the patch.  Will this patch be maintained in rsync-patches-3.0.6.tar.gz ?

On Oct 28, 2009, at 1:20 AM, Mike Bombich wrote:

HFS compression can be preserved as long as the relevant xattr(s) and flags on those files are preserved.  A compressed file has the compressed data in a hidden xattr (com.apple.decmpfs if < 4Kb, com.apple.ResourceFork if more), and has the UF_COMPRESSED flag set (decimal 40).  When rsync encounters a file like this, it should ignore the data fork of the file, which will appear to contain normal, uncompressed data.  It should also pass a special flag to the xattr calls to expose the decmpfs xattrs.

I've already implemented this in rsync (3.0.6), I just hadn't taken the time to craft the HFS-compression-specific changes into a patch.  I did that this evening and attached it below.  These are changes against the 3.0.6 base, plus the crtimes, fileflags, and backup-dir-dels patches.  It should work, at minimum, against the 3.0.6 base plus the fileflags patch (that patch is required).

Let me know if it doesn't work for you, it's entirely possible that I overlooked something in the extraction.

Mike

<rsync_3.0.6-hfs-compression_20091027.diff>

On Oct 27, 2009, at 11:08 PM, Matt McCutchen wrote:

On Tue, 2009-10-27 at 23:38 -0400, Tony wrote:

When rsync 3.0.6 copies files with HFS+ File Compression, the new  

extended attribute decmpfs is not preserved, and the UF_COMPRESSED  

flag is not set on the destination and the destination file is not  

compressed.

I examined the destination file as described in ars technica  (with ls  

and xattr from a 10.5 Leopard boot), and the compressed data is moved  

from the resource fork to the data fork, and the extended attributes  

'@' are removed from the file.

As far as I know, only ditto in 10.6 can handle HFS+ File  

Compression.  (I even tested a 'clone' with disk utility (file copy,  

not block), and it also failed (block copy, of course works).

Rsync is just reading and writing files via the filesystem API; it has
no access to any of the flags or xattrs used to implement the
compression.

I guess the filesystem doesn't compress new files by default.  If it had
an API to request compression, rsync could use that API when writing the
destination files.  Unfortunately, the API ditto is using appears to be
private to Apple.  See the post from brkirch beginning "The first thing
that I tried to do" on this page:

http://www.macosxhints.com/article.php?story=20090902223042255

So anyone interested in making rsync compress the destination files
would probably have to copy the relevant code from afsctool.  This could
be shared as a patch; I feel quite sure it would not be adopted in the
main version of rsync.

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html