losing meaningful whitespaces in an encrypted file

[paladino]

Hi, I'm sorry to jump right in with a dumb question, but I've tried doing some research myself and I have to confess to much of this being way over my head.

I work for a University that uses GnuPG to encrypt files to send out to various vendors.

We're having a very odd situation right now with one of our files.

We are sending a file that has a header line that ends with 13 spaces. We are encrypting the file from the command line, on a unix machine, with GnuPG.  Here's the actual command our guys are using:
/usr/local/bin/gpg -v -r XXXXXXXX -f &filename

We are then transmitting the file to a vendor who is unencrypting it with the windows version of PGP.

When I look at the file here, immediately before it is encrypted, the 13 white spaces are still there. When I look at the file at the vendor, immediately after decryption, the 13 spaces are gone.

I haven't had any luck with getting more information from the vendor about what kind of options they are using.  I do know they are using a windows version, and the guy says he basically just double clicks on it, types in a password, and it unencrypts the file.

Is there anything obvious that could be causing something like this?  Which end is it more likely the problem is at?

I've been reading about pgp and gpg all day, and while I've learned alot about both, I'm no closer to a solution for this one than when I started.  Thanks for any help at all.

[Robert J. Hansen-3]

paladino wrote:
> When I look at the file here, immediately before it is encrypted, the 13
> white spaces are still there. When I look at the file at the vendor,
> immediately after decryption, the 13 spaces are gone.

Have you tried a test decryption on your end?  E.g., encrypt the file
with your own public key and then decrypt that, and see whether the 13
spaces are present?

Also, version numbers would be very useful--both GnuPG on your end and
PGP on the vendor's end.

This may very well be a PGP problem as opposed to a GnuPG problem, in
which case you may be better served on a PGP list such as PGP-Basics at
Yahoo! Groups.

> Is there anything obvious that could be causing something like this?  Which
> end is it more likely the problem is at?

Impossible to say without more information.  My inclination is to think
it's probably on the vendor's end, especially if you're using a recent
version of GnuPG.  There are a lot of PGP 5.0 and 6.5.8 installations
out there, and both of them substantially predate the OpenPGP standard
which GnuPG conforms to.

_______________________________________________
Gnupg-users mailing list
Gnupg-users@...
http://lists.gnupg.org/mailman/listinfo/gnupg-users

[David Shaw]

On Thu, Sep 06, 2007 at 06:55:59AM -0700, paladino wrote:

>
> Hi, I'm sorry to jump right in with a dumb question, but I've tried doing
> some research myself and I have to confess to much of this being way over my
> head.
>
> I work for a University that uses GnuPG to encrypt files to send out to
> various vendors.
>
> We're having a very odd situation right now with one of our files.
>
> We are sending a file that has a header line that ends with 13 spaces. We
> are encrypting the file from the command line, on a unix machine, with
> GnuPG.  Here's the actual command our guys are using:
> /usr/local/bin/gpg -v -r XXXXXXXX -f &filename

Are you sure about that?  There is no '-f' option in GPG.

> We are then transmitting the file to a vendor who is unencrypting it with
> the windows version of PGP.
>
> When I look at the file here, immediately before it is encrypted, the 13
> white spaces are still there. When I look at the file at the vendor,
> immediately after decryption, the 13 spaces are gone.
>
> I haven't had any luck with getting more information from the vendor about
> what kind of options they are using.  I do know they are using a windows
> version, and the guy says he basically just double clicks on it, types in a
> password, and it unencrypts the file.
>
> Is there anything obvious that could be causing something like this?  Which
> end is it more likely the problem is at?

This is a mini-bug sort of thing, based on historical practice in the
OpenPGP community and two different versions of the OpenPGP.
Basically, OpenPGP specifies both binary and text transports.  Binary
is just that - a binary image of the file, with no changes.  Text is
canonical text, and there are rules for the canonicalization (change
line endings to CRLF, etc).

When the OpenPGP spec was written, a piece of 'incorrect' language
crept in: that whitespace at the end of a line would not be included
in canonical text.  PGP already worked fine and never changed their
code to match the language in the spec.  GPG came along later and
followed the spec exactly.  For various reasons, this mismatch in
canonicalization wasn't really a problem in practice, but
nevertheless, in the process of writing the updated OpenPGP spec
(which is with the RFC editor now, so it'll be out soon), this was
resolved.  The new canonicalization rule is the historical one: change
line endings, and do nothing to trailing whitespace.

It is hard for me to give you exact advice on what to do from here
without a few piece of information:

1) What version of GPG are you using?  (gpg --version)
2) The command line you are using (I'm fairly sure the one above is
   not correct, as GPG has no '-f' option).

David

_______________________________________________
Gnupg-users mailing list
Gnupg-users@...
http://lists.gnupg.org/mailman/listinfo/gnupg-users

[Paladino, Vanda K]

Thanks for your quick replies.  I actually drafted that message last
week but just managed to get it to go through today, so I do have some
more information.

I've gotten someone over here to help me a  bit, and we've run some
tests.

Our file is being encrypted with gpg version 1.2.6

We had them send us an encrypted file and we decrypted it using gpg
version 1.4.5 and the spaces were missing.  We decrypted it with PGP as
well, and the spaces were also missing, not sure what version of PGP
that was, I can find out.

I did originally go to the PGP people for help, and they, of course,
sent me over here :)

But it is starting to seem like the problem is on our side, which would
be the gpg side of the issue.

Vanda

 

-----Original Message-----
From: Robert J. Hansen [mailto:rjh@...]
Sent: Thursday, September 06, 2007 10:41 AM
To: Paladino, Vanda K
Cc: gnupg-users@...
Subject: Re: losing meaningful whitespaces in an encrypted file

paladino wrote:
> When I look at the file here, immediately before it is encrypted, the
> 13 white spaces are still there. When I look at the file at the
> vendor, immediately after decryption, the 13 spaces are gone.

Have you tried a test decryption on your end?  E.g., encrypt the file
with your own public key and then decrypt that, and see whether the 13
spaces are present?

Also, version numbers would be very useful--both GnuPG on your end and
PGP on the vendor's end.

This may very well be a PGP problem as opposed to a GnuPG problem, in
which case you may be better served on a PGP list such as PGP-Basics at
Yahoo! Groups.

> Is there anything obvious that could be causing something like this?  
> Which end is it more likely the problem is at?

Impossible to say without more information.  My inclination is to think
it's probably on the vendor's end, especially if you're using a recent
version of GnuPG.  There are a lot of PGP 5.0 and 6.5.8 installations
out there, and both of them substantially predate the OpenPGP standard
which GnuPG conforms to.

_______________________________________________
Gnupg-users mailing list
Gnupg-users@...
http://lists.gnupg.org/mailman/listinfo/gnupg-users

[Paladino, Vanda K]

I'm pretty sure it's the correct command line.  I asked them twice, and
they sent me a line of copy and paste straight out of the UC4 job.  I
had it in my head that the -f indicated the filename, but now that I
look, I'm not sure where I read that.  I'm checking with our Production
Control guys now on that one.

Thanks for all the suggestions and comments so far, any help is good
help at this point!

Vanda Paladino

_______________________________________________
Gnupg-users mailing list
Gnupg-users@...
http://lists.gnupg.org/mailman/listinfo/gnupg-users

[David Shaw]

Again, this is not a bug, but a documented part of the protocol.
There are ways around it, and the details on this will be changing in
the future, but at least for today, if you send files as text, you
will lose end-of-line whitespace.

David

On Thu, Sep 06, 2007 at 10:52:18AM -0400, Paladino, Vanda K wrote:

> Thanks for your quick replies.  I actually drafted that message last
> week but just managed to get it to go through today, so I do have some
> more information.
>
> I've gotten someone over here to help me a  bit, and we've run some
> tests.
>
> Our file is being encrypted with gpg version 1.2.6
>
> We had them send us an encrypted file and we decrypted it using gpg
> version 1.4.5 and the spaces were missing.  We decrypted it with PGP as
> well, and the spaces were also missing, not sure what version of PGP
> that was, I can find out.
>
> I did originally go to the PGP people for help, and they, of course,
> sent me over here :)
>
> But it is starting to seem like the problem is on our side, which would
> be the gpg side of the issue.
>
> Vanda
>
>  
>
> -----Original Message-----
> From: Robert J. Hansen [mailto:rjh@...]
> Sent: Thursday, September 06, 2007 10:41 AM
> To: Paladino, Vanda K
> Cc: gnupg-users@...
> Subject: Re: losing meaningful whitespaces in an encrypted file
>
> paladino wrote:
> > When I look at the file here, immediately before it is encrypted, the
> > 13 white spaces are still there. When I look at the file at the
> > vendor, immediately after decryption, the 13 spaces are gone.
>
> Have you tried a test decryption on your end?  E.g., encrypt the file
> with your own public key and then decrypt that, and see whether the 13
> spaces are present?
>
> Also, version numbers would be very useful--both GnuPG on your end and
> PGP on the vendor's end.
>
> This may very well be a PGP problem as opposed to a GnuPG problem, in
> which case you may be better served on a PGP list such as PGP-Basics at
> Yahoo! Groups.
>
> > Is there anything obvious that could be causing something like this?  
> > Which end is it more likely the problem is at?
>
> Impossible to say without more information.  My inclination is to think
> it's probably on the vendor's end, especially if you're using a recent
> version of GnuPG.  There are a lot of PGP 5.0 and 6.5.8 installations
> out there, and both of them substantially predate the OpenPGP standard
> which GnuPG conforms to.
>
> _______________________________________________
> Gnupg-users mailing list
> Gnupg-users@...
> http://lists.gnupg.org/mailman/listinfo/gnupg-users
>

_______________________________________________
Gnupg-users mailing list
Gnupg-users@...
http://lists.gnupg.org/mailman/listinfo/gnupg-users

[Paladino, Vanda K]

Sorry I don't know why that took so long to get through, and I also
apologize for the multiple posts of the same message.  I was using a
funky web interface to your mailing list and it seems to not like me
very much.

As clarification on the original issue, we weren't sending it as text,
we were sending it as a binary file.

However, we did finally at least partially identify our problem.

Version 1.2.6 for Linux strips our end of line white spaces.
Version 1.2 for Unix does not.

We'll be going back to using our Unix machine to do the encrypting until
we can upgrade or downgrade as necessary to find a linux version that
behaves properly.

Thanks to everyone here who helped me out via private email to narrow
down the solution.

Vanda Paladino

-----Original Message-----
From: David Shaw [mailto:dshaw@...]
Sent: Monday, September 10, 2007 8:55 PM
To: Paladino, Vanda K
Cc: Robert J. Hansen; gnupg-users@...
Subject: Re: losing meaningful whitespaces in an encrypted file

Again, this is not a bug, but a documented part of the protocol.
There are ways around it, and the details on this will be changing in
the future, but at least for today, if you send files as text, you will
lose end-of-line whitespace.

David

On Thu, Sep 06, 2007 at 10:52:18AM -0400, Paladino, Vanda K wrote:
> Thanks for your quick replies.  I actually drafted that message last
> week but just managed to get it to go through today, so I do have some

> more information.
>
> I've gotten someone over here to help me a  bit, and we've run some
> tests.
>
> Our file is being encrypted with gpg version 1.2.6
>
> We had them send us an encrypted file and we decrypted it using gpg
> version 1.4.5 and the spaces were missing.  We decrypted it with PGP
> as well, and the spaces were also missing, not sure what version of
> PGP that was, I can find out.
>
> I did originally go to the PGP people for help, and they, of course,
> sent me over here :)
>
> But it is starting to seem like the problem is on our side, which
> would be the gpg side of the issue.
>
> Vanda
>
>  
>
> -----Original Message-----
> From: Robert J. Hansen [mailto:rjh@...]
> Sent: Thursday, September 06, 2007 10:41 AM
> To: Paladino, Vanda K
> Cc: gnupg-users@...
> Subject: Re: losing meaningful whitespaces in an encrypted file
>
> paladino wrote:
> > When I look at the file here, immediately before it is encrypted,
> > the
> > 13 white spaces are still there. When I look at the file at the
> > vendor, immediately after decryption, the 13 spaces are gone.
>
> Have you tried a test decryption on your end?  E.g., encrypt the file
> with your own public key and then decrypt that, and see whether the 13

> spaces are present?
>
> Also, version numbers would be very useful--both GnuPG on your end and

> PGP on the vendor's end.
>
> This may very well be a PGP problem as opposed to a GnuPG problem, in
> which case you may be better served on a PGP list such as PGP-Basics
> at Yahoo! Groups.
>
> > Is there anything obvious that could be causing something like this?

> > Which end is it more likely the problem is at?
>
> Impossible to say without more information.  My inclination is to
> think it's probably on the vendor's end, especially if you're using a
> recent version of GnuPG.  There are a lot of PGP 5.0 and 6.5.8
> installations out there, and both of them substantially predate the
> OpenPGP standard which GnuPG conforms to.
>
> _______________________________________________
> Gnupg-users mailing list
> Gnupg-users@...
> http://lists.gnupg.org/mailman/listinfo/gnupg-users
>

_______________________________________________
Gnupg-users mailing list
Gnupg-users@...
http://lists.gnupg.org/mailman/listinfo/gnupg-users

[Werner Koch]

On Tue, 11 Sep 2007 13:20, vbushfield@... said:

> apologize for the multiple posts of the same message.  I was using a
> funky web interface to your mailing list and it seems to not like me
> very much.

If you are not subscribed at this address, modertaor approval is
required.  That may take some time.  BTW, thanks to the volunteers who
to this moderating for so many years now.

Also make sure that you don't send out HTML mails, or mails withy HTML
parts of ZIP files etc.

> However, we did finally at least partially identify our problem.
>
> Version 1.2.6 for Linux strips our end of line white spaces.
> Version 1.2 for Unix does not.

Hmmm, Linux is as much a Unix as any other Unix.  What do you mean by
version 1.2? gnupg 1.2.0?  All Unix versions are identical in particular
in the parts of the code handling the bulk data and doing white space
removal.  The only problem I recall are those with large files; that is
the usual 2 GB problem on some machines.


Salam-Shalom,

   Werner

--
Die Gedanken sind frei.  Auschnahme regelt ein Bundeschgesetz.


_______________________________________________
Gnupg-users mailing list
Gnupg-users@...
http://lists.gnupg.org/mailman/listinfo/gnupg-users

[Paladino, Vanda K]

I only know what my guys tell me :)  They say we have 1.2.6 on one
machine (the linux machine) and 1.2.0 on the other (the other unix
machine).  And someone else told me that Unix and Linux had different
versions of gpg, I didn't check that out, so apologies if it isn't the
case!

-----Original Message-----
From: Werner Koch [mailto:wk@...]
Sent: Wednesday, September 12, 2007 12:45 PM
To: Paladino, Vanda K
Cc: David Shaw; gnupg-users@...
Subject: Re: losing meaningful whitespaces in an encrypted file

On Tue, 11 Sep 2007 13:20, vbushfield@... said:

> apologize for the multiple posts of the same message.  I was using a
> funky web interface to your mailing list and it seems to not like me
> very much.

If you are not subscribed at this address, modertaor approval is
required.  That may take some time.  BTW, thanks to the volunteers who
to this moderating for so many years now.

Also make sure that you don't send out HTML mails, or mails withy HTML
parts of ZIP files etc.

> However, we did finally at least partially identify our problem.
>
> Version 1.2.6 for Linux strips our end of line white spaces.
> Version 1.2 for Unix does not.

Hmmm, Linux is as much a Unix as any other Unix.  What do you mean by
version 1.2? gnupg 1.2.0?  All Unix versions are identical in particular
in the parts of the code handling the bulk data and doing white space
removal.  The only problem I recall are those with large files; that is
the usual 2 GB problem on some machines.


Salam-Shalom,

   Werner

--
Die Gedanken sind frei.  Auschnahme regelt ein Bundeschgesetz.


_______________________________________________
Gnupg-users mailing list
Gnupg-users@...
http://lists.gnupg.org/mailman/listinfo/gnupg-users

[vedaal]

Paladino, Vanda K  ( vbushfield at purdue.edu )
wrote on Tue Sep 11 13:20:12 CEST 2007


>As clarification on the original issue, we weren't sending it as
text,
>we were sending it as a binary file

>However, we did finally at least partially identify our problem

>Version 1.2.6 for Linux strips our end of line white spaces.
>Version 1.2 for Unix does not


while this may be a possible source of the problem,
it sounds unlikely

try this workaround:

armor the file
(add '-a' before the '-r' if you are encrypting,
or before the '-u' if you are just signing

armoring the file preserves the plaintext and all trailing
whitespaces
(i don't know why this should be, but in my repeated testing with  
earlier versions of both gnupg and pgp, have found it to be so,
i don't like or use pgp9.x, so i can't vouch for that version,
but usually,
pgp gets 'more accepting' with every version, not less,
so i would expect it to work there too)

if this workaround 'does work'
then try it again with the linux and unix versions you used

a simple test would be to use a textfile of one line:
12345 followed by 5 blank spaces

check the size of the original textfile, and then
the size of 'saved form' of the armored signed text once it is
verified
and saved as plaintext

if the sizes are the same, the blank spaces have been kept
(this can then be confirmed by copying test line and pasting it
into a new textfile consisting of only one character, making sure
to paste it 'in front' of the character,
the resulting line will have the same number of blank spaces in
front of the single character, as in the original file)


vedaal

--
Click for a credit repair consultation, raise your FICO score.
http://tagline.hushmail.com/fc/Ioyw6h4d7lwT9b30ut4I2EHsT7GVTKJdKW9jIKLl8rgVT7pJOqLxvB/


_______________________________________________
Gnupg-users mailing list
Gnupg-users@...
http://lists.gnupg.org/mailman/listinfo/gnupg-users