Seg fault in x64dll.asm in x64 Release configuration

[Andrew Bromba]

Hello,

I'm experiencing strange behaviour using AES 256Bit encryption in my
VS2005SP1 Professional projects. I'm running a Core 2 Quad Q6600 with
Windows Vista Business 64Bit. My crypto++ calling code looks like
this:

CFB_Mode<AES>::Encryption aes(keyBuffer, keyBufferLength, IV);

(using CBC_Mode leads to the same result). Everything works fine in
x64 Debug version, but in x64 Release version a segmentation fault
occurs in x64dll.asm on line 93. The problem seems to be, that

mov rdi, QWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]

in line 69 fills "rdi" with 0x4079dbf000000040  instead of
0x0000000000000040. Also note, that the seg fault occurs upon
instantiation of the AES encryption object (see stack trace).

I saw that g_cacheLineSize is a word32 - could this be the problem? Am
I missing a preprocessor definition?

Inserting the line

printf("c: %x %x %x\n", *(((unsigned int *) &g_cacheLineSize) - 1),
(unsigned int) g_cacheLineSize, *(((unsigned int *) &g_cacheLineSize)
+ 1));

in rijndael.cpp:974 revealed, that 0x4079dbf0 is right next to
g_cacheLineSize == 0x00000040 in memory, the output on the command
line was
"c: 0 40 4079dbe0".

What am I doing wrong?

Kind regards,
Andrew Bromba


Stack trace:
> iddedit.exe!CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(const unsigned char * inBlocks=0x0000000000a2f260, const unsigned char * xorBlocks=0x0000000000000000, unsigned char * outBlocks=0x0000000000a2f2b0, unsigned __int64 length=0x0000000000000010, unsigned int flags=0x00000000)  Line 977 C++
  iddedit.exe!CryptoPP::Rijndael::Enc::ProcessAndXorBlock(const
unsigned char * inBlock=0x0000000000a2f260, const unsigned char *
xorBlock=0x0000000000000000, unsigned char *
outBlock=0x0000000000a2f2b0)  Line 289 C++
  iddedit.exe!CryptoPP::CFB_ModePolicy::TransformRegister()  Line 52 C+
+
  iddedit.exe!CryptoPP::CFB_ModePolicy::CipherResynchronize(const
unsigned char * iv=0x000000000012f660, unsigned __int64
length=0x0000000000000010)  Line 61 C++
  iddedit.exe!
CryptoPP::CFB_CipherTemplate<CryptoPP::AbstractPolicyHolder<CryptoPP::CFB_CipherAbstractPolicy,CryptoPP::CFB_ModePolicy>
>::UncheckedSetKey(const unsigned char * key=0x0000000000a2b870,
unsigned int length=0x00000020, const CryptoPP::NameValuePairs &
params={...})  Line 165 C++
  iddedit.exe!CryptoPP::SimpleKeyingInterface::SetKey(const unsigned
char * key=0x0000000000a2b870, unsigned __int64
length=0x0000000000000020, const CryptoPP::NameValuePairs & params=
{...})  Line 57 C++
  iddedit.exe!
CryptoPP::CipherModeFinalTemplate_CipherHolder<CryptoPP::BlockCipherFinal<0,CryptoPP::Rijndael::Enc>,CryptoPP::ConcretePolicyHolder<CryptoPP::Empty,CryptoPP::CFB_EncryptionTemplate<CryptoPP::AbstractPolicyHolder<CryptoPP::CFB_CipherAbstractPolicy,CryptoPP::CFB_ModePolicy>
>,CryptoPP::CFB_CipherAbstractPolicy>
>::CipherModeFinalTemplate_CipherHolder<CryptoPP::BlockCipherFinal<0,CryptoPP::Rijndael::Enc>,CryptoPP::ConcretePolicyHolder<CryptoPP::Empty,CryptoPP::CFB_EncryptionTemplate<CryptoPP::AbstractPolicyHolder<CryptoPP::CFB_CipherAbstractPolicy,CryptoPP::CFB_ModePolicy>
>,CryptoPP::CFB_CipherAbstractPolicy> >(const unsigned char *
key=0x0000000000a2b870, unsigned __int64 length=0x0000000000000020,
const unsigned char * iv=0x000000000012f660)  Line 264 + 0xae bytes C+
+
...




--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the "Crypto++ Users" Google Group.
To unsubscribe, send an email to cryptopp-users-unsubscribe@....
More information about Crypto++ and this group is available at http://www.cryptopp.com.
-~----------~----~----~----~------~----~------~--~---

[Andrew Bromba]

Hello again,

changing the line 69 in x64dll.asm from

mov rdi, QWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]

to

mov edi, DWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]

solved the problem for me. Please confirm that what I am doing makes
sense, because I am not used to assembly language:

- g_cacheLineSize is a word32 -> DWORD PTR [...] tells the compiler
about it
- edi is 32Bit access to rdi register -> the high order bits in rdi
are zero-padded which is what I want

Best regards,

Andrew Bromba
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the "Crypto++ Users" Google Group.
To unsubscribe, send an email to cryptopp-users-unsubscribe@....
More information about Crypto++ and this group is available at http://www.cryptopp.com.
-~----------~----~----~----~------~----~------~--~---

[Wei Dai]

Yes, that should be the right fix. Thanks for reporting this bug and
providing the fix.

--------------------------------------------------
From: "Andrew Bromba" <Andrew.Bromba@...>
Sent: Friday, July 03, 2009 2:59 AM
To: "Crypto++ Users" <cryptopp-users@...>
Subject: Re: Seg fault in x64dll.asm in x64 Release configuration

>
> Hello again,
>
> changing the line 69 in x64dll.asm from
>
> mov rdi, QWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]
>
> to
>
> mov edi, DWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]
>
> solved the problem for me. Please confirm that what I am doing makes
> sense, because I am not used to assembly language:
>
> - g_cacheLineSize is a word32 -> DWORD PTR [...] tells the compiler
> about it
> - edi is 32Bit access to rdi register -> the high order bits in rdi
> are zero-padded which is what I want
>
> Best regards,
>
> Andrew Bromba
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the "Crypto++ Users" Google Group.
To unsubscribe, send an email to cryptopp-users-unsubscribe@....
More information about Crypto++ and this group is available at http://www.cryptopp.com.
-~----------~----~----~----~------~----~------~--~---

[Robert Roessler]

Wei Dai wrote:
> Yes, that should be the right fix. Thanks for reporting this bug and
> providing the fix.

Is this change going into the SVN repo?

> --------------------------------------------------
> From: "Andrew Bromba"<Andrew.Bromba@...>
> Sent: Friday, July 03, 2009 2:59 AM
> To: "Crypto++ Users"<cryptopp-users@...>
> Subject: Re: Seg fault in x64dll.asm in x64 Release configuration
>
>>
>> Hello again,
>>
>> changing the line 69 in x64dll.asm from
>>
>> mov rdi, QWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]
>>
>> to
>>
>> mov edi, DWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]
>>
>> solved the problem for me. Please confirm that what I am doing makes
>> sense, because I am not used to assembly language:

Robert

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the "Crypto++ Users" Google Group.
To unsubscribe, send an email to cryptopp-users-unsubscribe@....
More information about Crypto++ and this group is available at http://www.cryptopp.com.
-~----------~----~----~----~------~----~------~--~---

[Wei Dai]

I've just checked in the fix.

--------------------------------------------------
From: "Robert Roessler" <roessler@...>
Sent: Sunday, July 05, 2009 12:45 PM
To: "Wei Dai" <weidai@...>
Cc: "Crypto++ Users" <cryptopp-users@...>
Subject: Re: Seg fault in x64dll.asm in x64 Release configuration

>
> Wei Dai wrote:
>> Yes, that should be the right fix. Thanks for reporting this bug and
>> providing the fix.
>
> Is this change going into the SVN repo?
>
>> --------------------------------------------------
>> From: "Andrew Bromba"<Andrew.Bromba@...>
>> Sent: Friday, July 03, 2009 2:59 AM
>> To: "Crypto++ Users"<cryptopp-users@...>
>> Subject: Re: Seg fault in x64dll.asm in x64 Release configuration
>>
>>>
>>> Hello again,
>>>
>>> changing the line 69 in x64dll.asm from
>>>
>>> mov rdi, QWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]
>>>
>>> to
>>>
>>> mov edi, DWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]
>>>
>>> solved the problem for me. Please confirm that what I am doing makes
>>> sense, because I am not used to assembly language:
>
> Robert
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the "Crypto++ Users" Google Group.
To unsubscribe, send an email to cryptopp-users-unsubscribe@....
More information about Crypto++ and this group is available at http://www.cryptopp.com.
-~----------~----~----~----~------~----~------~--~---