Extended characters in a web service result

[Johan_Braun]

Does anybody have a clue why this result do not show as it should? I do not
know if this is a strange behavior from 4D or from the web service itself.

We do a call to a in-house developed web-service that does a lookup in the
Active Directory. The soap call is working ok and we get the expected result
exept when we try to receive extended characters like ÅÄÖ....

The web service is of the "complex" type so we create a xml request document
send it to the web service and the receive the result as a XML.document.
Everything works fine until the web service transfers extended characters
like the Swedish characters å ä and ö (ÅÄÖ) Well everything above char code
127 gets wrong.

Like the following:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<ns2:getShortInfoInternalResponse xmlns:ns2="
http://main.accounts.alfamoving.se/">
  <return>
    <Attributes>
      <Attribute key="mailNickname" value="anjo1"/>
      <Attribute key="displayName" value="Anette J√∂rnevi"/>
      <Attribute key="memberOf" value="CN=Ovsersea,OU=AlfaAssist,OU=Central
IT,DC=alfamoving,DC=com"/>
      <Attribute key="givenName" value="Anette"/>
      <Attribute key="name" value="Anette J√∂rnevi"/> (This SHOULD be Anette
Jörnevi)
      <Attribute key="sn" value="J√∂rnevi"/>
      <Attribute key="mail" value="Anette.Jornevi@..."/>
    </Attributes>
  </return>
</ns2:getShortInfoInternalResponse>




The actual call to the webservice looks like
CALL WEB SERVICE("http://192.168.10.81:8080/accounts/
main";"";"getShortInfoInternal";"http://main.accounts.alfamoving.se/";Web
Service Manual )

When we test the webservice outside 4D everything looks OK and we get the
response we expect with the right characters.It seems like we receive a four
byte character as the character "ö" instead of a two byte character as UTF-8
should give you. Why do we get this strange mix between UTF-8 and UTF-16
when we are inside 4D and not when we call the webservice manual?


Best regards

Johan Braun
Generera Datautveckling AB
**********************************************************************
Get the speed and power of 4D v11 SQL
before upgrade prices increase - http://www.4d.com


4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4D.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4D_Tech-Unsubscribe@...
**********************************************************************

[Johan_Braun]

>
> BUMP!
>
 Have anyone had a problem with receiving extended chars in a web service
call and been able to do anything to get a correct solution?


>
> Best regards
>
> Johan Braun
> Generera Datautveckling AB
>
>
>
**********************************************************************
Get the speed and power of 4D v11 SQL
before upgrade prices increase - http://www.4d.com
   

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4D.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4D_Tech-Unsubscribe@...
**********************************************************************

[MIYAKO]

Hi,

you posted your CALL code,
would it be possible to ask for the GET code,
the DOM code (if any),
Unicode mode (if applicable)
and version information (sounds like v11 but just be sure) ?

miyako

On 2009/11/09, at 21:59, Johan Braun wrote:

> CALL WEB SERVICE("http://192.168.10.81:8080/accounts/
> main";"";"getShortInfoInternal";"http://
> main.accounts.alfamoving.se/";Web
> Service Manual )

**********************************************************************
Get the speed and power of 4D v11 SQL
before upgrade prices increase - http://www.4d.com
   

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4D.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4D_Tech-Unsubscribe@...
**********************************************************************

[Johan_Braun]

Thank you for your response Miyako. I had the pleasure to attend to one of
your Unicode classes at the devcon in LA last year. I have to say, i was
stunned by your knowledge in these matters. An excellent class.

Yes i am sorry. I was of hoping that someone instantly could say what was
wrong and what kind of stupid faults i have made. Here comes the full call
to the web service and with the creation of the request, the request itself
and the extract of result. As i said before, i get the expected result
except that all extended characters is looking strange. (It seems like i
receive 4 bytes characters that 4D interprets into two 2 byte characters).

And yes Miyako, we are running v11.4 hf6.

        `Preparation of the call
         $name:="anjo1"

>        $namespace:="http://main.accounts.alfamoving.se/"
>        $root:=DOM Create XML Ref("main:getShortInfoInternal";$namespace)
>        $subelem:=DOM Create XML element($root;"username")
>        DOM SET XML ELEMENT VALUE($subelem;$name)
>        SET WEB SERVICE PARAMETER("XMLIn";$root)
>
>       `Call the web service
>        CALL WEB SERVICE("http://192.168.10.81:8080/accounts/main";"";"
> getShortInfoInternal";"http://main.accounts.alfamoving.se/";Web Service
> Manual )
>
>        If (OK=1) & (lError=0)
>                `Get the result
>                GET WEB SERVICE RESULT($blob;"XMLOut";*)
>
>                $xmlSource:=DOM Parse XML variable($blob)

              `And here comes the rest of the XML parsing.....
**********************************************************************
Get the speed and power of 4D v11 SQL
before upgrade prices increase - http://www.4d.com
   

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4D.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4D_Tech-Unsubscribe@...
**********************************************************************

[MIYAKO]

Hello Johan,

could it be possible that the returned xml lacks an encoding="..."  
declaration
thus misleading the DOM Parse command?

what do you see if you BLOB TO DOCUMENT the xml and open it with a  
text editor?

miyako

P.S.

not sure if it helps but I do have an Internet Command wrapper for  
making a soap call.
(you need to construct your own soap envelope...)

miyako

http://forums.4d.fr/4DBB_Main/x_User/298210/files/3095213.zip

   `Preparation of the call
$name:="anjo1"

$namespace:="http://main.accounts.alfamoving.se/"
$root:=DOM Create XML Ref("main:getShortInfoInternal";$namespace)
$subelem:=DOM Create XML element($root;"username")
DOM SET XML ELEMENT VALUE($subelem;$name)

C_BLOB($xml)
DOM EXPORT TO VAR($root;$xml)
DOM CLOSE XML($root)

$endpoint_t:="http://192.168.10.81:8080/accounts/main"
$soap_action_t:=""
$soap_name_t:="getShortInfoInternal"

$0:=soap call ($endpoint_t;$soap_name_t;$xml)

On 2009/11/09, at 21:59, Johan Braun wrote:

> know if this is a strange behavior from 4D or from the web service  
> itself.

**********************************************************************
Get the speed and power of 4D v11 SQL
before upgrade prices increase - http://www.4d.com
   

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4D.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4D_Tech-Unsubscribe@...
**********************************************************************

[Johan_Braun]

The resulting XML do have the encoding tags as i should expect.
This is the first lines of code that i receive.

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<ns2:getShortInfoInternalResponse xmlns:ns2="
http://main.accounts.alfamoving.se/">
  <return>
    <Attributes>
      <Attribute key="mailNickname" value="anjo1"/>
      <Attribute key="sn" value="J√∂rnevi"/> `Should be Jörnevi ....

Could this have something to do with base64 encoding? If we run the
webservice in another application outside 4D we get the expected result.
i will have a look at your example.


2009/11/10 miyako <miyako@...>

> Hello Johan,
>
> could it be possible that the returned xml lacks an encoding="..."
> declaration
> thus misleading the DOM Parse command?
>
> what do you see if you BLOB TO DOCUMENT the xml and open it with a text
> editor?
>
> miyako
>
> P.S.
>
> not sure if it helps but I do have an Internet Command wrapper for making a
> soap call.
> (you need to construct your own soap envelope...)
>
> miyako
>
> http://forums.4d.fr/4DBB_Main/x_User/298210/files/3095213.zip
>
>
>  `Preparation of the call
> $name:="anjo1"
>
> $namespace:="http://main.accounts.alfamoving.se/"
> $root:=DOM Create XML Ref("main:getShortInfoInternal";$namespace)
> $subelem:=DOM Create XML element($root;"username")
> DOM SET XML ELEMENT VALUE($subelem;$name)
>
> C_BLOB($xml)
> DOM EXPORT TO VAR($root;$xml)
> DOM CLOSE XML($root)
>
> $endpoint_t:="http://192.168.10.81:8080/accounts/main"
> $soap_action_t:=""
> $soap_name_t:="getShortInfoInternal"
>
> $0:=soap call ($endpoint_t;$soap_name_t;$xml)
>
>
> On 2009/11/09, at 21:59, Johan Braun wrote:
>
>  know if this is a strange behavior from 4D or from the web service itself.
>>
>
> **********************************************************************
> Get the speed and power of 4D v11 SQL
> before upgrade prices increase - http://www.4d.com
>
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4D.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4D_Tech-Unsubscribe@...
> **********************************************************************
>

**********************************************************************
Get the speed and power of 4D v11 SQL
before upgrade prices increase - http://www.4d.com


4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4D.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4D_Tech-Unsubscribe@...
**********************************************************************

[MIYAKO]

Than you Johan,

there was another report on the Forums,
which seems to suggest that GET WEB SERVICE RESULT
with Manual Out might be making a BLOB to text-ish legacy call
(it said the text is trimmed to 32000)
which also means that the encoding is dependent on the system  
language. (but never utf-8)

I think we can confirm whether this is the case by making a simple  
call to 4D itself.

I will keep you posted...

miyako

On 2009/11/11, at 0:05, Johan Braun wrote:

> The resulting XML do have the encoding tags as i should expect.
> This is the first lines of code that i receive.
**********************************************************************
Get the speed and power of 4D v11 SQL
before upgrade prices increase - http://www.4d.com
   

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4D.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4D_Tech-Unsubscribe@...
**********************************************************************

[MIYAKO]

Hello Johan,

I was able to create a sample that illustrates the problem.  

http://forums.4d.fr/4DBB_Main/x_User/298210/files/3097349.zip

you can compare 4D vs 4D Internet Commands.

run the method,
see how extended characters are affected by the database language.
(preferences/database/interational)

in your case, changing it to "english" will solve the corrupt characters...
but of course, that is not a answer you expect.

I think GET WEB SERVICE RESULT should be independent of the database language,
and use "UTF-8" by default,

especially when
A : Unicode mode is on,
and
B :the response header has an explicit encoding declaration.

yes, it will fail if the soap was not utf-8,
but that kind of API is an increasing minority,
and even if that was the case (like Amazon),
modern services match the encoding with that of the request.

P.S.
you might wonder why the command doesn't return the raw data as blob,
apparently the it needs to convert the response xml, to strip the soap envelope.

miyako

On 2009/11/11, at 0:05, Johan Braun wrote:

> The resulting XML do have the encoding tags as i should expect.
> This is the first lines of code that i receive.

**********************************************************************
Get the speed and power of 4D v11 SQL
before upgrade prices increase - http://www.4d.com


4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4D.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4D_Tech-Unsubscribe@...
**********************************************************************

[Johan_Braun]

Tanks for you help Miyako. It is obvious that something is very wrong  
here. I find it very strange that i am the only one that have found  
this problem. There must be some workaround. Yes of course should not  
"GET WEB SERVICE RESULT" return anything else than the response from  
the call itself. I guess this is another example where 4D is to  
"helpful" and stumble on its own feets.

However i have tried your more hardcore call to the webservice and so  
far i could not get that to work either. The response i get is the  
following:

HTTP/1.1 500 Internal Server Error
Server: Apache-Coyote/1.1
Content-Type: text/xml;charset=UTF-8
Content-Length: 324
Date: Wed, 11 Nov 2009 09:24:41 GMT
Connection: close

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/ 
envelope/"><soap:Body><soap:Fault><faultcode>soap:VersionMismatch</
faultcode><faultstring>"http://main.accounts.alfamoving.se/", the  
namespace on the "getShortInfoInternal" element, is not a valid SOAP  
version.</faultstring></soap:Fault></soap:Body></soap:Envelope>




Johan Braun, Senior consultant and partner.

Generera Datautveckling AB
j.braun@...
[phone]

11 nov 2009 kl. 06.06 skrev miyako:

> Hello Johan,
>
> I was able to create a sample that illustrates the problem.
>
> http://forums.4d.fr/4DBB_Main/x_User/298210/files/3097349.zip
>
> you can compare 4D vs 4D Internet Commands.
>
> run the method,
> see how extended characters are affected by the database language.
> (preferences/database/interational)
>
> in your case, changing it to "english" will solve the corrupt  
> characters...
> but of course, that is not a answer you expect.
>
> I think GET WEB SERVICE RESULT should be independent of the database  
> language,
> and use "UTF-8" by default,
>
> especially when
> A : Unicode mode is on,
> and
> B :the response header has an explicit encoding declaration.
>
> yes, it will fail if the soap was not utf-8,
> but that kind of API is an increasing minority,
> and even if that was the case (like Amazon),
> modern services match the encoding with that of the request.
>
> P.S.
> you might wonder why the command doesn't return the raw data as blob,
> apparently the it needs to convert the response xml, to strip the  
> soap envelope.
>
> miyako
>
> On 2009/11/11, at 0:05, Johan Braun wrote:
>
>> The resulting XML do have the encoding tags as i should expect.
>> This is the first lines of code that i receive.
>
> **********************************************************************
> Get the speed and power of 4D v11 SQL
> before upgrade prices increase - http://www.4d.com
>
>
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4D.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4D_Tech-Unsubscribe@...
> **********************************************************************

**********************************************************************
Get the speed and power of 4D v11 SQL
before upgrade prices increase - http://www.4d.com
   

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4D.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4D_Tech-Unsubscribe@...
**********************************************************************

[MIYAKO]

Hi Johan,

the SOAP Header has an extra level in 1.2, I think,
I assumed your SOAP server was 1.1...
I will try and fix the method to be compatible with either version.
thank you for the feed back.

http://www.4d.com/docs/CMU/CMU00901.HTM

miyako

On 2009/11/11, at 18:35, Johan Braun wrote:

> <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><soap:Fault><faultcode>soap:VersionMismatch</faultcode><faultstring>"http://main.accounts.alfamoving.se/", the namespace on the "getShortInfoInternal" element, is not a valid SOAP version.</faultstring></soap:Fault></soap:Body></soap:Envelope>
**********************************************************************
Get the speed and power of 4D v11 SQL
before upgrade prices increase - http://www.4d.com


4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4D.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4D_Tech-Unsubscribe@...
**********************************************************************

[MIYAKO]

Johan,

here is link to the w3c documentation on "soap:VersionMismatch"

http://www.w3.org/TR/2001/WD-soap12-part1-20011002/#faultcodes

meanwhile the problem is reported to engineering as;

ACI0063949: GET WEB SERVICE RESULT (Web Service Manual) changes extended characters

Regards,

miyako

On 2009/11/11, at 18:35, Johan Braun wrote:

> <faultcode>soap:VersionMismatch</faultcode><faultstring>"http://main.accounts.alfamoving.se/", the namespace on the "getShortInfoInternal" element, is not a valid SOAP version.</faultstring>
**********************************************************************
Get up to $600 to spend on Amazon.com
this holiday season - http://www.4d.com/serverpromo.html


4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4D.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4D_Tech-Unsubscribe@...
**********************************************************************

[Johan_Braun]

Tank you Miyako


Johan Braun, Senior consultant and partner.

Generera Datautveckling AB
j.braun@...
[phone]

12 nov 2009 kl. 06.30 skrev miyako:

> Johan,
>
> here is link to the w3c documentation on "soap:VersionMismatch"
>
> http://www.w3.org/TR/2001/WD-soap12-part1-20011002/#faultcodes
>
> meanwhile the problem is reported to engineering as;
>
> ACI0063949: GET WEB SERVICE RESULT (Web Service Manual) changes  
> extended characters
>
> Regards,
>
> miyako
>
> On 2009/11/11, at 18:35, Johan Braun wrote:
>
>> <faultcode>soap:VersionMismatch</faultcode><faultstring>"http://main.accounts.alfamoving.se/ 
>> ", the namespace on the "getShortInfoInternal" element, is not a  
>> valid SOAP version.</faultstring>
> **********************************************************************
> Get up to $600 to spend on Amazon.com
> this holiday season - http://www.4d.com/serverpromo.html
>
>
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4D.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4D_Tech-Unsubscribe@...
> **********************************************************************

**********************************************************************
Get up to $600 to spend on Amazon.com
this holiday season - http://www.4d.com/serverpromo.html
   

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4D.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4D_Tech-Unsubscribe@...
**********************************************************************

[Johan_Braun]

Miyako!
Just so i understand your right so i can have the proper discussion with the developer of the web service. What you have found is that 4D will always interpret the returning XML with the current characterset of the database. So if we run the database in unicode mode (as we do) 4D will internally interpret the return XML as UTF-16. This as long as the returning XML does not have and explicit declaration of the character set. 4D will NOT interpret the return soap envelope accordingly.

This is the header of the return soap request

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/xml;charset=UTF-8
Content-Length: 2070
Date: Tue, 17 Nov 2009 13:01:59 GM

Johan Braun, Senior consultant and partner.

Generera Datautveckling AB
j.braun@...
[phone]

10 nov 2009 kl. 14.56 skrev miyako:

> Hello Johan,
>
> could it be possible that the returned xml lacks an encoding="..." declaration
> thus misleading the DOM Parse command?
>
> what do you see if you BLOB TO DOCUMENT the xml and open it with a text editor?
>
> miyako
>
> P.S.
>
> not sure if it helps but I do have an Internet Command wrapper for making a soap call.
> (you need to construct your own soap envelope...)
>
> miyako
>
> http://forums.4d.fr/4DBB_Main/x_User/298210/files/3095213.zip
>
>  `Preparation of the call
> $name:="anjo1"
>
> $namespace:="http://main.accounts.alfamoving.se/"
> $root:=DOM Create XML Ref("main:getShortInfoInternal";$namespace)
> $subelem:=DOM Create XML element($root;"username")
> DOM SET XML ELEMENT VALUE($subelem;$name)
>
> C_BLOB($xml)
> DOM EXPORT TO VAR($root;$xml)
> DOM CLOSE XML($root)
>
> $endpoint_t:="http://192.168.10.81:8080/accounts/main"
> $soap_action_t:=""
> $soap_name_t:="getShortInfoInternal"
>
> $0:=soap call ($endpoint_t;$soap_name_t;$xml)
>
> On 2009/11/09, at 21:59, Johan Braun wrote:
>
>> know if this is a strange behavior from 4D or from the web service itself.
>
> **********************************************************************
> Get the speed and power of 4D v11 SQL
> before upgrade prices increase - http://www.4d.com
>
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4D.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4D_Tech-Unsubscribe@...
> **********************************************************************

**********************************************************************
Get up to $600 to spend on Amazon.com
this holiday season - http://www.4d.com/serverpromo.html


4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4D.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4D_Tech-Unsubscribe@...
**********************************************************************