API URL encoding

[M Lubratt]

Hello!

I have an application that communicates with my LSMB server via the POST method (with a content type of "x-www-form-urlencoded") to the general ledger moduel (gl.pl).  I'm having difficulty with some of my URL encoding and how LSMB is decoding my POST data.  For example...  The following POST data works and my transaction is correctly posted.

login=TradingRecord&password=joseph1n3&path=bin/mozilla&action=post&transdate=10-31-2009&reference=10312009-Pit&description=Daily%20Pit%20Activity&accno_1=1051--Open%20Equity%20-%20Pit%20(unrealized)&debit_1=300.00&accno_2=4470--Change%20in%20Pit%20Open%20Equity&credit_2=300.00&accno_3=5770--Subscriptions&debit_3=25&accno_4=1054--Pit%20Trading%20Account&credit_4=25&rowcount=4

However, this POST data produces the error shown below eventhough it should be equivalent, if not even more proper than the above URL.

login=TradingRecord&password=joseph1n3&path=bin%2Fmozilla&action=post&transdate=10%2D31%2D2009&reference=10312009%2DPit&description=Daily+Pit+Activity&accno%5F1=1051%2D%2DOpen+Equity+%2D+Pit+%28unrealized%29&debit%5F1=300.00&accno%5F2=4470%2D%2DChange+in+Pit+Open+Equity&credit%5F2=300.00&accno%5F3=5770%2D%2DSubscriptions&debit%5F3=25&accno%5F4=1054%2D%2DPit+Trading+Account&credit%5F4=25&rowcount=4

DBD::Pg::st execute failed: ERROR:  null value in column "chart_id" violates not-null constraint

Error!

INSERT INTO acc_trans 

           (trans_id, chart_id, amount, 

           transdate, source, project_id, 

           fx_transaction, memo, cleared)

   VALUES  (?, (SELECT id

                  FROM chart

                 WHERE accno = ?),

          ?, ?, ?, ?, ?, ?, ?)

ERROR:  null value in column "chart_id" violates not-null constraint

This tells me that the LedgerSMB/Form.pm module isn't properly decoding the URL encoding.  I found the following lines in LedgerSMB/Form.pm:

$str =~ tr/+/ /;

$str =~ s/%([0-9a-fA-Z]{2})/pack("c",hex($1))/eg;

First, I think the regexp should be [0-9a-fA-F] and not [0-9a-fA-Z].  Anyway, this would appear to properly decode the URL.  But, for some reason it doesn't.  If I replace the "%5F"s in the problem URL and replace all of them with "_".  Then the above URL works.  

So, it would appear that somewhere in the LSMB code "%5F" isn't being properly decoded.

Can someone please help me?

Thanks!

Mark

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Ledger-smb-users mailing list
Ledger-smb-users@...
https://lists.sourceforge.net/lists/listinfo/ledger-smb-users

[Luke-74]

Two things:

1.  It might be helpful to put this on the devel list.

2.  One thing you could do, is to try printing the contents of the post
data before it experiences any processing.  Keep moving the commands to do
that deeper and deeper into the program cycle, until you find where it is
changed.  Maybe send it out to a debug file.

You could start right before the replacement code you quoted, and again
right after, and see what the various data looks like.

I had noticed that sometimes SQL-Ledger (on which I used the API far more
regularly) did unexpected things with certain "proper" encodings.

Just for kicks, have you tried processing this through bin/lynx instead of
bin/mozilla?  It probably doesn't make a difference for this part of the
process, but anything's worth a try.

Luke


 On Sat, 24 Oct
2009, M Lubratt wrote:

> Hello!
>
> I have an application that communicates with my LSMB server via the POST
> method (with a content type of "x-www-form-urlencoded") to the general
> ledger moduel (gl.pl).  I'm having difficulty with some of my URL encoding
> and how LSMB is decoding my POST data.  For example...  The following POST
> data works and my transaction is correctly posted.
>
> login=TradingRecord&password=joseph1n3&path=bin/mozilla&action=post&transdate=10-31-2009&reference=10312009-Pit&description=Daily%20Pit%20Activity&accno_1=1051--Open%20Equity%20-%20Pit%20(unrealized)&debit_1=300.00&accno_2=4470--Change%20in%20Pit%20Open%20Equity&credit_2=300.00&accno_3=5770--Subscriptions&debit_3=25&accno_4=1054--Pit%20Trading%20Account&credit_4=25&rowcount=4
>
> However, this POST data produces the error shown below eventhough it should
> be equivalent, if not even more proper than the above URL.
>
> login=TradingRecord&password=joseph1n3&path=bin%2Fmozilla&action=post&transdate=10%2D31%2D2009&reference=10312009%2DPit&description=Daily+Pit+Activity&accno%5F1=1051%2D%2DOpen+Equity+%2D+Pit+%28unrealized%29&debit%5F1=300.00&accno%5F2=4470%2D%2DChange+in+Pit+Open+Equity&credit%5F2=300.00&accno%5F3=5770%2D%2DSubscriptions&debit%5F3=25&accno%5F4=1054%2D%2DPit+Trading+Account&credit%5F4=25&rowcount=4
>
> DBD::Pg::st execute failed: ERROR:  null value in column "chart_id" violates
> not-null constraint
> Error!
> INSERT INTO acc_trans
>             (trans_id, chart_id, amount,
>             transdate, source, project_id,
>             fx_transaction, memo, cleared)
>     VALUES  (?, (SELECT id
>                    FROM chart
>                   WHERE accno = ?),
>            ?, ?, ?, ?, ?, ?, ?)
> ERROR:  null value in column "chart_id" violates not-null constraint
>
> This tells me that the LedgerSMB/Form.pm module isn't properly decoding the
> URL encoding.  I found the following lines in LedgerSMB/Form.pm:
>
>
> $str =~ tr/+/ /;
> $str =~ s/%([0-9a-fA-Z]{2})/pack("c",hex($1))/eg;
>
> First, I think the regexp should be [0-9a-fA-F] and not [0-9a-fA-Z].
>  Anyway, this would appear to properly decode the URL.  But, for some reason
> it doesn't.  If I replace the "%5F"s in the problem URL and replace all of
> them with "_".  Then the above URL works.
>
> So, it would appear that somewhere in the LSMB code "%5F" isn't being
> properly decoded.
>
> Can someone please help me?
>
> Thanks!
> Mark
>

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Ledger-smb-users mailing list
Ledger-smb-users@...
https://lists.sourceforge.net/lists/listinfo/ledger-smb-users

[John Locke]

Hi, Mark,

Can't speak to whether it's more correct or not, but the LSMB (and SQL
Ledger) API is not all that robust. I think you're right--the problem is
%5F, the _ character.

And the reason is that it's in the key values you pass:

acct%5F1
acct%5F2

... these aren't getting decoded, and they're used to determine which
chart to apply the transaction to.

My guess is that there's no URL decoding going on in the keys, only the
values. So don't URL-escape the keys! Use acct_1, acct_2 instead, and
URL-encode the data. Same thing applies to debit_* and credit_*.

Cheers,
John Locke
http://freelock.com

-------- Original Message  --------
Subject: [Ledger-smb-users] API URL encoding
From: M Lubratt <mplubratt@...>
To: ledger-smb-users <ledger-smb-users@...>
Date: Sat 24 Oct 2009 06:42:05 PM PDT

> Hello!
>
> I have an application that communicates with my LSMB server via the
> POST method (with a content type of "x-www-form-urlencoded") to the
> general ledger moduel (gl.pl <http://gl.pl>).  I'm having difficulty
> with some of my URL encoding and how LSMB is decoding my POST data.
>  For example...  The following POST data works and my transaction is
> correctly posted.
>
> login=TradingRecord&password=joseph1n3&path=bin/mozilla&action=post&transdate=10-31-2009&reference=10312009-Pit&description=Daily%20Pit%20Activity&accno_1=1051--Open%20Equity%20-%20Pit%20(unrealized)&debit_1=300.00&accno_2=4470--Change%20in%20Pit%20Open%20Equity&credit_2=300.00&accno_3=5770--Subscriptions&debit_3=25&accno_4=1054--Pit%20Trading%20Account&credit_4=25&rowcount=4
>
> However, this POST data produces the error shown below eventhough it
> should be equivalent, if not even more proper than the above URL.
>
> login=TradingRecord&password=joseph1n3&path=bin%2Fmozilla&action=post&transdate=10%2D31%2D2009&reference=10312009%2DPit&description=Daily+Pit+Activity&accno%5F1=1051%2D%2DOpen+Equity+%2D+Pit+%28unrealized%29&debit%5F1=300.00&accno%5F2=4470%2D%2DChange+in+Pit+Open+Equity&credit%5F2=300.00&accno%5F3=5770%2D%2DSubscriptions&debit%5F3=25&accno%5F4=1054%2D%2DPit+Trading+Account&credit%5F4=25&rowcount=4
>
> DBD::Pg::st execute failed: ERROR:  null value in column "chart_id"
> violates not-null constraint
> Error!
> INSERT INTO acc_trans
>            (trans_id, chart_id, amount,
>            transdate, source, project_id,
>            fx_transaction, memo, cleared)
>    VALUES  (?, (SELECT id
>                   FROM chart
>                  WHERE accno = ?),
>           ?, ?, ?, ?, ?, ?, ?)
> ERROR:  null value in column "chart_id" violates not-null constraint
>
> This tells me that the LedgerSMB/Form.pm module isn't properly
> decoding the URL encoding.  I found the following lines in
> LedgerSMB/Form.pm:
>
>
> $str =~ tr/+/ /;
> $str =~ s/%([0-9a-fA-Z]{2})/pack("c",hex($1))/eg;
>
> First, I think the regexp should be [0-9a-fA-F] and not [0-9a-fA-Z].
>  Anyway, this would appear to properly decode the URL.  But, for some
> reason it doesn't.  If I replace the "%5F"s in the problem URL and
> replace all of them with "_".  Then the above URL works.  
>
> So, it would appear that somewhere in the LSMB code "%5F" isn't being
> properly decoded.
>
> Can someone please help me?
>
> Thanks!
> Mark
>
>
>
> !DSPAM:4ae3aec7248233678446365!
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
>
> !DSPAM:4ae3aec7248233678446365!
>  
> ------------------------------------------------------------------------
>
> _______________________________________________
> Ledger-smb-users mailing list
> Ledger-smb-users@...
> https://lists.sourceforge.net/lists/listinfo/ledger-smb-users
>
>
> !DSPAM:4ae3aec7248233678446365!
>  

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Ledger-smb-users mailing list
Ledger-smb-users@...
https://lists.sourceforge.net/lists/listinfo/ledger-smb-users

[M Lubratt]

Hello, John,

Interesting thought.  And looking at Form.pm, it looks like you're right!  The 'unscaping' looks like it happens after the URL is split up into a hash.  It should happen before.  I'm going to try to work up a patch for the devel folks.

Thanks!

Mark

On Sun, Oct 25, 2009 at 12:54 PM, John Locke <mail@...> wrote:

Hi, Mark,

Can't speak to whether it's more correct or not, but the LSMB (and SQL
Ledger) API is not all that robust. I think you're right--the problem is
%5F, the _ character.

And the reason is that it's in the key values you pass:

acct%5F1
acct%5F2

... these aren't getting decoded, and they're used to determine which
chart to apply the transaction to.

My guess is that there's no URL decoding going on in the keys, only the
values. So don't URL-escape the keys! Use acct_1, acct_2 instead, and
URL-encode the data. Same thing applies to debit_* and credit_*.

Cheers,
John Locke
http://freelock.com

-------- Original Message  --------
Subject: [Ledger-smb-users] API URL encoding
From: M Lubratt <mplubratt@...>
To: ledger-smb-users <ledger-smb-users@...>
Date: Sat 24 Oct 2009 06:42:05 PM PDT
> Hello!
>
> I have an application that communicates with my LSMB server via the
> POST method (with a content type of "x-www-form-urlencoded") to the

> general ledger moduel (gl.pl <http://gl.pl>).  I'm having difficulty
> with some of my URL encoding and how LSMB is decoding my POST data.
>  For example...  The following POST data works and my transaction is
> correctly posted.
>
> login=TradingRecord&password=joseph1n3&path=bin/mozilla&action=post&transdate=10-31-2009&reference=10312009-Pit&description=Daily%20Pit%20Activity&accno_1=1051--Open%20Equity%20-%20Pit%20(unrealized)&debit_1=300.00&accno_2=4470--Change%20in%20Pit%20Open%20Equity&credit_2=300.00&accno_3=5770--Subscriptions&debit_3=25&accno_4=1054--Pit%20Trading%20Account&credit_4=25&rowcount=4
>
> However, this POST data produces the error shown below eventhough it
> should be equivalent, if not even more proper than the above URL.
>
> login=TradingRecord&password=joseph1n3&path=bin%2Fmozilla&action=post&transdate=10%2D31%2D2009&reference=10312009%2DPit&description=Daily+Pit+Activity&accno%5F1=1051%2D%2DOpen+Equity+%2D+Pit+%28unrealized%29&debit%5F1=300.00&accno%5F2=4470%2D%2DChange+in+Pit+Open+Equity&credit%5F2=300.00&accno%5F3=5770%2D%2DSubscriptions&debit%5F3=25&accno%5F4=1054%2D%2DPit+Trading+Account&credit%5F4=25&rowcount=4
>
> DBD::Pg::st execute failed: ERROR:  null value in column "chart_id"
> violates not-null constraint
> Error!
> INSERT INTO acc_trans
>            (trans_id, chart_id, amount,
>            transdate, source, project_id,
>            fx_transaction, memo, cleared)
>    VALUES  (?, (SELECT id
>                   FROM chart
>                  WHERE accno = ?),
>           ?, ?, ?, ?, ?, ?, ?)
> ERROR:  null value in column "chart_id" violates not-null constraint
>
> This tells me that the LedgerSMB/Form.pm module isn't properly
> decoding the URL encoding.  I found the following lines in
> LedgerSMB/Form.pm:
>
>
> $str =~ tr/+/ /;
> $str =~ s/%([0-9a-fA-Z]{2})/pack("c",hex($1))/eg;
>
> First, I think the regexp should be [0-9a-fA-F] and not [0-9a-fA-Z].
>  Anyway, this would appear to properly decode the URL.  But, for some
> reason it doesn't.  If I replace the "%5F"s in the problem URL and
> replace all of them with "_".  Then the above URL works.
>
> So, it would appear that somewhere in the LSMB code "%5F" isn't being
> properly decoded.
>
> Can someone please help me?
>
> Thanks!
> Mark
>
>
>

> !DSPAM:4ae3aec7248233678446365!
> ------------------------------------------------------------------------

>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
>

> !DSPAM:4ae3aec7248233678446365!
>
> ------------------------------------------------------------------------

>
> _______________________________________________
> Ledger-smb-users mailing list
> Ledger-smb-users@...
> https://lists.sourceforge.net/lists/listinfo/ledger-smb-users
>
>

> !DSPAM:4ae3aec7248233678446365!

>



------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Ledger-smb-users mailing list
Ledger-smb-users@...
https://lists.sourceforge.net/lists/listinfo/ledger-smb-users

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Ledger-smb-users mailing list
Ledger-smb-users@...
https://lists.sourceforge.net/lists/listinfo/ledger-smb-users