binary objects

[John-681]

Hi,

Currently I don't believe Dabo supports binary objects.  For example update
does not work.

What steps are suggested to work with binary objects such as images.  Please
don't say "don't" because there are many very valid reasons for storing
images in a DB.

thanks in advance


Johnf


_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/200910271722.42560.jfabiani@...

[Ed Leafe]

On Oct 27, 2009, at 8:22 PM, John wrote:

> Currently I don't believe Dabo supports binary objects.  For example  
> update
> does not work.

        It sure does; I demonstrated that with the original port of the old  
Northwind database and the images that came along with that.

> What steps are suggested to work with binary objects such as  
> images.  Please
> don't say "don't" because there are many very valid reasons for  
> storing
> images in a DB.


        Look in dabo.ui.uiwx.__init__.py for the bitmapFromData() and  
imageFromData() methods. Those should get you started.


-- Ed Leafe





_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/8267D6EF-A573-4829-A3CC-9B7807DC925F@...

[John-681]

On Tuesday 27 October 2009 05:42:57 pm Ed Leafe wrote:
> Look in dabo.ui.uiwx.__init__.py for the bitmapFromData() and  
> imageFromData() methods. Those should get you started.

Wow - I check it out immediately.

Johnf


_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/200910271747.14139.jfabiani@...

[lalong1-2]

Ed,

We can get the image data converted to a string, but when we try to save the
data via the bizobj text data type field, it barfs when daabo tries to
format the data before saving it to the back end.

Is there some other way we need to save the image?

Thanks,
Larry

> -----Original Message-----
> From: dabo-users-bounces@...
> [mailto:dabo-users-bounces@...] On Behalf Of Ed Leafe
> Sent: Tuesday, October 27, 2009 8:43 PM
> To: Dabo Users list
> Subject: Re: [dabo-users] binary objects
>
> On Oct 27, 2009, at 8:22 PM, John wrote:
>
> > Currently I don't believe Dabo supports binary objects.  
> For example
> > update does not work.
>
> It sure does; I demonstrated that with the original
> port of the old Northwind database and the images that came
> along with that.
>
> > What steps are suggested to work with binary objects such
> as images.  
> > Please don't say "don't" because there are many very valid
> reasons for
> > storing images in a DB.
>
>
> Look in dabo.ui.uiwx.__init__.py for the bitmapFromData() and
> imageFromData() methods. Those should get you started.
>
>
> -- Ed Leafe
>
>
>
>
>

[excessive quoting removed by server]

_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/!~!UENERkVCMDkAAQACAAAAAAAAAAAAAAAAABgAAAAAAAAAafA2fnYuPUOMNFpIYnBEQcKAAAAQAAAAatQCuHumZUywwDpy3mVwCgEAAAAA@...

[John-681]

On Tuesday 27 October 2009 05:42:57 pm Ed Leafe wrote:
> On Oct 27, 2009, at 8:22 PM, John wrote:
> > Currently I don't believe Dabo supports binary objects.  For example
> > update
> > does not work.
>
> It sure does; I demonstrated that with the original port of the old
> Northwind database and the images that came along with that.

>
>
> -- Ed Leafe

For the life of me I can't figure out how you were able to get the image data
into the table.  Any chance the code is lying around that demo's the steps.

I was able to create a simple file using img2py.py as suggested by the doc
string in dabo/dabo/ui/uiwx/__init__.py.

I'm able to use the file created (from img2py).  I can display the image data
etc... by using the file created by img2py.


 The trouble comes when I need to save the image data to the database. Either
Dabo complains or I get a database error.  The Dabo attempts to convert the
data to a quoted string and if I attempt to by pass Dabo by making changes
then the database complains that I have an encoding problem.  


So if that code is around I sure would like to see it.

Thanks,

Johnf





_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/200910281453.52760.jfabiani@...

[Paul McNett]

John wrote:

> On Tuesday 27 October 2009 05:42:57 pm Ed Leafe wrote:
>> On Oct 27, 2009, at 8:22 PM, John wrote:
>>> Currently I don't believe Dabo supports binary objects.  For example
>>> update
>>> does not work.
>> It sure does; I demonstrated that with the original port of the old
>> Northwind database and the images that came along with that.
>
>>
>> -- Ed Leafe
>
> For the life of me I can't figure out how you were able to get the image data
> into the table.  Any chance the code is lying around that demo's the steps.
>
> I was able to create a simple file using img2py.py as suggested by the doc
> string in dabo/dabo/ui/uiwx/__init__.py.
>
> I'm able to use the file created (from img2py).  I can display the image data
> etc... by using the file created by img2py.
>
>
>  The trouble comes when I need to save the image data to the database. Either
> Dabo complains or I get a database error.  The Dabo attempts to convert the
> data to a quoted string and if I attempt to by pass Dabo by making changes
> then the database complains that I have an encoding problem.  
>
>
> So if that code is around I sure would like to see it.

Just a quick check: is the field defined as a BLOB in your datastructure? (Type L for
bLob).

Paul


_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/4AE8BEBF.1000109@...

[John-681]

On Wednesday 28 October 2009 02:59:27 pm Paul McNett wrote:

> John wrote:
> > On Tuesday 27 October 2009 05:42:57 pm Ed Leafe wrote:
> >> On Oct 27, 2009, at 8:22 PM, John wrote:
> >>> Currently I don't believe Dabo supports binary objects.  For example
> >>> update
> >>> does not work.
> >>
> >> It sure does; I demonstrated that with the original port of the old
> >> Northwind database and the images that came along with that.
> >>
> >>
> >> -- Ed Leafe
> >
> > For the life of me I can't figure out how you were able to get the image
> > data into the table.  Any chance the code is lying around that demo's the
> > steps.
> >
> > I was able to create a simple file using img2py.py as suggested by the
> > doc string in dabo/dabo/ui/uiwx/__init__.py.
> >
> > I'm able to use the file created (from img2py).  I can display the image
> > data etc... by using the file created by img2py.
> >
> >
> >  The trouble comes when I need to save the image data to the database.
> > Either Dabo complains or I get a database error.  The Dabo attempts to
> > convert the data to a quoted string and if I attempt to by pass Dabo by
> > making changes then the database complains that I have an encoding
> > problem.
> >
> >
> > So if that code is around I sure would like to see it.
>
> Just a quick check: is the field defined as a BLOB in your datastructure?
> (Type L for bLob).
>
> Paul

I tried both 'Text' and 'bytea' (large Binary).  No success.  

Johnf




_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/200910281504.56938.jfabiani@...

[Paul McNett]

John wrote:
> On Wednesday 28 October 2009 02:59:27 pm Paul McNett wrote:
>> Just a quick check: is the field defined as a BLOB in your datastructure?
>> (Type L for bLob).
>>
>
> I tried both 'Text' and 'bytea' (large Binary).  No success.  

I'll ask again: what is the contents of biz.DataStructure? Do you see a datatype code
of "L" for the binary field?

Paul


_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/4AE8C3AD.1080106@...

[John-681]

On Wednesday 28 October 2009 03:20:29 pm Paul McNett wrote:

> John wrote:
> > On Wednesday 28 October 2009 02:59:27 pm Paul McNett wrote:
> >> Just a quick check: is the field defined as a BLOB in your
> >> datastructure? (Type L for bLob).
> >
> > I tried both 'Text' and 'bytea' (large Binary).  No success.
>
> I'll ask again: what is the contents of biz.DataStructure? Do you see a
> datatype code of "L" for the binary field?
>
> Paul

Sorry about that - I immediately thought about what dbPostgres sets the
columns too.  I missed the point yes it's 'L' for the binary as reported by
bizobj.DataSource.  But we have attempted both 'M' and 'L'.  Larry tried all
day to use binary tools (using 'L' for the column).  I tried using string
tools (using 'M' for the column).   In the end Larry tried both.  

Maybe we just haven't come up with the right combination.  

Johnf






_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/200910281602.42567.jfabiani@...

[Paul McNett]

John wrote:

> On Wednesday 28 October 2009 03:20:29 pm Paul McNett wrote:
>> John wrote:
>>> On Wednesday 28 October 2009 02:59:27 pm Paul McNett wrote:
>>>> Just a quick check: is the field defined as a BLOB in your
>>>> datastructure? (Type L for bLob).
>>> I tried both 'Text' and 'bytea' (large Binary).  No success.
>> I'll ask again: what is the contents of biz.DataStructure? Do you see a
>> datatype code of "L" for the binary field?
>>
>> Paul
>
> Sorry about that - I immediately thought about what dbPostgres sets the
> columns too.  I missed the point yes it's 'L' for the binary as reported by
> bizobj.DataSource.  But we have attempted both 'M' and 'L'.  Larry tried all
> day to use binary tools (using 'L' for the column).  I tried using string
> tools (using 'M' for the column).   In the end Larry tried both.  
>
> Maybe we just haven't come up with the right combination.  

Well, I've racked my brain and I don't think I've ever stored binary stuff in a
database, so I can't say for certain that it works.

Paul



_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/4AE8CF92.4010105@...

[John-681]

On Wednesday 28 October 2009 04:11:14 pm Paul McNett wrote:

> John wrote:
> > On Wednesday 28 October 2009 03:20:29 pm Paul McNett wrote:
> >> John wrote:
> >>> On Wednesday 28 October 2009 02:59:27 pm Paul McNett wrote:
> >>>> Just a quick check: is the field defined as a BLOB in your
> >>>> datastructure? (Type L for bLob).
> >>>
> >>> I tried both 'Text' and 'bytea' (large Binary).  No success.
> >>
> >> I'll ask again: what is the contents of biz.DataStructure? Do you see a
> >> datatype code of "L" for the binary field?
> >>
> >> Paul
> >
> > Sorry about that - I immediately thought about what dbPostgres sets the
> > columns too.  I missed the point yes it's 'L' for the binary as reported
> > by bizobj.DataSource.  But we have attempted both 'M' and 'L'.  Larry
> > tried all day to use binary tools (using 'L' for the column).  I tried
> > using string tools (using 'M' for the column).   In the end Larry tried
> > both.
> >
> > Maybe we just haven't come up with the right combination.
>
> Well, I've racked my brain and I don't think I've ever stored binary stuff
> in a database, so I can't say for certain that it works.
>
> Paul

I just realized that I have not actually provided the traceback.  As you can
see I'm just attempting an save (update).  I'm saving to a Dabo "L" column -
and a postgres column type 'bytea' (binary).

The steps I'm using.

I have a file that was created using img2py.py.  

newdata = my_images.jflogosmall.GetData()
# type(newdata) = str
pic = dabo.ui.imageFromData(newdata)
#type(pic) = wx._core.Image
self.Logo.Picture = pic # this work fine
self.PrimaryBizobj.setFieldVal('sys_company_logo1', newdata)
now a simple Form.save()

If I attempt to save the 'pic' as in

self.PrimaryBizobj.setFieldVal('sys_company_logo1', pic)

Then Dabo asks  escQuote() to process the data which of course is wrong.

I don't know what I'm doing wrong?


Traceback (most recent call last):
  File "/home/johnf/dabo/dabo/ui/uiwx/dControlMixin.py", line 27, in _onWxHit
    self.raiseEvent(dEvents.Hit, evt, *args, **kwargs)
  File "/home/johnf/dabo/dabo/ui/uiwx/dPemMixin.py", line 952, in raiseEvent
    super(dPemMixin, self).raiseEvent(eventClass, nativeEvent, *args,
**kwargs)
  File "/home/johnf/dabo/dabo/lib/eventMixin.py", line 93, in raiseEvent
    bindingFunction(event)
  File "/home/johnf/pyProject/court_ordered/SystemClasses.py", line 339, in
onSave
    self.Form.save()
  File "/home/johnf/dabo/dabo/ui/uiwx/dForm.py", line 383, in save
    bizobj.saveAll()
  File "/home/johnf/dabo/dabo/biz/dBizobj.py", line 361, in saveAll
    startTransaction=False)
  File "/home/johnf/dabo/dabo/biz/dBizobj.py", line 786, in scanChangedRows
    func(*args, **kwargs)
  File "/home/johnf/dabo/dabo/biz/dBizobj.py", line 419, in save
    cursor.save()
  File "/home/johnf/dabo/dabo/db/dCursorMixin.py", line 1310, in save
    saverow(self.RowNumber)
  File "/home/johnf/dabo/dabo/db/dCursorMixin.py", line 1277, in saverow
    self.__saverow(row)
  File "/home/johnf/dabo/dabo/db/dCursorMixin.py", line 1376, in __saverow
    updClause, pkWhere)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x89 in position 23:
ordinal not in range(128)

Johnf





_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/200910282034.10073.jfabiani@...

[Paul McNett]

John wrote:

> On Wednesday 28 October 2009 04:11:14 pm Paul McNett wrote:
>> John wrote:
>>> On Wednesday 28 October 2009 03:20:29 pm Paul McNett wrote:
>>>> John wrote:
>>>>> On Wednesday 28 October 2009 02:59:27 pm Paul McNett wrote:
>>>>>> Just a quick check: is the field defined as a BLOB in your
>>>>>> datastructure? (Type L for bLob).
>>>>> I tried both 'Text' and 'bytea' (large Binary).  No success.
>>>> I'll ask again: what is the contents of biz.DataStructure? Do you see a
>>>> datatype code of "L" for the binary field?
>>>>
>>>> Paul
>>> Sorry about that - I immediately thought about what dbPostgres sets the
>>> columns too.  I missed the point yes it's 'L' for the binary as reported
>>> by bizobj.DataSource.  But we have attempted both 'M' and 'L'.  Larry
>>> tried all day to use binary tools (using 'L' for the column).  I tried
>>> using string tools (using 'M' for the column).   In the end Larry tried
>>> both.
>>>
>>> Maybe we just haven't come up with the right combination.
>> Well, I've racked my brain and I don't think I've ever stored binary stuff
>> in a database, so I can't say for certain that it works.
>>
>> Paul
>
> I just realized that I have not actually provided the traceback.  As you can
> see I'm just attempting an save (update).  I'm saving to a Dabo "L" column -
> and a postgres column type 'bytea' (binary).
>
> The steps I'm using.
>
> I have a file that was created using img2py.py.  
>
> newdata = my_images.jflogosmall.GetData()
> # type(newdata) = str
> pic = dabo.ui.imageFromData(newdata)
> #type(pic) = wx._core.Image
> self.Logo.Picture = pic # this work fine
> self.PrimaryBizobj.setFieldVal('sys_company_logo1', newdata)
> now a simple Form.save()
>
> If I attempt to save the 'pic' as in
>
> self.PrimaryBizobj.setFieldVal('sys_company_logo1', pic)
>
> Then Dabo asks  escQuote() to process the data which of course is wrong.
>
> I don't know what I'm doing wrong?
>
>
> Traceback (most recent call last):
>   File "/home/johnf/dabo/dabo/ui/uiwx/dControlMixin.py", line 27, in _onWxHit
>     self.raiseEvent(dEvents.Hit, evt, *args, **kwargs)
>   File "/home/johnf/dabo/dabo/ui/uiwx/dPemMixin.py", line 952, in raiseEvent
>     super(dPemMixin, self).raiseEvent(eventClass, nativeEvent, *args,
> **kwargs)
>   File "/home/johnf/dabo/dabo/lib/eventMixin.py", line 93, in raiseEvent
>     bindingFunction(event)
>   File "/home/johnf/pyProject/court_ordered/SystemClasses.py", line 339, in
> onSave
>     self.Form.save()
>   File "/home/johnf/dabo/dabo/ui/uiwx/dForm.py", line 383, in save
>     bizobj.saveAll()
>   File "/home/johnf/dabo/dabo/biz/dBizobj.py", line 361, in saveAll
>     startTransaction=False)
>   File "/home/johnf/dabo/dabo/biz/dBizobj.py", line 786, in scanChangedRows
>     func(*args, **kwargs)
>   File "/home/johnf/dabo/dabo/biz/dBizobj.py", line 419, in save
>     cursor.save()
>   File "/home/johnf/dabo/dabo/db/dCursorMixin.py", line 1310, in save
>     saverow(self.RowNumber)
>   File "/home/johnf/dabo/dabo/db/dCursorMixin.py", line 1277, in saverow
>     self.__saverow(row)
>   File "/home/johnf/dabo/dabo/db/dCursorMixin.py", line 1376, in __saverow
>     updClause, pkWhere)
> UnicodeDecodeError: 'ascii' codec can't decode byte 0x89 in position 23:
> ordinal not in range(128)

Thanks for the traceback. It really helps. Does this patch help
(untested, and watch for line-wrap):

Index: db/dBackend.py
===================================================================
--- db/dBackend.py (revision 5507)
+++ db/dBackend.py (working copy)
@@ -72,7 +72,7 @@
  return cursorClass(self._connection)


- def formatForQuery(self, val):
+ def formatForQuery(self, val, fieldType=None):
  if isinstance(val, (datetime.date, datetime.datetime)):
  # Some databases have specific rules for formatting date values.
  return self.formatDateTime(val)
@@ -82,6 +82,8 @@
  return str(val)
  elif isinstance(val, dNoEscQuoteStr):
  return val
+ elif fieldType == "L":
+ return val
  elif val is None:
  return self.formatNone()
  else:
Index: db/dbSQLite.py
===================================================================
--- db/dbSQLite.py (revision 5507)
+++ db/dbSQLite.py (working copy)
@@ -75,7 +75,7 @@
  return self._dictCursorClass
 

- def formatForQuery(self, val):
+ def formatForQuery(self, val, fieldType=None):
  if isinstance(val, bool):
  return str(int(val))
  else:
Index: db/dCursorMixin.py
===================================================================
--- db/dCursorMixin.py (revision 5507)
+++ db/dCursorMixin.py (working copy)
@@ -1355,7 +1355,8 @@
  # Append the field and its value.
  flds += ", " + self.BackendObject.encloseNames(kk, aq)
  # add value to expression
- vals += ", %s" % (self.formatForQuery(vv[1]),)
+ fieldType = [ds[1] for ds in self.DataStructure if ds[0] == kk][0]
+ vals += ", %s" % (self.formatForQuery(vv[1]), fieldType)

  # Trim leading comma-space from the strings
  flds = flds[2:]
@@ -1945,7 +1946,8 @@
  continue
  if ret:
  ret += ", "
- ret += tblPrefix + bo.encloseNames(fld, aq) + " = " +
self.formatForQuery(new_val)
+ fieldType = [ds[1] for ds in self.DataStructure if ds[0] == fld][0]
+ ret += tblPrefix + bo.encloseNames(fld, aq) + " = " +
self.formatForQuery(new_val, fieldType)
  return ret

Paul


_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/4AE9126D.50102@...

[lalong1-2]

Paul,

In line 1377 of dCursorMixin where we are building the sql update statement
there is a problem handling binary data.  Converting string data to unicode
won't work for string data for byte values > 128.

For example, u'\x55' is OK but u'\xFF' is not allowed.  

When we do the string substitution, if any string is unicode, then the
entire string is implicitly converted to unicode.  In our case it's
something like ...sql="set %s = %s"%(u'fieldname','\xFF') so it barfs.

I think if we wrap the first parameter with a str() command that it may work
assuming that the field name should never need to be in unicode.

What cha think?
Larry

> -----Original Message-----
> From: dabo-users-bounces@...
> [mailto:dabo-users-bounces@...] On Behalf Of Paul McNett
> Sent: Wednesday, October 28, 2009 11:56 PM
> To: Dabo Users list
> Subject: Re: [dabo-users] binary objects
>
> John wrote:
> > On Wednesday 28 October 2009 04:11:14 pm Paul McNett wrote:
> >> John wrote:
> >>> On Wednesday 28 October 2009 03:20:29 pm Paul McNett wrote:
> >>>> John wrote:
> >>>>> On Wednesday 28 October 2009 02:59:27 pm Paul McNett wrote:
> >>>>>> Just a quick check: is the field defined as a BLOB in your
> >>>>>> datastructure? (Type L for bLob).
> >>>>> I tried both 'Text' and 'bytea' (large Binary).  No success.
> >>>> I'll ask again: what is the contents of
> biz.DataStructure? Do you
> >>>> see a datatype code of "L" for the binary field?
> >>>>
> >>>> Paul
> >>> Sorry about that - I immediately thought about what
> dbPostgres sets
> >>> the columns too.  I missed the point yes it's 'L' for the
> binary as
> >>> reported by bizobj.DataSource.  But we have attempted
> both 'M' and
> >>> 'L'.  Larry tried all day to use binary tools (using 'L'
> for the column).  I tried
> >>> using string tools (using 'M' for the column).   In the
> end Larry tried
> >>> both.
> >>>
> >>> Maybe we just haven't come up with the right combination.
> >> Well, I've racked my brain and I don't think I've ever
> stored binary
> >> stuff in a database, so I can't say for certain that it works.
> >>
> >> Paul
> >
> > I just realized that I have not actually provided the
> traceback.  As
> > you can see I'm just attempting an save (update).  I'm saving to a
> > Dabo "L" column - and a postgres column type 'bytea' (binary).
> >
> > The steps I'm using.
> >
> > I have a file that was created using img2py.py.  
> >
> > newdata = my_images.jflogosmall.GetData() # type(newdata) =
> str pic =
> > dabo.ui.imageFromData(newdata)
> > #type(pic) = wx._core.Image
> > self.Logo.Picture = pic # this work fine
> > self.PrimaryBizobj.setFieldVal('sys_company_logo1', newdata) now a
> > simple Form.save()
> >
> > If I attempt to save the 'pic' as in
> >
> > self.PrimaryBizobj.setFieldVal('sys_company_logo1', pic)
> >
> > Then Dabo asks  escQuote() to process the data which of
> course is wrong.
> >
> > I don't know what I'm doing wrong?
> >
> >
> > Traceback (most recent call last):
> >   File "/home/johnf/dabo/dabo/ui/uiwx/dControlMixin.py",
> line 27, in _onWxHit
> >     self.raiseEvent(dEvents.Hit, evt, *args, **kwargs)
> >   File "/home/johnf/dabo/dabo/ui/uiwx/dPemMixin.py", line
> 952, in raiseEvent
> >     super(dPemMixin, self).raiseEvent(eventClass,
> nativeEvent, *args,
> > **kwargs)
> >   File "/home/johnf/dabo/dabo/lib/eventMixin.py", line 93,
> in raiseEvent
> >     bindingFunction(event)
> >   File "/home/johnf/pyProject/court_ordered/SystemClasses.py", line
> > 339, in onSave
> >     self.Form.save()
> >   File "/home/johnf/dabo/dabo/ui/uiwx/dForm.py", line 383, in save
> >     bizobj.saveAll()
> >   File "/home/johnf/dabo/dabo/biz/dBizobj.py", line 361, in saveAll
> >     startTransaction=False)
> >   File "/home/johnf/dabo/dabo/biz/dBizobj.py", line 786, in
> scanChangedRows
> >     func(*args, **kwargs)
> >   File "/home/johnf/dabo/dabo/biz/dBizobj.py", line 419, in save
> >     cursor.save()
> >   File "/home/johnf/dabo/dabo/db/dCursorMixin.py", line
> 1310, in save
> >     saverow(self.RowNumber)
> >   File "/home/johnf/dabo/dabo/db/dCursorMixin.py", line
> 1277, in saverow
> >     self.__saverow(row)
> >   File "/home/johnf/dabo/dabo/db/dCursorMixin.py", line
> 1376, in __saverow
> >     updClause, pkWhere)
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0x89 in
> position 23:
> > ordinal not in range(128)
>
> Thanks for the traceback. It really helps. Does this patch
> help (untested, and watch for line-wrap):
>
> Index: db/dBackend.py
> ===================================================================
> --- db/dBackend.py (revision 5507)
> +++ db/dBackend.py (working copy)
> @@ -72,7 +72,7 @@
>   return cursorClass(self._connection)
>
>
> - def formatForQuery(self, val):
> + def formatForQuery(self, val, fieldType=None):
>   if isinstance(val, (datetime.date, datetime.datetime)):
>   # Some databases have specific rules
> for formatting date values.
>   return self.formatDateTime(val)
> @@ -82,6 +82,8 @@
>   return str(val)
>   elif isinstance(val, dNoEscQuoteStr):
>   return val
> + elif fieldType == "L":
> + return val
>   elif val is None:
>   return self.formatNone()
>   else:
> Index: db/dbSQLite.py
> ===================================================================
> --- db/dbSQLite.py (revision 5507)
> +++ db/dbSQLite.py (working copy)
> @@ -75,7 +75,7 @@
>   return self._dictCursorClass
>  
>
> - def formatForQuery(self, val):
> + def formatForQuery(self, val, fieldType=None):
>   if isinstance(val, bool):
>   return str(int(val))
>   else:
> Index: db/dCursorMixin.py
> ===================================================================
> --- db/dCursorMixin.py (revision 5507)
> +++ db/dCursorMixin.py (working copy)
> @@ -1355,7 +1355,8 @@
>   # Append the field and
> its value.
>   flds += ", " +
> self.BackendObject.encloseNames(kk, aq)
>   # add value to expression
> - vals += ", %s" %
> (self.formatForQuery(vv[1]),)
> + fieldType = [ds[1] for
> ds in self.DataStructure if ds[0] == kk][0]
> + vals += ", %s" %
> (self.formatForQuery(vv[1]), fieldType)
>
>   # Trim leading comma-space from
> the strings
>   flds = flds[2:]
> @@ -1945,7 +1946,8 @@
>   continue
>   if ret:
>   ret += ", "
> - ret += tblPrefix + bo.encloseNames(fld,
> aq) + " = " +
> self.formatForQuery(new_val)
> + fieldType = [ds[1] for ds in
> self.DataStructure if ds[0] == fld][0]
> + ret += tblPrefix + bo.encloseNames(fld,
> aq) + " = " +
> self.formatForQuery(new_val, fieldType)
>   return ret
>
> Paul
>
>

[excessive quoting removed by server]

_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/!~!UENERkVCMDkAAQACAAAAAAAAAAAAAAAAABgAAAAAAAAAafA2fnYuPUOMNFpIYnBEQcKAAAAQAAAAMVm5xhZiBESuzzoD/+zo+QEAAAAA@...

[Paul McNett]

lalong1@... wrote:

> Paul,
>
> In line 1377 of dCursorMixin where we are building the sql update statement
> there is a problem handling binary data.  Converting string data to unicode
> won't work for string data for byte values > 128.
>
> For example, u'\x55' is OK but u'\xFF' is not allowed.  
>
> When we do the string substitution, if any string is unicode, then the
> entire string is implicitly converted to unicode.  In our case it's
> something like ...sql="set %s = %s"%(u'fieldname','\xFF') so it barfs.
>
> I think if we wrap the first parameter with a str() command that it may work
> assuming that the field name should never need to be in unicode.

I agree that we don't want to try to convert binary data to unicode, especially not
using the ascii codec.

> What cha think?

Did you try my patch?

Paul


_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/4AE9BC04.80807@...

[John-681]

On Thursday 29 October 2009 09:00:04 am Paul McNett wrote:

> lalong1@... wrote:
> > Paul,
> >
> > In line 1377 of dCursorMixin where we are building the sql update
> > statement there is a problem handling binary data.  Converting string
> > data to unicode won't work for string data for byte values > 128.
> >
> > For example, u'\x55' is OK but u'\xFF' is not allowed.
> >
> > When we do the string substitution, if any string is unicode, then the
> > entire string is implicitly converted to unicode.  In our case it's
> > something like ...sql="set %s = %s"%(u'fieldname','\xFF') so it barfs.
> >
> > I think if we wrap the first parameter with a str() command that it may
> > work assuming that the field name should never need to be in unicode.
>
> I agree that we don't want to try to convert binary data to unicode,
> especially not using the ascii codec.
>
> > What cha think?
>
> Did you try my patch?
>
> Paul

first in dCursorMixin.py  formatForQuery() needs fieldType.

def formatForQuery(self, val, fieldType=None):

We have been trying your patch all morning.  We sort of got it pass the sql
update statement by forcing it to str (not in your patch).  But now have
trouble with the database seeing it as a string and not binary info.  

I coming to the conclusion that we need a special method to handle binary data
in the DB adapter.  In the O'Reilly python cookbook it is suggested that
MySQL and Postgres require special functions to insert binary data (other
DB's may need special functions too).  I have confirmed the functions for
Postgres.  MySQL has a function "escape_string" that will accept a pickle to
allow inserts of binary data.

Now that I said the above I have to note that the book is old.  I did also
discover that the following will create the binary data
 picdata = buffer(open('whereareyou.jpg').read())

and a simple statement  

cursor.execute("INSERT INTO test_binary   VALUES ( %(img)s)", picdata)

works outside of Dabo.  I don't need any special functions for Postgres.

I just can't get it to work with Dabo.  

Johnf






_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/200910291022.18219.jfabiani@...

[John-681]

On Thursday 29 October 2009 10:22:18 am John wrote:
> cursor.execute("INSERT INTO test_binary   VALUES ( %(img)s)", picdata)

should be:

cursor.execute("INSERT INTO test_binary   VALUES ( %s)", picdata)



_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/200910291102.38550.jfabiani@...

[Paul McNett]

John wrote:
> On Thursday 29 October 2009 10:22:18 am John wrote:
>> cursor.execute("INSERT INTO test_binary   VALUES ( %(img)s)", picdata)
>
> should be:
>
> cursor.execute("INSERT INTO test_binary   VALUES ( %s)", picdata)

I figured, thanks. Ok... I just did some testing of my own, and can confirm that BLOB
updating and inserting isn't working. I used SQLite, using this script:

{{{
import dabo
from dabo.lib import getRandomUUID

picdata = buffer(open("ss/resources/icon_green_16px.png").read())

con = dabo.db.connect(":memory:")
cur = con.cursor()
cur.execute("create table test (id PRIMARY KEY, img BLOB)")

biz = dabo.biz.dBizobj(con, DataSource="test", KeyField="id")
biz.DefaultValues["id"] = getRandomUUID

biz.new()
biz.Record.img = picdata

print len(biz.Record.img)

biz.save()
biz.requery()

print len(biz.Record.img)

}}}

I went ahead and committed my patch, since it doesn't appear to break anything and I
think it is a step in the right direction.

Actually, I think we need to bite the bullet and convert our INSERT and UPDATE
statements to send the parameters to the dbapi drivers, instead of converting to
string ourselves. Carl had a patch for that a couple years ago, that couldn't be
applied because right after he submitted it, we made a lot of incompatible changes to
dCursorMixin, and we never got an updated patch from him.

I think that BLOB support is important, and am surprised it's gone this long without
anyone noticing it's been broken.

Please start a ticket.

Paul


_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/4AE9DFA0.7010703@...

[John-681]

On Thursday 29 October 2009 11:32:00 am Paul McNett wrote:

> {{{
> import dabo
> from dabo.lib import getRandomUUID
>
> picdata = buffer(open("ss/resources/icon_green_16px.png").read())
>
> con = dabo.db.connect(":memory:")
> cur = con.cursor()
> cur.execute("create table test (id PRIMARY KEY, img BLOB)")
>
> biz = dabo.biz.dBizobj(con, DataSource="test", KeyField="id")
> biz.DefaultValues["id"] = getRandomUUID
>
> biz.new()
> biz.Record.img = picdata
>
> print len(biz.Record.img)
>
> biz.save()
> biz.requery()
>
> print len(biz.Record.img)
>
> }}}

At least it's good to know that Larry and I were not complete.  I created a
ticket.

Thanks for you help.

Johnf


_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/200910291842.23086.jfabiani@...

[John-681]

On Thursday 29 October 2009 11:32:00 am Paul McNett wrote:

> I went ahead and committed my patch, since it doesn't appear to break
> anything and I think it is a step in the right direction.
>
> Actually, I think we need to bite the bullet and convert our INSERT and
> UPDATE statements to send the parameters to the dbapi drivers, instead of
> converting to string ourselves. Carl had a patch for that a couple years
> ago, that couldn't be applied because right after he submitted it, we made
> a lot of incompatible changes to dCursorMixin, and we never got an updated
> patch from him.
>
> I think that BLOB support is important, and am surprised it's gone this
> long without anyone noticing it's been broken.

Paul, Larry was able to get Dabo to accept the binary data.  However, we
followed the idea that each DB adapter had a "fixBinaryData()" method that
made the correct conversion to allow Dabo to save/update/insert.  Other code
changes were done too.  But my point is I still think in the end having a
backend method is going to be a requirment.

For example in the case of Postgres

def fixBinaryData(self, val):
  return psycopg2.Binary(val)



FireBird has something called BLOB_load()

Oracle appears to several different types of binary data types each with
different functions to store the data.

MsSQL has several ways too.  And so does MySQL.

I don't how we can avoid the not using the special backend methods.

Your thoughts?

Johnf










_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/200910300910.12178.jfabiani@...

[Paul McNett]

John wrote:

> On Thursday 29 October 2009 11:32:00 am Paul McNett wrote:
>> I went ahead and committed my patch, since it doesn't appear to break
>> anything and I think it is a step in the right direction.
>>
>> Actually, I think we need to bite the bullet and convert our INSERT and
>> UPDATE statements to send the parameters to the dbapi drivers, instead of
>> converting to string ourselves. Carl had a patch for that a couple years
>> ago, that couldn't be applied because right after he submitted it, we made
>> a lot of incompatible changes to dCursorMixin, and we never got an updated
>> patch from him.
>>
>> I think that BLOB support is important, and am surprised it's gone this
>> long without anyone noticing it's been broken.
>
> Paul, Larry was able to get Dabo to accept the binary data.  However, we
> followed the idea that each DB adapter had a "fixBinaryData()" method that
> made the correct conversion to allow Dabo to save/update/insert.  Other code
> changes were done too.  But my point is I still think in the end having a
> backend method is going to be a requirment.
>
> For example in the case of Postgres
>
> def fixBinaryData(self, val):
>   return psycopg2.Binary(val)
>
>
>
> FireBird has something called BLOB_load()
>
> Oracle appears to several different types of binary data types each with
> different functions to store the data.
>
> MsSQL has several ways too.  And so does MySQL.
>
> I don't how we can avoid the not using the special backend methods.
>
> Your thoughts?

This sounds like a quick and easy way to get binary support working. Ed?

Paul



_______________________________________________
Post Messages to: Dabo-users@...
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/4AEB12E8.1090603@...