Re: Charsets, Gtkmm and MySQL

> Hello,
>
> thanks for the code.
>
> result is an object of mysqlpp::StoreQueryResult in the sql-libs.
> It stores the query data.
> But in the docs of libmysql++ is described, that are the stored data
> handeld as a derived std::string. Also compatible to the original
> std::string.
>
> I inserted your code as you described and it returns the following
> output (2 textviews as example, one with shown data, one with empty)
> The output is taken from console:
>
> :::::::::::::::::: SNIP first example data :::::::::::::::::
>
> ** Message: UTF-8 valid: 0: Demozugang ist der gleiche.
>
> Auch f\xfcr gesch\xe4ftliche Aktivit\xe4ten (Streeting
> Unternehmensvertrieb, siehe Unterlagen).
>
> Vertriebsnummer: 00028
>
> Vertriebsseite:
> URL: http://vertrieb.streeting.com
> Benutzer: xxxx
> Kennwort: xxxx
>
> Abrechnungssystem (im Vertriebslogin):
> Benutzer: xxxx
> Kennwort: xxxx
>
> :::::::::::::::::: SNIP second example data :::::::::::::::::
>
> ** Message: UTF-8 valid: 1: Daten CySoTec:
> --------------
>
> Jabba (XMPP) Messenger Client:
> Benutzer: xxxx
> Kennwort: xxxx
> Server: cysotec.info
> Jabba ID: xrobbi@...
> [...]
>
> :::::::::::::::::: SNIP example data end :::::::::::::::::
>
> As far as I can see, the one with empty data is not UTF-8 valid
> and the other is.
> So, what can I do, to convert the Data from the SQL server to My app in
> a valid UTF-8, or on the other hand,
> what collation should I use in on My SQL server?
>
> Thank you alot
>
> Bernd
>
>
> Am Montag, den 29.12.2008, 00:47 +0100 schrieb Milosz Derezynski:
> > Of what type is 'result'?
> >
> > Also, could you add, before this line:
> >
> >    tmp.push_back((std::string)result[row][i]);
> >
> > this code:
> >
> >    Glib::ustring utf8_str(result[row][i]);
> >    g_message("UTF-8 valid: %d: %s", int(utf8_str.validate()),
> > utf8_str.c_str());
> >
> > and see what it gives?
> >
> >
> > M.
> >
> > P.S.:
> > Yes std::string and ustring are generally compatible, with the main
> > differences being that ustring works by-character not by-byte (e.g.
> > find(), operator[], etc), ustring has methods for validation of UTF-8
> > and related functionality; the main potential culprit here is
> > operator<< and operator>> of ustring. I had really big problems with
> > these using boost::format() until i realised that internally, it's a
> > stream to which the args are fed using operator<<() of ustring.
> >
> >
> > On Mon, Dec 29, 2008 at 12:01 AM, Bernd Robertz
> > <bernd.robertz@...> wrote:
> >         Thanks alot for your fast reply,
> >        
> >         I wrote a simple MySQL-Class wich is derived from the libsql++
> >         class.
> >        
> >         In this class I used std::string to get the columns of a row.
> >         The columns (strings) are stored in a vector<string> wich I
> >         return to
> >         Glib::ustring in My treeview-data-window.
> >        
> >         Hiere are some snippets:
> >        
> >         :::::::::::::::::::::::: SNIP ::::::::::::::::::::::
> >         database.cpp
> >         (My simple MySQL database class.)
> >        
> >         DataBase::Database()
> >         {
> >                // db-connect
> >                // query-setup
> >                // other connect related stuff
> >        
> >                mysqlpp::StoreQueryResult result; // Setup the result
> >         object
> >                int rows; // Keep track of result counts
> >         }
> >        
> >         void DataBase::returnrow(int dataid)
> >         {
> >                // Setup db-query with dataid to get requested db-row
> >                //
> >                mysqlpp::Query query = dbcon.query(queries.alldata); //
> >         DB Query
> >                result = query.store(); // Store the results
> >                rows = result.num_rows(); // Set the number of results
> >         }
> >        
> >         std::vector<std::string> DataBase::fetchdata(int row)
> >         {
> >                std::vector<std::string> tmp; // Temp vector for return
> >        
> >                for (int i = 0; i < result.field_names()->size(); ++i)
> >                {
> >                        // Every column will be stored in the temp
> >         vector
> >                        tmp.push_back((std::string)result[row][i]);
> >                }
> >                return tmp; // Return the temp-vector with the
> >         [columns]=data
> >         }
> >        
> >         ------------
> >        
> >         datawindow.cpp
> >         (The window in where the data is displayed after dbl-click on
> >         treeview)
> >        
> >         DataWindow::DataWindow()
> >         {
> >                // Widgets setup
> >                // Database connect via DataBase-Object
> >         }
> >        
> >         void DataWindow::getid(int dataid)
> >         {
> >                // From the treeview I call this method to set the
> >                // requested row-id
> >                mysql.returnrow(dataid);
> >        
> >                // Call this method to setup the entries and textview
> >                setdata();
> >         }
> >        
> >         void DataWindow::setdata()
> >         {
> >                l_dataid.set_text(mysql.fetchdata(0)[0]);
> >                Gtk::TreeModel::Row selrow = *(ls_cbox->prepend());
> >                selrow[c_mod.c_cat] = mysql.fetchdata(0)[1];
> >                cbox.set_active(0);
> >        
> >                // Here comes the problem, I get every column from My
> >                // database-object wich I store in the entries.
> >                e_name.set_text(mysql.fetchdata(0)[2]);
> >                e_number.set_text(mysql.fetchdata(0)[3]);
> >                e_user.set_text(mysql.fetchdata(0)[4]);
> >                e_pass.set_text(mysql.fetchdata(0)[5]);
> >                e_mail.set_text(mysql.fetchdata(0)[6]);
> >                e_url.set_text(mysql.fetchdata(0)[7]);
> >                e_other.set_text(mysql.fetchdata(0)[8]);
> >        
> >                // Here we have the main problem: the textview-buffer
> >                // Some rows are shown here, some not.
> >                b_descr->set_text(mysql.fetchdata(0)[9]);
> >         }
> >        
> >         ::::::::::::::::::::::::: SNIP
> >         END ::::::::::::::::::::::::::::
> >        
> >         That is the main code where the problem occours.
> >        
> >         The DataWindow is called on doubleclick on a treeview-row.
> >         With the call of the DataWindow I call the DataView::getid()
> >         wich receive the database row-id.
> >         Then I fill with the DataView::setdata() method the widgets
> >         in the new window.
> >        
> >         Some chars are, as I told, glyphs, but the textbuffer is the
> >         main
> >         problem.
> >        
> >         As far as I know, the entries and textbuffers are expecting
> >         Glib::ustring, but I thought the conversion from std::string
> >         to
> >         Glib::ustring is quite compatible between these types.
> >         I used std::string in the database class, because I wanted to
> >         isolated
> >         it from Gtkmm for other not Gtkmm-related projects.
> >        
> >         Thank you for helping.
> >        
> >         Bernd
> >        
> >        
> >         Am Sonntag, den 28.12.2008, 22:26 +0100 schrieb Milosz
> >         Derezynski:
> >        
> >         > Are you using Glib::ustring anywhere in the process of
> >         getting the
> >         > data from the SQL database to the view?
> >         >
> >         > Glib::ustring implicitly (and in my opinion still at least
> >         not well
> >         > enough documented/hinted at) converts from UTF-8
> >         > to the current locale with operators << and >>. If you are
> >         using a
> >         > stringstream anywhere in the process, or are
> >         > using operator<< or operator>> in any other way, this could
> >         then be
> >         > the cause of the problem.
> >         >
> >         > Otherwise, and in the above described case too actually it
> >         would help
> >         > very much if you could post (if possible, all of)
> >         > the relevant code online.
> >         >
> >         > Regards,
> >         > M.
> >         >
> >         > On Sun, Dec 28, 2008 at 8:53 PM, Bernd Robertz
> >         > <bernd.robertz@...> wrote:
> >         >         Hello list,
> >         >
> >         >         I'm not really sure, but I hope My question is in
> >         the right
> >         >         place here.
> >         >
> >         >         I wrote a simple database app like this:
> >         >
> >         >         Connect to a MySQL server via libmysql++,
> >         >         and then list the rows in a treeview.
> >         >         Doubleclick on a row shows a window with some
> >         entries and a
> >         >         textview
> >         >         with the content of the selected row.
> >         >         The textview is associated with a type text in the
> >         DB.
> >         >
> >         >         Everything is fine.
> >         >
> >         >         So but I ran really into problems with the charsets.
> >         >         german umlaute (ö, ä, etc.) and some other
> >         characters are
> >         >         glyphs in the
> >         >         treeview. Ok, thats not the main problem.
> >         >
> >         >         That is: The entries in the detailed view still
> >         shows the
> >         >         glyphs for
> >         >         some chars like ö, ä.
> >         >         But on some rows, the represented textbuffer for
> >         textview is
> >         >         not
> >         >         visible. Wich means the textview is in some rows
> >         empty.
> >         >
> >         >         I run My app from a console and I got these error
> >         messages:
> >         >
> >         >         :::::::::
> >         >         Gtk-CRITICAL **: gtk_text_buffer_emit_insert:
> >         assertion
> >         >         `g_utf8_validate
> >         >         (text, len, NULL)' failed
> >         >         :::::::::
> >         >
> >         >         In other rows, the textbuffer is correct shown,
> >         exepct some
> >         >         glyphs.
> >         >
> >         >         If I print the textbuffer as a simple .c_str() in a
> >         console it
> >         >         will be
> >         >         shown correctly (inklusive ä, ö, ü...).
> >         >         (Tried to show the textview as a .c_str(), without
> >         any
> >         >         effect.)
> >         >
> >         >         The SQL connection is serverside-configured as UTF-8
> >         Unicode.
> >         >         The SQL table collation is UTF-8 General CI (Case
> >         >         Insensitive).
> >         >         The Application runs on a DE_de@Euro (ISO-8859-15).
> >         >
> >         >         All characters are shown correctly in a PHP app wich
> >         does the
> >         >         same, just
> >         >         only online. :-)
> >         >
> >         >         I hope someone could turn on a light above my head,
> >         why I
> >         >         don't get the
> >         >         right charactes.
> >         >
> >         >         Thanks alot in advance
> >         >
> >         >         Bernd
> >         >
> >         >
> >         >         _______________________________________________
> >         >         gtkmm-list mailing list
> >         >         gtkmm-list@...
> >         >         http://mail.gnome.org/mailman/listinfo/gtkmm-list
> >         >
> >         >
> >         >
> >         > --
> >         > Please note that according to the German law on data
> >         retention,
> >         > information on every electronic information exchange with me
> >         is
> >         > retained for a period of six months.
> >         > [Bitte beachten Sie, dass dem Gesetz zur
> >         Vorratsdatenspeicherung
> >         > zufolge
> >         > jeder elektronische Kontakt mit mir sechs Monate lang
> >         gespeichert
> >         > wird.]
> >        
> >        
> >
> >
> >
> > --
> > Please note that according to the German law on data retention,
> > information on every electronic information exchange with me is
> > retained for a period of six months.
> > [Bitte beachten Sie, dass dem Gesetz zur Vorratsdatenspeicherung
> > zufolge
> > jeder elektronische Kontakt mit mir sechs Monate lang gespeichert
> > wird.]
>
> _______________________________________________
> gtkmm-list mailing list
> gtkmm-list@...
> http://mail.gnome.org/mailman/listinfo/gtkmm-list