Django 1.1 on IronPython 2.6rc2 findings

[Markus Törnqvist]

Hi!

I got my code to run far enough on IronPython that it sets and clears
the test cookies, stores sessions in MSSQL, executes a lot of queries,
 but dies eventually on
"Process is terminated due to StackOverflowException" :(

However, as the new IPy version was released like yesterday (thanks guys!)
I figured I'd try it out.

So I installed 2.6rc2, but unfortunately it didn't help.

At this point I run into my skill deficiency; as I'm presented with the
options to Debug or Close and Debug doesn't tell me anything, I dunno
how to debug this.

Any hints?

Someone else wanna confirm this?

PS.
Here is how I had to hack generic.py, it's the modification to the beginning
of the execute function:

    def execute(self, operation, parameters=None, parameter_names=None):
        ## Take from mssql.py because of the wrong cursor seems to get initialized
        #"""
        parameter_names = None
        if parameters:
            parameter_names = tuple('@p%i' % i for i in range(len(parameters)))
            operation = operation % parameter_names
        #"""    
        print "adonet2dbapi: executing '" + str(operation) + "' with '" + str(parameters) + "'"

Here's the connect function (in its entirety) from mssql.py

def connect(connstr):
    relevant_parts = [part for part in connstr.split(';') if not part.upper().startswith('PROVIDER')]
    connstr = ';'.join(relevant_parts)
    #return generic_connect("System.Data", "System.Data.SqlClient.SqlConnection", connstr)
    # Fake signature of generic_connect!
    assembly = "System.Data"
    typename = "System.Data.SqlClient.SqlConnection"
    data = connstr
    factory = None
    # Import relevant definitions
    from generic import _build_string, _load_type
    if isinstance(data, dict):
        data = _build_string(data)
    connector = _load_type(assembly, typename)
    connection = connector(data)
    retval = Connection(connection) if factory is None else factory(connection)
    print 'returning', retval
    return retval

Way too hacky for my taste but it gets the job kinda done.

--
mjt

_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Seo Sanghyeon-3]

Intro... I, together with Mark Rees, wrote most of adonet-dbapi code in 2006.

2009/10/28 Markus Törnqvist <mjt@...>:
> Here's the connect function (in its entirety) from mssql.py
>
> def connect(connstr):
>    relevant_parts = [part for part in connstr.split(';') if not part.upper().startswith('PROVIDER')]
>    connstr = ';'.join(relevant_parts)
>    #return generic_connect("System.Data", "System.Data.SqlClient.SqlConnection", connstr)
>
> Way too hacky for my taste but it gets the job kinda done.

Indeed. The question is, what underlying Python DB-API 2 driver does
django-mssql use? Is it http://pymssql.sourceforge.net/ ?

Argument to connect() function in Python DB-API 2 compliant drivers is
not standardized. In case of pymssql, it receives keyword arguments,
so it should be handled in a manner similar to MySQLdb.py wrapper in
adonet-dbapi. But from your hack it seems django-mssql's underlying
Python DB-API 2 driver expects a string argument, so django-mssql is
building string by itself -- this is ungood. Read psycopg.py wrapper
in adonet-dbapi to see how to handle this.

On the other hand, as I understand, Django does not handle paramstyle
difference in DB-API 2 drivers, so this is kind of a lost cause. How
do other Django DB backends deal with the issue? SQLAlchemy,
fortunately, handles paramstyle difference by itself.

--
Seo Sanghyeon
_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Markus Törnqvist]

On Wed, Oct 28, 2009 at 10:53:45PM +0900, Seo Sanghyeon wrote:
>Intro... I, together with Mark Rees, wrote most of adonet-dbapi code in 2006.

Hi :)

>2009/10/28 Markus Törnqvist <mjt@...>:
>> Here's the connect function (in its entirety) from mssql.py

>> def connect(connstr):
>>    relevant_parts = [part for part in connstr.split(';') if not part.upper().startswith('PROVIDER')]
>>    connstr = ';'.join(relevant_parts)
>>    #return generic_connect("System.Data", "System.Data.SqlClient.SqlConnection", connstr)

>> Way too hacky for my taste but it gets the job kinda done.
>
>Indeed. The question is, what underlying Python DB-API 2 driver does
>django-mssql use? Is it http://pymssql.sourceforge.net/ ?

It's on adonet-dbapi...

>Argument to connect() function in Python DB-API 2 compliant drivers is
>not standardized. In case of pymssql, it receives keyword arguments,
>so it should be handled in a manner similar to MySQLdb.py wrapper in
>adonet-dbapi. But from your hack it seems django-mssql's underlying
>Python DB-API 2 driver expects a string argument, so django-mssql is
>building string by itself -- this is ungood. Read psycopg.py wrapper
>in adonet-dbapi to see how to handle this.

Ok, something like that could be used instead.

>On the other hand, as I understand, Django does not handle paramstyle
>difference in DB-API 2 drivers, so this is kind of a lost cause. How
>do other Django DB backends deal with the issue? SQLAlchemy,
>fortunately, handles paramstyle difference by itself.

I'd have to create the param dict from the connect string, very
much like your psycopg :)

Django is intended to use the dbapi, I don't think there's any other
way of handling this?

Anyway, I think the StackOverflowException is a bigger issue :)

--
mjt

_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Jeff Hardy-4]

On Wed, Oct 28, 2009 at 7:53 AM, Seo Sanghyeon <sanxiyn@...> wrote:
>
> Indeed. The question is, what underlying Python DB-API 2 driver does
> django-mssql use? Is it http://pymssql.sourceforge.net/ ?

It uses an old, modified version to Vernon Cole's adodb library that
it ships with itself.

> On the other hand, as I understand, Django does not handle paramstyle
> difference in DB-API 2 drivers, so this is kind of a lost cause. How
> do other Django DB backends deal with the issue? SQLAlchemy,
> fortunately, handles paramstyle difference by itself.

Django always uses %s-style parameters; each backend converts that to
whatever the database driver wants (qmarks, @name, whatever).

- Jeff
_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Jeff Hardy-4]

2009/10/28 Markus Törnqvist <mjt@...>:
> At this point I run into my skill deficiency; as I'm presented with the
> options to Debug or Close and Debug doesn't tell me anything, I dunno
> how to debug this.
>
> Any hints?

If you run ipy with the -D flag (or set system.compilation/debug=true
for NWSGI) then IronPython will emit debug symbols; otherwise you
won't be able to actually debug the Python code. You can then attach
Visual Studio to the process (which clicking "Debug" should do) and
step through the Python code. Note that debugging isn't always
perfect, but it at least gives you an idea where to look.

- Jeff
_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Vernon Cole-3]

Markus:
  Would you be so kind as to zip up and email to me the adonet-dbapi as you now have it patched?
I will create a version which does not error out. (May take a few days.)
--
Vernon

2009/10/28 Markus Törnqvist <mjt@...>

On Wed, Oct 28, 2009 at 10:53:45PM +0900, Seo Sanghyeon wrote:
>Intro... I, together with Mark Rees, wrote most of adonet-dbapi code in 2006.

Hi :)

>2009/10/28 Markus Törnqvist <mjt@...>:
>> Here's the connect function (in its entirety) from mssql.py

>> def connect(connstr):
>>    relevant_parts = [part for part in connstr.split(';') if not part.upper().startswith('PROVIDER')]
>>    connstr = ';'.join(relevant_parts)
>>    #return generic_connect("System.Data", "System.Data.SqlClient.SqlConnection", connstr)

>> Way too hacky for my taste but it gets the job kinda done.
>
>Indeed. The question is, what underlying Python DB-API 2 driver does
>django-mssql use? Is it http://pymssql.sourceforge.net/ ?

It's on adonet-dbapi...

>Argument to connect() function in Python DB-API 2 compliant drivers is
>not standardized. In case of pymssql, it receives keyword arguments,
>so it should be handled in a manner similar to MySQLdb.py wrapper in
>adonet-dbapi. But from your hack it seems django-mssql's underlying
>Python DB-API 2 driver expects a string argument, so django-mssql is
>building string by itself -- this is ungood. Read psycopg.py wrapper
>in adonet-dbapi to see how to handle this.

Ok, something like that could be used instead.

>On the other hand, as I understand, Django does not handle paramstyle
>difference in DB-API 2 drivers, so this is kind of a lost cause. How
>do other Django DB backends deal with the issue? SQLAlchemy,
>fortunately, handles paramstyle difference by itself.

I'd have to create the param dict from the connect string, very
much like your psycopg :)

Django is intended to use the dbapi, I don't think there's any other
way of handling this?

Anyway, I think the StackOverflowException is a bigger issue :)

--
mjt

_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Dave Fugate]

Just a hunch, but you might want to try passing "-X:MaxRecursion 1000" to ipy.exe.  By default, IronPython has a very high recursion limit compared to CPython.  On occasion this can cause the entire interactive session/process to die with a StackOverflowException rather than emit a catchable Python RuntimeError.

The other thing is if you're running under an x64 OS, try ipy.exe instead of ipy64.exe.  ipy.exe is purely a 32-bit CLR process and hence needs less memory.

Hope that helps,

Dave

-----Original Message-----
From: users-bounces@... [mailto:users-bounces@...] On Behalf Of Jeff Hardy
Sent: Wednesday, October 28, 2009 7:30 AM
To: Discussion of IronPython
Subject: Re: [IronPython] Django 1.1 on IronPython 2.6rc2 findings

2009/10/28 Markus Törnqvist <mjt@...>:
> At this point I run into my skill deficiency; as I'm presented with the
> options to Debug or Close and Debug doesn't tell me anything, I dunno
> how to debug this.
>
> Any hints?

If you run ipy with the -D flag (or set system.compilation/debug=true
for NWSGI) then IronPython will emit debug symbols; otherwise you
won't be able to actually debug the Python code. You can then attach
Visual Studio to the process (which clicking "Debug" should do) and
step through the Python code. Note that debugging isn't always
perfect, but it at least gives you an idea where to look.

- Jeff
_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Jeff Hardy-4]

On Wed, Oct 28, 2009 at 10:23 AM, Vernon Cole <vernondcole@...> wrote:
> Markus:
>   Would you be so kind as to zip up and email to me the adonet-dbapi as you
> now have it patched?

A better option IMO would be to create a fork on bitbucket and push
changes there. That way I can pull back into my existing repo.

> I will create a version which does not error out. (May take a few days.)

How much overlap is there between your other adodb project and
adonet-dbapi? I don't think there's any need to have two libraries
that do the same thing for IronPython, so we might as well figure out
what's different and work towards one library.

- Jeff
_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Seo Sanghyeon-3]

2009/10/29 Jeff Hardy <jdhardy@...>:
> How much overlap is there between your other adodb project and
> adonet-dbapi? I don't think there's any need to have two libraries
> that do the same thing for IronPython, so we might as well figure out
> what's different and work towards one library.

My understanding is that adodb uses COM, so it can't be used with Mono
on Linux. That seals my position against any merge.

--
Seo Sanghyeon
_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Vernon Cole-3]

Yes, adodbapi does use COM.  My intention is to make an ADO.NET version of it, but I want to make one which will be a near drop in replacement for the fork which has been adapted for django -- which also uses COM. My idea is to make the django changes first, before making  the .NET changes. That way, if I must fork the code for .NET, the new version will have the same features as the COM version. I want to add Mono to the list of systems adodbapi will run on. The existing Linux ODBC module leaves a bit to be desired, and something better may be
needed to make MS-SQL access work on Linux.
--
Vernon Cole

On Wed, Oct 28, 2009 at 4:02 PM, Seo Sanghyeon <sanxiyn@...> wrote:

2009/10/29 Jeff Hardy <jdhardy@...>:

> How much overlap is there between your other adodb project and
> adonet-dbapi? I don't think there's any need to have two libraries
> that do the same thing for IronPython, so we might as well figure out
> what's different and work towards one library.

My understanding is that adodb uses COM, so it can't be used with Mono
on Linux. That seals my position against any merge.

--
Seo Sanghyeon

_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Vernon Cole-3]

Jeff:
   Thank you for your kind suggestion to use bitbucket.  I don't think that would be a good idea.  I already have to keep up three distributions of adodbapi -- on pywin32 and FePy as well as its own project. A fourth would be a bit too much. I would prefer to include django capability in the trunk, if possible.
--
Vernon

On Wed, Oct 28, 2009 at 11:15 AM, Jeff Hardy <jdhardy@...> wrote:

On Wed, Oct 28, 2009 at 10:23 AM, Vernon Cole <vernondcole@...> wrote:
> Markus:
>   Would you be so kind as to zip up and email to me the adonet-dbapi as you
> now have it patched?

A better option IMO would be to create a fork on bitbucket and push
changes there. That way I can pull back into my existing repo.

> I will create a version which does not error out. (May take a few days.)

How much overlap is there between your other adodb project and
adonet-dbapi? I don't think there's any need to have two libraries
that do the same thing for IronPython, so we might as well figure out
what's different and work towards one library.

- Jeff

_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Jeff Hardy-4]

On Thu, Oct 29, 2009 at 12:07 AM, Vernon Cole <vernondcole@...> wrote:
> Jeff:
>    Thank you for your kind suggestion to use bitbucket.  I don't think that
> would be a good idea.  I already have to keep up three distributions of
> adodbapi -- on pywin32 and FePy as well as its own project. A fourth would
> be a bit too much. I would prefer to include django capability in the trunk,
> if possible.

Hi Vernon,
Sorry, I wasn't clear - I meant any changes to adonet-dbapi, which is
already on bitbucket. I agree that moving adodbapi doean't make any
sense.

- Jeff
_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Vernon Cole-3]

I am cross posting here my inquiry sent to db-sig about how to change paramstyles, along with the only response.

If you feel that option #1 below is NOT the way to go, please say something now.
--
Vernon Cole

M.-A. Lemburg ✆  to me, db-sig

show details Sep 26

Vernon Cole wrote:
> I am planning to add, as an extension, the ability for adodbapi to change
> its paramstyle.  I personally like "named" much better than "qmark", and it
> happens that django expects to have "format", so I will add that, too.
>
> My question is, what would be an appropriate way to specify the altered
> paramstyle?
>
> 1) As an attribute of the connection:
>     con = adodbapi.connect('spam eggs")
>     con.paramstyle = "named"

Most other configuration parameters are made available on
connections and cursors (with the cursor setting overriding the
connection one), so I think that's the most DB-API
compatible way of implementing this.

> 2) Make the module global mutable:
>    adodbapi.adodbapi.paramstyle = "named"  # [ ick! ]
>
> 3) As a named parameter of the connection:
>   con = adodbapi.connect("spam eggs", paramstyle="named")
>
> Has anyone else done this?
> How did they do it?
> How will some speculated future db-api version 3.0 want it?

Probably in the way you mentioned above.

The module global would then provide to the default setting.

--
Marc-Andre Lemburg
eGenix.com

 

_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Seo Sanghyeon-3]

2009/10/30 Vernon Cole <vernondcole@...>

> I am cross posting here my inquiry sent to db-sig about how to change paramstyles, along with the only response.
>
> If you feel that option #1 below is NOT the way to go, please say something now.
>
>> M.-A. Lemburg ✆
>> Vernon Cole wrote:
>> > My question is, what would be an appropriate way to specify the altered
>> > paramstyle?
>> >
>> > 1) As an attribute of the connection:
>> >     con = adodbapi.connect('spam eggs")
>> >     con.paramstyle = "named"
>>
>> Most other configuration parameters are made available on
>> connections and cursors (with the cursor setting overriding the
>> connection one), so I think that's the most DB-API
>> compatible way of implementing this.

I agree with this. Sensible.

--
Seo Sanghyeon
_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com