The tale of a full HybridMapping and a not so tempfile

[Martin (gzlist)]

I'm trying to complete some work getting a big project running on
IronPython, but hit an issue where everything in the test suite would
start failing after a certain point with errors along the lines of:

Traceback (most recent call last):
...
  File "C:\Program Files\IronPython 2.6\Lib\tempfile.py", line 444, in
NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
  File "C:\Program Files\IronPython 2.6\Lib\tempfile.py", line 228, in
_mkstemp_inner
    fd = _os.open(file, flags, 0600)
OSError: [Errno -2146233079] HybridMapping is full

After a long boring trek over the hills of ignorance that may have
been shorter if I'd seen that traceback first rather than other more
obscure ones, the root cause turned out to be a failure to clean up
temporary files. After IronPython has created 4096 of them and not
removed any afterwards everything involving filenos starts to
throwing. (As an aside, I wonder if using non-integer unique tokens
like Jython mightn't be a better approach).

The following issue tracker entry covers the tempfile issue:

<http://ironpython.codeplex.com/WorkItem/View.aspx?WorkItemId=23936>

However the bug report gets it a little wrong, the problem is not that
sys.platform == "win32" is false under IronPython, but rather that
os.name == "nt" is still true, but IronPython silently ignores the
os.O_TEMPORARY flag.

A trivial work around is:

    if sys.platform == "cli":
        os.name = "nty"
        import tempfile
        os.name = "nt"

However fixing this properly is also pretty simple, in
Src/IronPython.Modules/nt.cs the open function just needs to make sure
that File.Open gets passed DeleteOnClose. There may as well be a
FileOptionsFromFlags as per the existing FileModeFromFlags and
FileAccessFromFlags so that the following mappings are correct as
well:

os.O_SHORT_LIVED -> ? -> FILE_ATTRIBUTE_TEMPORARY
os.O_TEMPORARY -> FileOptions.DeleteOnClose -> FILE_FLAG_DELETE_ON_CLOSE
os.O_RANDOM -> FileOptions.RandomAccess -> FILE_FLAG_RANDOM_ACCESS
os.O_SEQUENTIAL -> FileOptions.SequentialScan -> FILE_FLAG_SEQUENTIAL_SCAN

Martin
_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Dino Viehland]

Thanks for the great detail here.  As you said the fix is easy and I believe
I have it ready to go.  We're already kicking off our RC2 build so I don't
know that it'll make it until that build (or 2.6.0 for that matter) but it'll
be fixed by 2.6.1 at the latest.

> -----Original Message-----
> From: users-bounces@... [mailto:users-
> bounces@...] On Behalf Of Martin (gzlist)
> Sent: Thursday, October 22, 2009 2:50 PM
> To: users@...
> Subject: [IronPython] The tale of a full HybridMapping and a not so tempfile
>
> I'm trying to complete some work getting a big project running on
> IronPython, but hit an issue where everything in the test suite would
> start failing after a certain point with errors along the lines of:
>
> Traceback (most recent call last):
> ...
>   File "C:\Program Files\IronPython 2.6\Lib\tempfile.py", line 444, in
> NamedTemporaryFile
>     (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
>   File "C:\Program Files\IronPython 2.6\Lib\tempfile.py", line 228, in
> _mkstemp_inner
>     fd = _os.open(file, flags, 0600)
> OSError: [Errno -2146233079] HybridMapping is full
>
> After a long boring trek over the hills of ignorance that may have
> been shorter if I'd seen that traceback first rather than other more
> obscure ones, the root cause turned out to be a failure to clean up
> temporary files. After IronPython has created 4096 of them and not
> removed any afterwards everything involving filenos starts to
> throwing. (As an aside, I wonder if using non-integer unique tokens
> like Jython mightn't be a better approach).
>
> The following issue tracker entry covers the tempfile issue:
>
> <http://ironpython.codeplex.com/WorkItem/View.aspx?WorkItemId=23936>
>
> However the bug report gets it a little wrong, the problem is not that
> sys.platform == "win32" is false under IronPython, but rather that
> os.name == "nt" is still true, but IronPython silently ignores the
> os.O_TEMPORARY flag.
>
> A trivial work around is:
>
>     if sys.platform == "cli":
>         os.name = "nty"
>         import tempfile
>         os.name = "nt"
>
> However fixing this properly is also pretty simple, in
> Src/IronPython.Modules/nt.cs the open function just needs to make sure
> that File.Open gets passed DeleteOnClose. There may as well be a
> FileOptionsFromFlags as per the existing FileModeFromFlags and
> FileAccessFromFlags so that the following mappings are correct as
> well:
>
> os.O_SHORT_LIVED -> ? -> FILE_ATTRIBUTE_TEMPORARY
> os.O_TEMPORARY -> FileOptions.DeleteOnClose -> FILE_FLAG_DELETE_ON_CLOSE
> os.O_RANDOM -> FileOptions.RandomAccess -> FILE_FLAG_RANDOM_ACCESS
> os.O_SEQUENTIAL -> FileOptions.SequentialScan -> FILE_FLAG_SEQUENTIAL_SCAN
>
> Martin
> _______________________________________________
> Users mailing list
> Users@...
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Martin (gzlist)]

On 23/10/2009, Dino Viehland <dinov@...> wrote:
> Thanks for the great detail here.  As you said the fix is easy and I believe
> I have it ready to go.  We're already kicking off our RC2 build so I don't
> know that it'll make it until that build (or 2.6.0 for that matter) but
> it'll be fixed by 2.6.1 at the latest.

That's great, thanks, I can survive on the workaround till the first
minor update.

My only outstanding issue that doesn't involve implementing whole
modules is various things related to encodings, which I need to find
time to investigate further and report on.

Martin
_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com