IPY and multitasking

[Pavel Suhotyuk]

I have this collection of methods:
--------------------------------------------------------------------------------------------------
public static class IronPythonHelper
{
         public static ScriptEngine
CreateScriptEngine(IEnumerable<string> paths)
         {
             var langSetup = ScriptRuntimeSetup.ReadConfiguration();
             var engine = new
ScriptRuntime(langSetup).GetEngine("IronPython");
             var path = engine.GetSearchPaths();
             path.Extend(paths);
             engine.SetSearchPaths(path);
             engine.Execute("import citypay", engine.Runtime.Globals);
             engine.Execute("import cPickle", engine.Runtime.Globals);
             engine.Execute("from citypay.utils import Struct",
engine.Runtime.Globals);
             return engine;
         }

         public static object PackParams(ScriptEngine engine,
IDictionary<string, object> param)
         {
             var scope = engine.CreateScope();
             engine.Execute("from citypay.utils import Struct", scope);
             engine.Execute("def pack( params ) : return
Struct(**dict(params))", scope);
             var pack = scope.GetVariable<Func<IDictionary<string,
object>, object>>("pack");
             return pack(param);
         }

         public static byte[] CPickleSerialize(ScriptEngine engine,
object obj)
         {
             var scope = engine.CreateScope();
             engine.Execute("import cPickle", scope);
             engine.Execute("def serialize( obj ) : return
cPickle.dumps( obj )", scope);
             var serialize = scope.GetVariable<Func<object,
string>>("serialize");
             return Encoding.Unicode.GetBytes(serialize(obj));
         }
}
--------------------------------------------------------------------------------------------------
I have base class used for create needed Engine and Scope as:
--------------------------------------------------------------------------------------------------
public absract class SripterBase {
     private static readonly ScriptEngine engine =
IronPythonHelper.CreateScriptEngine(Settings.Default.PythonLibPaths.Cast<string>());

     ...
}
--------------------------------------------------------------------------------------------------
My problem in next: Scripter based classes can be created in many
threads in one time.
Scripter class used for get pickle'd data by CPickleSerialize(), from
something like:
--------------------------------------------------------------------------------------------------
class Struct ( object ) :

     def __init__ ( self, **kwds ) :
         for name, val in kwds.iteritems() :
             setattr( self, name, val )

     def _update ( self, **kwds ) :
         for name, val in kwds.iteritems() :
             setattr( self, name, val )

     def _enum ( self ) :
         return [( a, getattr( self, a ) ) for a in dir( self ) if
a.find( '__' ) == -1 and not a.startswith( '_' )]

     def __repr__ ( self ) :
         return '<pmm3.util.Struct %s>' % ' '.join( [ '%s=%s' % ( str( n
), repr( v ) ) for n, v in self._enum() ] )

     def __eq__ ( self, other ) :
         if not isinstance( other, type( self ) ) :
             return NotImplemented
         return self._enum() == other._enum()
--------------------------------------------------------------------------------------------------

Multithread code in some (not stable) cases throws 'strange' exceptions.
This problem exists only in multithreaded environment and not exist in
single thread.

How I can fix or avoid this problem ?


usmpsrv1 :: FATAL :: CityPay.Core.Utils.RecursiveException: Recursive
ended. ---> IronPython.Runtime.Exceptions.ImportException: Cannot import
name Struct
    in IronPython.Runtime.Importer.ImportFrom(CodeContext context,
Object from, String name)
    in Microsoft.Scripting.Utils.InvokeHelper`4.Invoke(Object arg0,
Object arg1, Object arg2)
    in
Microsoft.Scripting.Interpreter.CallInstruction.Run(InterpretedFrame frame)
    in
Microsoft.Scripting.Interpreter.Interpreter.RunInstructions(InterpretedFrame
frame)
    in Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame
frame)
    in Microsoft.Scripting.Interpreter.LightLambda.Run2[T0,T1,TRet](T0
arg0, T1 arg1)
    in IronPython.Compiler.PythonScriptCode.Run(Scope scope)
    in
IronPython.Compiler.RuntimeScriptCode.InvokeTarget(LambdaExpression
code, Scope scope)
    in Microsoft.Scripting.Hosting.ScriptSource.Execute(ScriptScope scope)
    in CityPay.Core.Utils.IronPythonHelper.PackParams(ScriptEngine
engine, IDictionary`2 param)
    in
CityPay.Providers.Helpers.Beeline.PaymentInfo.Serialize(ScriptEngine engine)
    in CityPay.Providers.Helpers.PaymentInfoBase.ToString(ScriptEngine
engine)
    in
CityPay.PaymentsBroker.Bridge.WinService.Workers.ProcessWorker.Request(Payment
payment)
    in CityPay.Core.Utils.Util.Recursive[TResult](Func`1 func, Int32 count)
    in CityPay.Core.Utils.Util.Recursive[TResult](Func`1 func, Int32 count)
    in CityPay.Core.Utils.Util.Recursive[TResult](Func`1 func, Int32 count)
    in CityPay.Core.Utils.Util.Recursive[TResult](Func`1 func, Int32 count)
    in CityPay.Core.Utils.Util.Recursive[TResult](Func`1 func, Int32 count)
    in CityPay.Core.Utils.Util.Recursive[TResult](Func`1 func, Int32 count)
    in
CityPay.PaymentsBroker.Bridge.WinService.Workers.ProcessWorker.Execute(Object
o)
Logger: CityPay.PaymentsBroker.Bridge.WinService.Workers.ProcessWorker
Thread: 4
Date: 2009-11-11 17:27:31,484
NDC: (null)

usmpsrv1 :: FATAL :: CityPay.Core.Utils.RecursiveException: Recursive
ended. ---> System.Exception: Can't pickle
IronPython.Runtime.Types.BuiltinFunction: it's not the same object as
copy_reg._reconstructor
_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Dino Viehland]

You're only using 1 ScriptEngine class?  That should be fine based upon
the code below but I just want to make sure I understand the scenario.

Is citypay.utils being ran multiple times when you run the code in the
multi-threaded scenario?  If you have a console app you could put a
print statement in utils.py to see if it's getting executed multiple
times.  If it's not a console app you could put some other logging
into it.  Executing Struct's definition multiple times might cause
the exceptions you're seeing but I don't know why it would get executed
multiple times.

> -----Original Message-----
> From: users-bounces@... [mailto:users-
> bounces@...] On Behalf Of Pavel Suhotyuk
> Sent: Thursday, November 12, 2009 7:23 AM
> To: Discussion of IronPython
> Subject: [IronPython] IPY and multitasking
>
> I have this collection of methods:
> -----------------------------------------------------------------------
> ---------------------------
> public static class IronPythonHelper
> {
>          public static ScriptEngine
> CreateScriptEngine(IEnumerable<string> paths)
>          {
>              var langSetup = ScriptRuntimeSetup.ReadConfiguration();
>              var engine = new
> ScriptRuntime(langSetup).GetEngine("IronPython");
>              var path = engine.GetSearchPaths();
>              path.Extend(paths);
>              engine.SetSearchPaths(path);
>              engine.Execute("import citypay", engine.Runtime.Globals);
>              engine.Execute("import cPickle", engine.Runtime.Globals);
>              engine.Execute("from citypay.utils import Struct",
> engine.Runtime.Globals);
>              return engine;
>          }
>
>          public static object PackParams(ScriptEngine engine,
> IDictionary<string, object> param)
>          {
>              var scope = engine.CreateScope();
>              engine.Execute("from citypay.utils import Struct", scope);
>              engine.Execute("def pack( params ) : return
> Struct(**dict(params))", scope);
>              var pack = scope.GetVariable<Func<IDictionary<string,
> object>, object>>("pack");
>              return pack(param);
>          }
>
>          public static byte[] CPickleSerialize(ScriptEngine engine,
> object obj)
>          {
>              var scope = engine.CreateScope();
>              engine.Execute("import cPickle", scope);
>              engine.Execute("def serialize( obj ) : return
> cPickle.dumps( obj )", scope);
>              var serialize = scope.GetVariable<Func<object,
> string>>("serialize");
>              return Encoding.Unicode.GetBytes(serialize(obj));
>          }
> }
> -----------------------------------------------------------------------
> ---------------------------
> I have base class used for create needed Engine and Scope as:
> -----------------------------------------------------------------------
> ---------------------------
> public absract class SripterBase {
>      private static readonly ScriptEngine engine =
> IronPythonHelper.CreateScriptEngine(Settings.Default.PythonLibPaths.Cas
> t<string>());
>
>      ...
> }
> -----------------------------------------------------------------------
> ---------------------------
> My problem in next: Scripter based classes can be created in many
> threads in one time.
> Scripter class used for get pickle'd data by CPickleSerialize(), from
> something like:
> -----------------------------------------------------------------------
> ---------------------------
> class Struct ( object ) :
>
>      def __init__ ( self, **kwds ) :
>          for name, val in kwds.iteritems() :
>              setattr( self, name, val )
>
>      def _update ( self, **kwds ) :
>          for name, val in kwds.iteritems() :
>              setattr( self, name, val )
>
>      def _enum ( self ) :
>          return [( a, getattr( self, a ) ) for a in dir( self ) if
> a.find( '__' ) == -1 and not a.startswith( '_' )]
>
>      def __repr__ ( self ) :
>          return '<pmm3.util.Struct %s>' % ' '.join( [ '%s=%s' % ( str(
> n
> ), repr( v ) ) for n, v in self._enum() ] )
>
>      def __eq__ ( self, other ) :
>          if not isinstance( other, type( self ) ) :
>              return NotImplemented
>          return self._enum() == other._enum()
> -----------------------------------------------------------------------
> ---------------------------
>
> Multithread code in some (not stable) cases throws 'strange'
> exceptions.
> This problem exists only in multithreaded environment and not exist in
> single thread.
>
> How I can fix or avoid this problem ?
>
>
> usmpsrv1 :: FATAL :: CityPay.Core.Utils.RecursiveException: Recursive
> ended. ---> IronPython.Runtime.Exceptions.ImportException: Cannot
> import
> name Struct
>     in IronPython.Runtime.Importer.ImportFrom(CodeContext context,
> Object from, String name)
>     in Microsoft.Scripting.Utils.InvokeHelper`4.Invoke(Object arg0,
> Object arg1, Object arg2)
>     in
> Microsoft.Scripting.Interpreter.CallInstruction.Run(InterpretedFrame
> frame)
>     in
> Microsoft.Scripting.Interpreter.Interpreter.RunInstructions(Interpreted
> Frame
> frame)
>     in Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame
> frame)
>     in Microsoft.Scripting.Interpreter.LightLambda.Run2[T0,T1,TRet](T0
> arg0, T1 arg1)
>     in IronPython.Compiler.PythonScriptCode.Run(Scope scope)
>     in
> IronPython.Compiler.RuntimeScriptCode.InvokeTarget(LambdaExpression
> code, Scope scope)
>     in Microsoft.Scripting.Hosting.ScriptSource.Execute(ScriptScope
> scope)
>     in CityPay.Core.Utils.IronPythonHelper.PackParams(ScriptEngine
> engine, IDictionary`2 param)
>     in
> CityPay.Providers.Helpers.Beeline.PaymentInfo.Serialize(ScriptEngine
> engine)
>     in CityPay.Providers.Helpers.PaymentInfoBase.ToString(ScriptEngine
> engine)
>     in
> CityPay.PaymentsBroker.Bridge.WinService.Workers.ProcessWorker.Request(
> Payment
> payment)
>     in CityPay.Core.Utils.Util.Recursive[TResult](Func`1 func, Int32
> count)
>     in CityPay.Core.Utils.Util.Recursive[TResult](Func`1 func, Int32
> count)
>     in CityPay.Core.Utils.Util.Recursive[TResult](Func`1 func, Int32
> count)
>     in CityPay.Core.Utils.Util.Recursive[TResult](Func`1 func, Int32
> count)
>     in CityPay.Core.Utils.Util.Recursive[TResult](Func`1 func, Int32
> count)
>     in CityPay.Core.Utils.Util.Recursive[TResult](Func`1 func, Int32
> count)
>     in
> CityPay.PaymentsBroker.Bridge.WinService.Workers.ProcessWorker.Execute(
> Object
> o)
> Logger: CityPay.PaymentsBroker.Bridge.WinService.Workers.ProcessWorker
> Thread: 4
> Date: 2009-11-11 17:27:31,484
> NDC: (null)
>
> usmpsrv1 :: FATAL :: CityPay.Core.Utils.RecursiveException: Recursive
> ended. ---> System.Exception: Can't pickle
> IronPython.Runtime.Types.BuiltinFunction: it's not the same object as
> copy_reg._reconstructor
> _______________________________________________
> Users mailing list
> Users@...
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

[Pavel Suhotyuk]

I'm prepare simple test for problem demonstration.
http://files.roinet.net/DLRTest.zip

Module utils imported 6-7 times on 2x Dual-Core Opteron 2216 machine,
but method IronPythonHelper.CreateScript() called one time. On Core 2
Quad this problem has detected 4 times. On Core 2 Duo problem has not
detected.

In file out.txt console output with exceptions and logging information.

Dino Viehland wrote:

> You're only using 1 ScriptEngine class?  That should be fine based upon
> the code below but I just want to make sure I understand the scenario.
>
> Is citypay.utils being ran multiple times when you run the code in the
> multi-threaded scenario?  If you have a console app you could put a
> print statement in utils.py to see if it's getting executed multiple
> times.  If it's not a console app you could put some other logging
> into it.  Executing Struct's definition multiple times might cause
> the exceptions you're seeing but I don't know why it would get executed
> multiple times.
>
> _______________________________________________
> Users mailing list
> Users@...
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

_______________________________________________
Users mailing list
Users@...
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com