script evaluation

[Mia Gipson]

Greetings,

We're currently using Janino as a script evaluator.  We've noticed that as the size of the parameters passed into the "evaluate" method increases, the performance of the evaluation decreases.  For example, passing in a HashMap with 10 key/value pairs (with each value having only about 20 characters) takes significantly longer than passing in one with 5 pairs.  We're not seeing the same performance degradation when running tests outside of Janino (e.g., testing how long it takes to perform a pass-by-reference), and moving to the "fast" script evaluation (with the createFastScriptEvaluator method) doesn't seem to help.

The issue is that we run evaluations in a large loop, and are trying to optimize performance, so even a difference of a few milliseconds per evaluation matters.

Is this correct behavior, or is this a limitation of Janino?  If the former, would it be possible to get a technical explanation for why this is?

I greatly appreciate any assistance that can be provided!

 

Best,

--
Mai

[Philipp Hagemeister]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Mia,

can you provide a test case of the scenario? I'm no expert on Janino by
any means, but I just can't reproduce it. I attached my test case. The
results on my machine are as following:

(called an empty method with a map the size of : run time)
~     1: 669    ms
~     5: 672    ms
~    10: 670    ms
~   100: 680    ms
~  1000: 676    ms
~ 10000: 670    ms

i.e. the performance of the evaluate method does not depend on the size
of the arguments on the heap.

Regards,

Philipp Hagemeister

Mia Gipson wrote:
| Greetings,
|
| We're currently using Janino as a script evaluator.  We've noticed that as
| the size of the parameters passed into the "evaluate" method
increases, the
| performance of the evaluation decreases.  For example, passing in a
HashMap
| with 10 key/value pairs (with each value having only about 20 characters)
| takes significantly longer than passing in one with 5 pairs.  We're not
| seeing the same performance degradation when running tests outside of
Janino
| (e.g., testing how long it takes to perform a pass-by-reference), and
moving
| to the "fast" script evaluation (with the createFastScriptEvaluator
method)
| doesn't seem to help.
|
| The issue is that we run evaluations in a large loop, and are trying to
| optimize performance, so even a difference of a few milliseconds per
| evaluation matters.
|
| Is this correct behavior, or is this a limitation of Janino?  If the
former,
| would it be possible to get a technical explanation for why this is?
|
| I greatly appreciate any assistance that can be provided!
|
|
|
| Best,
|
|
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEAREKAAYFAkgSWeYACgkQ9eq1gvr7CFxQNQCgj6Xk6c3baWZ4gyFcI5vUjMLb
Q/MAni/Hj6hVir51GJZs6ke1haA58ThW
=XmuM
-----END PGP SIGNATURE-----

import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
import java.util.UUID;

import org.codehaus.janino.ScriptEvaluator;

public class TestEvaluatePerformance {
        protected static final int CALLNUMS_PER_TEST = 1000;
       
        public static void main(String[] args) throws Exception {
                int[] tests = { 1, 5, 10, 100, 1000, 10000 };
                long[] bestTimes = new long[tests.length];
                Arrays.fill(bestTimes, Integer.MAX_VALUE);
               
                for (int i = 0;i < 10;i++) {
                        for (int j = 0;j < tests.length;j++) {
                                long time = testEvaluation(tests[j]);
                               
                                if (time < bestTimes[j]) {
                                        bestTimes[j] = time;
                                }
                        }
                }
               
                for (int i = 0;i < tests.length;i++) {
                        System.out.println(String.format("%6d", tests[i]) + ": " +
                                                String.format("%-6d", bestTimes[i] / 1000) + " ms");
                }
        }
       
        protected static long testEvaluation(int mapSize) throws Exception {
                Map<String,String> testMap = genMap(mapSize);
               
                ScriptEvaluator se = new ScriptEvaluator(
                                "return map;",
                                Map.class,
                                new String[] {"map"},
                                new Class[] {Map.class}
                                );
               
                System.gc();
               
                long startTime = System.nanoTime();
               
                for (int i = 0;i < CALLNUMS_PER_TEST;i++) {
                        se.evaluate(new Object[] {testMap});
                }
               
                long runTime = System.nanoTime() - startTime;
               
                System.gc();
               
                return runTime;
        }

        protected static Map<String, String> genMap(int mapSize) {
                Map<String,String> res = new HashMap<String, String>(mapSize);
               
                while (mapSize-- > 0) {
                        res.put(UUID.randomUUID().toString(), UUID.randomUUID().toString());
                }
               
                return res;
        }
}


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Quartz]

Hi,

Are you filling the map in your java code before calling evaluate, or are you initializing the
hashmap in the script? I couldn't understand that part of your question.

If your hashmap is initialised once only and passed in argument to thousands of calls to evaluate,
you should not have to pay more than once the nearly-linear cost of adding to a map. As of using
the map, with a load factor of 0.25f instead of the default 0.75f, you should be able to avoid
hash collisions. You may need to look at the hashcode of your map keys, if they are custom
objects. Breakpoint your code right after you filled the map, and examine the map 'table' array in
the IDE: if you see an array of 32 elements and that you added more element to the map than you
see non-nulls in that array, then you have hash collisions. The bigger the difference in the
non-null count to your map size, the more collisions you have, which would explain why your
hasmap.get() are  sluggish.

Evaluating script everytime is likely trashing the permanent heap with as many ad-hoc classes. Not
sure how janino works in that aspect, but I can tell you how I use it.
I make a real java interface, let's say "IMyInterface", with a method, let's say "public MyResult
myMethod(String s, Map m)".

Then you get janino to compile a class implementing that interface (no need to save on file, just
stick with memory as usual). Then you use clazz.newInstance() to get an instance, cast it to
IMyInterface and voila, ready to use.

As you can guess, I don't like interpreted language. But here, your usage requires performance of
fully compiled code, but somehow you need dynamically generated code. I am not sure you needed
janino for your task, maybe you can explain your goal.


--- Mia Gipson <mia.gipson@...> wrote:

> Greetings,
>
> We're currently using Janino as a script evaluator.  We've noticed that as
> the size of the parameters passed into the "evaluate" method increases, the
> performance of the evaluation decreases.  For example, passing in a HashMap
> with 10 key/value pairs (with each value having only about 20 characters)
> takes significantly longer than passing in one with 5 pairs.  We're not
> seeing the same performance degradation when running tests outside of Janino
> (e.g., testing how long it takes to perform a pass-by-reference), and moving
> to the "fast" script evaluation (with the createFastScriptEvaluator method)
> doesn't seem to help.
>
> The issue is that we run evaluations in a large loop, and are trying to
> optimize performance, so even a difference of a few milliseconds per
> evaluation matters.
>
> Is this correct behavior, or is this a limitation of Janino?  If the former,
> would it be possible to get a technical explanation for why this is?
>
> I greatly appreciate any assistance that can be provided!
>
>
>
> Best,
>
>
> --
> Mai
>

      ____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Arno Unkrig]

Hello everybody,

Quartz wrote:
> Hi,
...
> I make a real java interface, let's say "IMyInterface", with a method, let's say "public MyResult
> myMethod(String s, Map m)".
>
> Then you get janino to compile a class implementing that interface (no need to save on file, just
> stick with memory as usual). Then you use clazz.newInstance() to get an instance, cast it to
> IMyInterface and voila, ready to use.

That's exactly what "ScriptEvaluator.createFastEvaluator()" does:

     interface IMyInterface {
         void foo(int x); // Declare exactly ONE method here.
     }
...
     IMyInterface mine = (IMyInterface)
             ScriptEvaluator.createFastEvaluator(
         "System.out.println(x);",
         IMyInterface.class,
         new String[] { "x" }
     );
     mine.foo(999); // Prints "999"
     mine.foo(9999); // Prints "9999"


CU

Arno

> As you can guess, I don't like interpreted language. But here, your usage requires performance of
> fully compiled code, but somehow you need dynamically generated code. I am not sure you needed
> janino for your task, maybe you can explain your goal.
>
>
> --- Mia Gipson <mia.gipson@...> wrote:
>
>> Greetings,
>>
>> We're currently using Janino as a script evaluator.  We've noticed that as
>> the size of the parameters passed into the "evaluate" method increases, the
>> performance of the evaluation decreases.  For example, passing in a HashMap
>> with 10 key/value pairs (with each value having only about 20 characters)
>> takes significantly longer than passing in one with 5 pairs.  We're not
>> seeing the same performance degradation when running tests outside of Janino
>> (e.g., testing how long it takes to perform a pass-by-reference), and moving
>> to the "fast" script evaluation (with the createFastScriptEvaluator method)
>> doesn't seem to help.
>>
>> The issue is that we run evaluations in a large loop, and are trying to
>> optimize performance, so even a difference of a few milliseconds per
>> evaluation matters.
>>
>> Is this correct behavior, or is this a limitation of Janino?  If the former,
>> would it be possible to get a technical explanation for why this is?
>>
>> I greatly appreciate any assistance that can be provided!
>>
>>
>>
>> Best,
>>
>>
>> --
>> Mai
>>
>
>
>
>       ____________________________________________________________________________________
> Be a better friend, newshound, and
> know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Arno Unkrig]

Hi Mia,

I don't see why the SIZE OF THE HASHMAP that you pass affects the
performance. The NUMBER OF THE PARAMETERS is surely significant, both
when cooking and evaluating the script.

You wrote:

 > We're not
 > seeing the same performance degradation when running tests outside of
 > Janino (e.g., testing how long it takes to perform a
 > pass-by-reference), and moving to the "fast" script evaluation (with
 > the createFastScriptEvaluator method) doesn't seem to help.

What do you mean with "perform a pass-by-reference"? You already passed
the HashMap by reference to the JANINO ScriptEvaluator, sisn't you?


CU

Philipp Hagemeister schrieb:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Mia,
>
> can you provide a test case of the scenario? I'm no expert on Janino by
> any means, but I just can't reproduce it. I attached my test case. The
> results on my machine are as following:
>
> (called an empty method with a map the size of : run time)
> ~     1: 669    ms
> ~     5: 672    ms
> ~    10: 670    ms
> ~   100: 680    ms
> ~  1000: 676    ms
> ~ 10000: 670    ms
>
> i.e. the performance of the evaluate method does not depend on the size
> of the arguments on the heap.
>
> Regards,
>
> Philipp Hagemeister
>
> Mia Gipson wrote:
> | Greetings,
> |
> | We're currently using Janino as a script evaluator.  We've noticed
> that as
> | the size of the parameters passed into the "evaluate" method
> increases, the
> | performance of the evaluation decreases.  For example, passing in a
> HashMap
> | with 10 key/value pairs (with each value having only about 20 characters)
> | takes significantly longer than passing in one with 5 pairs.  We're not
> | seeing the same performance degradation when running tests outside of
> Janino
> | (e.g., testing how long it takes to perform a pass-by-reference), and
> moving
> | to the "fast" script evaluation (with the createFastScriptEvaluator
> method)
> | doesn't seem to help.
> |
> | The issue is that we run evaluations in a large loop, and are trying to
> | optimize performance, so even a difference of a few milliseconds per
> | evaluation matters.
> |
> | Is this correct behavior, or is this a limitation of Janino?  If the
> former,
> | would it be possible to get a technical explanation for why this is?
> |
> | I greatly appreciate any assistance that can be provided!
> |
> |
> |
> | Best,
> |
> |
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (GNU/Linux)
>
> iEYEAREKAAYFAkgSWeYACgkQ9eq1gvr7CFxQNQCgj6Xk6c3baWZ4gyFcI5vUjMLb
> Q/MAni/Hj6hVir51GJZs6ke1haA58ThW
> =XmuM
> -----END PGP SIGNATURE-----
>
>
> ------------------------------------------------------------------------
>
> import java.util.Arrays;
> import java.util.HashMap;
> import java.util.Map;
> import java.util.UUID;
>
> import org.codehaus.janino.ScriptEvaluator;
>
> public class TestEvaluatePerformance {
> protected static final int CALLNUMS_PER_TEST = 1000;
>
> public static void main(String[] args) throws Exception {
> int[] tests = { 1, 5, 10, 100, 1000, 10000 };
> long[] bestTimes = new long[tests.length];
> Arrays.fill(bestTimes, Integer.MAX_VALUE);
>
> for (int i = 0;i < 10;i++) {
> for (int j = 0;j < tests.length;j++) {
> long time = testEvaluation(tests[j]);
>
> if (time < bestTimes[j]) {
> bestTimes[j] = time;
> }
> }
> }
>
> for (int i = 0;i < tests.length;i++) {
> System.out.println(String.format("%6d", tests[i]) + ": " +
> String.format("%-6d", bestTimes[i] / 1000) + " ms");
> }
> }
>
> protected static long testEvaluation(int mapSize) throws Exception {
> Map<String,String> testMap = genMap(mapSize);
>
> ScriptEvaluator se = new ScriptEvaluator(
> "return map;",
> Map.class,
> new String[] {"map"},
> new Class[] {Map.class}
> );
>
> System.gc();
>
> long startTime = System.nanoTime();
>
> for (int i = 0;i < CALLNUMS_PER_TEST;i++) {
> se.evaluate(new Object[] {testMap});
> }
>
> long runTime = System.nanoTime() - startTime;
>
> System.gc();
>
> return runTime;
> }
>
> protected static Map<String, String> genMap(int mapSize) {
> Map<String,String> res = new HashMap<String, String>(mapSize);
>
> while (mapSize-- > 0) {
> res.put(UUID.randomUUID().toString(), UUID.randomUUID().toString());
> }
>
> return res;
> }
> }
>
>
>
> ------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email