Groovy runs code in static initializers during compile

[Robert Stroud]

On 3 May 2007, at 23:41, Jochen Theodorou wrote:

> Robert Stroud schrieb:
> [...]
>>         URL sourceURL = null;
>>         URL objectURL = null;
>>         try {
>>             sourceURL = gcl.getResourceLoader().loadGroovySource
>> (name);
>>             if ( sourceURL != null)
>>             {
>>                 objectURL = gcl.getResourceLoader
>> ().loadGroovyObject(name);
>>             }

N.B. loadGroovySource and loadGroovyObject are essentially the same,  
except that one looks for a file extension of .groovy whilst the  
other looks for a file extension of .class...

>>         } catch (MalformedURLException e) {
>>             // fall through and let the URL be null
>>         }
>
> shouldn't sourceURL and objectURL be the same after this?

No, the idea is that sourceURL points to the source file, and  
objectURL points to the compiled class file (if it exists). Either or  
both could be null I think, although I suspect there are only three  
interesting cases:

sourceURL == null

sourceURL != null, targetURL == null

sourceURL != null, targetURL != null

I'm not sure that the  fourth case makes sense:

sourceURL == null,  targetURL != null

That would mean that a class file existed without any corresponding  
source code - I guess that is possible, and would presumably mean  
that you couldn't tell whether a re-compilation was necessary or not.

Anyway, this is just a quick and dirty hack - it took about 5 minutes  
to implement, and I don't really know what's going on here. However,  
it seems to work, or at least it passes all the tests, and fixes  
Aaron's bug. It's intended as a proof of concept that you can use  
file modification times rather than reflection to find out whether a  
class needs to be re-compiled...

I'm not sure what would happen if the class file was more recent than  
the source file, but didn't correspond to the source code - my guess  
is that you'd get some sort of Java error at load time (or link  
time). This shouldn't happen unless someone tries to trick the  
compiler, but what does the Java compiler do in this situation? If it  
fails in the same way, that's fine - otherwise, I think you'd need to  
extract the necessary data from the class file to make a more  
sophisticated version check in the kind of way that Aaron has suggested.

It would be interesting to look at the Java compiler, and see how it  
deals with this issue, but failing at load time (or link time) seems  
like a reasonable compromise to me, providing it doesn't break  
anything else...

Robert

> bye blackdrag
>
> --
> Jochen "blackdrag" Theodorou
> Groovy Tech Lead (http://groovy.codehaus.org)
> http://blackdragsview.blogspot.com/
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>    http://xircles.codehaus.org/manage_email
>

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Aaron Digulla]

Quoting Robert Stroud <R.J.Stroud@...>:


> It would be interesting to look at the Java compiler, and see how it
> deals with this issue, but failing at load time (or link time) seems
> like a reasonable compromise to me, providing it doesn't break anything
> else...

You can look at the source of the Eclipse Java Compiler. Here are the  
pointers I've got from the JDT guys:

------------- Walter Harley -------------
You can start with
org.eclipse.jdt.internal.compiler.classfmt.ClassFileReader, and then look at
the classes that call its methods.  Also look at
org.eclipse.jdt.internal.compiler.lookup.BinaryTypeBinding and its
superclasses.
------------- Walter Harley -------------

The full thread is here:  
http://www.eclipse.org/newsportal/article.php?id=20541&group=eclipse.tools.jdt#20541

Or use a news reader on news://news.eclipse.org/eclipse.tools.jdt

Regards,

--
Aaron "Optimizer" Digulla a.k.a. Philmann Dark
"It's not the universe that's limited, it's our imagination.
Follow me and I'll show you something beyond the limits."
http://www.pdark.de/

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Robert Stroud]

On 4 May 2007, at 07:54, Aaron Digulla wrote:

> The full thread is here: http://www.eclipse.org/newsportal/ 
> article.php?id=20541&group=eclipse.tools.jdt#20541

Unfortunately, this seems to be password protected...

Robert


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Robert Stroud]

On 4 May 2007, at 07:54, Aaron Digulla wrote:

> You can look at the source of the Eclipse Java Compiler. Here are  
> the pointers I've got from the JDT guys:

Thanks - since it's not exactly obvious how to find the relevant  
source in the Eclipse repository, here's a clue for anyone else who  
wants to take a look - start here and work your way down...

http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.jdt.core/compiler/

Robert

>
> ------------- Walter Harley -------------
> You can start with
> org.eclipse.jdt.internal.compiler.classfmt.ClassFileReader, and  
> then look at
> the classes that call its methods.  Also look at
> org.eclipse.jdt.internal.compiler.lookup.BinaryTypeBinding and its
> superclasses.
> ------------- Walter Harley -------------
>
> The full thread is here: http://www.eclipse.org/newsportal/ 
> article.php?id=20541&group=eclipse.tools.jdt#20541
>
> Or use a news reader on news://news.eclipse.org/eclipse.tools.jdt
>
> Regards,
>
> --
> Aaron "Optimizer" Digulla a.k.a. Philmann Dark
> "It's not the universe that's limited, it's our imagination.
> Follow me and I'll show you something beyond the limits."
> http://www.pdark.de/
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>    http://xircles.codehaus.org/manage_email
>

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Aaron Digulla]

Quoting Robert Stroud <R.J.Stroud@...>:

>> The full thread is here:  
>> http://www.eclipse.org/newsportal/article.php?id=20541&group=eclipse.tools.jdt#20541
>
> Unfortunately, this seems to be password protected...

It's just spam protected :-) If you have no news reader, go here to  
get a password:

http://www.eclipse.org/newsgroups/register.php

Regards,

--
Aaron "Optimizer" Digulla a.k.a. Philmann Dark
"It's not the universe that's limited, it's our imagination.
Follow me and I'll show you something beyond the limits."
http://www.pdark.de/

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Robert Stroud]

On 3 May 2007, at 23:25, Robert Stroud wrote:

>> But it looks to me as though the relevant code only compares the  
>> modification times if the source file exists - if the code has  
>> been generated on the fly, then there won't be a source file, so  
>> there will be nothing to compare...
>> Of course, there may be other reasons why the Groovy compiler  
>> needs to initialize classes during compilation, but I don't see  
>> why it's necessary in this particular situation.
>
> I'll try and figure out how to generate a patch if someone confirms  
> that my approach is reasonable...

Sorry for the delay - I didn't realise how easy it was to type "svn  
diff". I've now added my patch to

http://jira.codehaus.org/browse/GROOVY-1863

I'd be grateful if someone could take a look and give it a try - it  
certainly fixes the bug in Aaron's simple example, but I don't know  
how it would work in more complicated examples...

Thanks,

Robert

> BTW, I've not looked to see whether the duplicate code in  
> GroovyClassLoader can also be rewritten in this way, but it should  
> be possible, and that needs to be part of the patch too...
>
> Thanks.
>
> Robert


---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Robert Stroud]

On 4 May 2007, at 09:37, Aaron Digulla wrote:

> Quoting Robert Stroud <R.J.Stroud@...>:
>
>>> The full thread is here:  http://www.eclipse.org/newsportal/ 
>>> article.php?id=20541&group=eclipse.tools.jdt#20541
>>
>> Unfortunately, this seems to be password protected...
>
> It's just spam protected :-) If you have no news reader, go here to  
> get a password:
>
> http://www.eclipse.org/newsgroups/register.php

Thanks - I read the thread, but I don't think it's very helpful, and  
it's also a bit confused.

The first branch says that the Eclipse compiler uses its own class  
reader - fair enough, but I still don't see why this is necessary  
(although it simplifies bootstrapping)

The second branch correctly states that there is a distinction  
between loading and initializing a class, but then claims that you  
can only examine the public members of a class that has been loaded  
but not initialized. This is not true - you can obtain type  
information about both the public and private members of a class that  
has been loaded but not initialized, but the class has to be  
initialized in order to access the values of these properties. You  
can bypass security checks to access the values of private fields,  
but I don't think you can access the value of a field in an  
uninitialized class. The author is confusing accessing the value of a  
property with obtaining type information about a property.

The third branch points out that you can use Class.forName to obtain  
a Class object without initialising the class, but that this doesn't  
help you access the value of a field in the class, which is correct.

Hence, I don't think there's anything wrong with Groovy's use of the  
Java ClassLoader to obtain type information about classes - the  
difficulty lies with the use of a timestamp to record the last  
modification time of the class object as a static constant. From  
within Java, the value of this constant can only be accessed by  
initializing the class, which in turn causes static initializers to  
be invoked during compile time.

Robert

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Aaron Digulla]

Quoting Robert Stroud <R.J.Stroud@...>:

> Hence, I don't think there's anything wrong with Groovy's use of the
> Java ClassLoader to obtain type information about classes - the
> difficulty lies with the use of a timestamp to record the last
> modification time of the class object as a static constant. From within
> Java, the value of this constant can only be accessed by initializing
> the class, which in turn causes static initializers to be invoked
> during compile time.

The Eclipse compiler must have solved this somehow because it can read  
constant values from classes without initializing them. I have no idea  
how they do it but it apparently works.

Also, AFAIK, the reflection API gives no gurantee that it will not  
initialize the class when you read the methods and fields. That is  
probably the reason why the JDT developers chose to write their own  
code. I'm not sure if Groovy needs to do this, too. It's a whole lot  
of work with little benefit if Java 1.4, 1.5 and 6 can read the class  
information without initializing them.

Regards,

--
Aaron "Optimizer" Digulla a.k.a. Philmann Dark
"It's not the universe that's limited, it's our imagination.
Follow me and I'll show you something beyond the limits."
http://www.pdark.de/

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Robert Stroud]

On 10 May 2007, at 10:47, Aaron Digulla wrote:

> Quoting Robert Stroud <R.J.Stroud@...>:
>
>> Hence, I don't think there's anything wrong with Groovy's use of the
>> Java ClassLoader to obtain type information about classes - the
>> difficulty lies with the use of a timestamp to record the last
>> modification time of the class object as a static constant. From  
>> within
>> Java, the value of this constant can only be accessed by initializing
>> the class, which in turn causes static initializers to be invoked
>> during compile time.
>
> The Eclipse compiler must have solved this somehow because it can  
> read constant values from classes without initializing them. I have  
> no idea how they do it but it apparently works.

I'm a bit confused - how do you know that the Eclipse compiler can  
read constant values from classes without initializing them? Your  
test program only fails when you run it through a Groovy compiler,  
not when you run it through a Java compiler. A Java compiler has no  
need to read constant values from classes, whether initialized or not...

However, to answer your question, the Eclipse compiler doesn't use  
the Java class loader to obtain type information about classes, it  
uses its own class file reader, and builds its own versions of the  
java.lang.reflect objects, in particular, FieldInfo and MethodInfo  
objects, and FieldInfo provides a getConstant method that reads a  
constant value for the field directly out of the attribute table in  
the class file.

So if you write your own class file reader, you can do what you like  
within reason, including reading the values of constant fields.

However, constant fields are clearly a special case - their value is  
stored in the class file and doesn't need to be computed at run-time,  
which would require initializing the class.

The difficulty is that Java's reflective Field class doesn't treat  
constant fields or fields whose initial value is a compile-time  
constant as a special case, so the Field.get method always  
initializes the class before accessing the value of the field, since  
this is the only way of dealing with the general case.

Robert

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Aaron Digulla]

Quoting Robert Stroud <R.J.Stroud@...>:

>> The Eclipse compiler must have solved this somehow because it can  
>> read constant values from classes without initializing them. I have  
>>  no idea how they do it but it apparently works.
>
> I'm a bit confused - how do you know that the Eclipse compiler can read
> constant values from classes without initializing them?

Write a short test and compile it:

public class A
{
        public final static int FOO = 1;
        public final static A FOO2 = new A();

        static
        {
                if (true)
                        throw new RuntimeException ("Bad!");
        }
}

public class B
{
        public final static int BAR = A.FOO;
        public final static A BAR2 = A.FOO2;
}

When you look at the bytecode, you'll see that it says:

public class B
{

     public B()
     {
     //    0    0:aload_0
     //    1    1:invokespecial   #24  <Method void Object()>
     //    2    4:return
     }

     public static final int BAR = 1;
     public static final A BAR2;

     static
     {
         BAR2 = A.FOO2;
     //    0    0:getstatic       #14  <Field A A.FOO2>
     //    1    3:putstatic       #19  <Field A BAR2>
     //*   2    6:return
     }
}

This means the Eclipse compiler can resolve constants in dangerous  
classes without using reflection.

This output got me thinking: Why are Groovy timestamps set in the  
static code? It seems that the JVM offers a way to define constants  
which can be read without initializing the class.

Regards,

--
Aaron "Optimizer" Digulla a.k.a. Philmann Dark
"It's not the universe that's limited, it's our imagination.
Follow me and I'll show you something beyond the limits."
http://www.pdark.de/

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Robert Stroud]

Aaron,

Thanks for the example - I see what you mean. I'll have to have a  
think about how that works in practice - A.FOO is a compile-time  
constant, but the compiler has somehow figured out the value from A  
whilst compiling B, and copied it from A to B...

> This output got me thinking: Why are Groovy timestamps set in the  
> static code?

Good question (see discussion elsewhere) - but given that they are...

> It seems that the JVM offers a way to define constants which can be  
> read without initializing the class.

Hmm - I don't think it's the JVM though, it's the compiler. If you  
stick within the Java API and use the Java class loader, I don't  
think you can access the value of compile-time constants in class  
files, and this is arguably a limitation of the API and the language  
definition, since the two are somewhat intertwined in this area.

However, it looks as though you may have found the reason why a Java  
compiler can't use a Java class loader to compile Java, but has to  
use its own class file reader instead...

Robert

On 10 May 2007, at 13:51, Aaron Digulla wrote:

> Quoting Robert Stroud <R.J.Stroud@...>:
>
>>> The Eclipse compiler must have solved this somehow because it  
>>> can  read constant values from classes without initializing them.  
>>> I have  no idea how they do it but it apparently works.
>>
>> I'm a bit confused - how do you know that the Eclipse compiler can  
>> read
>> constant values from classes without initializing them?
>
> Write a short test and compile it:
>
> public class A
> {
> public final static int FOO = 1;
> public final static A FOO2 = new A();
>
> static
> {
> if (true)
> throw new RuntimeException ("Bad!");
> }
> }
>
> public class B
> {
> public final static int BAR = A.FOO;
> public final static A BAR2 = A.FOO2;
> }
>
> When you look at the bytecode, you'll see that it says:
>
> public class B
> {
>
>     public B()
>     {
>     //    0    0:aload_0
>     //    1    1:invokespecial   #24  <Method void Object()>
>     //    2    4:return
>     }
>
>     public static final int BAR = 1;
>     public static final A BAR2;
>
>     static
>     {
>         BAR2 = A.FOO2;
>     //    0    0:getstatic       #14  <Field A A.FOO2>
>     //    1    3:putstatic       #19  <Field A BAR2>
>     //*   2    6:return
>     }
> }
>
> This means the Eclipse compiler can resolve constants in dangerous  
> classes without using reflection.
>
> This output got me thinking: Why are Groovy timestamps set in the  
> static code? It seems that the JVM offers a way to define constants  
> which can be read without initializing the class.
>
> Regards,
>
> --
> Aaron "Optimizer" Digulla a.k.a. Philmann Dark
> "It's not the universe that's limited, it's our imagination.
> Follow me and I'll show you something beyond the limits."
> http://www.pdark.de/
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>    http://xircles.codehaus.org/manage_email
>

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email