checking out a large file

[Andrew Aucott-2]

When checking out large files it seems like the entire file is read  
into memory before it's written to disk.

Is there a configuration option which limits the amount read into  
memory before writing to disk?

Thanks
Andrew

[Alexander Kitaev-3]

Hello Andrew,

 > When checking out large files it seems like the entire file is read into
 > memory before it's written to disk.
That shouldn't be true. During checkout SVNKit processes "windows" not
large than 100kb and subsequently writes result of processing to the
target file. What code do you use for checking out?

Alexander Kitaev,
TMate Software,
http://svnkit.com/ - Java [Sub]Versioning Library!

Andrew Aucott wrote:
> When checking out large files it seems like the entire file is read into
> memory before it's written to disk.
>
> Is there a configuration option which limits the amount read into memory
> before writing to disk?
>
> Thanks
> Andrew
>

[Andrew Aucott-2]

Hi Alexander, 

I've pasted in some example code at the bottom of this email that re-creates the problem. I guess it depends on what the JVM memory is set to but I consistently get an OutOfMemory exception within the doCheckout call.

Thanks

Andrew

That shouldn't be true. During checkout SVNKit processes "windows" not large than 100kb and subsequently writes result of processing to the target file. What code do you use for checking out?

Alexander Kitaev,

TMate Software,

http://svnkit.com/ - Java [Sub]Versioning Library!

package test;

import java.io.File;

import junit.framework.TestCase;

import org.tmatesoft.svn.core.SVNURL;

import org.tmatesoft.svn.core.internal.io.fs.FSRepositoryFactory;

import org.tmatesoft.svn.core.io.SVNRepositoryFactory;

import org.tmatesoft.svn.core.wc.SVNClientManager;

import org.tmatesoft.svn.core.wc.SVNRevision;

public class SVNCheckoutTest extends TestCase {

    private final static File FILE = new File("eclipse-SDK-3.2.1-macosx-carbon.tar.gz");

    private SVNURL url;

    private File repoFile;

    static {

        FSRepositoryFactory.setup();

    }

    

    public void setUp() throws Exception {

        this.repoFile = File.createTempFile("SVN", "Test");

        delete(repoFile);

        url = SVNRepositoryFactory.createLocalRepository(repoFile, true, true);

   }

    

    public void tearDown() {

        delete(repoFile);

    }

    

    public void testCheckOut() throws Exception {

        SVNClientManager mgr = SVNClientManager.newInstance();

        SVNURL fileURL = url.appendPath(FILE.getName(), false);

        mgr.getCommitClient().doImport(FILE, fileURL, "commit big file", false);

        System.out.println("checking out...");

        File tmp = new File(System.getProperty("java.io.tmpdir"), getName());

        delete(tmp);

        tmp.mkdir();

        try {

            mgr.getUpdateClient().doCheckout(url, tmp, SVNRevision.HEAD, SVNRevision.HEAD, false);

        } finally {

            delete(tmp);

        }

    }

    

    

    private static boolean delete(File file) {

        boolean success = true;

        if (file.exists()) {

            if (file.isDirectory()) {

                File[] files = file.listFiles();

                for (int i = 0; i < files.length; i++) {

                    success = success && delete(files[i]);

                }

            }

            success = success && file.delete();

        }

        return success;

    }

}

[Alexander Kitaev-3]

Hello Andrew,

Thank you very much for the code! I've managed to reproduce the problem
with 1.1.4 version and it is specific for local repository access. I
hope we will fix it in a few days. Thanks!

Alexander Kitaev,
TMate Software,
http://svnkit.com/ - Java [Sub]Versioning Library!

Andrew Aucott wrote:

>
> Hi Alexander,
>
> I've pasted in some example code at the bottom of this email that
> re-creates the problem. I guess it depends on what the JVM memory is set
> to but I consistently get an OutOfMemory exception within the doCheckout
> call.
>
> Thanks
> Andrew
>
>
>> That shouldn't be true. During checkout SVNKit processes "windows" not
>> large than 100kb and subsequently writes result of processing to the
>> target file. What code do you use for checking out?
>>
>> Alexander Kitaev,
>> TMate Software,
>> http://svnkit.com/ - Java [Sub]Versioning Library!
>
> package test;
>
> import java.io.File;
>
> import junit.framework.TestCase;
>
> import org.tmatesoft.svn.core.SVNURL;
> import org.tmatesoft.svn.core.internal.io.fs.FSRepositoryFactory;
> import org.tmatesoft.svn.core.io.SVNRepositoryFactory;
> import org.tmatesoft.svn.core.wc.SVNClientManager;
> import org.tmatesoft.svn.core.wc.SVNRevision;
>
> public class SVNCheckoutTest extends TestCase {
>     private final static File FILE = new
> File("eclipse-SDK-3.2.1-macosx-carbon.tar.gz");
>     private SVNURL url;
>     private File repoFile;
>     static {
>         FSRepositoryFactory.setup();
>     }
>
>    
>
>     public void setUp() throws Exception {
>         this.repoFile = File.createTempFile("SVN", "Test");
>         delete(repoFile);
>         url = SVNRepositoryFactory.createLocalRepository(repoFile, true,
> true);
>    }
>
>    
>
>     public void tearDown() {
>         delete(repoFile);
>     }
>
>    
>
>     public void testCheckOut() throws Exception {
>
>         SVNClientManager mgr = SVNClientManager.newInstance();
>         SVNURL fileURL = url.appendPath(FILE.getName(), false);
>         mgr.getCommitClient().doImport(FILE, fileURL, "commit big file",
> false);
>         System.out.println("checking out...");
>         File tmp = new File(System.getProperty("java.io.tmpdir"),
> getName());
>         delete(tmp);
>         tmp.mkdir();
>         try {
>             mgr.getUpdateClient().doCheckout(url, tmp, SVNRevision.HEAD,
> SVNRevision.HEAD, false);
>         } finally {
>             delete(tmp);
>         }
>     }
>
>    
>
>    
>
>     private static boolean delete(File file) {
>         boolean success = true;
>         if (file.exists()) {
>             if (file.isDirectory()) {
>                 File[] files = file.listFiles();
>                 for (int i = 0; i < files.length; i++) {
>                     success = success && delete(files[i]);
>                 }
>             }
>             success = success && file.delete();
>         }
>         return success;
>     }
> }
>

[Alexander Kitaev-3]

Hello Andrew,

I've just fixed the problem - we forgot to remove debug code that was
storing whole file contents to byte[] array! There even was TODO comment
like "remove me" :(

Now this code is removed in 1.1.x branch (rev 3366) and I was able to
checkout large file (140MB) with -XMx16M (max heap size of 16 megabytes).

Alexander Kitaev,
TMate Software,
http://svnkit.com/ - Java [Sub]Versioning Library!

Andrew Aucott wrote:

>
> Hi Alexander,
>
> I've pasted in some example code at the bottom of this email that
> re-creates the problem. I guess it depends on what the JVM memory is set
> to but I consistently get an OutOfMemory exception within the doCheckout
> call.
>
> Thanks
> Andrew
>
>
>> That shouldn't be true. During checkout SVNKit processes "windows" not
>> large than 100kb and subsequently writes result of processing to the
>> target file. What code do you use for checking out?
>>
>> Alexander Kitaev,
>> TMate Software,
>> http://svnkit.com/ - Java [Sub]Versioning Library!
>
> package test;
>
> import java.io.File;
>
> import junit.framework.TestCase;
>
> import org.tmatesoft.svn.core.SVNURL;
> import org.tmatesoft.svn.core.internal.io.fs.FSRepositoryFactory;
> import org.tmatesoft.svn.core.io.SVNRepositoryFactory;
> import org.tmatesoft.svn.core.wc.SVNClientManager;
> import org.tmatesoft.svn.core.wc.SVNRevision;
>
> public class SVNCheckoutTest extends TestCase {
>     private final static File FILE = new
> File("eclipse-SDK-3.2.1-macosx-carbon.tar.gz");
>     private SVNURL url;
>     private File repoFile;
>     static {
>         FSRepositoryFactory.setup();
>     }
>
>    
>
>     public void setUp() throws Exception {
>         this.repoFile = File.createTempFile("SVN", "Test");
>         delete(repoFile);
>         url = SVNRepositoryFactory.createLocalRepository(repoFile, true,
> true);
>    }
>
>    
>
>     public void tearDown() {
>         delete(repoFile);
>     }
>
>    
>
>     public void testCheckOut() throws Exception {
>
>         SVNClientManager mgr = SVNClientManager.newInstance();
>         SVNURL fileURL = url.appendPath(FILE.getName(), false);
>         mgr.getCommitClient().doImport(FILE, fileURL, "commit big file",
> false);
>         System.out.println("checking out...");
>         File tmp = new File(System.getProperty("java.io.tmpdir"),
> getName());
>         delete(tmp);
>         tmp.mkdir();
>         try {
>             mgr.getUpdateClient().doCheckout(url, tmp, SVNRevision.HEAD,
> SVNRevision.HEAD, false);
>         } finally {
>             delete(tmp);
>         }
>     }
>
>    
>
>    
>
>     private static boolean delete(File file) {
>         boolean success = true;
>         if (file.exists()) {
>             if (file.isDirectory()) {
>                 File[] files = file.listFiles();
>                 for (int i = 0; i < files.length; i++) {
>                     success = success && delete(files[i]);
>                 }
>             }
>             success = success && file.delete();
>         }
>         return success;
>     }
> }
>

[Andrew Aucott-2]

Hi Alexander,

I also get the problem when using https. This is where I first encountered the out of memory error. 

I've just tested and can confirm that I still get out of memory when using https. Memory used by the java process gradually reaches its maximum and then sits at that value for quite some time before throwing the out of memory error.

Here is the code im using to replicate the error.

    public void testDavCheckout() throws Exception {

        url = SVNURL.parseURIDecoded("https://yoururl");

        

        SVNClientManager mgr = SVNClientManager.newInstance();

        File tmp = new File(System.getProperty("java.io.tmpdir"), getName());

        delete(tmp);

        tmp.mkdir();

        try {

            mgr.getUpdateClient().doCheckout(url, tmp, SVNRevision.HEAD, SVNRevision.HEAD, false);

        } finally {

            delete(tmp);

        }

        

    }

Thanks

Andrew

On 14 Aug 2007, at 23:22, Alexander Kitaev wrote:

Hello Andrew,

I've just fixed the problem - we forgot to remove debug code that was storing whole file contents to byte[] array! There even was TODO comment like "remove me" :(

Now this code is removed in 1.1.x branch (rev 3366) and I was able to checkout large file (140MB) with -XMx16M (max heap size of 16 megabytes).

Alexander Kitaev,

TMate Software,

http://svnkit.com/ - Java [Sub]Versioning Library!

Andrew Aucott wrote:

Hi Alexander, I've pasted in some example code at the bottom of this email that re-creates the problem. I guess it depends on what the JVM memory is set to but I consistently get an OutOfMemory exception within the doCheckout call.

Thanks

Andrew

That shouldn't be true. During checkout SVNKit processes "windows" not large than 100kb and subsequently writes result of processing to the target file. What code do you use for checking out?

Alexander Kitaev,

TMate Software,

http://svnkit.com/ - Java [Sub]Versioning Library!

package test;

import java.io.File;

import junit.framework.TestCase;

import org.tmatesoft.svn.core.SVNURL;

import org.tmatesoft.svn.core.internal.io.fs.FSRepositoryFactory;

import org.tmatesoft.svn.core.io.SVNRepositoryFactory;

import org.tmatesoft.svn.core.wc.SVNClientManager;

import org.tmatesoft.svn.core.wc.SVNRevision;

public class SVNCheckoutTest extends TestCase {

    private final static File FILE = new File("eclipse-SDK-3.2.1-macosx-carbon.tar.gz");

    private SVNURL url;

    private File repoFile;

    static {

        FSRepositoryFactory.setup();

    }

        public void setUp() throws Exception {

        this.repoFile = File.createTempFile("SVN", "Test");

        delete(repoFile);

        url = SVNRepositoryFactory.createLocalRepository(repoFile, true, true);

   }

        public void tearDown() {

        delete(repoFile);

    }

        public void testCheckOut() throws Exception {

        SVNClientManager mgr = SVNClientManager.newInstance();

        SVNURL fileURL = url.appendPath(FILE.getName(), false);

        mgr.getCommitClient().doImport(FILE, fileURL, "commit big file", false);

        System.out.println("checking out...");

        File tmp = new File(System.getProperty("java.io.tmpdir"), getName());

        delete(tmp);

        tmp.mkdir();

        try {

            mgr.getUpdateClient().doCheckout(url, tmp, SVNRevision.HEAD, SVNRevision.HEAD, false);

        } finally {

            delete(tmp);

        }

    }

            private static boolean delete(File file) {

        boolean success = true;

        if (file.exists()) {

            if (file.isDirectory()) {

                File[] files = file.listFiles();

                for (int i = 0; i < files.length; i++) {

                    success = success && delete(files[i]);

                }

            }

            success = success && file.delete();

        }

        return success;

    }

}

[Andrew Aucott-2]

Its possible when running with JVM 1.6 to get a stack trace. This may help tracking down the problem.

Regards

Andrew

Exception in thread "Thread-14" java.lang.OutOfMemoryError: Java heap space

        at org.apache.xerces.readers.CharReader.fillCurrentChunk(CharReader.java:255)

        at org.apache.xerces.readers.AbstractCharReader.slowLoadNextChar(AbstractCharReader.java:1370)

        at org.apache.xerces.readers.AbstractCharReader.skipAsciiCharData(AbstractCharReader.java:1287)

        at org.apache.xerces.readers.AbstractCharReader.scanContent(AbstractCharReader.java:1028)

        at org.apache.xerces.framework.XMLDocumentScanner$ContentDispatcher.dispatch(XMLDocumentScanner.java:1094)

        at org.apache.xerces.framework.XMLDocumentScanner.parseSome(XMLDocumentScanner.java:381)

        at org.apache.xerces.framework.XMLParser.parse(XMLParser.java:1098)

        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:634)

        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:597)

        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.dispatch(HTTPRequest.java:197)

        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:285)

        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:230)

        at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:218)

        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:219)

        at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:211)

        at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:608)

        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:162)

        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:344)

On 15 Aug 2007, at 12:50, Andrew Aucott wrote:

Hi Alexander,

I also get the problem when using https. This is where I first encountered the out of memory error. 

I've just tested and can confirm that I still get out of memory when using https. Memory used by the java process gradually reaches its maximum and then sits at that value for quite some time before throwing the out of memory error.

Here is the code im using to replicate the error.

    public void testDavCheckout() throws Exception {

        url = SVNURL.parseURIDecoded("https://yoururl");

        

        SVNClientManager mgr = SVNClientManager.newInstance();

        File tmp = new File(System.getProperty("java.io.tmpdir"), getName());

        delete(tmp);

        tmp.mkdir();

        try {

            mgr.getUpdateClient().doCheckout(url, tmp, SVNRevision.HEAD, SVNRevision.HEAD, false);

        } finally {

            delete(tmp);

        }

        

    }

Thanks

Andrew

On 14 Aug 2007, at 23:22, Alexander Kitaev wrote:

Hello Andrew,

I've just fixed the problem - we forgot to remove debug code that was storing whole file contents to byte[] array! There even was TODO comment like "remove me" :(

Now this code is removed in 1.1.x branch (rev 3366) and I was able to checkout large file (140MB) with -XMx16M (max heap size of 16 megabytes).

Alexander Kitaev,

TMate Software,

http://svnkit.com/ - Java [Sub]Versioning Library!

Andrew Aucott wrote:

Hi Alexander, I've pasted in some example code at the bottom of this email that re-creates the problem. I guess it depends on what the JVM memory is set to but I consistently get an OutOfMemory exception within the doCheckout call.

Thanks

Andrew

That shouldn't be true. During checkout SVNKit processes "windows" not large than 100kb and subsequently writes result of processing to the target file. What code do you use for checking out?

Alexander Kitaev,

TMate Software,

http://svnkit.com/ - Java [Sub]Versioning Library!

package test;

import java.io.File;

import junit.framework.TestCase;

import org.tmatesoft.svn.core.SVNURL;

import org.tmatesoft.svn.core.internal.io.fs.FSRepositoryFactory;

import org.tmatesoft.svn.core.io.SVNRepositoryFactory;

import org.tmatesoft.svn.core.wc.SVNClientManager;

import org.tmatesoft.svn.core.wc.SVNRevision;

public class SVNCheckoutTest extends TestCase {

    private final static File FILE = new File("eclipse-SDK-3.2.1-macosx-carbon.tar.gz");

    private SVNURL url;

    private File repoFile;

    static {

        FSRepositoryFactory.setup();

    }

        public void setUp() throws Exception {

        this.repoFile = File.createTempFile("SVN", "Test");

        delete(repoFile);

        url = SVNRepositoryFactory.createLocalRepository(repoFile, true, true);

   }

        public void tearDown() {

        delete(repoFile);

    }

        public void testCheckOut() throws Exception {

        SVNClientManager mgr = SVNClientManager.newInstance();

        SVNURL fileURL = url.appendPath(FILE.getName(), false);

        mgr.getCommitClient().doImport(FILE, fileURL, "commit big file", false);

        System.out.println("checking out...");

        File tmp = new File(System.getProperty("java.io.tmpdir"), getName());

        delete(tmp);

        tmp.mkdir();

        try {

            mgr.getUpdateClient().doCheckout(url, tmp, SVNRevision.HEAD, SVNRevision.HEAD, false);

        } finally {

            delete(tmp);

        }

    }

            private static boolean delete(File file) {

        boolean success = true;

        if (file.exists()) {

            if (file.isDirectory()) {

                File[] files = file.listFiles();

                for (int i = 0; i < files.length; i++) {

                    success = success && delete(files[i]);

                }

            }

            success = success && file.delete();

        }

        return success;

    }

}

[Alexander Kitaev-3]

Hello Andrew,

Does this problem occurs only with https or with plain http as well? I
couldn't reproduce it locally with http so far. Will try with https as
soon as I'll upload large file to the server.

Alexander Kitaev,
TMate Software,
http://svnkit.com/ - Java [Sub]Versioning Library!

Andrew Aucott wrote:

> Its possible when running with JVM 1.6 to get a stack trace. This may
> help tracking down the problem.
>
> Regards
> Andrew
>
> Exception in thread "Thread-14" java.lang.OutOfMemoryError: Java heap space
>         at
> org.apache.xerces.readers.CharReader.fillCurrentChunk(CharReader.java:255)
>         at
> org.apache.xerces.readers.AbstractCharReader.slowLoadNextChar(AbstractCharReader.java:1370)
>         at
> org.apache.xerces.readers.AbstractCharReader.skipAsciiCharData(AbstractCharReader.java:1287)
>         at
> org.apache.xerces.readers.AbstractCharReader.scanContent(AbstractCharReader.java:1028)
>         at
> org.apache.xerces.framework.XMLDocumentScanner$ContentDispatcher.dispatch(XMLDocumentScanner.java:1094)
>         at
> org.apache.xerces.framework.XMLDocumentScanner.parseSome(XMLDocumentScanner.java:381)
>         at org.apache.xerces.framework.XMLParser.parse(XMLParser.java:1098)
>         at
> org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:634)
>         at
> org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:597)
>         at
> org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.dispatch(HTTPRequest.java:197)
>         at
> org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:285)
>         at
> org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:230)
>         at
> org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:218)
>         at
> org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:219)
>         at
> org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:211)
>         at
> org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:608)
>         at
> org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:162)
>         at
> org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:344)
> On 15 Aug 2007, at 12:50, Andrew Aucott wrote:
>
>> Hi Alexander,
>>
>> I also get the problem when using https. This is where I first
>> encountered the out of memory error.
>>
>> I've just tested and can confirm that I still get out of memory when
>> using https. Memory used by the java process gradually reaches its
>> maximum and then sits at that value for quite some time before
>> throwing the out of memory error.
>>
>> Here is the code im using to replicate the error.
>>
>>     public void testDavCheckout() throws Exception {
>>         url = SVNURL.parseURIDecoded("https://yoururl");
>>        
>>         SVNClientManager mgr = SVNClientManager.newInstance();
>>
>>         File tmp = new File(System.getProperty("java.io.tmpdir"),
>> getName());
>>         delete(tmp);
>>         tmp.mkdir();
>>         try {
>>             mgr.getUpdateClient().doCheckout(url, tmp,
>> SVNRevision.HEAD, SVNRevision.HEAD, false);
>>         } finally {
>>             delete(tmp);
>>         }
>>        
>>     }
>>
>> Thanks
>> Andrew
>>
>>
>> On 14 Aug 2007, at 23:22, Alexander Kitaev wrote:
>>
>>> Hello Andrew,
>>>
>>> I've just fixed the problem - we forgot to remove debug code that was
>>> storing whole file contents to byte[] array! There even was TODO
>>> comment like "remove me" :(
>>>
>>> Now this code is removed in 1.1.x branch (rev 3366) and I was able to
>>> checkout large file (140MB) with -XMx16M (max heap size of 16 megabytes).
>>>
>>> Alexander Kitaev,
>>> TMate Software,
>>> http://svnkit.com/ - Java [Sub]Versioning Library!
>>>
>>> Andrew Aucott wrote:
>>>> Hi Alexander, I've pasted in some example code at the bottom of this
>>>> email that re-creates the problem. I guess it depends on what the
>>>> JVM memory is set to but I consistently get an OutOfMemory exception
>>>> within the doCheckout call.
>>>> Thanks
>>>> Andrew
>>>>> That shouldn't be true. During checkout SVNKit processes "windows"
>>>>> not large than 100kb and subsequently writes result of processing
>>>>> to the target file. What code do you use for checking out?
>>>>>
>>>>> Alexander Kitaev,
>>>>> TMate Software,
>>>>> http://svnkit.com/ - Java [Sub]Versioning Library!
>>>> package test;
>>>> import java.io.File;
>>>> import junit.framework.TestCase;
>>>> import org.tmatesoft.svn.core.SVNURL;
>>>> import org.tmatesoft.svn.core.internal.io.fs.FSRepositoryFactory;
>>>> import org.tmatesoft.svn.core.io.SVNRepositoryFactory;
>>>> import org.tmatesoft.svn.core.wc.SVNClientManager;
>>>> import org.tmatesoft.svn.core.wc.SVNRevision;
>>>> public class SVNCheckoutTest extends TestCase {
>>>>     private final static File FILE = new
>>>> File("eclipse-SDK-3.2.1-macosx-carbon.tar.gz");
>>>>     private SVNURL url;
>>>>     private File repoFile;
>>>>     static {
>>>>         FSRepositoryFactory.setup();
>>>>     }
>>>>         public void setUp() throws Exception {
>>>>         this.repoFile = File.createTempFile("SVN", "Test");
>>>>         delete(repoFile);
>>>>         url = SVNRepositoryFactory.createLocalRepository(repoFile,
>>>> true, true);
>>>>    }
>>>>         public void tearDown() {
>>>>         delete(repoFile);
>>>>     }
>>>>         public void testCheckOut() throws Exception {
>>>>         SVNClientManager mgr = SVNClientManager.newInstance();
>>>>         SVNURL fileURL = url.appendPath(FILE.getName(), false);
>>>>         mgr.getCommitClient().doImport(FILE, fileURL, "commit big
>>>> file", false);
>>>>         System.out.println("checking out...");
>>>>         File tmp = new File(System.getProperty("java.io.tmpdir"),
>>>> getName());
>>>>         delete(tmp);
>>>>         tmp.mkdir();
>>>>         try {
>>>>             mgr.getUpdateClient().doCheckout(url, tmp,
>>>> SVNRevision.HEAD, SVNRevision.HEAD, false);
>>>>         } finally {
>>>>             delete(tmp);
>>>>         }
>>>>     }
>>>>             private static boolean delete(File file) {
>>>>         boolean success = true;
>>>>         if (file.exists()) {
>>>>             if (file.isDirectory()) {
>>>>                 File[] files = file.listFiles();
>>>>                 for (int i = 0; i < files.length; i++) {
>>>>                     success = success && delete(files[i]);
>>>>                 }
>>>>             }
>>>>             success = success && file.delete();
>>>>         }
>>>>         return success;
>>>>     }
>>>> }
>>
>

[Alexander Kitaev-3]

Hello Andrew,

I ran test on https://svn.svnkit.com/repos/test/ - it contains one large
(140Mb) file. Both http and https checkout worked fine with -XMx16M
option - I've ran them on jdk 1.6.0_02 and jdk 1.4.2_15 (Windows Vista).

Is it possible to get RO access to your repository? It could be that
problem only appears with specific version of Subversion (or HTTP)
server, particular file or client OS (what OS you're running checkout on?).

Alexander Kitaev,
TMate Software,
http://svnkit.com/ - Java [Sub]Versioning Library!

Andrew Aucott wrote:

> Its possible when running with JVM 1.6 to get a stack trace. This may
> help tracking down the problem.
>
> Regards
> Andrew
>
> Exception in thread "Thread-14" java.lang.OutOfMemoryError: Java heap space
>         at
> org.apache.xerces.readers.CharReader.fillCurrentChunk(CharReader.java:255)
>         at
> org.apache.xerces.readers.AbstractCharReader.slowLoadNextChar(AbstractCharReader.java:1370)
>         at
> org.apache.xerces.readers.AbstractCharReader.skipAsciiCharData(AbstractCharReader.java:1287)
>         at
> org.apache.xerces.readers.AbstractCharReader.scanContent(AbstractCharReader.java:1028)
>         at
> org.apache.xerces.framework.XMLDocumentScanner$ContentDispatcher.dispatch(XMLDocumentScanner.java:1094)
>         at
> org.apache.xerces.framework.XMLDocumentScanner.parseSome(XMLDocumentScanner.java:381)
>         at org.apache.xerces.framework.XMLParser.parse(XMLParser.java:1098)
>         at
> org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:634)
>         at
> org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:597)
>         at
> org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.dispatch(HTTPRequest.java:197)
>         at
> org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:285)
>         at
> org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:230)
>         at
> org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:218)
>         at
> org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:219)
>         at
> org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:211)
>         at
> org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:608)
>         at
> org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:162)
>         at
> org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:344)
> On 15 Aug 2007, at 12:50, Andrew Aucott wrote:
>
>> Hi Alexander,
>>
>> I also get the problem when using https. This is where I first
>> encountered the out of memory error.
>>
>> I've just tested and can confirm that I still get out of memory when
>> using https. Memory used by the java process gradually reaches its
>> maximum and then sits at that value for quite some time before
>> throwing the out of memory error.
>>
>> Here is the code im using to replicate the error.
>>
>>     public void testDavCheckout() throws Exception {
>>         url = SVNURL.parseURIDecoded("https://yoururl");
>>        
>>         SVNClientManager mgr = SVNClientManager.newInstance();
>>
>>         File tmp = new File(System.getProperty("java.io.tmpdir"),
>> getName());
>>         delete(tmp);
>>         tmp.mkdir();
>>         try {
>>             mgr.getUpdateClient().doCheckout(url, tmp,
>> SVNRevision.HEAD, SVNRevision.HEAD, false);
>>         } finally {
>>             delete(tmp);
>>         }
>>        
>>     }
>>
>> Thanks
>> Andrew
>>
>>
>> On 14 Aug 2007, at 23:22, Alexander Kitaev wrote:
>>
>>> Hello Andrew,
>>>
>>> I've just fixed the problem - we forgot to remove debug code that was
>>> storing whole file contents to byte[] array! There even was TODO
>>> comment like "remove me" :(
>>>
>>> Now this code is removed in 1.1.x branch (rev 3366) and I was able to
>>> checkout large file (140MB) with -XMx16M (max heap size of 16 megabytes).
>>>
>>> Alexander Kitaev,
>>> TMate Software,
>>> http://svnkit.com/ - Java [Sub]Versioning Library!
>>>
>>> Andrew Aucott wrote:
>>>> Hi Alexander, I've pasted in some example code at the bottom of this
>>>> email that re-creates the problem. I guess it depends on what the
>>>> JVM memory is set to but I consistently get an OutOfMemory exception
>>>> within the doCheckout call.
>>>> Thanks
>>>> Andrew
>>>>> That shouldn't be true. During checkout SVNKit processes "windows"
>>>>> not large than 100kb and subsequently writes result of processing
>>>>> to the target file. What code do you use for checking out?
>>>>>
>>>>> Alexander Kitaev,
>>>>> TMate Software,
>>>>> http://svnkit.com/ - Java [Sub]Versioning Library!
>>>> package test;
>>>> import java.io.File;
>>>> import junit.framework.TestCase;
>>>> import org.tmatesoft.svn.core.SVNURL;
>>>> import org.tmatesoft.svn.core.internal.io.fs.FSRepositoryFactory;
>>>> import org.tmatesoft.svn.core.io.SVNRepositoryFactory;
>>>> import org.tmatesoft.svn.core.wc.SVNClientManager;
>>>> import org.tmatesoft.svn.core.wc.SVNRevision;
>>>> public class SVNCheckoutTest extends TestCase {
>>>>     private final static File FILE = new
>>>> File("eclipse-SDK-3.2.1-macosx-carbon.tar.gz");
>>>>     private SVNURL url;
>>>>     private File repoFile;
>>>>     static {
>>>>         FSRepositoryFactory.setup();
>>>>     }
>>>>         public void setUp() throws Exception {
>>>>         this.repoFile = File.createTempFile("SVN", "Test");
>>>>         delete(repoFile);
>>>>         url = SVNRepositoryFactory.createLocalRepository(repoFile,
>>>> true, true);
>>>>    }
>>>>         public void tearDown() {
>>>>         delete(repoFile);
>>>>     }
>>>>         public void testCheckOut() throws Exception {
>>>>         SVNClientManager mgr = SVNClientManager.newInstance();
>>>>         SVNURL fileURL = url.appendPath(FILE.getName(), false);
>>>>         mgr.getCommitClient().doImport(FILE, fileURL, "commit big
>>>> file", false);
>>>>         System.out.println("checking out...");
>>>>         File tmp = new File(System.getProperty("java.io.tmpdir"),
>>>> getName());
>>>>         delete(tmp);
>>>>         tmp.mkdir();
>>>>         try {
>>>>             mgr.getUpdateClient().doCheckout(url, tmp,
>>>> SVNRevision.HEAD, SVNRevision.HEAD, false);
>>>>         } finally {
>>>>             delete(tmp);
>>>>         }
>>>>     }
>>>>             private static boolean delete(File file) {
>>>>         boolean success = true;
>>>>         if (file.exists()) {
>>>>             if (file.isDirectory()) {
>>>>                 File[] files = file.listFiles();
>>>>                 for (int i = 0; i < files.length; i++) {
>>>>                     success = success && delete(files[i]);
>>>>                 }
>>>>             }
>>>>             success = success && file.delete();
>>>>         }
>>>>         return success;
>>>>     }
>>>> }
>>
>

[Andrew Aucott]

On 15 Aug 2007, at 17:27, Alexander Kitaev wrote:

> Hello Andrew,
>
> I ran test on https://svn.svnkit.com/repos/test/ - it contains one  
> large (140Mb) file. Both http and https checkout worked fine with -
> XMx16M option - I've ran them on jdk 1.6.0_02 and jdk 1.4.2_15  
> (Windows Vista).
>
> Is it possible to get RO access to your repository? It could be  
> that problem only appears with specific version of Subversion (or  
> HTTP) server, particular file or client OS (what OS you're running  
> checkout on?).
>

Hi Alexander,

Ill get the RO access setup.

BTW we are using release 1.1.2 of svnkit, dont know if this makes a  
difference but ill try using the 1.1.x branch.

I get the issue running the checkout on both Windows 2003 server and  
OSX 10.4.9.

Andrew

[Alexander Kitaev-3]

Hello Andrew,

 > Ill get the RO access setup.
Thanks, it will help to reproduce and fix the problem.

 > BTW we are using release 1.1.2 of svnkit, dont know if this makes a
 > difference but ill try using the 1.1.x branch.
Upgrading could make a huge difference! You may get latest SVNKit binary
from 1.1.x branch at https://teamcity.svnkit.com/ - follow 'Artifacts'
link in "1.1.x" build configuration to get binaries.

Alexander Kitaev,
TMate Software,
http://svnkit.com/ - Java [Sub]Versioning Library!

Andrew Aucott wrote:

>
> On 15 Aug 2007, at 17:27, Alexander Kitaev wrote:
>
>> Hello Andrew,
>>
>> I ran test on https://svn.svnkit.com/repos/test/ - it contains one
>> large (140Mb) file. Both http and https checkout worked fine with
>> -XMx16M option - I've ran them on jdk 1.6.0_02 and jdk 1.4.2_15
>> (Windows Vista).
>>
>> Is it possible to get RO access to your repository? It could be that
>> problem only appears with specific version of Subversion (or HTTP)
>> server, particular file or client OS (what OS you're running checkout
>> on?).
>>
> Hi Alexander,
>
> Ill get the RO access setup.
>
> BTW we are using release 1.1.2 of svnkit, dont know if this makes a
> difference but ill try using the 1.1.x branch.
>
> I get the issue running the checkout on both Windows 2003 server and OSX
> 10.4.9.
>
> Andrew
>