listFiles produce multiple entrys for some files..

[michael.studer]

Hi there

Im working since yesterday with jcifs and i faced a small issue. When i listFiles() on a smb file, i get some files more than once.
It even has a patter.. Its like:
1 File - list has 1 File
2 Files - list has 3 Files
3 Files - list has 6 Files

and so on

Is this a possible bug in the API or did i set any settings wrong?

thanks for your help

Michael

[Michael B Allen]

On Thu, Oct 8, 2009 at 6:07 AM,  <michael.studer@...> wrote:

>
> Hi there
>
> Im working since yesterday with jcifs and i faced a small issue. When i
> listFiles() on a smb file, i get some files more than once.
> It even has a patter.. Its like:
> 1 File - list has 1 File
> 2 Files - list has 3 Files
> 3 Files - list has 6 Files
>
> and so on
>
> Is this a possible bug in the API or did i set any settings wrong?

Hi Michael,

I don't know. What is the server? Post your code. Does the problem
also occur with the examples/ListFiles.java example?

Mike

--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

[michael.studer]

Hi again

I figured out, the problem seems to occur only on AS400 shares. I did not face it on windows servers.

Again, i try to explain the problem. When i use the "smbFile.listFiles()" method on a file, which is located inside a AS400 share, the amount of returned files is multiple times the actual amount it should return.
For example: 2 Files in the share returns me 3 Files, where one File is twice in the collection. This would not be such a big deal, its just getting nasty when you have folders with a lot of Files inside.
While i tested, i faced some folders with arround 5000+ Files. This results in arround 250'000 returned SMBFile Objects, with a total used amount of like 500+MB of memory and a runtime out of memory.

Heres the full code for my directory reading method(s):

        public HashSet<String> getFilesList(String relativePath, boolean recursive, boolean includeFolders) throws MalformedURLException, SmbException, ExsException {
               
                HashSet<String> collection = new HashSet<String>();
                HashSet<String> unreadable = new HashSet<String>();
               
                getFilesList(relativePath,recursive,includeFolders,false,collection, unreadable);        
                       
                return collection;
               
        }
       
        private void getFilesList(String path, boolean recursive, boolean includeFolders, boolean isFullPath, HashSet<String> collection, HashSet<String> unreadable) throws MalformedURLException, SmbException, ExsException {

                if(path == null) {
                        path = "";
                }
               
                if(path.trim().length() > 0 && (!path.trim().endsWith("/") && !path.trim().endsWith("\\"))) {
                        path = path + "/";
                }
               
                String url = "";
               
                if(isFullPath) {
                        url = path;
                } else {
                        url = generateURLString(path);                
                }

                SmbFile smbFolder = new SmbFile(url);

                if(!smbFolder.exists() || !smbFolder.isDirectory()) {
                        throw new ExsException("Error: Folder cannot be read from smb share: " + smbFolder.getPath());
                }
               
                if(!smbFolder.canRead()) {
                        unreadable.add(smbFolder.getPath());
                        return;
                }

                // this helps preventing heap overflows but it wont prevent it when to many files at once are opend
                HashSet<SmbFile> uniqueList = new HashSet<SmbFile>();
                for(SmbFile currentSmbFile : smbFolder.listFiles()) {
                        uniqueList.add(currentSmbFile);
                }
               
                for(SmbFile currentSmbFile : uniqueList) {
                                               
                        String currentPath = currentSmbFile.getPath();
               
                        if(currentSmbFile.isFile()) {
                                currentSmbFile = null;
                                collection.add(currentPath);
                               
                        } else if(currentSmbFile.isDirectory()) {
                                int size = 0;
                                try {
                                        String[] list = currentSmbFile.list();
                                        size = list.length;
                                        list = null;
                                } catch(Throwable t) {
                                        continue;
                                }
                               
                                currentSmbFile = null;
                                if(includeFolders) {
                                        collection.add(currentPath);
                                }
                                if(recursive) {
                                        if(size > 0) {
                                                getFilesList(currentPath, recursive, includeFolders, true, collection, unreadable);
                                        }
                                }                
                        }                        
                }
               
                uniqueList.clear();
                uniqueList = null;
        }

[Michael B Allen]

You could send me a packet capture. But I don't have access to AS400
and I don't really have "Free" time so unless it's an obvious fix this
could sit for a while. But I know JCIFS has worked with mainframes
before so I have to wonder why this is showing up now. Is AS400 really
old or really new or unmaintained?

Mike

On Mon, Oct 12, 2009 at 3:17 AM,  <michael.studer@...> wrote:

>
> Hi again
>
> I figured out, the problem seems to occur only on AS400 shares. I did not
> face it on windows servers.
>
> Again, i try to explain the problem. When i use the "smbFile.listFiles()"
> method on a file, which is located inside a AS400 share, the amount of
> returned files is multiple times the actual amount it should return.
> For example: 2 Files in the share returns me 3 Files, where one File is
> twice in the collection. This would not be such a big deal, its just getting
> nasty when you have folders with a lot of Files inside.
> While i tested, i faced some folders with arround 5000+ Files. This results
> in arround 250'000 returned SMBFile Objects, with a total used amount of
> like 500+MB of memory and a runtime out of memory.
>
> Heres the full code for my directory reading method(s):
>
>         public HashSet<String> getFilesList(String relativePath, boolean
> recursive, boolean includeFolders) throws MalformedURLException,
> SmbException, ExsException {
>
>                 HashSet<String> collection = new HashSet<String>();
>                 HashSet<String> unreadable = new HashSet<String>();
>
>
> getFilesList(relativePath,recursive,includeFolders,false,collection,
> unreadable);
>
>                 return collection;
>
>         }
>
>         private void getFilesList(String path, boolean recursive, boolean
> includeFolders, boolean isFullPath, HashSet<String> collection,
> HashSet<String> unreadable) throws MalformedURLException, SmbException,
> ExsException {
>
>                 if(path == null) {
>                         path = "";
>                 }
>
>                 if(path.trim().length() > 0 && (!path.trim().endsWith("/")
> && !path.trim().endsWith("\\"))) {
>                         path = path + "/";
>                 }
>
>                 String url = "";
>
>                 if(isFullPath) {
>                         url = path;
>                 } else {
>                         url = generateURLString(path);
>                 }
>
>                 SmbFile smbFolder = new SmbFile(url);
>
>                 if(!smbFolder.exists() || !smbFolder.isDirectory()) {
>                         throw new ExsException("Error: Folder cannot be read
> from smb share: " + smbFolder.getPath());
>                 }
>
>                 if(!smbFolder.canRead()) {
>                         unreadable.add(smbFolder.getPath());
>                         return;
>                 }
>
>                 // this helps preventing heap overflows but it wont prevent
> it when to many files at once are opend
>                 HashSet<SmbFile> uniqueList = new HashSet<SmbFile>();
>                 for(SmbFile currentSmbFile : smbFolder.listFiles()) {
>                         uniqueList.add(currentSmbFile);
>                 }
>
>                 for(SmbFile currentSmbFile : uniqueList) {
>
>                         String currentPath = currentSmbFile.getPath();
>
>                         if(currentSmbFile.isFile()) {
>                                 currentSmbFile = null;
>                                 collection.add(currentPath);
>
>                         } else if(currentSmbFile.isDirectory()) {
>                                 int size = 0;
>                                 try {
>                                         String[] list =
> currentSmbFile.list();
>                                         size = list.length;
>                                         list = null;
>                                 } catch(Throwable t) {
>                                         continue;
>                                 }
>
>                                 currentSmbFile = null;
>                                 if(includeFolders) {
>                                         collection.add(currentPath);
>                                 }
>                                 if(recursive) {
>                                         if(size > 0) {
>                                                 getFilesList(currentPath,
> recursive, includeFolders, true, collection, unreadable);
>                                         }
>                                 }
>                         }
>                 }
>
>                 uniqueList.clear();
>                 uniqueList = null;
>         }
>
>
>

--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

[Paul Clapham]

I have access to AS400 and I'm willing to work on this problem. I asked michael.studer (accidentally off-list) to send me a short compilable program which I could modify and test. So far I just got a large lot of code which was way too much to deal with. Seems to me that a program to test this should be less than 50 lines of code.

PC2

-----Original Message-----
From: jcifs-bounces@... [mailto:jcifs-bounces@...] On Behalf Of Michael B Allen
Sent: October 26, 2009 08:25
To: michael.studer@...
Cc: jcifs@...
Subject: Re: [jcifs] listFiles produce multiple entrys for some files..

You could send me a packet capture. But I don't have access to AS400 and I don't really have "Free" time so unless it's an obvious fix this could sit for a while. But I know JCIFS has worked with mainframes before so I have to wonder why this is showing up now. Is AS400 really old or really new or unmaintained?

Mike

On Mon, Oct 12, 2009 at 3:17 AM,  <michael.studer@...> wrote:

>
> Hi again
>
> I figured out, the problem seems to occur only on AS400 shares. I did
> not face it on windows servers.
>
> Again, i try to explain the problem. When i use the "smbFile.listFiles()"
> method on a file, which is located inside a AS400 share, the amount of
> returned files is multiple times the actual amount it should return.
> For example: 2 Files in the share returns me 3 Files, where one File
> is twice in the collection. This would not be such a big deal, its
> just getting nasty when you have folders with a lot of Files inside.
> While i tested, i faced some folders with arround 5000+ Files. This
> results in arround 250'000 returned SMBFile Objects, with a total used
> amount of like 500+MB of memory and a runtime out of memory.
>
> Heres the full code for my directory reading method(s):
>
>         public HashSet<String> getFilesList(String relativePath,
> boolean recursive, boolean includeFolders) throws
> MalformedURLException, SmbException, ExsException {
>
>                 HashSet<String> collection = new HashSet<String>();
>                 HashSet<String> unreadable = new HashSet<String>();
>
>
> getFilesList(relativePath,recursive,includeFolders,false,collection,
> unreadable);
>
>                 return collection;
>
>         }
>
>         private void getFilesList(String path, boolean recursive,
> boolean includeFolders, boolean isFullPath, HashSet<String>
> collection, HashSet<String> unreadable) throws MalformedURLException,
> SmbException, ExsException {
>
>                 if(path == null) {
>                         path = "";
>                 }
>
>                 if(path.trim().length() > 0 &&
> (!path.trim().endsWith("/") && !path.trim().endsWith("\\"))) {
>                         path = path + "/";
>                 }
>
>                 String url = "";
>
>                 if(isFullPath) {
>                         url = path;
>                 } else {
>                         url = generateURLString(path);
>                 }
>
>                 SmbFile smbFolder = new SmbFile(url);
>
>                 if(!smbFolder.exists() || !smbFolder.isDirectory()) {
>                         throw new ExsException("Error: Folder cannot
> be read from smb share: " + smbFolder.getPath());
>                 }
>
>                 if(!smbFolder.canRead()) {
>                         unreadable.add(smbFolder.getPath());
>                         return;
>                 }
>
>                 // this helps preventing heap overflows but it wont
> prevent it when to many files at once are opend
>                 HashSet<SmbFile> uniqueList = new HashSet<SmbFile>();
>                 for(SmbFile currentSmbFile : smbFolder.listFiles()) {
>                         uniqueList.add(currentSmbFile);
>                 }
>
>                 for(SmbFile currentSmbFile : uniqueList) {
>
>                         String currentPath = currentSmbFile.getPath();
>
>                         if(currentSmbFile.isFile()) {
>                                 currentSmbFile = null;
>                                 collection.add(currentPath);
>
>                         } else if(currentSmbFile.isDirectory()) {
>                                 int size = 0;
>                                 try {
>                                         String[] list =
> currentSmbFile.list();
>                                         size = list.length;
>                                         list = null;
>                                 } catch(Throwable t) {
>                                         continue;
>                                 }
>
>                                 currentSmbFile = null;
>                                 if(includeFolders) {
>                                         collection.add(currentPath);
>                                 }
>                                 if(recursive) {
>                                         if(size > 0) {
>                                                
> getFilesList(currentPath, recursive, includeFolders, true, collection,
> unreadable);
>                                         }
>                                 }
>                         }
>                 }
>
>                 uniqueList.clear();
>                 uniqueList = null;
>         }
>
>
>

--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

[Michael B Allen]

Hi Paul and Michael,

Better still, everyone should be using examples/ListFiles.java to
start so that we are all on the same page.

Thanks for helping out.

Mike

On Mon, Oct 26, 2009 at 11:29 AM, Clapham, Paul <pclapham@...> wrote:

> I have access to AS400 and I'm willing to work on this problem. I asked michael.studer (accidentally off-list) to send me a short compilable program which I could modify and test. So far I just got a large lot of code which was way too much to deal with. Seems to me that a program to test this should be less than 50 lines of code.
>
> PC2
>
> -----Original Message-----
> From: jcifs-bounces@... [mailto:jcifs-bounces@...] On Behalf Of Michael B Allen
> Sent: October 26, 2009 08:25
> To: michael.studer@...
> Cc: jcifs@...
> Subject: Re: [jcifs] listFiles produce multiple entrys for some files..
>
> You could send me a packet capture. But I don't have access to AS400 and I don't really have "Free" time so unless it's an obvious fix this could sit for a while. But I know JCIFS has worked with mainframes before so I have to wonder why this is showing up now. Is AS400 really old or really new or unmaintained?
>
> Mike
>
> On Mon, Oct 12, 2009 at 3:17 AM,  <michael.studer@...> wrote:
>>
>> Hi again
>>
>> I figured out, the problem seems to occur only on AS400 shares. I did
>> not face it on windows servers.
>>
>> Again, i try to explain the problem. When i use the "smbFile.listFiles()"
>> method on a file, which is located inside a AS400 share, the amount of
>> returned files is multiple times the actual amount it should return.
>> For example: 2 Files in the share returns me 3 Files, where one File
>> is twice in the collection. This would not be such a big deal, its
>> just getting nasty when you have folders with a lot of Files inside.
>> While i tested, i faced some folders with arround 5000+ Files. This
>> results in arround 250'000 returned SMBFile Objects, with a total used
>> amount of like 500+MB of memory and a runtime out of memory.
>>
>> Heres the full code for my directory reading method(s):
>>
>>         public HashSet<String> getFilesList(String relativePath,
>> boolean recursive, boolean includeFolders) throws
>> MalformedURLException, SmbException, ExsException {
>>
>>                 HashSet<String> collection = new HashSet<String>();
>>                 HashSet<String> unreadable = new HashSet<String>();
>>
>>
>> getFilesList(relativePath,recursive,includeFolders,false,collection,
>> unreadable);
>>
>>                 return collection;
>>
>>         }
>>
>>         private void getFilesList(String path, boolean recursive,
>> boolean includeFolders, boolean isFullPath, HashSet<String>
>> collection, HashSet<String> unreadable) throws MalformedURLException,
>> SmbException, ExsException {
>>
>>                 if(path == null) {
>>                         path = "";
>>                 }
>>
>>                 if(path.trim().length() > 0 &&
>> (!path.trim().endsWith("/") && !path.trim().endsWith("\\"))) {
>>                         path = path + "/";
>>                 }
>>
>>                 String url = "";
>>
>>                 if(isFullPath) {
>>                         url = path;
>>                 } else {
>>                         url = generateURLString(path);
>>                 }
>>
>>                 SmbFile smbFolder = new SmbFile(url);
>>
>>                 if(!smbFolder.exists() || !smbFolder.isDirectory()) {
>>                         throw new ExsException("Error: Folder cannot
>> be read from smb share: " + smbFolder.getPath());
>>                 }
>>
>>                 if(!smbFolder.canRead()) {
>>                         unreadable.add(smbFolder.getPath());
>>                         return;
>>                 }
>>
>>                 // this helps preventing heap overflows but it wont
>> prevent it when to many files at once are opend
>>                 HashSet<SmbFile> uniqueList = new HashSet<SmbFile>();
>>                 for(SmbFile currentSmbFile : smbFolder.listFiles()) {
>>                         uniqueList.add(currentSmbFile);
>>                 }
>>
>>                 for(SmbFile currentSmbFile : uniqueList) {
>>
>>                         String currentPath = currentSmbFile.getPath();
>>
>>                         if(currentSmbFile.isFile()) {
>>                                 currentSmbFile = null;
>>                                 collection.add(currentPath);
>>
>>                         } else if(currentSmbFile.isDirectory()) {
>>                                 int size = 0;
>>                                 try {
>>                                         String[] list =
>> currentSmbFile.list();
>>                                         size = list.length;
>>                                         list = null;
>>                                 } catch(Throwable t) {
>>                                         continue;
>>                                 }
>>
>>                                 currentSmbFile = null;
>>                                 if(includeFolders) {
>>                                         collection.add(currentPath);
>>                                 }
>>                                 if(recursive) {
>>                                         if(size > 0) {
>>
>> getFilesList(currentPath, recursive, includeFolders, true, collection,
>> unreadable);
>>                                         }
>>                                 }
>>                         }
>>                 }
>>
>>                 uniqueList.clear();
>>                 uniqueList = null;
>>         }
>>
>>
>>
>
>
>
> --
> Michael B Allen
> Java Active Directory Integration
> http://www.ioplex.com/
>

--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

[Paul Clapham]

Ah, okay. It was 2004 when I first integrated jcifs into my application, so I'd forgotten that directory even existed. Let me see what I can do with ListFiles.

PC2

-----Original Message-----
From: Michael B Allen [mailto:ioplex@...]
Sent: October 26, 2009 08:55
To: Clapham, Paul
Cc: michael.studer@...; jcifs@...
Subject: Re: [jcifs] listFiles produce multiple entrys for some files..

Hi Paul and Michael,

Better still, everyone should be using examples/ListFiles.java to start so that we are all on the same page.

Thanks for helping out.

Mike

On Mon, Oct 26, 2009 at 11:29 AM, Clapham, Paul <pclapham@...> wrote:

> I have access to AS400 and I'm willing to work on this problem. I asked michael.studer (accidentally off-list) to send me a short compilable program which I could modify and test. So far I just got a large lot of code which was way too much to deal with. Seems to me that a program to test this should be less than 50 lines of code.
>
> PC2
>
> -----Original Message-----
> From: jcifs-bounces@...
> [mailto:jcifs-bounces@...] On Behalf Of Michael B Allen
> Sent: October 26, 2009 08:25
> To: michael.studer@...
> Cc: jcifs@...
> Subject: Re: [jcifs] listFiles produce multiple entrys for some files..
>
> You could send me a packet capture. But I don't have access to AS400 and I don't really have "Free" time so unless it's an obvious fix this could sit for a while. But I know JCIFS has worked with mainframes before so I have to wonder why this is showing up now. Is AS400 really old or really new or unmaintained?
>
> Mike
>
> On Mon, Oct 12, 2009 at 3:17 AM,  <michael.studer@...> wrote:
>>
>> Hi again
>>
>> I figured out, the problem seems to occur only on AS400 shares. I did
>> not face it on windows servers.
>>
>> Again, i try to explain the problem. When i use the "smbFile.listFiles()"
>> method on a file, which is located inside a AS400 share, the amount
>> of returned files is multiple times the actual amount it should return.
>> For example: 2 Files in the share returns me 3 Files, where one File
>> is twice in the collection. This would not be such a big deal, its
>> just getting nasty when you have folders with a lot of Files inside.
>> While i tested, i faced some folders with arround 5000+ Files. This
>> results in arround 250'000 returned SMBFile Objects, with a total
>> used amount of like 500+MB of memory and a runtime out of memory.
>>
>> Heres the full code for my directory reading method(s):
>>
>>         public HashSet<String> getFilesList(String relativePath,
>> boolean recursive, boolean includeFolders) throws
>> MalformedURLException, SmbException, ExsException {
>>
>>                 HashSet<String> collection = new HashSet<String>();
>>                 HashSet<String> unreadable = new HashSet<String>();
>>
>>
>> getFilesList(relativePath,recursive,includeFolders,false,collection,
>> unreadable);
>>
>>                 return collection;
>>
>>         }
>>
>>         private void getFilesList(String path, boolean recursive,
>> boolean includeFolders, boolean isFullPath, HashSet<String>
>> collection, HashSet<String> unreadable) throws MalformedURLException,
>> SmbException, ExsException {
>>
>>                 if(path == null) {
>>                         path = "";
>>                 }
>>
>>                 if(path.trim().length() > 0 &&
>> (!path.trim().endsWith("/") && !path.trim().endsWith("\\"))) {
>>                         path = path + "/";
>>                 }
>>
>>                 String url = "";
>>
>>                 if(isFullPath) {
>>                         url = path;
>>                 } else {
>>                         url = generateURLString(path);
>>                 }
>>
>>                 SmbFile smbFolder = new SmbFile(url);
>>
>>                 if(!smbFolder.exists() || !smbFolder.isDirectory()) {
>>                         throw new ExsException("Error: Folder cannot
>> be read from smb share: " + smbFolder.getPath());
>>                 }
>>
>>                 if(!smbFolder.canRead()) {
>>                         unreadable.add(smbFolder.getPath());
>>                         return;
>>                 }
>>
>>                 // this helps preventing heap overflows but it wont
>> prevent it when to many files at once are opend
>>                 HashSet<SmbFile> uniqueList = new HashSet<SmbFile>();
>>                 for(SmbFile currentSmbFile : smbFolder.listFiles()) {
>>                         uniqueList.add(currentSmbFile);
>>                 }
>>
>>                 for(SmbFile currentSmbFile : uniqueList) {
>>
>>                         String currentPath =
>> currentSmbFile.getPath();
>>
>>                         if(currentSmbFile.isFile()) {
>>                                 currentSmbFile = null;
>>                                 collection.add(currentPath);
>>
>>                         } else if(currentSmbFile.isDirectory()) {
>>                                 int size = 0;
>>                                 try {
>>                                         String[] list =
>> currentSmbFile.list();
>>                                         size = list.length;
>>                                         list = null;
>>                                 } catch(Throwable t) {
>>                                         continue;
>>                                 }
>>
>>                                 currentSmbFile = null;
>>                                 if(includeFolders) {
>>                                         collection.add(currentPath);
>>                                 }
>>                                 if(recursive) {
>>                                         if(size > 0) {
>>
>> getFilesList(currentPath, recursive, includeFolders, true,
>> collection, unreadable);
>>                                         }
>>                                 }
>>                         }
>>                 }
>>
>>                 uniqueList.clear();
>>                 uniqueList = null;
>>         }
>>
>>
>>
>
>
>
> --
> Michael B Allen
> Java Active Directory Integration
> http://www.ioplex.com/
>

--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

[Paul Clapham]

Hi Mike,

I sent you a log from that program off-list.

PC2

-----Original Message-----
From: Michael B Allen [mailto:ioplex@...]
Sent: October 26, 2009 08:55
To: Clapham, Paul
Cc: michael.studer@...; jcifs@...
Subject: Re: [jcifs] listFiles produce multiple entrys for some files..

Hi Paul and Michael,

Better still, everyone should be using examples/ListFiles.java to start so that we are all on the same page.

Thanks for helping out.

Mike

On Mon, Oct 26, 2009 at 11:29 AM, Clapham, Paul <pclapham@...> wrote:

> I have access to AS400 and I'm willing to work on this problem. I asked michael.studer (accidentally off-list) to send me a short compilable program which I could modify and test. So far I just got a large lot of code which was way too much to deal with. Seems to me that a program to test this should be less than 50 lines of code.
>
> PC2
>
> -----Original Message-----
> From: jcifs-bounces@...
> [mailto:jcifs-bounces@...] On Behalf Of Michael B Allen
> Sent: October 26, 2009 08:25
> To: michael.studer@...
> Cc: jcifs@...
> Subject: Re: [jcifs] listFiles produce multiple entrys for some files..
>
> You could send me a packet capture. But I don't have access to AS400 and I don't really have "Free" time so unless it's an obvious fix this could sit for a while. But I know JCIFS has worked with mainframes before so I have to wonder why this is showing up now. Is AS400 really old or really new or unmaintained?
>
> Mike
>
> On Mon, Oct 12, 2009 at 3:17 AM,  <michael.studer@...> wrote:
>>
>> Hi again
>>
>> I figured out, the problem seems to occur only on AS400 shares. I did
>> not face it on windows servers.
>>
>> Again, i try to explain the problem. When i use the "smbFile.listFiles()"
>> method on a file, which is located inside a AS400 share, the amount
>> of returned files is multiple times the actual amount it should return.
>> For example: 2 Files in the share returns me 3 Files, where one File
>> is twice in the collection. This would not be such a big deal, its
>> just getting nasty when you have folders with a lot of Files inside.
>> While i tested, i faced some folders with arround 5000+ Files. This
>> results in arround 250'000 returned SMBFile Objects, with a total
>> used amount of like 500+MB of memory and a runtime out of memory.
>>
>> Heres the full code for my directory reading method(s):
>>
>>         public HashSet<String> getFilesList(String relativePath,
>> boolean recursive, boolean includeFolders) throws
>> MalformedURLException, SmbException, ExsException {
>>
>>                 HashSet<String> collection = new HashSet<String>();
>>                 HashSet<String> unreadable = new HashSet<String>();
>>
>>
>> getFilesList(relativePath,recursive,includeFolders,false,collection,
>> unreadable);
>>
>>                 return collection;
>>
>>         }
>>
>>         private void getFilesList(String path, boolean recursive,
>> boolean includeFolders, boolean isFullPath, HashSet<String>
>> collection, HashSet<String> unreadable) throws MalformedURLException,
>> SmbException, ExsException {
>>
>>                 if(path == null) {
>>                         path = "";
>>                 }
>>
>>                 if(path.trim().length() > 0 &&
>> (!path.trim().endsWith("/") && !path.trim().endsWith("\\"))) {
>>                         path = path + "/";
>>                 }
>>
>>                 String url = "";
>>
>>                 if(isFullPath) {
>>                         url = path;
>>                 } else {
>>                         url = generateURLString(path);
>>                 }
>>
>>                 SmbFile smbFolder = new SmbFile(url);
>>
>>                 if(!smbFolder.exists() || !smbFolder.isDirectory()) {
>>                         throw new ExsException("Error: Folder cannot
>> be read from smb share: " + smbFolder.getPath());
>>                 }
>>
>>                 if(!smbFolder.canRead()) {
>>                         unreadable.add(smbFolder.getPath());
>>                         return;
>>                 }
>>
>>                 // this helps preventing heap overflows but it wont
>> prevent it when to many files at once are opend
>>                 HashSet<SmbFile> uniqueList = new HashSet<SmbFile>();
>>                 for(SmbFile currentSmbFile : smbFolder.listFiles()) {
>>                         uniqueList.add(currentSmbFile);
>>                 }
>>
>>                 for(SmbFile currentSmbFile : uniqueList) {
>>
>>                         String currentPath =
>> currentSmbFile.getPath();
>>
>>                         if(currentSmbFile.isFile()) {
>>                                 currentSmbFile = null;
>>                                 collection.add(currentPath);
>>
>>                         } else if(currentSmbFile.isDirectory()) {
>>                                 int size = 0;
>>                                 try {
>>                                         String[] list =
>> currentSmbFile.list();
>>                                         size = list.length;
>>                                         list = null;
>>                                 } catch(Throwable t) {
>>                                         continue;
>>                                 }
>>
>>                                 currentSmbFile = null;
>>                                 if(includeFolders) {
>>                                         collection.add(currentPath);
>>                                 }
>>                                 if(recursive) {
>>                                         if(size > 0) {
>>
>> getFilesList(currentPath, recursive, includeFolders, true,
>> collection, unreadable);
>>                                         }
>>                                 }
>>                         }
>>                 }
>>
>>                 uniqueList.clear();
>>                 uniqueList = null;
>>         }
>>
>>
>>
>
>
>
> --
> Michael B Allen
> Java Active Directory Integration
> http://www.ioplex.com/
>

--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

[Michael B Allen]

Oh yeah. Wow. That's pretty messed up. Can you send me a packet capture?

On Mon, Oct 26, 2009 at 12:49 PM, Clapham, Paul <pclapham@...> wrote:

> Hi Mike,
>
> I sent you a log from that program off-list.
>
> PC2
>
> -----Original Message-----
> From: Michael B Allen [mailto:ioplex@...]
> Sent: October 26, 2009 08:55
> To: Clapham, Paul
> Cc: michael.studer@...; jcifs@...
> Subject: Re: [jcifs] listFiles produce multiple entrys for some files..
>
> Hi Paul and Michael,
>
> Better still, everyone should be using examples/ListFiles.java to start so that we are all on the same page.
>
> Thanks for helping out.
>
> Mike
>
> On Mon, Oct 26, 2009 at 11:29 AM, Clapham, Paul <pclapham@...> wrote:
>> I have access to AS400 and I'm willing to work on this problem. I asked michael.studer (accidentally off-list) to send me a short compilable program which I could modify and test. So far I just got a large lot of code which was way too much to deal with. Seems to me that a program to test this should be less than 50 lines of code.
>>
>> PC2
>>
>> -----Original Message-----
>> From: jcifs-bounces@...
>> [mailto:jcifs-bounces@...] On Behalf Of Michael B Allen
>> Sent: October 26, 2009 08:25
>> To: michael.studer@...
>> Cc: jcifs@...
>> Subject: Re: [jcifs] listFiles produce multiple entrys for some files..
>>
>> You could send me a packet capture. But I don't have access to AS400 and I don't really have "Free" time so unless it's an obvious fix this could sit for a while. But I know JCIFS has worked with mainframes before so I have to wonder why this is showing up now. Is AS400 really old or really new or unmaintained?
>>
>> Mike
>>
>> On Mon, Oct 12, 2009 at 3:17 AM,  <michael.studer@...> wrote:
>>>
>>> Hi again
>>>
>>> I figured out, the problem seems to occur only on AS400 shares. I did
>>> not face it on windows servers.
>>>
>>> Again, i try to explain the problem. When i use the "smbFile.listFiles()"
>>> method on a file, which is located inside a AS400 share, the amount
>>> of returned files is multiple times the actual amount it should return.
>>> For example: 2 Files in the share returns me 3 Files, where one File
>>> is twice in the collection. This would not be such a big deal, its
>>> just getting nasty when you have folders with a lot of Files inside.
>>> While i tested, i faced some folders with arround 5000+ Files. This
>>> results in arround 250'000 returned SMBFile Objects, with a total
>>> used amount of like 500+MB of memory and a runtime out of memory.
>>>
>>> Heres the full code for my directory reading method(s):
>>>
>>>         public HashSet<String> getFilesList(String relativePath,
>>> boolean recursive, boolean includeFolders) throws
>>> MalformedURLException, SmbException, ExsException {
>>>
>>>                 HashSet<String> collection = new HashSet<String>();
>>>                 HashSet<String> unreadable = new HashSet<String>();
>>>
>>>
>>> getFilesList(relativePath,recursive,includeFolders,false,collection,
>>> unreadable);
>>>
>>>                 return collection;
>>>
>>>         }
>>>
>>>         private void getFilesList(String path, boolean recursive,
>>> boolean includeFolders, boolean isFullPath, HashSet<String>
>>> collection, HashSet<String> unreadable) throws MalformedURLException,
>>> SmbException, ExsException {
>>>
>>>                 if(path == null) {
>>>                         path = "";
>>>                 }
>>>
>>>                 if(path.trim().length() > 0 &&
>>> (!path.trim().endsWith("/") && !path.trim().endsWith("\\"))) {
>>>                         path = path + "/";
>>>                 }
>>>
>>>                 String url = "";
>>>
>>>                 if(isFullPath) {
>>>                         url = path;
>>>                 } else {
>>>                         url = generateURLString(path);
>>>                 }
>>>
>>>                 SmbFile smbFolder = new SmbFile(url);
>>>
>>>                 if(!smbFolder.exists() || !smbFolder.isDirectory()) {
>>>                         throw new ExsException("Error: Folder cannot
>>> be read from smb share: " + smbFolder.getPath());
>>>                 }
>>>
>>>                 if(!smbFolder.canRead()) {
>>>                         unreadable.add(smbFolder.getPath());
>>>                         return;
>>>                 }
>>>
>>>                 // this helps preventing heap overflows but it wont
>>> prevent it when to many files at once are opend
>>>                 HashSet<SmbFile> uniqueList = new HashSet<SmbFile>();
>>>                 for(SmbFile currentSmbFile : smbFolder.listFiles()) {
>>>                         uniqueList.add(currentSmbFile);
>>>                 }
>>>
>>>                 for(SmbFile currentSmbFile : uniqueList) {
>>>
>>>                         String currentPath =
>>> currentSmbFile.getPath();
>>>
>>>                         if(currentSmbFile.isFile()) {
>>>                                 currentSmbFile = null;
>>>                                 collection.add(currentPath);
>>>
>>>                         } else if(currentSmbFile.isDirectory()) {
>>>                                 int size = 0;
>>>                                 try {
>>>                                         String[] list =
>>> currentSmbFile.list();
>>>                                         size = list.length;
>>>                                         list = null;
>>>                                 } catch(Throwable t) {
>>>                                         continue;
>>>                                 }
>>>
>>>                                 currentSmbFile = null;
>>>                                 if(includeFolders) {
>>>                                         collection.add(currentPath);
>>>                                 }
>>>                                 if(recursive) {
>>>                                         if(size > 0) {
>>>
>>> getFilesList(currentPath, recursive, includeFolders, true,
>>> collection, unreadable);
>>>                                         }
>>>                                 }
>>>                         }
>>>                 }
>>>
>>>                 uniqueList.clear();
>>>                 uniqueList = null;
>>>         }
>>>
>>>
>>>
>>
>>
>>
>> --
>> Michael B Allen
>> Java Active Directory Integration
>> http://www.ioplex.com/
>>
>
>
>
> --
> Michael B Allen
> Java Active Directory Integration
> http://www.ioplex.com/
>

--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

[Paul Clapham]

I should be able to do that, after I download some packet-capturing product and learn how to use it. Yes, I have seen the docs page about how to get a packet capture so I will start from there.

PC2

-----Original Message-----
From: Michael B Allen [mailto:ioplex@...]
Sent: October 26, 2009 10:09
To: Clapham, Paul
Cc: michael.studer@...; jcifs@...
Subject: Re: [jcifs] listFiles produce multiple entrys for some files..

Oh yeah. Wow. That's pretty messed up. Can you send me a packet capture?

On Mon, Oct 26, 2009 at 12:49 PM, Clapham, Paul <pclapham@...> wrote:

> Hi Mike,
>
> I sent you a log from that program off-list.
>
> PC2
>
> -----Original Message-----
> From: Michael B Allen [mailto:ioplex@...]
> Sent: October 26, 2009 08:55
> To: Clapham, Paul
> Cc: michael.studer@...; jcifs@...
> Subject: Re: [jcifs] listFiles produce multiple entrys for some files..
>
> Hi Paul and Michael,
>
> Better still, everyone should be using examples/ListFiles.java to start so that we are all on the same page.
>
> Thanks for helping out.
>
> Mike
>
> On Mon, Oct 26, 2009 at 11:29 AM, Clapham, Paul <pclapham@...> wrote:
>> I have access to AS400 and I'm willing to work on this problem. I asked michael.studer (accidentally off-list) to send me a short compilable program which I could modify and test. So far I just got a large lot of code which was way too much to deal with. Seems to me that a program to test this should be less than 50 lines of code.
>>
>> PC2
>>
>> -----Original Message-----
>> From: jcifs-bounces@...
>> [mailto:jcifs-bounces@...] On Behalf Of Michael B Allen
>> Sent: October 26, 2009 08:25
>> To: michael.studer@...
>> Cc: jcifs@...
>> Subject: Re: [jcifs] listFiles produce multiple entrys for some files..
>>
>> You could send me a packet capture. But I don't have access to AS400 and I don't really have "Free" time so unless it's an obvious fix this could sit for a while. But I know JCIFS has worked with mainframes before so I have to wonder why this is showing up now. Is AS400 really old or really new or unmaintained?
>>
>> Mike
>>
>> On Mon, Oct 12, 2009 at 3:17 AM,  <michael.studer@...> wrote:
>>>
>>> Hi again
>>>
>>> I figured out, the problem seems to occur only on AS400 shares. I
>>> did not face it on windows servers.
>>>
>>> Again, i try to explain the problem. When i use the "smbFile.listFiles()"
>>> method on a file, which is located inside a AS400 share, the amount
>>> of returned files is multiple times the actual amount it should return.
>>> For example: 2 Files in the share returns me 3 Files, where one File
>>> is twice in the collection. This would not be such a big deal, its
>>> just getting nasty when you have folders with a lot of Files inside.
>>> While i tested, i faced some folders with arround 5000+ Files. This
>>> results in arround 250'000 returned SMBFile Objects, with a total
>>> used amount of like 500+MB of memory and a runtime out of memory.
>>>
>>> Heres the full code for my directory reading method(s):
>>>
>>>         public HashSet<String> getFilesList(String relativePath,
>>> boolean recursive, boolean includeFolders) throws
>>> MalformedURLException, SmbException, ExsException {
>>>
>>>                 HashSet<String> collection = new HashSet<String>();
>>>                 HashSet<String> unreadable = new HashSet<String>();
>>>
>>>
>>> getFilesList(relativePath,recursive,includeFolders,false,collection,
>>> unreadable);
>>>
>>>                 return collection;
>>>
>>>         }
>>>
>>>         private void getFilesList(String path, boolean recursive,
>>> boolean includeFolders, boolean isFullPath, HashSet<String>
>>> collection, HashSet<String> unreadable) throws
>>> MalformedURLException, SmbException, ExsException {
>>>
>>>                 if(path == null) {
>>>                         path = "";
>>>                 }
>>>
>>>                 if(path.trim().length() > 0 &&
>>> (!path.trim().endsWith("/") && !path.trim().endsWith("\\"))) {
>>>                         path = path + "/";
>>>                 }
>>>
>>>                 String url = "";
>>>
>>>                 if(isFullPath) {
>>>                         url = path;
>>>                 } else {
>>>                         url = generateURLString(path);
>>>                 }
>>>
>>>                 SmbFile smbFolder = new SmbFile(url);
>>>
>>>                 if(!smbFolder.exists() || !smbFolder.isDirectory())
>>> {
>>>                         throw new ExsException("Error: Folder cannot
>>> be read from smb share: " + smbFolder.getPath());
>>>                 }
>>>
>>>                 if(!smbFolder.canRead()) {
>>>                         unreadable.add(smbFolder.getPath());
>>>                         return;
>>>                 }
>>>
>>>                 // this helps preventing heap overflows but it wont
>>> prevent it when to many files at once are opend
>>>                 HashSet<SmbFile> uniqueList = new
>>> HashSet<SmbFile>();
>>>                 for(SmbFile currentSmbFile : smbFolder.listFiles())
>>> {
>>>                         uniqueList.add(currentSmbFile);
>>>                 }
>>>
>>>                 for(SmbFile currentSmbFile : uniqueList) {
>>>
>>>                         String currentPath =
>>> currentSmbFile.getPath();
>>>
>>>                         if(currentSmbFile.isFile()) {
>>>                                 currentSmbFile = null;
>>>                                 collection.add(currentPath);
>>>
>>>                         } else if(currentSmbFile.isDirectory()) {
>>>                                 int size = 0;
>>>                                 try {
>>>                                         String[] list =
>>> currentSmbFile.list();
>>>                                         size = list.length;
>>>                                         list = null;
>>>                                 } catch(Throwable t) {
>>>                                         continue;
>>>                                 }
>>>
>>>                                 currentSmbFile = null;
>>>                                 if(includeFolders) {
>>>                                         collection.add(currentPath);
>>>                                 }
>>>                                 if(recursive) {
>>>                                         if(size > 0) {
>>>
>>> getFilesList(currentPath, recursive, includeFolders, true,
>>> collection, unreadable);
>>>                                         }
>>>                                 }
>>>                         }
>>>                 }
>>>
>>>                 uniqueList.clear();
>>>                 uniqueList = null;
>>>         }
>>>
>>>
>>>
>>
>>
>>
>> --
>> Michael B Allen
>> Java Active Directory Integration
>> http://www.ioplex.com/
>>
>
>
>
> --
> Michael B Allen
> Java Active Directory Integration
> http://www.ioplex.com/
>

--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/