How to get Objects by specifying bucket path

[MarkAtHarvest]

    def s3files =  s3.listObjects("trial")
works great and returns all the s3files within the trial bucket, I am able to use the same to retrieve the files by speficifying the bucket path

e.g to get all the files in docs folder, which is inside the trial, can I use something like
   def s3files = s3.listObjects("trial//docs")

pls help

[James Murty-2]

Please see the documentation for the S3Service#listObjects method that accepts a prefix argument, this will allow you to limit the listed objects based on an object prefix string:
http://jets3t.s3.amazonaws.com/api/org/jets3t/service/S3Service.html#listObjects(org.jets3t.service.model.S3Bucket,%20java.lang.String,%20java.lang.String)

There is also a basic example in the JetS3t code samples here:
http://jets3t.s3.amazonaws.com/toolkit/code-samples.html#listing

James

On Wed, Jan 14, 2009 at 3:07 AM, MarkAtHarvest <mark@...> wrote:

   def s3files =  s3.listObjects("trial")
works great and returns all the s3files within the trial bucket, I am able
to use the same to retrieve the files by speficifying the bucket path

e.g to get all the files in docs folder, which is inside the trial, can I
use something like
  def s3files = s3.listObjects("trial//docs")

pls help
--
View this message in context: http://www.nabble.com/How-to-get-Objects-by-specifying-bucket-path-tp21438812p21438812.html
Sent from the JetS3t Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[MarkAtHarvest]

Hi James,
Went though documentation link you gave and came up with solution to list all the objects in particular subdirectory
e.g to list all the files in /trial/docs folder,

        S3Bucket[] buckList = s3.listAllBuckets()
 buckList?.each{
            log.debug("Bucket Name is $it.name")
            if(it.name=="trial")
            {
                log.debug("Foound $it.name")
                String prefix = "docs/";
                String delimiter = "/"; // Refer to the S3 guide for more information on delimiters
                S3Object[] filteredObjects = s3.listObjects(it, prefix, delimiter);
                filteredObjects.each{
                    log.debug("File Name is $it.key ")
                    def getUrl = S3Service.createSignedGetUrl(it.bucketName,it.key, awsCredentials, expiryDate, false)
                    log.debug("GET URL IS $getUrl")
                }

            }

Let me try the listObjectsChunked method, mentioned on your reply to Amy, thank you

James Murty-2 wrote:

Please see the documentation for the S3Service#listObjects method that
accepts a prefix argument, this will allow you to limit the listed objects
based on an object prefix string:
http://jets3t.s3.amazonaws.com/api/org/jets3t/service/S3Service.html#listObjects(org.jets3t.service.model.S3Bucket,%20java.lang.String,%20java.lang.String)

There is also a basic example in the JetS3t code samples here:
http://jets3t.s3.amazonaws.com/toolkit/code-samples.html#listing

James


On Wed, Jan 14, 2009 at 3:07 AM, MarkAtHarvest <[e-mail]>wrote:

>
>    def s3files =  s3.listObjects("trial")
> works great and returns all the s3files within the trial bucket, I am able
> to use the same to retrieve the files by speficifying the bucket path
>
> e.g to get all the files in docs folder, which is inside the trial, can I
> use something like
>   def s3files = s3.listObjects("trial//docs")
>
> pls help
> --
> View this message in context:
> http://www.nabble.com/How-to-get-Objects-by-specifying-bucket-path-tp21438812p21438812.html
> Sent from the JetS3t Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [e-mail]
> For additional commands, e-mail: [e-mail]
>
>

[MarkAtHarvest]

Dear James,

I tried the get Objects chunked method, I do get all the files, but my common prefixes has nothing in it
following is my directory structure
harvesttrial (has 14 files and 2 folders)
harvesttrial\abc.pdf (it has 14 files in it)
harvesttrial\..
harvesttrial\docs\emp\pay\pay1.pdf
harvesttrial\docs\emp\admin

I get total 16 S3 Objects back when queried on harvesttrial, but my common prefix length is 0, here is the sample code

                String prefix=null
                S3ObjectsChunk chunk = s3.listObjectsChunked("harvesttrial",
                    prefix, null, Constants.DEFAULT_OBJECT_LIST_CHUNK_SIZE, null, true);
                S3Object[] s3files = chunk.getObjects();
                log.debug("s3files returned $s3files.length files")
                String[] commonPrefixes = chunk.getCommonPrefixes();
                log.debug("returned commonsPrefixes $commonPrefixes.length")
                commonPrefixes.each{
                    log.debug("Chunk Commmon Prefixes are $it ")
                }

regards
Mark

MarkAtHarvest wrote:

    def s3files =  s3.listObjects("trial")
works great and returns all the s3files within the trial bucket, I am able to use the same to retrieve the files by speficifying the bucket path

e.g to get all the files in docs folder, which is inside the trial, can I use something like
   def s3files = s3.listObjects("trial//docs")

pls help

[James Murty-2]

You will only get common prefixes when you use a non-null delimiter string, and that delimiter string occurs before the end of the object key names.

So, to find the "subdirectory" paths under harvesttrial, you need to use prefix="harvesttrial" and delimiter="/".

---
http://www.jamesmurty.com

On Thu, Jan 15, 2009 at 10:21 PM, MarkAtHarvest <mark@...> wrote:

Dear James,

I tried the get Objects chunked method, I do get all the files, but my
common prefixes has nothing in it
following is my directory structure
harvesttrial (has 14 files and 2 folders)
harvesttrial\abc.pdf (it has 14 files in it)
harvesttrial\..
harvesttrial\docs\emp\pay\pay1.pdf
harvesttrial\docs\emp\admin

I get total 16 S3 Objects back when queried on harvesttrial, but my common
prefix length is 0, here is the sample code

               String prefix=null
               S3ObjectsChunk chunk = s3.listObjectsChunked("harvesttrial",
                   prefix, null, Constants.DEFAULT_OBJECT_LIST_CHUNK_SIZE,
null, true);
               S3Object[] s3files = chunk.getObjects();
               log.debug("s3files returned $s3files.length files")
               String[] commonPrefixes = chunk.getCommonPrefixes();
               log.debug("returned commonsPrefixes $commonPrefixes.length")
               commonPrefixes.each{
                   log.debug("Chunk Commmon Prefixes are $it ")
               }

regards
Mark

MarkAtHarvest wrote:
>
>     def s3files =  s3.listObjects("trial")
> works great and returns all the s3files within the trial bucket, I am able
> to use the same to retrieve the files by speficifying the bucket path
>
> e.g to get all the files in docs folder, which is inside the trial, can I
> use something like
>    def s3files = s3.listObjects("trial//docs")
>
> pls help
>

--

View this message in context: http://www.nabble.com/How-to-get-Objects-by-specifying-bucket-path-tp21438812p21475736.html

Sent from the JetS3t Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[MarkAtHarvest]

Thank you James, everything works good as mentioned

        S3ObjectsChunk chunk = s3.listObjectsChunked("harvesttrial",null, "/", 1000, null,true);
        String[] commonPrefixes = chunk.getCommonPrefixes();
        commonPrefixes.each{
                   S3ObjectsChunk subchunk = s3.listObjectsChunked("harvesttrial",it, "/", 1000, null,true);
                  S3Object[] s3files = chunk.getObjects();
                  log.debug("Files in $subFolderPrefix returned $s3files.length files")
       }
One small question is, if the sub folder has again subfolders, I do get those folders as S3Objects and also get them in common prefixes.
so e.g harvestrial/docs/emp1
         harvestrial/docs/emp2
When I print the list of files, I get,

File Name is docs/employees/emp2_$folder$

I can ignore these files, which have $folder$, but I thought we should not even get them in files, when we are getting them in commonPrefixes.

Thanks again.

James Murty-2 wrote:

Please see the documentation for the S3Service#listObjects method that
accepts a prefix argument, this will allow you to limit the listed objects
based on an object prefix string:
http://jets3t.s3.amazonaws.com/api/org/jets3t/service/S3Service.html#listObjects(org.jets3t.service.model.S3Bucket,%20java.lang.String,%20java.lang.String)

There is also a basic example in the JetS3t code samples here:
http://jets3t.s3.amazonaws.com/toolkit/code-samples.html#listing

James


On Wed, Jan 14, 2009 at 3:07 AM, MarkAtHarvest <[e-mail]>wrote:

>
>    def s3files =  s3.listObjects("trial")
> works great and returns all the s3files within the trial bucket, I am able
> to use the same to retrieve the files by speficifying the bucket path
>
> e.g to get all the files in docs folder, which is inside the trial, can I
> use something like
>   def s3files = s3.listObjects("trial//docs")
>
> pls help
> --
> View this message in context:
> http://www.nabble.com/How-to-get-Objects-by-specifying-bucket-path-tp21438812p21438812.html
> Sent from the JetS3t Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [e-mail]
> For additional commands, e-mail: [e-mail]
>
>

[James Murty-3]

My guess is that the "docs/employees/emp2_$folder$" objects you are seeing are indeed real objects, and were probably created by the tool you used to upload your files. They are most likely place-holder objects that your upload program uses to keep track of your directory structure.

If you look at your bucket using the JetS3t Cockpit tool, it will show you exactly what objects are present in your bucket. Looking through this listing will allow you to easily see whether or not your listing results are unusual.

HTH,
James

On Sat, Jan 17, 2009 at 6:52 PM, MarkAtHarvest <mark@...> wrote:

Thank you James, everything works good as mentioned

       S3ObjectsChunk chunk = s3.listObjectsChunked("harvesttrial",null,
"/", 1000, null,true);
       String[] commonPrefixes = chunk.getCommonPrefixes();
       commonPrefixes.each{
                  S3ObjectsChunk subchunk =
s3.listObjectsChunked("harvesttrial",it, "/", 1000, null,true);
                 S3Object[] s3files = chunk.getObjects();
                 log.debug("Files in $subFolderPrefix returned
$s3files.length files")
      }
One small question is, if the sub folder has again subfolders, I do get
those folders as S3Objects and also get them in common prefixes.
so e.g harvestrial/docs/emp1
        harvestrial/docs/emp2
When I print the list of files, I get,

File Name is docs/employees/emp2_$folder$

I can ignore these files, which have $folder$, but I thought we should not
even get them in files, when we are getting them in commonPrefixes.

Thanks again.

James Murty-2 wrote:
>
> Please see the documentation for the S3Service#listObjects method that
> accepts a prefix argument, this will allow you to limit the listed objects
> based on an object prefix string:
> http://jets3t.s3.amazonaws.com/api/org/jets3t/service/S3Service.html#listObjects(org.jets3t.service.model.S3Bucket,%20java.lang.String,%20java.lang.String)
>
> There is also a basic example in the JetS3t code samples here:
> http://jets3t.s3.amazonaws.com/toolkit/code-samples.html#listing
>
> James
>
>
> On Wed, Jan 14, 2009 at 3:07 AM, MarkAtHarvest
> <mark@...>wrote:
>
>>
>>    def s3files =  s3.listObjects("trial")
>> works great and returns all the s3files within the trial bucket, I am
>> able
>> to use the same to retrieve the files by speficifying the bucket path
>>
>> e.g to get all the files in docs folder, which is inside the trial, can I
>> use something like
>>   def s3files = s3.listObjects("trial//docs")
>>
>> pls help
>> --
>> View this message in context:
>> http://www.nabble.com/How-to-get-Objects-by-specifying-bucket-path-tp21438812p21438812.html
>> Sent from the JetS3t Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@...
>> For additional commands, e-mail: users-help@...
>>
>>
>
>

--
View this message in context: http://www.nabble.com/How-to-get-Objects-by-specifying-bucket-path-tp21438812p21513966.html

Sent from the JetS3t Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[MarkAtHarvest]

I checked the cockpit tool, and it explains it all, the $folder$ is comingfrom my S3 Fox plugin for FireFox, which I was using to create directories. Jets3t kit is right in exposing it.

Thank you very much for all the help.  

James Murty-3 wrote:

My guess is that the "docs/employees/emp2_$folder$" objects you are seeing
are indeed real objects, and were probably created by the tool you used to
upload your files. They are most likely place-holder objects that your
upload program uses to keep track of your directory structure.

If you look at your bucket using the JetS3t Cockpit tool, it will show you
exactly what objects are present in your bucket. Looking through this
listing will allow you to easily see whether or not your listing results are
unusual.

HTH,
James


On Sat, Jan 17, 2009 at 6:52 PM, MarkAtHarvest <[e-mail]>wrote:

>
> Thank you James, everything works good as mentioned
>
>        S3ObjectsChunk chunk = s3.listObjectsChunked("harvesttrial",null,
> "/", 1000, null,true);
>        String[] commonPrefixes = chunk.getCommonPrefixes();
>        commonPrefixes.each{
>                   S3ObjectsChunk subchunk =
> s3.listObjectsChunked("harvesttrial",it, "/", 1000, null,true);
>                  S3Object[] s3files = chunk.getObjects();
>                  log.debug("Files in $subFolderPrefix returned
> $s3files.length files")
>       }
> One small question is, if the sub folder has again subfolders, I do get
> those folders as S3Objects and also get them in common prefixes.
> so e.g harvestrial/docs/emp1
>         harvestrial/docs/emp2
> When I print the list of files, I get,
>
> File Name is docs/employees/emp2_$folder$
>
> I can ignore these files, which have $folder$, but I thought we should not
> even get them in files, when we are getting them in commonPrefixes.
>
> Thanks again.
>
>
> James Murty-2 wrote:
> >
> > Please see the documentation for the S3Service#listObjects method that
> > accepts a prefix argument, this will allow you to limit the listed
> objects
> > based on an object prefix string:
> >
> http://jets3t.s3.amazonaws.com/api/org/jets3t/service/S3Service.html#listObjects(org.jets3t.service.model.S3Bucket,%20java.lang.String,%20java.lang.String)<http://jets3t.s3.amazonaws.com/api/org/jets3t/service/S3Service.html#listObjects%28org.jets3t.service.model.S3Bucket,%20java.lang.String,%20java.lang.String%29>
> >
> > There is also a basic example in the JetS3t code samples here:
> > http://jets3t.s3.amazonaws.com/toolkit/code-samples.html#listing
> >
> > James
> >
> >
> > On Wed, Jan 14, 2009 at 3:07 AM, MarkAtHarvest
> > <[e-mail]>wrote:
> >
> >>
> >>    def s3files =  s3.listObjects("trial")
> >> works great and returns all the s3files within the trial bucket, I am
> >> able
> >> to use the same to retrieve the files by speficifying the bucket path
> >>
> >> e.g to get all the files in docs folder, which is inside the trial, can
> I
> >> use something like
> >>   def s3files = s3.listObjects("trial//docs")
> >>
> >> pls help
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/How-to-get-Objects-by-specifying-bucket-path-tp21438812p21438812.html
> >> Sent from the JetS3t Users mailing list archive at Nabble.com.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [e-mail]
> >> For additional commands, e-mail: [e-mail]
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/How-to-get-Objects-by-specifying-bucket-path-tp21438812p21513966.html
> Sent from the JetS3t Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [e-mail]
> For additional commands, e-mail: [e-mail]
>
>