apparent issue with S3Service.listObjects and delimiter

[amygdala]

I am having trouble getting the listObjects method to work as expected when I pass it a delimiter.  Surely if this method was broken someone else would have complained :), but I have consulted the AWS documentation and can not see what I am doing wrong.

I am calling listObjects with a prefix of: acc2/proj32/
If I call it with the delimiter set to null, I get this list (of strange keys generated for testing purposes):

 acc2/proj32/folder3/
 acc2/proj32/folder3/26841802.jpg
 acc2/proj32/folder3/obama_youth_04.jpg
 acc2/proj32/temp.txt/1/temp.txt
 acc2/proj32/temp.txt/2/temp.txt
 acc2/proj32/testing.java/1/testing.java
 acc2/proj32/testing.java/2/testing.java
 acc2/proj32/testing.java/3/infinite_cats.jpg
 acc2/proj32/testing.java/3/testing.java
 acc2/proj32/testing.java/abc/
 acc2/proj32/testing.java/abc/anewell.JPG
 acc2/proj32/testing.java/abc/cuttlefish.png
 
However, if I call it with the delimiter set to "/", I get 0 results returned.
(I would have expected to see a 'subdirectory' listing).
I tried removing the trailing "/" from the prefix, but still get no results returned.

Any advice would be appreciated!

  thanks,
  Amy

[James Murty-2]

Hi Amy,

The S3 object listing behaviour with delimiters is a bit unexpected sometimes, but it makes sense when you realise that S3 will only ever return complete keys as objects. When the delimiter causes key strings to be cut short into "subdirectory" paths, these paths are returned as something called Common Prefixes, not as objects.

The upshot of all of this is that the standard S3Service#listObjects method won't do what you need.

You need to use the more advanced S3Service#listObjectsChunked methods instead, because these return S3ObjectsChunk objects from which you can extract both the normal S3Objects returned by S3, and the CommonPrefix strings that represent any partial "subdirectory" paths.

Something like the following should do the trick:

           S3ObjectsChunk completeChunk = s3Service.listObjectsChunked(bucket.getName(),
                prefix, null, Constants.DEFAULT_OBJECT_LIST_CHUNK_SIZE, null, true);
           S3Object[] objects = chunk.getObjects();

String[] commonPrefixes = chunk.getCommonPrefixes();

Hope this helps,
James

---
http://www.jamesmurty.com

On Thu, Jan 15, 2009 at 12:09 PM, amygdala <info@...> wrote:

I am having trouble getting the listObjects method to work as expected when I
pass it a delimiter.  Surely if this method was broken someone else would
have complained :), but I have consulted the AWS documentation and can not
see what I am doing wrong.

I am calling listObjects with a prefix of: acc2/proj32/
If I call it with the delimiter set to null, I get this list (of strange
keys generated for testing purposes):

 acc2/proj32/folder3/
 acc2/proj32/folder3/26841802.jpg
 acc2/proj32/folder3/obama_youth_04.jpg
 acc2/proj32/temp.txt/1/temp.txt
 acc2/proj32/temp.txt/2/temp.txt
 acc2/proj32/testing.java/1/testing.java
 acc2/proj32/testing.java/2/testing.java
 acc2/proj32/testing.java/3/infinite_cats.jpg
 acc2/proj32/testing.java/3/testing.java
 acc2/proj32/testing.java/abc/
 acc2/proj32/testing.java/abc/anewell.JPG
 acc2/proj32/testing.java/abc/cuttlefish.png

However, if I call it with the delimiter set to "/", I get 0 results
returned.
(I would have expected to see a 'subdirectory' listing).
I tried removing the trailing "/" from the prefix, but still get no results
returned.

Any advice would be appreciated!

 thanks,
 Amy
--
View this message in context: http://www.nabble.com/apparent-issue-with-S3Service.listObjects-and-delimiter-tp21469102p21469102.html
Sent from the JetS3t Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[MarkAtHarvest]

Hi Amy,

Also pls review the code in
http://www.nabble.com/How-to-get-Objects-by-specifying-bucket-path-td21438812.html#a21475283

thanks
Mark

James Murty-2 wrote:

Hi Amy,

The S3 object listing behaviour with delimiters is a bit unexpected
sometimes, but it makes sense when you realise that S3 will only ever return
complete keys as objects. When the delimiter causes key strings to be cut
short into "subdirectory" paths, these paths are returned as something
called Common Prefixes, not as objects.

The upshot of all of this is that the standard S3Service#listObjects method
won't do what you need.

You need to use the more advanced S3Service#listObjectsChunked methods
instead, because these return S3ObjectsChunk objects from which you can
extract both the normal S3Objects returned by S3, and the CommonPrefix
strings that represent any partial "subdirectory" paths.

Something like the following should do the trick:

           S3ObjectsChunk completeChunk =
s3Service.listObjectsChunked(bucket.getName(),
                prefix, null, Constants.DEFAULT_OBJECT_LIST_CHUNK_SIZE,
null, true);
           S3Object[] objects = chunk.getObjects();
String[] commonPrefixes = chunk.getCommonPrefixes();

Hope this helps,
James

---
http://www.jamesmurty.com


On Thu, Jan 15, 2009 at 12:09 PM, amygdala <[e-mail]> wrote:

>
> I am having trouble getting the listObjects method to work as expected when
> I
> pass it a delimiter.  Surely if this method was broken someone else would
> have complained :), but I have consulted the AWS documentation and can not
> see what I am doing wrong.
>
> I am calling listObjects with a prefix of: acc2/proj32/
> If I call it with the delimiter set to null, I get this list (of strange
> keys generated for testing purposes):
>
>  acc2/proj32/folder3/
>  acc2/proj32/folder3/26841802.jpg
>  acc2/proj32/folder3/obama_youth_04.jpg
>  acc2/proj32/temp.txt/1/temp.txt
>  acc2/proj32/temp.txt/2/temp.txt
>  acc2/proj32/testing.java/1/testing.java
>  acc2/proj32/testing.java/2/testing.java
>  acc2/proj32/testing.java/3/infinite_cats.jpg
>  acc2/proj32/testing.java/3/testing.java
>  acc2/proj32/testing.java/abc/
>  acc2/proj32/testing.java/abc/anewell.JPG
>  acc2/proj32/testing.java/abc/cuttlefish.png
>
> However, if I call it with the delimiter set to "/", I get 0 results
> returned.
> (I would have expected to see a 'subdirectory' listing).
> I tried removing the trailing "/" from the prefix, but still get no results
> returned.
>
> Any advice would be appreciated!
>
>  thanks,
>  Amy
> --
> View this message in context:
> http://www.nabble.com/apparent-issue-with-S3Service.listObjects-and-delimiter-tp21469102p21469102.html
> Sent from the JetS3t Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [e-mail]
> For additional commands, e-mail: [e-mail]
>
>

[James Murty-2]

I should note that my example code above will not work as intended because I forgot to include a non-null delimiter string. You will need to supply the string "/" as the delimiter argument.

On Thu, Jan 15, 2009 at 2:43 PM, James Murty <jmurty@...> wrote:

Hi Amy,

The S3 object listing behaviour with delimiters is a bit unexpected sometimes, but it makes sense when you realise that S3 will only ever return complete keys as objects. When the delimiter causes key strings to be cut short into "subdirectory" paths, these paths are returned as something called Common Prefixes, not as objects.

The upshot of all of this is that the standard S3Service#listObjects method won't do what you need.

You need to use the more advanced S3Service#listObjectsChunked methods instead, because these return S3ObjectsChunk objects from which you can extract both the normal S3Objects returned by S3, and the CommonPrefix strings that represent any partial "subdirectory" paths.

Something like the following should do the trick:

           S3ObjectsChunk completeChunk = s3Service.listObjectsChunked(bucket.getName(),
                prefix, null, Constants.DEFAULT_OBJECT_LIST_CHUNK_SIZE, null, true);
           S3Object[] objects = chunk.getObjects();

String[] commonPrefixes = chunk.getCommonPrefixes();

Hope this helps,
James

---
http://www.jamesmurty.com

On Thu, Jan 15, 2009 at 12:09 PM, amygdala <info@...> wrote:

I am having trouble getting the listObjects method to work as expected when I
pass it a delimiter.  Surely if this method was broken someone else would
have complained :), but I have consulted the AWS documentation and can not
see what I am doing wrong.

I am calling listObjects with a prefix of: acc2/proj32/
If I call it with the delimiter set to null, I get this list (of strange
keys generated for testing purposes):

 acc2/proj32/folder3/
 acc2/proj32/folder3/26841802.jpg
 acc2/proj32/folder3/obama_youth_04.jpg
 acc2/proj32/temp.txt/1/temp.txt
 acc2/proj32/temp.txt/2/temp.txt
 acc2/proj32/testing.java/1/testing.java
 acc2/proj32/testing.java/2/testing.java
 acc2/proj32/testing.java/3/infinite_cats.jpg
 acc2/proj32/testing.java/3/testing.java
 acc2/proj32/testing.java/abc/
 acc2/proj32/testing.java/abc/anewell.JPG
 acc2/proj32/testing.java/abc/cuttlefish.png

However, if I call it with the delimiter set to "/", I get 0 results
returned.
(I would have expected to see a 'subdirectory' listing).
I tried removing the trailing "/" from the prefix, but still get no results
returned.

Any advice would be appreciated!

 thanks,
 Amy
--
View this message in context: http://www.nabble.com/apparent-issue-with-S3Service.listObjects-and-delimiter-tp21469102p21469102.html
Sent from the JetS3t Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...