|
»
EC2->S3 Upload Problems?
|
View:
New views
10 Messages
—
Rating Filter:
Alert me
|
 |
EC2->S3 Upload Problems?
by Patrick Lightbody-3
::
Rate this Message:
I'm having some trouble uploading some files to S3, though the problem
isn't reproducible locally, but I've seen it happen at least twice (outlined below) on EC2. I'm going to run some more tests to try to reproduce, but I'm skeptical that this test ran exactly at the time when EC2->S3 connectivity went wacky. Any thoughts would be appreciated! It looks like the underlying HttpClient is causing S3 to reject some of the content I'm trying to upload to S3. It's odd because if I grab the same exact file and upload it locally with my code (and jets3t) it works just fine. But I just saw this runtime error on one of my EC2 instances: INFO 10/07 16:13:36 o.a.c.h.HttpMethodB~ - 100 (continue) read timeout. Resume sending the request WARN 10/07 16:13:36 o.j.s.i.r.h.RestS3S~ - Response '/xxx' - Unexpected response code 505, expected 200 WARN 10/07 16:13:36 o.a.c.h.HttpMethodB~ - Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended. WARN 10/07 16:13:36 c.b.a.Foo - Could not upload logs, continuing on org.jets3t.service.S3ServiceException: S3 PUT request failed for '/xxx' - ResponseCode=505, ResponseMessage=HTTP Version Not Supported at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:485) at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestPut(RestS3Service.java:734) at org.jets3t.service.impl.rest.httpclient.RestS3Service.putAclImpl(RestS3Service.java:1205) at org.jets3t.service.impl.rest.httpclient.RestS3Service.createObjectImpl(RestS3Service.java:1373) at org.jets3t.service.impl.rest.httpclient.RestS3Service.putObjectImpl(RestS3Service.java:1275) at org.jets3t.service.S3Service.putObject(S3Service.java:1470) at org.jets3t.service.S3Service.putObject(S3Service.java:1722) at com.xxx.Foo.upload(Foo.java:116) at com.xxx.Foo.run(Foo.java:64) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) Caused by: org.jets3t.service.impl.rest.HttpException at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:483) ... 14 more What is weird is that in another case (right around the same time, but on a different EC2 machine) it appeared I was able to upload successfully, as per my logs. Note that there was a warning and a HttpClient retry, but the call to S3Service.putObject() did return, since that last log statement is immediately after the putObject call. WARN 10/07 16:13:42 o.j.s.i.r.h.RestS3S~ - Retrying PUT request with path '/yyy' - attempt 1 of 5 INFO 10/07 16:13:42 o.a.c.h.HttpMethodD~ - I/O exception (java.net.SocketException) caught when processing request: Socket Closed INFO 10/07 16:13:42 o.a.c.h.HttpMethodD~ - Retrying request INFO 10/07 16:13:42 c.b.c.Bar - Files uploaded to S3 Unfortunately, in both cases xxx and yyy were incorrectly uploaded. Both source files are very small (4K) and both pieces of code were using commons-httpclient 3.1, yet there the files that show up in S3 have a content length of 0 bytes. When I download them, they are 0 bytes locally too. Any recommended debugging steps/tips I should follow? I did some searching on the 505 error and it looks like that can happen when the client doesn't speak exactly the same dialect of HTTP that S3 is expecting, but I would have expected jets3t to deal with most of that as the wrapper. Thanks! Patrick --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: EC2->S3 Upload Problems?
by James Murty-3
::
Rate this Message:
Hi Patrick,
The fact that both of these failures occurred at the same time is a bit suspicious. The fact that one upload apparently succeeded, but actually created an invalid empty object, may indicate a problem with one of Amazon's outer S3 servers (load-balancer or similar) which corrupted the data in transit. Unfortunately, it's very difficult to be certain after the fact. If possible, you could turn on "wire" debugging while you continue testing in case the issue recurs. With this debug mode you can see all the data travelling to and from S3 via JetS3t/HttpClient. As a general recommendation, you should always use S3's hashing mechanisms to ensure the data that arrives at an S3 storage node is exactly the same as the data you uploaded. If you are using JetS3t 0.6.1, the toolkit will automatically perform an ETag comparison check after uploading data and will notify you if there is a mismatch. With earlier versions of JetS3t, you need to do this checking in your own code by either 1) calculating the MD5 hash of data before uploading and including it as the Content-MD5 header in the S3Object, or 2) reading the ETag value from the S3 server response after an upload and comparing it to the MD5 hash value calculated from your data. James --- http://www.jamesmurty.com On Wed, Oct 8, 2008 at 7:48 AM, Patrick Lightbody <plightbo@...> wrote: I'm having some trouble uploading some files to S3, though the problem |
|
|
Re: EC2->S3 Upload Problems?
by Patrick Lightbody-3
::
Rate this Message:
James,
I am indeed using 0.6.1 and now errors were thrown, though I wasn't aware of that new feature, so I also just added this for additional safety: String localHash = obj.getMd5HashAsHex(); service.putObject(bucket, obj); String remoteHash = service.getObjectDetails(bucket, key).getMd5HashAsHex(); boolean success = localHash.equals(remoteHash); I added some logic to try the upload up to 3 times, mostly for the purposes of seeing if the error happens and then "fixes" itself. I'll let you know if I find anything. I just ran 10 more attempts and everything looked good, so perhaps it was a freak anomaly! Patrick On Tue, Oct 7, 2008 at 2:26 PM, James Murty <james@...> wrote: > Hi Patrick, > > The fact that both of these failures occurred at the same time is a bit > suspicious. The fact that one upload apparently succeeded, but actually > created an invalid empty object, may indicate a problem with one of Amazon's > outer S3 servers (load-balancer or similar) which corrupted the data in > transit. > > Unfortunately, it's very difficult to be certain after the fact. If > possible, you could turn on "wire" debugging while you continue testing in > case the issue recurs. With this debug mode you can see all the data > travelling to and from S3 via JetS3t/HttpClient. > > As a general recommendation, you should always use S3's hashing mechanisms > to ensure the data that arrives at an S3 storage node is exactly the same as > the data you uploaded. If you are using JetS3t 0.6.1, the toolkit will > automatically perform an ETag comparison check after uploading data and will > notify you if there is a mismatch. With earlier versions of JetS3t, you need > to do this checking in your own code by either 1) calculating the MD5 hash > of data before uploading and including it as the Content-MD5 header in the > S3Object, or 2) reading the ETag value from the S3 server response after an > upload and comparing it to the MD5 hash value calculated from your data. > > James > > --- > http://www.jamesmurty.com > > > On Wed, Oct 8, 2008 at 7:48 AM, Patrick Lightbody <plightbo@...> > wrote: >> >> I'm having some trouble uploading some files to S3, though the problem >> isn't reproducible locally, but I've seen it happen at least twice >> (outlined below) on EC2. I'm going to run some more tests to try to >> reproduce, but I'm skeptical that this test ran exactly at the time >> when EC2->S3 connectivity went wacky. Any thoughts would be >> appreciated! >> >> It looks like the underlying HttpClient is causing S3 to reject some >> of the content I'm trying to upload to S3. It's odd because if I grab >> the same exact file and upload it locally with my code (and jets3t) it >> works just fine. But I just saw this runtime error on one of my EC2 >> instances: >> >> INFO 10/07 16:13:36 o.a.c.h.HttpMethodB~ - 100 (continue) read >> timeout. Resume sending the request >> WARN 10/07 16:13:36 o.j.s.i.r.h.RestS3S~ - Response '/xxx' - >> Unexpected response code 505, expected 200 >> WARN 10/07 16:13:36 o.a.c.h.HttpMethodB~ - Going to buffer response >> body of large or unknown size. Using getResponseBodyAsStream instead >> is recommended. >> WARN 10/07 16:13:36 c.b.a.Foo - Could not upload logs, >> continuing on >> org.jets3t.service.S3ServiceException: S3 PUT request failed for >> '/xxx' - ResponseCode=505, ResponseMessage=HTTP Version Not Supported >> at >> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:485) >> at >> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestPut(RestS3Service.java:734) >> at >> org.jets3t.service.impl.rest.httpclient.RestS3Service.putAclImpl(RestS3Service.java:1205) >> at >> org.jets3t.service.impl.rest.httpclient.RestS3Service.createObjectImpl(RestS3Service.java:1373) >> at >> org.jets3t.service.impl.rest.httpclient.RestS3Service.putObjectImpl(RestS3Service.java:1275) >> at org.jets3t.service.S3Service.putObject(S3Service.java:1470) >> at org.jets3t.service.S3Service.putObject(S3Service.java:1722) >> at com.xxx.Foo.upload(Foo.java:116) >> at com.xxx.Foo.run(Foo.java:64) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >> at java.lang.Thread.run(Thread.java:619) >> Caused by: org.jets3t.service.impl.rest.HttpException >> at >> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:483) >> ... 14 more >> >> What is weird is that in another case (right around the same time, but >> on a different EC2 machine) it appeared I was able to upload >> successfully, as per my logs. Note that there was a warning and a >> HttpClient retry, but the call to S3Service.putObject() did return, >> since that last log statement is immediately after the putObject call. >> >> WARN 10/07 16:13:42 o.j.s.i.r.h.RestS3S~ - Retrying PUT request with >> path '/yyy' - attempt 1 of 5 >> INFO 10/07 16:13:42 o.a.c.h.HttpMethodD~ - I/O exception >> (java.net.SocketException) caught when processing request: Socket >> Closed >> INFO 10/07 16:13:42 o.a.c.h.HttpMethodD~ - Retrying request >> INFO 10/07 16:13:42 c.b.c.Bar - Files uploaded to S3 >> >> Unfortunately, in both cases xxx and yyy were incorrectly uploaded. >> Both source files are very small (4K) and both pieces of code were >> using commons-httpclient 3.1, yet there the files that show up in S3 >> have a content length of 0 bytes. When I download them, they are 0 >> bytes locally too. >> >> Any recommended debugging steps/tips I should follow? I did some >> searching on the 505 error and it looks like that can happen when the >> client doesn't speak exactly the same dialect of HTTP that S3 is >> expecting, but I would have expected jets3t to deal with most of that >> as the wrapper. >> >> Thanks! >> >> Patrick >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscribe@... >> For additional commands, e-mail: users-help@... >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: EC2->S3 Upload Problems?
by James Murty-2
::
Rate this Message:
Definitely let me know how your testing proceeds. It's the first time I've heard of JetS3t producing a 505 error, and in the AWS S3 forums the most recent discussions about 505s are from more than a year ago, so hopefully it was just a temporary blip.
Regarding your new code to check the local and remote hash values, I think it's a wise approach. You could skip the S3Service#getObjectDetails step because the S3Service#putObject method will return S3's calculated hash value, however in the interests of absolute certainty (and slight paranoia) stick with what you've got until you can be sure the issue won't recur. Cheers, James --- http://www.jamesmurty.com On Wed, Oct 8, 2008 at 8:41 AM, Patrick Lightbody <plightbo@...> wrote: James, |
|
|
Re: EC2->S3 Upload Problems?
by Patrick Lightbody-3
::
Rate this Message:
James,
I did try using the S3Object returned by putObject, but I was finding that getMd5HashAsHex() was returning null in my tests, which also surprised me. That's why I added the getObjectDetails() call in fact. Any thoughts? Patrick On Tue, Oct 7, 2008 at 3:20 PM, James Murty <jmurty@...> wrote: > Definitely let me know how your testing proceeds. It's the first time I've > heard of JetS3t producing a 505 error, and in the AWS S3 forums the most > recent discussions about 505s are from more than a year ago, so hopefully it > was just a temporary blip. > > Regarding your new code to check the local and remote hash values, I think > it's a wise approach. You could skip the S3Service#getObjectDetails step > because the S3Service#putObject method will return S3's calculated hash > value, however in the interests of absolute certainty (and slight paranoia) > stick with what you've got until you can be sure the issue won't recur. > > Cheers, > James > > --- > http://www.jamesmurty.com > > > On Wed, Oct 8, 2008 at 8:41 AM, Patrick Lightbody <plightbo@...> > wrote: >> >> James, >> I am indeed using 0.6.1 and now errors were thrown, though I wasn't >> aware of that new feature, so I also just added this for additional >> safety: >> >> String localHash = obj.getMd5HashAsHex(); >> service.putObject(bucket, obj); >> String remoteHash = service.getObjectDetails(bucket, >> key).getMd5HashAsHex(); >> boolean success = localHash.equals(remoteHash); >> >> I added some logic to try the upload up to 3 times, mostly for the >> purposes of seeing if the error happens and then "fixes" itself. I'll >> let you know if I find anything. I just ran 10 more attempts and >> everything looked good, so perhaps it was a freak anomaly! >> >> Patrick >> >> On Tue, Oct 7, 2008 at 2:26 PM, James Murty <james@...> wrote: >> > Hi Patrick, >> > >> > The fact that both of these failures occurred at the same time is a bit >> > suspicious. The fact that one upload apparently succeeded, but actually >> > created an invalid empty object, may indicate a problem with one of >> > Amazon's >> > outer S3 servers (load-balancer or similar) which corrupted the data in >> > transit. >> > >> > Unfortunately, it's very difficult to be certain after the fact. If >> > possible, you could turn on "wire" debugging while you continue testing >> > in >> > case the issue recurs. With this debug mode you can see all the data >> > travelling to and from S3 via JetS3t/HttpClient. >> > >> > As a general recommendation, you should always use S3's hashing >> > mechanisms >> > to ensure the data that arrives at an S3 storage node is exactly the >> > same as >> > the data you uploaded. If you are using JetS3t 0.6.1, the toolkit will >> > automatically perform an ETag comparison check after uploading data and >> > will >> > notify you if there is a mismatch. With earlier versions of JetS3t, you >> > need >> > to do this checking in your own code by either 1) calculating the MD5 >> > hash >> > of data before uploading and including it as the Content-MD5 header in >> > the >> > S3Object, or 2) reading the ETag value from the S3 server response after >> > an >> > upload and comparing it to the MD5 hash value calculated from your data. >> > >> > James >> > >> > --- >> > http://www.jamesmurty.com >> > >> > >> > On Wed, Oct 8, 2008 at 7:48 AM, Patrick Lightbody <plightbo@...> >> > wrote: >> >> >> >> I'm having some trouble uploading some files to S3, though the problem >> >> isn't reproducible locally, but I've seen it happen at least twice >> >> (outlined below) on EC2. I'm going to run some more tests to try to >> >> reproduce, but I'm skeptical that this test ran exactly at the time >> >> when EC2->S3 connectivity went wacky. Any thoughts would be >> >> appreciated! >> >> >> >> It looks like the underlying HttpClient is causing S3 to reject some >> >> of the content I'm trying to upload to S3. It's odd because if I grab >> >> the same exact file and upload it locally with my code (and jets3t) it >> >> works just fine. But I just saw this runtime error on one of my EC2 >> >> instances: >> >> >> >> INFO 10/07 16:13:36 o.a.c.h.HttpMethodB~ - 100 (continue) read >> >> timeout. Resume sending the request >> >> WARN 10/07 16:13:36 o.j.s.i.r.h.RestS3S~ - Response '/xxx' - >> >> Unexpected response code 505, expected 200 >> >> WARN 10/07 16:13:36 o.a.c.h.HttpMethodB~ - Going to buffer response >> >> body of large or unknown size. Using getResponseBodyAsStream instead >> >> is recommended. >> >> WARN 10/07 16:13:36 c.b.a.Foo - Could not upload logs, >> >> continuing on >> >> org.jets3t.service.S3ServiceException: S3 PUT request failed for >> >> '/xxx' - ResponseCode=505, ResponseMessage=HTTP Version Not Supported >> >> at >> >> >> >> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:485) >> >> at >> >> >> >> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestPut(RestS3Service.java:734) >> >> at >> >> >> >> org.jets3t.service.impl.rest.httpclient.RestS3Service.putAclImpl(RestS3Service.java:1205) >> >> at >> >> >> >> org.jets3t.service.impl.rest.httpclient.RestS3Service.createObjectImpl(RestS3Service.java:1373) >> >> at >> >> >> >> org.jets3t.service.impl.rest.httpclient.RestS3Service.putObjectImpl(RestS3Service.java:1275) >> >> at org.jets3t.service.S3Service.putObject(S3Service.java:1470) >> >> at org.jets3t.service.S3Service.putObject(S3Service.java:1722) >> >> at com.xxx.Foo.upload(Foo.java:116) >> >> at com.xxx.Foo.run(Foo.java:64) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> >> at >> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> >> at >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) >> >> at >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >> >> at java.lang.Thread.run(Thread.java:619) >> >> Caused by: org.jets3t.service.impl.rest.HttpException >> >> at >> >> >> >> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:483) >> >> ... 14 more >> >> >> >> What is weird is that in another case (right around the same time, but >> >> on a different EC2 machine) it appeared I was able to upload >> >> successfully, as per my logs. Note that there was a warning and a >> >> HttpClient retry, but the call to S3Service.putObject() did return, >> >> since that last log statement is immediately after the putObject call. >> >> >> >> WARN 10/07 16:13:42 o.j.s.i.r.h.RestS3S~ - Retrying PUT request with >> >> path '/yyy' - attempt 1 of 5 >> >> INFO 10/07 16:13:42 o.a.c.h.HttpMethodD~ - I/O exception >> >> (java.net.SocketException) caught when processing request: Socket >> >> Closed >> >> INFO 10/07 16:13:42 o.a.c.h.HttpMethodD~ - Retrying request >> >> INFO 10/07 16:13:42 c.b.c.Bar - Files uploaded to S3 >> >> >> >> Unfortunately, in both cases xxx and yyy were incorrectly uploaded. >> >> Both source files are very small (4K) and both pieces of code were >> >> using commons-httpclient 3.1, yet there the files that show up in S3 >> >> have a content length of 0 bytes. When I download them, they are 0 >> >> bytes locally too. >> >> >> >> Any recommended debugging steps/tips I should follow? I did some >> >> searching on the 505 error and it looks like that can happen when the >> >> client doesn't speak exactly the same dialect of HTTP that S3 is >> >> expecting, but I would have expected jets3t to deal with most of that >> >> as the wrapper. >> >> >> >> Thanks! >> >> >> >> Patrick >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: users-unsubscribe@... >> >> For additional commands, e-mail: users-help@... >> >> >> > >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscribe@... >> For additional commands, e-mail: users-help@... >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: EC2->S3 Upload Problems?
by James Murty-2
::
Rate this Message:
I don't have the code handy to check this, but it's probably the case that only the ETag header value is set in the result from the S3Service#putObject method. The #getMd5HashAsHex method sources its value from elsewhere, so setting the ETag may not be enough to get a result from this method. I'll look into this later and get back to you. |
|
|
Re: EC2->S3 Upload Problems?
by Patrick Lightbody-3
::
Rate this Message:
James,
I'm seeing some _very_ strange behavior. Files are still being uploaded and "verified" successfully, but the file still ends up being a zero byte file. When I re-run the code and verify the hash, the local hash is different from the remote one, despite the fact that the local has was correct the first time. Let me share first the code that originally uploads the file: String key = "xxx/yyy/zzz"; S3Object obj = new S3Object(bucket, new File(LOGS_DUMP_LOC)); obj.setKey(key); AccessControlList acl = new AccessControlList(); acl.setOwner(bucket.getOwner()); acl.grantPermission(GroupGrantee.ALL_USERS, Permission.PERMISSION_READ); obj.setAcl(acl); String localHash = obj.getMd5HashAsHex(); LOG.info("Got local S3 object hash of %s", localHash); service.putObject(bucket, obj); String remoteHash = service.getObjectDetails(bucket, key).getMd5HashAsHex(); LOG.info("Got remote S3 object hash of %s", remoteHash); boolean success = localHash.equals(remoteHash); The logs show this: INFO 10/07 19:16:34 c.b.a.Foo - Got local S3 object hash of d41d8cd98f00b204e9800998ecf8427e INFO 10/07 19:16:34 c.b.a.Foo - Got remote S3 object hash of d41d8cd98f00b204e9800998ecf8427e So everything appears to be good. However, the file ends up being garbage. When I re-run similar code (a simple main() class I am writing by hand on my EC2 machine), I get different hashes. The code I run is: S3Object obj = new S3Object(bucket, new File(LOGS_DUMP_LOC)); System.out.println(obj.getMd5HashAsHex()); System.out.println(service.getObjectDetails(bucket, "xxx/yyy/zz").getMd5HashAsHex()); I get the following output: f05f59594f9c33c945b8d14c128eb87a d41d8cd98f00b204e9800998ecf8427e So, as you can see, the hash reported by S3 is staying consistent, but the hash constructed locally is changing, despite the fact that the file hasn't changed (as far as I can tell). My guess is, considering that the code for computing the MD5 is really straight forward, that the file is in fact changing and I have a bug. I'm going to be chasing down that possibility, but I wanted to share my findings so far in case you had any thoughts. Patrick On Tue, Oct 7, 2008 at 3:36 PM, James Murty <jmurty@...> wrote: >> I did try using the S3Object returned by putObject, but I was finding >> that getMd5HashAsHex() was returning null in my tests > > I don't have the code handy to check this, but it's probably the case that > only the ETag header value is set in the result from the S3Service#putObject > method. The #getMd5HashAsHex method sources its value from elsewhere, so > setting the ETag may not be enough to get a result from this method. > > I'll look into this later and get back to you. > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: EC2->S3 Upload Problems?
by Patrick Lightbody-3
::
Rate this Message:
James,
Just a followup on this. It looks like it was indeed a bug in my code in which I was uploading the file before it was fully written to. Really sorry about the false alarm. My guess is that it was the root cause for all the weird behavior I was seeing (0 bytes and 505s). As a side note unrelated to jets3t, what is interested is that the process writing the file out was an external Process (gzip) and I didn't start the S3 upload until Process.waitFor() returned. I didn't investigate it further, but perhaps the file didn't get flushed or closed until just after the gzip ended? Beats me, but I'm all set now with a different approach. Thanks for creating jets3t - it's working great! Patrick On Tue, Oct 7, 2008 at 4:29 PM, Patrick Lightbody <plightbo@...> wrote: > James, > I'm seeing some _very_ strange behavior. Files are still being > uploaded and "verified" successfully, but the file still ends up being > a zero byte file. When I re-run the code and verify the hash, the > local hash is different from the remote one, despite the fact that the > local has was correct the first time. > > Let me share first the code that originally uploads the file: > > String key = "xxx/yyy/zzz"; > > S3Object obj = new S3Object(bucket, new File(LOGS_DUMP_LOC)); > obj.setKey(key); > > AccessControlList acl = new AccessControlList(); > acl.setOwner(bucket.getOwner()); > acl.grantPermission(GroupGrantee.ALL_USERS, Permission.PERMISSION_READ); > obj.setAcl(acl); > > String localHash = obj.getMd5HashAsHex(); > LOG.info("Got local S3 object hash of %s", localHash); > > service.putObject(bucket, obj); > > String remoteHash = service.getObjectDetails(bucket, > key).getMd5HashAsHex(); > LOG.info("Got remote S3 object hash of %s", remoteHash); > > boolean success = localHash.equals(remoteHash); > > The logs show this: > > INFO 10/07 19:16:34 c.b.a.Foo - Got local S3 object hash of > d41d8cd98f00b204e9800998ecf8427e > INFO 10/07 19:16:34 c.b.a.Foo - Got remote S3 object hash of > d41d8cd98f00b204e9800998ecf8427e > > So everything appears to be good. However, the file ends up being > garbage. When I re-run similar code (a simple main() class I am > writing by hand on my EC2 machine), I get different hashes. The code I > run is: > > S3Object obj = new S3Object(bucket, new File(LOGS_DUMP_LOC)); > System.out.println(obj.getMd5HashAsHex()); > System.out.println(service.getObjectDetails(bucket, > "xxx/yyy/zz").getMd5HashAsHex()); > > I get the following output: > > f05f59594f9c33c945b8d14c128eb87a > d41d8cd98f00b204e9800998ecf8427e > > So, as you can see, the hash reported by S3 is staying consistent, but > the hash constructed locally is changing, despite the fact that the > file hasn't changed (as far as I can tell). > > My guess is, considering that the code for computing the MD5 is really > straight forward, that the file is in fact changing and I have a bug. > I'm going to be chasing down that possibility, but I wanted to share > my findings so far in case you had any thoughts. > > Patrick > > On Tue, Oct 7, 2008 at 3:36 PM, James Murty <jmurty@...> wrote: >>> I did try using the S3Object returned by putObject, but I was finding >>> that getMd5HashAsHex() was returning null in my tests >> >> I don't have the code handy to check this, but it's probably the case that >> only the ETag header value is set in the result from the S3Service#putObject >> method. The #getMd5HashAsHex method sources its value from elsewhere, so >> setting the ETag may not be enough to get a result from this method. >> >> I'll look into this later and get back to you. >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: EC2->S3 Upload Problems?
by James Murty-3
::
Rate this Message:
Hi Patrick,
I'm glad you have found the cause of the hash mismatches, that was a worrying problem. Uploading the file before it is fully written and flushed would cause the hashes to change between the beginning of the upload and the end, although I don't think it explains the 505 error. It's interesting that you can't be sure the file write is complete when you invoke an external gzip process. You might want to consider doing the gzip-ping in code instead of spawning an external process. Java includes Input and Output streams to do this, and JetS3t includes supplemental iostream classes in the org.jets3t.service.io package. Cheers, James --- http://www.jamesmurty.com On Thu, Oct 9, 2008 at 12:03 AM, Patrick Lightbody <plightbo@...> wrote: James, |
|
|
Re: EC2->S3 Upload Problems?
by Patrick Lightbody-3
::
Rate this Message:
James,
I thought about using Java's built in zipping capabilities, but I need the resulting file to be accessible directly and able to be unzipped using standard desktop tools. I didn't investigate it beyond this, but I was under the impression that gzip and the Java zipping classes weren't compatible... Patrick On Wed, Oct 8, 2008 at 6:16 AM, James Murty <james@...> wrote: > Hi Patrick, > > I'm glad you have found the cause of the hash mismatches, that was a > worrying problem. Uploading the file before it is fully written and flushed > would cause the hashes to change between the beginning of the upload and the > end, although I don't think it explains the 505 error. > > It's interesting that you can't be sure the file write is complete when you > invoke an external gzip process. You might want to consider doing the > gzip-ping in code instead of spawning an external process. Java includes > Input and Output streams to do this, and JetS3t includes supplemental > iostream classes in the org.jets3t.service.io package. > > Cheers, > James > > --- > http://www.jamesmurty.com > > > On Thu, Oct 9, 2008 at 12:03 AM, Patrick Lightbody <plightbo@...> > wrote: >> >> James, >> Just a followup on this. It looks like it was indeed a bug in my code >> in which I was uploading the file before it was fully written to. >> Really sorry about the false alarm. My guess is that it was the root >> cause for all the weird behavior I was seeing (0 bytes and 505s). >> >> As a side note unrelated to jets3t, what is interested is that the >> process writing the file out was an external Process (gzip) and I >> didn't start the S3 upload until Process.waitFor() returned. I didn't >> investigate it further, but perhaps the file didn't get flushed or >> closed until just after the gzip ended? >> >> Beats me, but I'm all set now with a different approach. Thanks for >> creating jets3t - it's working great! >> >> Patrick >> >> On Tue, Oct 7, 2008 at 4:29 PM, Patrick Lightbody <plightbo@...> >> wrote: >> > James, >> > I'm seeing some _very_ strange behavior. Files are still being >> > uploaded and "verified" successfully, but the file still ends up being >> > a zero byte file. When I re-run the code and verify the hash, the >> > local hash is different from the remote one, despite the fact that the >> > local has was correct the first time. >> > >> > Let me share first the code that originally uploads the file: >> > >> > String key = "xxx/yyy/zzz"; >> > >> > S3Object obj = new S3Object(bucket, new File(LOGS_DUMP_LOC)); >> > obj.setKey(key); >> > >> > AccessControlList acl = new AccessControlList(); >> > acl.setOwner(bucket.getOwner()); >> > acl.grantPermission(GroupGrantee.ALL_USERS, >> > Permission.PERMISSION_READ); >> > obj.setAcl(acl); >> > >> > String localHash = obj.getMd5HashAsHex(); >> > LOG.info("Got local S3 object hash of %s", localHash); >> > >> > service.putObject(bucket, obj); >> > >> > String remoteHash = service.getObjectDetails(bucket, >> > key).getMd5HashAsHex(); >> > LOG.info("Got remote S3 object hash of %s", remoteHash); >> > >> > boolean success = localHash.equals(remoteHash); >> > >> > The logs show this: >> > >> > INFO 10/07 19:16:34 c.b.a.Foo - Got local S3 object hash of >> > d41d8cd98f00b204e9800998ecf8427e >> > INFO 10/07 19:16:34 c.b.a.Foo - Got remote S3 object hash of >> > d41d8cd98f00b204e9800998ecf8427e >> > >> > So everything appears to be good. However, the file ends up being >> > garbage. When I re-run similar code (a simple main() class I am >> > writing by hand on my EC2 machine), I get different hashes. The code I >> > run is: >> > >> > S3Object obj = new S3Object(bucket, new File(LOGS_DUMP_LOC)); >> > System.out.println(obj.getMd5HashAsHex()); >> > System.out.println(service.getObjectDetails(bucket, >> > "xxx/yyy/zz").getMd5HashAsHex()); >> > >> > I get the following output: >> > >> > f05f59594f9c33c945b8d14c128eb87a >> > d41d8cd98f00b204e9800998ecf8427e >> > >> > So, as you can see, the hash reported by S3 is staying consistent, but >> > the hash constructed locally is changing, despite the fact that the >> > file hasn't changed (as far as I can tell). >> > >> > My guess is, considering that the code for computing the MD5 is really >> > straight forward, that the file is in fact changing and I have a bug. >> > I'm going to be chasing down that possibility, but I wanted to share >> > my findings so far in case you had any thoughts. >> > >> > Patrick >> > >> > On Tue, Oct 7, 2008 at 3:36 PM, James Murty <jmurty@...> wrote: >> >>> I did try using the S3Object returned by putObject, but I was finding >> >>> that getMd5HashAsHex() was returning null in my tests >> >> >> >> I don't have the code handy to check this, but it's probably the case >> >> that >> >> only the ETag header value is set in the result from the >> >> S3Service#putObject >> >> method. The #getMd5HashAsHex method sources its value from elsewhere, >> >> so >> >> setting the ETag may not be enough to get a result from this method. >> >> >> >> I'll look into this later and get back to you. >> >> >> >> >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscribe@... >> For additional commands, e-mail: users-help@... >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
| Free embeddable forum powered by Nabble | Forum Help |
