Nabble - JetS3t Users

Re: Updating ACLs for existing objects in S3

2009-12-06T22:32:43Z

botemout wrote:

... I'll be doing very large buckets so hopefully this can be done with the multi* functions.

Hi,

Why you bother about the sample code ? Using BucketExplorer, you can set the acls for very large buckets. You can ask BucketExplorer to start multiple threads to update acl parallelly. It also provides the control over the process.

Suppose, you started the process to update acl for 5 lac objects. You can stop the process in between and restart it from the point you stopped lasttime. It also gives u the statistics, how many objects have been updated and remaining.

Updating ACLs for existing objects in S3

2009-12-04T11:39:36Z

Greetings,

I've looked through the jets3t code (the sample code and Synchronize.java, mostly) and it's not obvious to me how I can, efficiently and, hopefully, quickly, add a new user permission (using CanonicalUser), to the objects that are already up there. It must be doable (Bucket Explorer, s3fox, etc... manage it ;-)

Could someone point me to some example code? I'll be doing very large buckets so hopefully this can be done with the multi* functions.

Thanks
JR

Re: NullPointer Exception in RestS3Service

2009-08-29T17:18:15Z

I have applied a fix for this issue and made the S3ServiceException error reporting a bit more accurate in general. When response information is not available, such as when there is no Internet access, the S3ServiceException class will not print out response attributes for which it has no meaningful information.

Thanks very much for the bug report and suggested fix.

James

---
http://www.jamesmurty.com

On Wed, Aug 26, 2009 at 10:59 PM, Jawahar Nayak <jlal@...> wrote:

In the performRequest() method of Rests3Service class if any exception is
thrown then it constructs the instance of S3Exception. Suppose if the
network is disabled then the exception is thrown. In catch block it
constructs the S3ServiceException

as

<Code>

s3ServiceException = new S3ServiceException("Request Error.", t);

s3ServiceException.setRequestVerb(httpMethod.getName());

s3ServiceException.setRequestPath(httpMethod.getPath());

s3ServiceException.setResponseCode(httpMethod.getStatusCode());
// This statement is throwing NullPointerException

s3ServiceException.setResponseStatus(httpMethod.getStatusText());

</Code>

The following is the exception
<Exception>

java.lang.NullPointerException
at
org.apache.commons.httpclient.HttpMethodBase.getStatusCode(HttpMethodBase.java:570)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:1064)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestHead(RestS3Service.java:1637)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:3373)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectDetailsImpl(RestS3Service.java:3245)
at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1990)
at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1335

</Exception>

if we are running upload, we always get

s3ServiceEventPerformed(CreateObjectsEvent event) {

// We always get event.getEventCode() == CreateObjectsEvent.EVENT_COMPLETED
if the network is disabled.

}

I fixed the problem in this way.
try{

s3ServiceException.setResponseCode(httpMethod.getStatusCode());
}catch(NullPointerException ne){
s3ServiceException.setResponseCode(404);
}

try{

s3ServiceException.setResponseStatus(httpMethod.getStatusText());
}catch(NullPointerException ne){
s3ServiceException.setResponseStatus("Network Problem");
}

Its working for me.
--
View this message in context: http://www.nabble.com/NullPointer-Exception-in-RestS3Service-tp25166654p25166654.html
Sent from the JetS3t Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

NullPointer Exception in RestS3Service

2009-08-26T22:59:06Z

In the performRequest() method of Rests3Service class if any exception is thrown then it constructs the instance of S3Exception. Suppose if the network is disabled then the exception is thrown. In catch block it constructs the S3ServiceException

as

<Code>

s3ServiceException = new S3ServiceException("Request Error.", t);

s3ServiceException.setRequestVerb(httpMethod.getName());

s3ServiceException.setRequestPath(httpMethod.getPath());

s3ServiceException.setResponseCode(httpMethod.getStatusCode());
// This statement is throwing NullPointerException

s3ServiceException.setResponseStatus(httpMethod.getStatusText());

</Code>

The following is the exception
<Exception>

java.lang.NullPointerException
at org.apache.commons.httpclient.HttpMethodBase.getStatusCode(HttpMethodBase.java:570)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:1064)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestHead(RestS3Service.java:1637)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:3373)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectDetailsImpl(RestS3Service.java:3245)
at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1990)
at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1335

</Exception>

if we are running upload, we always get

s3ServiceEventPerformed(CreateObjectsEvent event) {

// We always get event.getEventCode() == CreateObjectsEvent.EVENT_COMPLETED
if the network is disabled.

}

I fixed the problem in this way.
try{
s3ServiceException.setResponseCode(httpMethod.getStatusCode());
}catch(NullPointerException ne){
s3ServiceException.setResponseCode(404);
}

try{
s3ServiceException.setResponseStatus(httpMethod.getStatusText());
}catch(NullPointerException ne){
s3ServiceException.setResponseStatus("Network Problem");
}

Its working for me.

Re: Changes in Jets3t code for TargetGrant in BucketLogging

2009-08-02T23:39:24Z

Hi Jawahar,

I have added support for TargetBuckets to JetS3t's codebase in CVS. Note that the code changes are slightly different from the code you provided.

Please confirm whether this update works for you.

Thanks,
James

---
http://www.jamesmurty.com

On Fri, Jul 31, 2009 at 4:32 AM, Jawahar Nayak <jlal@...> wrote:

Hi James,

You can include my submission in the JetS3t.

Thanks

James Murty-3 wrote:
>
> Hi Jawahar,
>
> Thanks for the code submission, I will look into adding TargetGrants
> support
> into JetS3t over the next few days.
>
> Can you please confirm for me that you grant all rights for your
> submission
> to be included in JetS3t and distributed under the Apache 2.0 license?
>
> Thanks,
> James
>
> ---
> http://www.jamesmurty.com
>
>
> On Thu, Jul 30, 2009 at 3:07 AM, Jawahar Nayak <jlal@...> wrote:
>
>>
>>
>> Hi,
>>
>> I am Jawahar Lal Nayak. I am using the latest version of Jets3t and I
>> made
>> changes regarding TragetGrants for bucket logging in jets3t source:
>>
>> -----------------------------------------------------------------------------------------------
>> Class: org\jets3t\service\model\S3BucketLoggingStatus.java
>>
>> -----------------------------------------------------------------------------------------------
>>
>> //Added one instance varriable:
>>
>> private Set trangetGrants;
>>
>> // Added set and get methods for trangetGrants;
>>
>> public void setTrangetGrants(Set targetgrants) {
>> trangetGrants= targetgrants;
>> }
>>
>> public Set getTrangetGrants() {
>> return trangetGrants;
>> }
>>
>>
>> // Modified the toString() method
>>
>> public String toXml() {
>> boolean bAddGrantACL = false;
>> StringBuffer sb = new StringBuffer();
>> StringBuffer strGrantACL = new StringBuffer();
>> if (isLoggingEnabled() && (trangetGrants != null &&
>> trangetGrants.size() > 0)) {
>> bAddGrantACL = true;
>> strGrantACL.append("<TargetGrants>");
>> Iterator grantIter = trangetGrants.iterator();
>> while (grantIter.hasNext()) {
>> GrantAndPermission gap = (GrantAndPermission)
>> grantIter.next();
>> GranteeInterface grantee = gap.getGrantee();
>> Permission permission = gap.getPermission();
>> strGrantACL.append(
>> "<Grant>" +
>> grantee.toXml() +
>> "<Permission>" + permission + "</Permission>" +
>> "</Grant>");
>> }
>> strGrantACL.append("</TargetGrants>");
>> }
>> sb.append(
>> "<BucketLoggingStatus xmlns=\"" + Constants.XML_NAMESPACE
>> +
>> "\">" +
>> (!isLoggingEnabled() ? "" : "<LoggingEnabled>" +
>> "<TargetBucket>" + getTargetBucketName() +
>> "</TargetBucket>"
>> +
>> "<TargetPrefix>" + getLogfilePrefix() + "</TargetPrefix>"
>> +
>> (bAddGrantACL ? strGrantACL.toString() : "") +
>> "</LoggingEnabled>") +
>> "</BucketLoggingStatus>");
>> return sb.toString();
>> }
>>
>>
>>
>>
>> ---------------------------------------------------------------------------------------------------
>> Class: org\jets3t\service\impl\rest\XmlResponsesSaxParser.java
>>
>> ---------------------------------------------------------------------------------------------------
>>
>> // Following is the complete class with changes
>>
>> public class BucketLoggingStatusHandler extends DefaultHandler {
>>
>> private S3BucketLoggingStatus bucketLoggingStatus = null;
>> private String targetBucket = null;
>> private String targetPrefix = null;
>> private StringBuffer currText = null;
>> private GranteeInterface currentGrantee = null;
>> private Permission currentPermission = null;
>> private GrantAndPermission currentGrantAndPermission=null;
>> private Set trangetGrants = null;
>>
>> public BucketLoggingStatusHandler() {
>> super();
>> this.currText = new StringBuffer();
>> }
>>
>> /**
>> * @return
>> * an object representing the bucket's LoggingStatus document.
>> */
>> public S3BucketLoggingStatus getBucketLoggingStatus() {
>>
>> return bucketLoggingStatus;
>> }
>>
>> public void startDocument() {
>> }
>>
>> public void endDocument() {
>> }
>>
>> public void startElement(String uri, String name, String qName,
>> Attributes attrs) {
>> if (name.equals("BucketLoggingStatus")) {
>> bucketLoggingStatus = new S3BucketLoggingStatus();
>> } else if (name.equals("TargetGrants")) {
>> trangetGrants = new HashSet();
>> }
>> }
>>
>> public void endElement(String uri, String name, String qName) {
>> String elementText = this.currText.toString();
>> if (name.equals("TargetBucket")) {
>> targetBucket = elementText;
>> } else if (name.equals("TargetPrefix")) {
>> targetPrefix = elementText;
>> } else if (name.equals("LoggingEnabled")) {
>> bucketLoggingStatus.setTargetBucketName(targetBucket);
>> bucketLoggingStatus.setLogfilePrefix(targetPrefix);
>> } else if (name.equals("ID")) {
>> currentGrantee = new CanonicalGrantee();
>> currentGrantee.setIdentifier(elementText);
>> } else if (name.equals("EmailAddress")) {
>> currentGrantee = new EmailAddressGrantee();
>> currentGrantee.setIdentifier(elementText);
>> } else if (name.equals("DisplayName")) {
>> ((CanonicalGrantee)
>> currentGrantee).setDisplayName(elementText);
>> } else if (name.equals("Permission")) {
>> currentPermission =
>> Permission.parsePermission(elementText);
>> }else if (name.equals("Grant")) {
>> currentGrantAndPermission = new
>> GrantAndPermission(currentGrantee, currentPermission);
>> trangetGrants.add(currentGrantAndPermission);
>> }else if(name.equals("TargetGrants")){
>> bucketLoggingStatus.setGrants(trangetGrants);
>> }
>> this.currText = new StringBuffer();
>> }
>>
>> public void characters(char ch[], int start, int length) {
>> this.currText.append(ch, start, length);
>> }
>> }
>>
>> Thanks
>> --
>> View this message in context:
>> http://www.nabble.com/Changes-in-Jets3t-code-for-TargetGrant-in-BucketLogging-tp24735589p24735589.html
>> Sent from the JetS3t Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@...
>> For additional commands, e-mail: users-help@...
>>
>>
>
>

--
View this message in context: http://www.nabble.com/Changes-in-Jets3t-code-for-TargetGrant-in-BucketLogging-tp24735589p24754953.html

Sent from the JetS3t Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Changes in Jets3t code for TargetGrant in BucketLogging

2009-07-31T04:32:51Z

Hi James,

You can include my submission in the JetS3t.

Thanks

James Murty-3 wrote:

Hi Jawahar,

Thanks for the code submission, I will look into adding TargetGrants support
into JetS3t over the next few days.

Can you please confirm for me that you grant all rights for your submission
to be included in JetS3t and distributed under the Apache 2.0 license?

Thanks,
James

---
http://www.jamesmurty.com

On Thu, Jul 30, 2009 at 3:07 AM, Jawahar Nayak <jlal@chambal.com> wrote:

>
>
> Hi,
>
> I am Jawahar Lal Nayak. I am using the latest version of Jets3t and I made
> changes regarding TragetGrants for bucket logging in jets3t source:
>
> -----------------------------------------------------------------------------------------------
> Class: org\jets3t\service\model\S3BucketLoggingStatus.java
>
> -----------------------------------------------------------------------------------------------
>
> //Added one instance varriable:
>
> private Set trangetGrants;
>
> // Added set and get methods for trangetGrants;
>
> public void setTrangetGrants(Set targetgrants) {
> trangetGrants= targetgrants;
> }
>
> public Set getTrangetGrants() {
> return trangetGrants;
> }
>
>
> // Modified the toString() method
>
> public String toXml() {
> boolean bAddGrantACL = false;
> StringBuffer sb = new StringBuffer();
> StringBuffer strGrantACL = new StringBuffer();
> if (isLoggingEnabled() && (trangetGrants != null &&
> trangetGrants.size() > 0)) {
> bAddGrantACL = true;
> strGrantACL.append("<TargetGrants>");
> Iterator grantIter = trangetGrants.iterator();
> while (grantIter.hasNext()) {
> GrantAndPermission gap = (GrantAndPermission)
> grantIter.next();
> GranteeInterface grantee = gap.getGrantee();
> Permission permission = gap.getPermission();
> strGrantACL.append(
> "<Grant>" +
> grantee.toXml() +
> "<Permission>" + permission + "</Permission>" +
> "</Grant>");
> }
> strGrantACL.append("</TargetGrants>");
> }
> sb.append(
> "<BucketLoggingStatus xmlns=\"" + Constants.XML_NAMESPACE +
> "\">" +
> (!isLoggingEnabled() ? "" : "<LoggingEnabled>" +
> "<TargetBucket>" + getTargetBucketName() + "</TargetBucket>"
> +
> "<TargetPrefix>" + getLogfilePrefix() + "</TargetPrefix>" +
> (bAddGrantACL ? strGrantACL.toString() : "") +
> "</LoggingEnabled>") +
> "</BucketLoggingStatus>");
> return sb.toString();
> }
>
>
>
>
> ---------------------------------------------------------------------------------------------------
> Class: org\jets3t\service\impl\rest\XmlResponsesSaxParser.java
>
> ---------------------------------------------------------------------------------------------------
>
> // Following is the complete class with changes
>
> public class BucketLoggingStatusHandler extends DefaultHandler {
>
> private S3BucketLoggingStatus bucketLoggingStatus = null;
> private String targetBucket = null;
> private String targetPrefix = null;
> private StringBuffer currText = null;
> private GranteeInterface currentGrantee = null;
> private Permission currentPermission = null;
> private GrantAndPermission currentGrantAndPermission=null;
> private Set trangetGrants = null;
>
> public BucketLoggingStatusHandler() {
> super();
> this.currText = new StringBuffer();
> }
>
> /**
> * @return
> * an object representing the bucket's LoggingStatus document.
> */
> public S3BucketLoggingStatus getBucketLoggingStatus() {
>
> return bucketLoggingStatus;
> }
>
> public void startDocument() {
> }
>
> public void endDocument() {
> }
>
> public void startElement(String uri, String name, String qName,
> Attributes attrs) {
> if (name.equals("BucketLoggingStatus")) {
> bucketLoggingStatus = new S3BucketLoggingStatus();
> } else if (name.equals("TargetGrants")) {
> trangetGrants = new HashSet();
> }
> }
>
> public void endElement(String uri, String name, String qName) {
> String elementText = this.currText.toString();
> if (name.equals("TargetBucket")) {
> targetBucket = elementText;
> } else if (name.equals("TargetPrefix")) {
> targetPrefix = elementText;
> } else if (name.equals("LoggingEnabled")) {
> bucketLoggingStatus.setTargetBucketName(targetBucket);
> bucketLoggingStatus.setLogfilePrefix(targetPrefix);
> } else if (name.equals("ID")) {
> currentGrantee = new CanonicalGrantee();
> currentGrantee.setIdentifier(elementText);
> } else if (name.equals("EmailAddress")) {
> currentGrantee = new EmailAddressGrantee();
> currentGrantee.setIdentifier(elementText);
> } else if (name.equals("DisplayName")) {
> ((CanonicalGrantee)
> currentGrantee).setDisplayName(elementText);
> } else if (name.equals("Permission")) {
> currentPermission = Permission.parsePermission(elementText);
> }else if (name.equals("Grant")) {
> currentGrantAndPermission = new
> GrantAndPermission(currentGrantee, currentPermission);
> trangetGrants.add(currentGrantAndPermission);
> }else if(name.equals("TargetGrants")){
> bucketLoggingStatus.setGrants(trangetGrants);
> }
> this.currText = new StringBuffer();
> }
>
> public void characters(char ch[], int start, int length) {
> this.currText.append(ch, start, length);
> }
> }
>
> Thanks
> --
> View this message in context:
> http://www.nabble.com/Changes-in-Jets3t-code-for-TargetGrant-in-BucketLogging-tp24735589p24735589.html
> Sent from the JetS3t Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@jets3t.dev.java.net
> For additional commands, e-mail: users-help@jets3t.dev.java.net
>
>

Re: Changes in Jets3t code for TargetGrant in BucketLogging

2009-07-30T23:30:07Z

Hi Jawahar,

Thanks for the code submission, I will look into adding TargetGrants support into JetS3t over the next few days.

Can you please confirm for me that you grant all rights for your submission to be included in JetS3t and distributed under the Apache 2.0 license?

Thanks,
James

---
http://www.jamesmurty.com

On Thu, Jul 30, 2009 at 3:07 AM, Jawahar Nayak <jlal@...> wrote:

Hi,

I am Jawahar Lal Nayak. I am using the latest version of Jets3t and I made
changes regarding TragetGrants for bucket logging in jets3t source:
-----------------------------------------------------------------------------------------------
Class: org\jets3t\service\model\S3BucketLoggingStatus.java
-----------------------------------------------------------------------------------------------

//Added one instance varriable:

private Set trangetGrants;

// Added set and get methods for trangetGrants;

public void setTrangetGrants(Set targetgrants) {
trangetGrants= targetgrants;
}

public Set getTrangetGrants() {
return trangetGrants;
}

// Modified the toString() method

public String toXml() {
boolean bAddGrantACL = false;
StringBuffer sb = new StringBuffer();
StringBuffer strGrantACL = new StringBuffer();
if (isLoggingEnabled() && (trangetGrants != null &&
trangetGrants.size() > 0)) {
bAddGrantACL = true;
strGrantACL.append("<TargetGrants>");
Iterator grantIter = trangetGrants.iterator();
while (grantIter.hasNext()) {
GrantAndPermission gap = (GrantAndPermission)
grantIter.next();
GranteeInterface grantee = gap.getGrantee();
Permission permission = gap.getPermission();
strGrantACL.append(
"<Grant>" +
grantee.toXml() +
"<Permission>" + permission + "</Permission>" +
"</Grant>");
}
strGrantACL.append("</TargetGrants>");
}
sb.append(
"<BucketLoggingStatus xmlns=\"" + Constants.XML_NAMESPACE +
"\">" +
(!isLoggingEnabled() ? "" : "<LoggingEnabled>" +
"<TargetBucket>" + getTargetBucketName() + "</TargetBucket>"
+
"<TargetPrefix>" + getLogfilePrefix() + "</TargetPrefix>" +
(bAddGrantACL ? strGrantACL.toString() : "") +
"</LoggingEnabled>") +
"</BucketLoggingStatus>");
return sb.toString();
}

---------------------------------------------------------------------------------------------------
Class: org\jets3t\service\impl\rest\XmlResponsesSaxParser.java
---------------------------------------------------------------------------------------------------

// Following is the complete class with changes

public class BucketLoggingStatusHandler extends DefaultHandler {

private S3BucketLoggingStatus bucketLoggingStatus = null;
private String targetBucket = null;
private String targetPrefix = null;
private StringBuffer currText = null;
private GranteeInterface currentGrantee = null;
private Permission currentPermission = null;
private GrantAndPermission currentGrantAndPermission=null;
private Set trangetGrants = null;

public BucketLoggingStatusHandler() {
super();
this.currText = new StringBuffer();
}

/**
* @return
* an object representing the bucket's LoggingStatus document.
*/
public S3BucketLoggingStatus getBucketLoggingStatus() {

return bucketLoggingStatus;
}

public void startDocument() {
}

public void endDocument() {
}

public void startElement(String uri, String name, String qName,
Attributes attrs) {
if (name.equals("BucketLoggingStatus")) {
bucketLoggingStatus = new S3BucketLoggingStatus();
} else if (name.equals("TargetGrants")) {
trangetGrants = new HashSet();
}
}

public void endElement(String uri, String name, String qName) {
String elementText = this.currText.toString();
if (name.equals("TargetBucket")) {
targetBucket = elementText;
} else if (name.equals("TargetPrefix")) {
targetPrefix = elementText;
} else if (name.equals("LoggingEnabled")) {
bucketLoggingStatus.setTargetBucketName(targetBucket);
bucketLoggingStatus.setLogfilePrefix(targetPrefix);
} else if (name.equals("ID")) {
currentGrantee = new CanonicalGrantee();
currentGrantee.setIdentifier(elementText);
} else if (name.equals("EmailAddress")) {
currentGrantee = new EmailAddressGrantee();
currentGrantee.setIdentifier(elementText);
} else if (name.equals("DisplayName")) {
((CanonicalGrantee)
currentGrantee).setDisplayName(elementText);
} else if (name.equals("Permission")) {
currentPermission = Permission.parsePermission(elementText);
}else if (name.equals("Grant")) {
currentGrantAndPermission = new
GrantAndPermission(currentGrantee, currentPermission);
trangetGrants.add(currentGrantAndPermission);
}else if(name.equals("TargetGrants")){
bucketLoggingStatus.setGrants(trangetGrants);
}
this.currText = new StringBuffer();
}

public void characters(char ch[], int start, int length) {
this.currText.append(ch, start, length);
}
}

Thanks
--
View this message in context: http://www.nabble.com/Changes-in-Jets3t-code-for-TargetGrant-in-BucketLogging-tp24735589p24735589.html
Sent from the JetS3t Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Changes in Jets3t code for TargetGrant in BucketLogging

2009-07-30T03:07:22Z

Hi,

I am Jawahar Lal Nayak. I am using the latest version of Jets3t and I made changes regarding TragetGrants for bucket logging in jets3t source:
-----------------------------------------------------------------------------------------------
Class: org\jets3t\service\model\S3BucketLoggingStatus.java
-----------------------------------------------------------------------------------------------

//Added one instance varriable:

private Set trangetGrants;

// Added set and get methods for trangetGrants;

public void setTrangetGrants(Set targetgrants) {
trangetGrants= targetgrants;
}

public Set getTrangetGrants() {
return trangetGrants;
}

// Modified the toString() method

public String toXml() {
boolean bAddGrantACL = false;
StringBuffer sb = new StringBuffer();
StringBuffer strGrantACL = new StringBuffer();
if (isLoggingEnabled() && (trangetGrants != null && trangetGrants.size() > 0)) {
bAddGrantACL = true;
strGrantACL.append("<TargetGrants>");
Iterator grantIter = trangetGrants.iterator();
while (grantIter.hasNext()) {
GrantAndPermission gap = (GrantAndPermission) grantIter.next();
GranteeInterface grantee = gap.getGrantee();
Permission permission = gap.getPermission();
strGrantACL.append(
"<Grant>" +
grantee.toXml() +
"<Permission>" + permission + "</Permission>" +
"</Grant>");
}
strGrantACL.append("</TargetGrants>");
}
sb.append(
"<BucketLoggingStatus xmlns=\"" + Constants.XML_NAMESPACE + "\">" +
(!isLoggingEnabled() ? "" : "<LoggingEnabled>" +
"<TargetBucket>" + getTargetBucketName() + "</TargetBucket>" +
"<TargetPrefix>" + getLogfilePrefix() + "</TargetPrefix>" +
(bAddGrantACL ? strGrantACL.toString() : "") +
"</LoggingEnabled>") +
"</BucketLoggingStatus>");
return sb.toString();
}

---------------------------------------------------------------------------------------------------
Class: org\jets3t\service\impl\rest\XmlResponsesSaxParser.java
---------------------------------------------------------------------------------------------------

// Following is the complete class with changes

public class BucketLoggingStatusHandler extends DefaultHandler {

private S3BucketLoggingStatus bucketLoggingStatus = null;
private String targetBucket = null;
private String targetPrefix = null;
private StringBuffer currText = null;
private GranteeInterface currentGrantee = null;
private Permission currentPermission = null;
private GrantAndPermission currentGrantAndPermission=null;
private Set trangetGrants = null;

public BucketLoggingStatusHandler() {
super();
this.currText = new StringBuffer();
}

/**
* @return
* an object representing the bucket's LoggingStatus document.
*/
public S3BucketLoggingStatus getBucketLoggingStatus() {

return bucketLoggingStatus;
}

public void startDocument() {
}

public void endDocument() {
}

public void startElement(String uri, String name, String qName, Attributes attrs) {
if (name.equals("BucketLoggingStatus")) {
bucketLoggingStatus = new S3BucketLoggingStatus();
} else if (name.equals("TargetGrants")) {
trangetGrants = new HashSet();
}
}

public void endElement(String uri, String name, String qName) {
String elementText = this.currText.toString();
if (name.equals("TargetBucket")) {
targetBucket = elementText;
} else if (name.equals("TargetPrefix")) {
targetPrefix = elementText;
} else if (name.equals("LoggingEnabled")) {
bucketLoggingStatus.setTargetBucketName(targetBucket);
bucketLoggingStatus.setLogfilePrefix(targetPrefix);
} else if (name.equals("ID")) {
currentGrantee = new CanonicalGrantee();
currentGrantee.setIdentifier(elementText);
} else if (name.equals("EmailAddress")) {
currentGrantee = new EmailAddressGrantee();
currentGrantee.setIdentifier(elementText);
} else if (name.equals("DisplayName")) {
((CanonicalGrantee) currentGrantee).setDisplayName(elementText);
} else if (name.equals("Permission")) {
currentPermission = Permission.parsePermission(elementText);
}else if (name.equals("Grant")) {
currentGrantAndPermission = new GrantAndPermission(currentGrantee, currentPermission);
trangetGrants.add(currentGrantAndPermission);
}else if(name.equals("TargetGrants")){
bucketLoggingStatus.setGrants(trangetGrants);
}
this.currText = new StringBuffer();
}

public void characters(char ch[], int start, int length) {
this.currText.append(ch, start, length);
}
}

Thanks

Eucalyptus 1.5.2

2009-07-17T14:40:25Z

Hello,

sorry about the spam, but thought some people might be interested.

Eucalyptus 1.5.2 is out with better API support.

http://forum.eucalyptus.com/forum/viewtopic.php?f=3&t=807

The demo Eucalyptus Public Cloud is still running an older version of
Eucalyptus and will upgraded soon (in case you are interested in
interacting with it).

Suggestions/feedback will be welcome.

thank you
neil

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Synchronize didn't error but created a single, empty file

2009-07-08T17:36:56Z

Brendan,

You will certainly need to test how well different software interacts if you plan to use different tools to store items in subfolders of the same bucket. For the most part things should be OK, in that all your data will be stored in S3 and will be accessible using the same tool you used to upload it. However, you may face problems if you later try using program X to download files uploaded with program Y, or if you use both X and Y to upload files to the same "subdirectory" in S3.

It would be very interesting if you could follow up here with your experiences using a few different tools, although that would make you something of a guinea pig. Depending on exactly how other tools work, it may be possible to add options to JetS3t to make it more compatible with them. I have done this already for one or two other common tools, though not S3Fox.

James

---
http://www.jamesmurty.com

On Wed, Jul 8, 2009 at 5:21 PM, Branden Makana <branden.makana@...> wrote:

Thanks James - that makes sense then. Now my concern is - do I need to make sure any files put anywhere in that bucket are done so by jets3t? I was planning on using one bucket for all my files, and then "folders" inside that bucket for each server I planned on backing up. It sounds like if I need to make sure to use jets3t or else various programs might step on each other in how they do folders, is that correct?

On Wed, Jul 8, 2009 at 5:03 PM, James Murty <james@...> wrote:

Hi Branden,

I'm afraid this problem is due to an incompatibility between the way JetS3t represents folders in S3 and the way the Firefox plugin does so. All your files will be present in S3 and completely accessible to JetS3t programs, but because JetS3t and Elasticfox use different methods to indicate when an S3 object really represents a folder you will need to give Elasticfox a little help.

Instead of relying on Elasticfox recognizing folders, you will need to type in the folder path into the object path field with an added slash character at the end, e.g. "myfolder/". When you add the slash character, you should get a listing of files inside that "folder".

As an alternative, you can use the JetS3t Cockpit application which will list all the objects in a bucket. However, Cockpit does not offer the same kind of folder-specific view that Elasticfox does.

It is unfortunate that there is not a standard way to represent folders stored in S3 and that different tools use different techniques. The underlying issue is that there is really no such thing as a folder object in S3, so we all use different tricks when storing folder hierarchies so we can recognize which objects represent folders and which represent files. All of which causes headaches for users.

I hope this helps,
James

---
http://www.jamesmurty.com

On Wed, Jul 8, 2009 at 4:12 PM, Branden Makana <branden.makana@...> wrote:
Hello,

I was trying out jets3t's Synchronize program to backup a folder structure to amazon's S3. I ran the batch file, and it appeared to run correctly - it says it copied 61,544 files, it took several hours, and no errors were spit out. However, when I use the Firefox plugin "S3 Firefox" (so that I can confirm the backup did run correctly), I see it made a single file that's 0 bytes. Now I'm trying to figure out what happened.

Here's how I ran jets3t, on a Windows 2003 server:

synchronize.bat -b UP "portentint/homerbak" e:\inetpub\wwwroot

portentint being my bucket on S3, and I wanted a directory inside that called homerbak. When I look in S3 Firefox, I see homerbak folder inside my bucket (so that worked), and then inside there I see "wwwroot" which looks like a file, and has a file size of 0.

What did I do wrong, and how could it have taken several hours copying 4-5GB but I only see this empty file?

Any help/comments are appreciated :)

Re: Synchronize didn't error but created a single, empty file

2009-07-08T17:21:14Z

Thanks James - that makes sense then. Now my concern is - do I need to make sure any files put anywhere in that bucket are done so by jets3t? I was planning on using one bucket for all my files, and then "folders" inside that bucket for each server I planned on backing up. It sounds like if I need to make sure to use jets3t or else various programs might step on each other in how they do folders, is that correct?

On Wed, Jul 8, 2009 at 5:03 PM, James Murty <james@...> wrote:

Hi Branden,

I'm afraid this problem is due to an incompatibility between the way JetS3t represents folders in S3 and the way the Firefox plugin does so. All your files will be present in S3 and completely accessible to JetS3t programs, but because JetS3t and Elasticfox use different methods to indicate when an S3 object really represents a folder you will need to give Elasticfox a little help.

Instead of relying on Elasticfox recognizing folders, you will need to type in the folder path into the object path field with an added slash character at the end, e.g. "myfolder/". When you add the slash character, you should get a listing of files inside that "folder".

As an alternative, you can use the JetS3t Cockpit application which will list all the objects in a bucket. However, Cockpit does not offer the same kind of folder-specific view that Elasticfox does.

It is unfortunate that there is not a standard way to represent folders stored in S3 and that different tools use different techniques. The underlying issue is that there is really no such thing as a folder object in S3, so we all use different tricks when storing folder hierarchies so we can recognize which objects represent folders and which represent files. All of which causes headaches for users.

I hope this helps,
James

---
http://www.jamesmurty.com

On Wed, Jul 8, 2009 at 4:12 PM, Branden Makana <branden.makana@...> wrote:
Hello,

I was trying out jets3t's Synchronize program to backup a folder structure to amazon's S3. I ran the batch file, and it appeared to run correctly - it says it copied 61,544 files, it took several hours, and no errors were spit out. However, when I use the Firefox plugin "S3 Firefox" (so that I can confirm the backup did run correctly), I see it made a single file that's 0 bytes. Now I'm trying to figure out what happened.

Here's how I ran jets3t, on a Windows 2003 server:

synchronize.bat -b UP "portentint/homerbak" e:\inetpub\wwwroot

portentint being my bucket on S3, and I wanted a directory inside that called homerbak. When I look in S3 Firefox, I see homerbak folder inside my bucket (so that worked), and then inside there I see "wwwroot" which looks like a file, and has a file size of 0.

What did I do wrong, and how could it have taken several hours copying 4-5GB but I only see this empty file?

Any help/comments are appreciated :)

Re: Synchronize didn't error but created a single, empty file

2009-07-08T17:17:25Z

Oops, substitute "S3Fox" wherever I said "Elasticfox" in my last message. Elasticfox is the EC2 Firefox add-on, not the S3 one.

James

On Wed, Jul 8, 2009 at 5:03 PM, James Murty <james@...> wrote:

Hi Branden,

I'm afraid this problem is due to an incompatibility between the way JetS3t represents folders in S3 and the way the Firefox plugin does so. All your files will be present in S3 and completely accessible to JetS3t programs, but because JetS3t and Elasticfox use different methods to indicate when an S3 object really represents a folder you will need to give Elasticfox a little help.

Instead of relying on Elasticfox recognizing folders, you will need to type in the folder path into the object path field with an added slash character at the end, e.g. "myfolder/". When you add the slash character, you should get a listing of files inside that "folder".

As an alternative, you can use the JetS3t Cockpit application which will list all the objects in a bucket. However, Cockpit does not offer the same kind of folder-specific view that Elasticfox does.

It is unfortunate that there is not a standard way to represent folders stored in S3 and that different tools use different techniques. The underlying issue is that there is really no such thing as a folder object in S3, so we all use different tricks when storing folder hierarchies so we can recognize which objects represent folders and which represent files. All of which causes headaches for users.

I hope this helps,
James

---
http://www.jamesmurty.com

On Wed, Jul 8, 2009 at 4:12 PM, Branden Makana <branden.makana@...> wrote:
Hello,

I was trying out jets3t's Synchronize program to backup a folder structure to amazon's S3. I ran the batch file, and it appeared to run correctly - it says it copied 61,544 files, it took several hours, and no errors were spit out. However, when I use the Firefox plugin "S3 Firefox" (so that I can confirm the backup did run correctly), I see it made a single file that's 0 bytes. Now I'm trying to figure out what happened.

Here's how I ran jets3t, on a Windows 2003 server:

synchronize.bat -b UP "portentint/homerbak" e:\inetpub\wwwroot

portentint being my bucket on S3, and I wanted a directory inside that called homerbak. When I look in S3 Firefox, I see homerbak folder inside my bucket (so that worked), and then inside there I see "wwwroot" which looks like a file, and has a file size of 0.

What did I do wrong, and how could it have taken several hours copying 4-5GB but I only see this empty file?

Any help/comments are appreciated :)

Re: Synchronize didn't error but created a single, empty file

2009-07-08T17:03:41Z

Hi Branden,

I'm afraid this problem is due to an incompatibility between the way JetS3t represents folders in S3 and the way the Firefox plugin does so. All your files will be present in S3 and completely accessible to JetS3t programs, but because JetS3t and Elasticfox use different methods to indicate when an S3 object really represents a folder you will need to give Elasticfox a little help.

Instead of relying on Elasticfox recognizing folders, you will need to type in the folder path into the object path field with an added slash character at the end, e.g. "myfolder/". When you add the slash character, you should get a listing of files inside that "folder".

As an alternative, you can use the JetS3t Cockpit application which will list all the objects in a bucket. However, Cockpit does not offer the same kind of folder-specific view that Elasticfox does.

It is unfortunate that there is not a standard way to represent folders stored in S3 and that different tools use different techniques. The underlying issue is that there is really no such thing as a folder object in S3, so we all use different tricks when storing folder hierarchies so we can recognize which objects represent folders and which represent files. All of which causes headaches for users.

I hope this helps,
James

---
http://www.jamesmurty.com

On Wed, Jul 8, 2009 at 4:12 PM, Branden Makana <branden.makana@...> wrote:

Hello,

I was trying out jets3t's Synchronize program to backup a folder structure to amazon's S3. I ran the batch file, and it appeared to run correctly - it says it copied 61,544 files, it took several hours, and no errors were spit out. However, when I use the Firefox plugin "S3 Firefox" (so that I can confirm the backup did run correctly), I see it made a single file that's 0 bytes. Now I'm trying to figure out what happened.

Here's how I ran jets3t, on a Windows 2003 server:

synchronize.bat -b UP "portentint/homerbak" e:\inetpub\wwwroot

portentint being my bucket on S3, and I wanted a directory inside that called homerbak. When I look in S3 Firefox, I see homerbak folder inside my bucket (so that worked), and then inside there I see "wwwroot" which looks like a file, and has a file size of 0.

What did I do wrong, and how could it have taken several hours copying 4-5GB but I only see this empty file?

Any help/comments are appreciated :)

Synchronize didn't error but created a single, empty file

2009-07-08T16:12:49Z

Hello,

I was trying out jets3t's Synchronize program to backup a folder structure to amazon's S3. I ran the batch file, and it appeared to run correctly - it says it copied 61,544 files, it took several hours, and no errors were spit out. However, when I use the Firefox plugin "S3 Firefox" (so that I can confirm the backup did run correctly), I see it made a single file that's 0 bytes. Now I'm trying to figure out what happened.

Here's how I ran jets3t, on a Windows 2003 server:

synchronize.bat -b UP "portentint/homerbak" e:\inetpub\wwwroot

portentint being my bucket on S3, and I wanted a directory inside that called homerbak. When I look in S3 Firefox, I see homerbak folder inside my bucket (so that worked), and then inside there I see "wwwroot" which looks like a file, and has a file size of 0.

What did I do wrong, and how could it have taken several hours copying 4-5GB but I only see this empty file?

Any help/comments are appreciated :)

Re: toolkit and crypto

2009-06-09T20:09:01Z

I should mention that it is difficult to achieve truly transparent
encryption of data sent to S3 because you need to know the exact size
of an S3 object before you upload it. If you simply run data through
an encryption cipher as it is being uploaded, the resultant data
probably won't match the size of the original data and the upload will
fail.

So if you do use the EncryptUtils#encrypt method to get an encrypting
cipher stream, make sure you write this data to a temporary file or
buffer first, and only upload it once you know the final size.

> am i now stuck with warnock's dilemma? only time will tell.

No dilemma. However, the JetS3t Users Google group is probably a
better place to ask questions because more people are listening there:
http://groups.google.com/group/jets3t-users

Cheers,
James

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: toolkit and crypto

2009-06-09T19:56:22Z

Hi Patrick,

You are correct. The "crypto.algorithm" property setting in
jets3t.properties only applies to the JetS3t applications, and will
not enable automatic encryption of uploaded objects in the toolkit.

The easiest way to encrypt items you upload to S3 is to use the
ObjectUtils#createObjectForUpload utility methods to transform a file
prior to upload. These methods take an EncryptionUtil argument where
you can specify a password and the encryption algorithm you wish to
use.

The ObjectUtils class has a corresponding #createPackageForDownload
method that make it simpler to download encrypted objects from S3,
decrypting the data as it is downloaded.

Alternately, you can use the EncrptionUtil #encrypt and #decrypt
methods to generate cipher input and output streams. These will
encrypt or decrypt data in the underlying streams you provide.
However, if you use these cipher streams directly, it would be wise to
include enough metadata information with your S3 objects to allow you
to decrypt them later. The advantage of the ObjectUtils methods is
that they automatically create and consume S3 metadata items that
indicate the algorithm that was used to encrypt data.

Hope this helps,
James

---
http://www.jamesmurty.com

On Tue, Jun 9, 2009 at 3:10 PM, Patrick Linehan<plinehan@...> wrote:

> i'm having trouble getting transparent encryption working when using the
> toolkit. i had assumed that setting the "crypto.algorithm" property would
> automatically enable encryption, but i was wrong.
> i've searched the docs, mailing lists and the web, but i can't find any
> discussion of this.
> i'm using the standard "PBEWithMD5AndDES" algorithm, and running
> EncryptionUtil.java's "main" method verifies that the cipher should be
> available on my machine.
> is there a setting i'm missing? does the toolkit even support this
> functionality, or is it limited to the applications?
> any help would be grealy appreciated.
> thanks!
> PAt

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: toolkit and crypto

2009-06-09T19:41:55Z

> is there a setting i'm missing? does the toolkit even support this functionality, or

> is it limited to the applications?

after doing some more reading and grepping of the source, i believe i can now answer my own question: the toolkit does not support "transparent" encryption, only the apps

it looks like i should use ObjectUtils and/or EncryptionUtil to do this work manually before uploading by blobs.

am i now stuck with warnock's dilemma? only time will tell.

On Tue, Jun 9, 2009 at 3:10 PM, Patrick Linehan <plinehan@...> wrote:

i'm having trouble getting transparent encryption working when using the toolkit. i had assumed that setting the "crypto.algorithm" property would automatically enable encryption, but i was wrong.

i've searched the docs, mailing lists and the web, but i can't find any discussion of this.

i'm using the standard "PBEWithMD5AndDES" algorithm, and running EncryptionUtil.java's "main" method verifies that the cipher should be available on my machine.

is there a setting i'm missing? does the toolkit even support this functionality, or is it limited to the applications?

any help would be grealy appreciated.

thanks!
PAt

toolkit and crypto

2009-06-09T15:10:47Z

i'm having trouble getting transparent encryption working when using the toolkit. i had assumed that setting the "crypto.algorithm" property would automatically enable encryption, but i was wrong.

i've searched the docs, mailing lists and the web, but i can't find any discussion of this.

i'm using the standard "PBEWithMD5AndDES" algorithm, and running EncryptionUtil.java's "main" method verifies that the cipher should be available on my machine.

is there a setting i'm missing? does the toolkit even support this functionality, or is it limited to the applications?

any help would be grealy appreciated.

thanks!

PAt

Re: S3 GET connection failed for

2009-06-01T18:53:16Z

Hi Joe,

I haven't come across this kind of error before so I'm afraid I don't have any solutions. I've done some Googling without finding a consistent explanation.

Are you running your application on Solaris with TCP No Delay enabled? If so, this bug report might be relevant:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6378870

I have come across a few other mentions of this issue but no solid fixes or work-arounds:
http://www.nabble.com/SocketException:-Connection-reset---Invalid-argument-td9478521.html
http://lists.apple.com/archives/Java-dev/2008/Jul/msg00089.html
http://www.innovation.ch/java/HTTPClient/errors.html

Reading between the lines of some of these discussions, it sounds like this exception *may* be something you can ignore as an intermittent connection problem -- despite the nasty exception message.

Some of the links even imply this exception indicates a server-side problem, though that seems unlikely given you're interacting with S3. I don't suppose there's a proxy between your client and S3? I notice that you're connecting with HTTPS, so probably not.

If you think it is acceptable to ignore these exceptions and just retry the operation, you may want to modify JetS3t RestS3Service class to catch these specific exceptions and retry the request using its built-in retry mechanism -- like it already does for Internal Server (500) errors from the S3 service. By default, this exception will be thrown by RestS3Service straight away and it won't retry the request.

Does anyone else have experience with this issue?

James

---
http://www.jamesmurty.com

On Mon, Jun 1, 2009 at 5:57 PM, Joe Moreno <joemoreno@...> wrote:

Hi,

I have some code that uploads thousands of objects into S3, using several threads at a time. For some reason, the code runs fine for for awhile and then it suddenly starts throwing exceptions that look like this:

[2009-06-01 02:19:34 PDT] <S3HTMLBucketThread-archive.adjix.com : 4xpd-20090501-20090531-713e20e95a60410499d9b79ec596e7a5.csv> archive.adjix.com Exception writing HTML file to bucket = org.jets3t.service.S3ServiceException: S3 GET connection failed for '/'
org.jets3t.service.S3ServiceException: S3 GET connection failed for '/'
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:516)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:752)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.listAllBucketsImpl(RestS3Service.java:1041)
at org.jets3t.service.S3Service.listAllBuckets(S3Service.java:1348)
at org.jets3t.service.S3Service.getBucket(S3Service.java:1578)
at org.jets3t.service.S3Service.getOrCreateBucket(S3Service.java:1601)
at com.woextras.S3Utilities.bucketNamed(S3Utilities.java:50)
at com.woextras.S3HTMLBucketThread.run(S3HTMLBucketThread.java:82)
Caused by: java.net.SocketException: Invalid argument
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:723)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1366)
at com.sun.net.ssl.internal.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:103)
at com.sun.net.ssl.internal.ssl.Handshaker.kickstart(Handshaker.java:528)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.kickstartHandshake(SSLSocketImpl.java:1120)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1029)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:622)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1565
)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:342)
... 7 more

Any help would be appreciated.

- Joe

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

S3 GET connection failed for

2009-06-01T17:57:13Z

Hi,

I have some code that uploads thousands of objects into S3, using
several threads at a time. For some reason, the code runs fine for for
awhile and then it suddenly starts throwing exceptions that look like
this:

[2009-06-01 02:19:34 PDT] <S3HTMLBucketThread-archive.adjix.com :
4xpd-20090501-20090531-713e20e95a60410499d9b79ec596e7a5.csv>
archive.adjix.com Exception writing HTML file to bucket =
org.jets3t.service.S3ServiceException: S3 GET connection failed for '/'
org.jets3t.service.S3ServiceException: S3 GET connection failed for '/'
at
org
.jets3t
.service
.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:
516)
at
org
.jets3t
.service
.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:
752)
at
org
.jets3t
.service
.impl
.rest.httpclient.RestS3Service.listAllBucketsImpl(RestS3Service.java:
1041)
at org.jets3t.service.S3Service.listAllBuckets(S3Service.java:1348)
at org.jets3t.service.S3Service.getBucket(S3Service.java:1578)
at org.jets3t.service.S3Service.getOrCreateBucket(S3Service.java:1601)
at com.woextras.S3Utilities.bucketNamed(S3Utilities.java:50)
at com.woextras.S3HTMLBucketThread.run(S3HTMLBucketThread.java:82)
Caused by: java.net.SocketException: Invalid argument
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at
com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at
com
.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:
723)
at
com
.sun
.net.ssl.internal.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:
1366)
at
com
.sun
.net.ssl.internal.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:
103)
at com.sun.net.ssl.internal.ssl.Handshaker.kickstart(Handshaker.java:
528)
at
com
.sun
.net
.ssl.internal.ssl.SSLSocketImpl.kickstartHandshake(SSLSocketImpl.java:
1120)
at
com
.sun
.net
.ssl
.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:
1029)
at
com
.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:
622)
at
com
.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:
65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at
org
.apache
.commons
.httpclient
.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager
$
HttpConnectionAdapter
.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1565
)
at
org
.apache
.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:
2116)
at
org
.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:
1096)
at
org
.apache
.commons
.httpclient
.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at
org
.apache
.commons
.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:
171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:
397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:
323)
at
org
.jets3t
.service
.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:
342)
... 7 more

Any help would be appreciated.

- Joe

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Questions about security and confidentiality

2009-05-29T19:57:42Z

Hi Mark,

Does Jets3t plans to have any APIs to encrypt/decrypt users data based on
AES256 standards/..

JetS3t includes support for encryption, and combined with the BouncyCastle encryption library that is packaged with the distribution you can use the AES cipher.

There is a section called "Encryption properties" on the configuration page that details how to configure service-wide encryption:
http://jets3t.s3.amazonaws.com/toolkit/configuration.html#jets3t

See also the documentation for the EncryptUtil utility class, which provides more fine-grained tools for encrypting/decrypting byte arrays and input streams:
http://jets3t.s3.amazonaws.com/api/org/jets3t/service/security/EncryptionUtil.html

The ObjectUtils class pulls everything together and makes it easy to prepare objects whose data will be encrypted prior to upload:
http://jets3t.s3.amazonaws.com/api/org/jets3t/service/utils/ObjectUtils.html

Thanks for making things clear..:-)

You're welcome, good luck with it.

James

Re: Questions about security and confidentiality

2009-05-29T19:23:54Z

Thanks James, for making things crystal clear. The more I think of it more things get complex..

I have the big picture now, here is what I am planning to do..
-- I do not have a choice but keep the data encrypted all times..
-- Rules to keep different buckets for different access points
-- Use s3fs/subcloud to mount S3 on EC2
-- Samba uses the Mount Point (how will samba decrypt and show the data to client needs to be formulated, I guess the decrypion will happen on Server..)
-- Put some 3rdParty/softwares , proper access controls and procedures to make sure admins accidently
do not access the data. but are aware with the hurdles, and logging, that they are accessing on purpose.
-- EBS is worth looking, but S3 webservices on JetS3T is so easy, and with new features getting added it makes life easier for programmers, with same code can be resused and extend to create different type of clients.
* I am not aware of any ready made APIs, of the opinion that there is lot of work from ground up to create a web application on top of it. And have to repeat the same..

Does Jets3t plans to have any APIs to encrypt/decrypt users data based on AES256 standards/..

Thanks for making things clear..

James Murty-3 wrote:

Mark, to make your objectives achievable I think you need to simplify things
as much as you can. You say that your requirements are simple, but I don't
think they are.

Here are some things you need to think about:

- If you want to protect your customer's data "at rest" from Amazon staff,
or from your AWS account being compromised, you must encrypt the data. You
have no choice, even though it makes everything more complicated.

- If you encrypt data, you need to decide where the encryption/decryption
will take place.
* You get the most security if client-side software does this, in which case
no-one but your clients can read the data. However the clients then need to
deal with the complexity of installing/managing software, and you need to
provide this software.
* If you want to provide web access to the data without any client-side
software, you will need to encrypt/decrypt on your server. This is less
secure but more user-friendly. You will also need to trust your admins.

- Trying to have multiple access points for writing to the same storage
space is a bad idea. Only do this if you will have some very smart
software/processes to manage (or avoid) data writing collisions, or if the
Cloudfront and JetS3t components use completely different S3 buckets. This
isn't an issue of compatibility between Cloudfront and JetS3t

- Providing SAMBA access to S3 objects sounds very difficult indeed. Are you
going to cache all or some of your S3 objects on an EC2 instance to make it
available via SAMBA? If not, how will you mediate between the SAMBA server
on EC2 and your S3 storage? If so, how and where will you handle
encryption/decryption of the data?

- Jon's comment on using EC2's Elastic Block Store drives is worth
considering.

James

On Wed, May 27, 2009 at 1:40 PM, MarkAtHarvest <mark@harvestinfotech.com>wrote:

>
> Thank you Jonathan , James.
> I think I need to work more on IT Standards which acceptable. Will comeback
> on it
>
> About dual access, I need to use S3, I am very comfortable with it, rather
> than going for a new solution. I think with Subcloud I can use Jets3t (need
> to confirm on it though)
>
>
>
> Jonathan Harlap wrote:
> >
> > Hi Mark,
> >
> > First off, I'm leaving aside what I think is an obvious issue you'll need
> > to
> > resolve about the two access methods conflicting (ie, users manipulating
> > the
> > same data via your web app and smb simultaneously). James already
> > mentioned
> > it and it's a property of your apparent design not of the AWS services.
> >
> > What strikes me as particularly interesting your statement that S3 will
> be
> > mounted in an EC2 instance to provide smb access. If you're doing that,
> > couldn't you just as well put your data in an EBS volume, mount that to
> > your
> > EC2 instance for smb access and *also* have your web app interact with
> the
> > data which now appears to exist on a local mount point? Then S3 (as an
> > API)
> > is out of the picture entirely and you have what sounds like a simpler
> > problem to solve. I put that aside now as well, as it's more a question
> > of
> > your design than anything S3 specific, which was the purpose of the
> > question, I believe.
> >
> > S3 signed urls are not related to jets3t, although jets3t does facilitate
> > their creation. However, I interpret your question to say that you lose
> > the
> > ability to point user's web browsers directly at a signed url, and this
> > would be correct if you encrypt the data in S3. If you encrypt the data
> > you
> > store, then you will need to provide a client (whether it be a web app or
> > a
> > thick client) that will perform the encryption/decryption for your
> > clients.
> >
> > Unfortunately, I don't think anyone outside your business can really
> > answer
> > the questions you're posing beyond what has already been said. We don't
> > know what your application does. We can't tell you whether to encrypt
> > your
> > data and, if so, how, nor how to adapt your app design accordingly.
> >
> > Good luck with your project.
> >
> > Cheers,
> > Jonathan
> >
> >
> > On Wed, May 27, 2009 at 1:58 PM, MarkAtHarvest
> > <mark@harvestinfotech.com>wrote:
> >
> >>
> >> Thanks Jonathan for the interesting links.
> >>
> >> My requirement is very simple, just want to build a application based on
> >> S3
> >> where I can store customers data, and give them a guarantee that your
> >> data
> >> is safe and secure, while at move and while at rest.
> >> 1. My JetS3t Web application will access the data for the customer as
> >> well
> >> as
> >> 2. S3 will get mounted on EC2 for a SAMBA access.
> >>
> >> Data while at move can be protected with SSL, I am more concerned with
> >> data
> >> while at rest, also about accessKey, SecretKey, TokenIDs.
> >>
> >> Questions which arise are
> >> 1. Do i need to encrypt the S3 data for such type of situations (If I
> >> encrypt, then i loose the JetS3t features like getSignedUrl, which
> >> creates
> >> a
> >> link to open the file directly from S3 instead of coming to our servers)
> >> 2. what are must must things I need to do.
> >>
> >> Thanks again for your inputs
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> Jonathan Harlap wrote:
> >> >
> >> > Mark,
> >> >
> >> > I suspect one source of confusion for you is the claim of HIPAA
> >> > compliance.
> >> > Firstly, AWS itself is not HIPAA compliant nor does it try to be. The
> >> > whitepaper in question discusses some of the strategies that might be
> >> used
> >> > by AWS customers to build HIPAA compliant systems that run on the AWS
> >> > cloud.
> >> >
> >> > As to crypto, high security, and so on - I don't find your need very
> >> > clear,
> >> > so I can't offer any specific advice other than to say that crypto is
> a
> >> > very
> >> > tricky thing to do right, and a very easy thing to do wrong, so for
> >> your
> >> > clients' sake, do it carefully. For a fun example of how the little
> >> > details
> >> > make a difference, see
> >> > http://www.codinghorror.com/blog/archives/001267.htmland
> >> > http://www.codinghorror.com/blog/archives/001268.html
> >> >
> >> > Cheers,
> >> > J
> >> >
> >> > On Wed, May 27, 2009 at 8:47 AM, MarkAtHarvest
> >> > <mark@harvestinfotech.com>wrote:
> >> >
> >> >>
> >> >> Thank you for clearing my doubt!
> >> >> But I see white papers saying Amazon AWS is HIPAA compliant, which is
> >> a
> >> >> stringent security standard.
> >> >> The confusion I have is
> >> >> 1. I have a webcient build on JetS3 application,soon be adding a
> >> DevPay
> >> >> support.
> >> >> 2. I also want to sync S3 files on EC2 using something like Subcloud
> >> and
> >> >> provide access to the data
> >> >> on clients windows explorer using SAMBA.
> >> >>
> >> >> Now if I do not encrypt, and use ACL to control access on the S3, so
> >> that
> >> >> only the respective client has an access to it.
> >> >> Subcloud will not be able to import the data , if it does not have
> >> >> permission, or it will be able to import the data using secret key.
> >> >>
> >> >> As per what you say, I need to keep that secretKey really secret and
> >> only
> >> >> one Admin can know it and there should be policies to access that
> key.
> >> >>
> >> >> So all the S3 products out there in market who use S3, administrator
> >> do
> >> >> have
> >> >> a access to data, which might be controlled by some measures. So how
> >> can
> >> >> be
> >> >> claim high security compliance
> >> >>
> >> >> Thanks for addressing this question, I am not able to get a clean
> head
> >> >> path
> >> >> on it
> >> >>
> >> >>
> >> >>
> >> >> Now is that acceptable in terms of Secu
> >> >>
> >> >>
> >> >>
> >> >> James Murty-3 wrote:
> >> >> >
> >> >> > The only way to allow clients to store encrypted data in S3 in such
> >> a
> >> >> way
> >> >> > that administrators cannot read it, is to provide some kind of
> >> >> application
> >> >> > the client can run on his/her own machine to do this work.
> >> >> >
> >> >> > If you don't trust your admins, there is little point encrypting a
> >> >> user's
> >> >> > files on your own server because admins will have simply be able to
> >> >> access
> >> >> > to the data there, rather than from S3 directly. Encrypting the
> data
> >> on
> >> >> > your
> >> >> > server would protect it from the Amazon admins who maintain S3, but
> >> not
> >> >> > from
> >> >> > your own server admins.
> >> >> >
> >> >> > A custom client app would allow your clients to automatically
> >> encrypt
> >> >> data
> >> >> > prior to uploading, and to decrypt it when downloading. This app
> >> could
> >> >> > also
> >> >> > be designed to interact with your server component to obtain signed
> >> >> URLs
> >> >> > to
> >> >> > gain access to S3.
> >> >> >
> >> >> > The problem is, such an app doesn't exist as far as I know. The
> >> >> > combination
> >> >> > of JetS3t's CockpitLite and Gatekeeper applications comes close,
> but
> >> >> > CockpitLite does not do any encryption.
> >> >> >
> >> >> > Hope this helps,
> >> >> > James
> >> >> >
> >> >> > ---
> >> >> > http://www.jamesmurty.com
> >> >> >
> >> >> >
> >> >> > On Tue, May 26, 2009 at 9:24 PM, MarkAtHarvest
> >> >> > <mark@harvestinfotech.com>wrote:
> >> >> >
> >> >> >>
> >> >> >> My jets3t client is working fine, current I am trying to upgrade
> it
> >> to
> >> >> >> DEV
> >> >> >> Pay account. I would like to ask if I can get little bit guidance
> >> on
> >> >> >> following
> >> >> >>
> >> >> >> I am using Amazon Dev Pay, then so as that an evil administrator
> is
> >> >> not
> >> >> >> able
> >> >> >> to see the S3 files of customers, what exactly do I need to do
> >> >> >> 1. can be to encrypt all the files stored in S3.
> >> >> >> The problem I see with that approach is,
> >> >> >> *. I cannot use Amazon HTTP Post to directly upload files on S3,
> >> >> >> without
> >> >> >> going through my server.
> >> >> >> *. I cannot use createSignedGetUrl(), to create signed URLs to
> >> >> expose
> >> >> >> links for a temporary time, as the files need to be brought to my
> >> >> sever
> >> >> >> before user can download it.
> >> >> >>
> >> >> >> Is there a way I can solve the above two problems..
> >> >> >>
> >> >> >> --
> >> >> >> View this message in context:
> >> >> >>
> >> >>
> >>
> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23735783.html
> >> >> >> Sent from the JetS3t Users mailing list archive at Nabble.com.
> >> >> >>
> >> >> >>
> >> >> >>
> >> ---------------------------------------------------------------------
> >> >> >> To unsubscribe, e-mail: users-unsubscribe@jets3t.dev.java.net
> >> >> >> For additional commands, e-mail: users-help@jets3t.dev.java.net
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23741804.html
> >> >> Sent from the JetS3t Users mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: users-unsubscribe@jets3t.dev.java.net
> >> >> For additional commands, e-mail: users-help@jets3t.dev.java.net
> >> >>
> >> >>
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23747665.html
> >> Sent from the JetS3t Users mailing list archive at Nabble.com.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@jets3t.dev.java.net
> >> For additional commands, e-mail: users-help@jets3t.dev.java.net
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23750391.html
> Sent from the JetS3t Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@jets3t.dev.java.net
> For additional commands, e-mail: users-help@jets3t.dev.java.net
>
>

Re: Questions about security and confidentiality

2009-05-28T12:01:57Z

Mark, to make your objectives achievable I think you need to simplify things as much as you can. You say that your requirements are simple, but I don't think they are.

Here are some things you need to think about:

- If you want to protect your customer's data "at rest" from Amazon staff, or from your AWS account being compromised, you must encrypt the data. You have no choice, even though it makes everything more complicated.

- If you encrypt data, you need to decide where the encryption/decryption will take place.
* You get the most security if client-side software does this, in which case no-one but your clients can read the data. However the clients then need to deal with the complexity of installing/managing software, and you need to provide this software.
* If you want to provide web access to the data without any client-side software, you will need to encrypt/decrypt on your server. This is less secure but more user-friendly. You will also need to trust your admins.

- Trying to have multiple access points for writing to the same storage space is a bad idea. Only do this if you will have some very smart software/processes to manage (or avoid) data writing collisions, or if the Cloudfront and JetS3t components use completely different S3 buckets. This isn't an issue of compatibility between Cloudfront and JetS3t

- Providing SAMBA access to S3 objects sounds very difficult indeed. Are you going to cache all or some of your S3 objects on an EC2 instance to make it available via SAMBA? If not, how will you mediate between the SAMBA server on EC2 and your S3 storage? If so, how and where will you handle encryption/decryption of the data?

- Jon's comment on using EC2's Elastic Block Store drives is worth considering.

James

On Wed, May 27, 2009 at 1:40 PM, MarkAtHarvest <mark@...> wrote:

Thank you Jonathan , James.
I think I need to work more on IT Standards which acceptable. Will comeback
on it

About dual access, I need to use S3, I am very comfortable with it, rather
than going for a new solution. I think with Subcloud I can use Jets3t (need
to confirm on it though)

Jonathan Harlap wrote:
>
> Hi Mark,
>
> First off, I'm leaving aside what I think is an obvious issue you'll need
> to
> resolve about the two access methods conflicting (ie, users manipulating
> the
> same data via your web app and smb simultaneously). James already
> mentioned
> it and it's a property of your apparent design not of the AWS services.
>
> What strikes me as particularly interesting your statement that S3 will be
> mounted in an EC2 instance to provide smb access. If you're doing that,
> couldn't you just as well put your data in an EBS volume, mount that to
> your
> EC2 instance for smb access and *also* have your web app interact with the
> data which now appears to exist on a local mount point? Then S3 (as an
> API)
> is out of the picture entirely and you have what sounds like a simpler
> problem to solve. I put that aside now as well, as it's more a question
> of
> your design than anything S3 specific, which was the purpose of the
> question, I believe.
>
> S3 signed urls are not related to jets3t, although jets3t does facilitate
> their creation. However, I interpret your question to say that you lose
> the
> ability to point user's web browsers directly at a signed url, and this
> would be correct if you encrypt the data in S3. If you encrypt the data
> you
> store, then you will need to provide a client (whether it be a web app or
> a
> thick client) that will perform the encryption/decryption for your
> clients.
>
> Unfortunately, I don't think anyone outside your business can really
> answer
> the questions you're posing beyond what has already been said. We don't
> know what your application does. We can't tell you whether to encrypt
> your
> data and, if so, how, nor how to adapt your app design accordingly.
>
> Good luck with your project.
>
> Cheers,
> Jonathan
>
>
> On Wed, May 27, 2009 at 1:58 PM, MarkAtHarvest
> <mark@...>wrote:
>
>>
>> Thanks Jonathan for the interesting links.
>>
>> My requirement is very simple, just want to build a application based on
>> S3
>> where I can store customers data, and give them a guarantee that your
>> data
>> is safe and secure, while at move and while at rest.
>> 1. My JetS3t Web application will access the data for the customer as
>> well
>> as
>> 2. S3 will get mounted on EC2 for a SAMBA access.
>>
>> Data while at move can be protected with SSL, I am more concerned with
>> data
>> while at rest, also about accessKey, SecretKey, TokenIDs.
>>
>> Questions which arise are
>> 1. Do i need to encrypt the S3 data for such type of situations (If I
>> encrypt, then i loose the JetS3t features like getSignedUrl, which
>> creates
>> a
>> link to open the file directly from S3 instead of coming to our servers)
>> 2. what are must must things I need to do.
>>
>> Thanks again for your inputs
>>
>>
>>
>>
>>
>>
>>
>> Jonathan Harlap wrote:
>> >
>> > Mark,
>> >
>> > I suspect one source of confusion for you is the claim of HIPAA
>> > compliance.
>> > Firstly, AWS itself is not HIPAA compliant nor does it try to be. The
>> > whitepaper in question discusses some of the strategies that might be
>> used
>> > by AWS customers to build HIPAA compliant systems that run on the AWS
>> > cloud.
>> >
>> > As to crypto, high security, and so on - I don't find your need very
>> > clear,
>> > so I can't offer any specific advice other than to say that crypto is a
>> > very
>> > tricky thing to do right, and a very easy thing to do wrong, so for
>> your
>> > clients' sake, do it carefully. For a fun example of how the little
>> > details
>> > make a difference, see
>> > http://www.codinghorror.com/blog/archives/001267.htmland
>> > http://www.codinghorror.com/blog/archives/001268.html
>> >
>> > Cheers,
>> > J
>> >
>> > On Wed, May 27, 2009 at 8:47 AM, MarkAtHarvest
>> > <mark@...>wrote:
>> >
>> >>
>> >> Thank you for clearing my doubt!
>> >> But I see white papers saying Amazon AWS is HIPAA compliant, which is
>> a
>> >> stringent security standard.
>> >> The confusion I have is
>> >> 1. I have a webcient build on JetS3 application,soon be adding a
>> DevPay
>> >> support.
>> >> 2. I also want to sync S3 files on EC2 using something like Subcloud
>> and
>> >> provide access to the data
>> >> on clients windows explorer using SAMBA.
>> >>
>> >> Now if I do not encrypt, and use ACL to control access on the S3, so
>> that
>> >> only the respective client has an access to it.
>> >> Subcloud will not be able to import the data , if it does not have
>> >> permission, or it will be able to import the data using secret key.
>> >>
>> >> As per what you say, I need to keep that secretKey really secret and
>> only
>> >> one Admin can know it and there should be policies to access that key.
>> >>
>> >> So all the S3 products out there in market who use S3, administrator
>> do
>> >> have
>> >> a access to data, which might be controlled by some measures. So how
>> can
>> >> be
>> >> claim high security compliance
>> >>
>> >> Thanks for addressing this question, I am not able to get a clean head
>> >> path
>> >> on it
>> >>
>> >>
>> >>
>> >> Now is that acceptable in terms of Secu
>> >>
>> >>
>> >>
>> >> James Murty-3 wrote:
>> >> >
>> >> > The only way to allow clients to store encrypted data in S3 in such
>> a
>> >> way
>> >> > that administrators cannot read it, is to provide some kind of
>> >> application
>> >> > the client can run on his/her own machine to do this work.
>> >> >
>> >> > If you don't trust your admins, there is little point encrypting a
>> >> user's
>> >> > files on your own server because admins will have simply be able to
>> >> access
>> >> > to the data there, rather than from S3 directly. Encrypting the data
>> on
>> >> > your
>> >> > server would protect it from the Amazon admins who maintain S3, but
>> not
>> >> > from
>> >> > your own server admins.
>> >> >
>> >> > A custom client app would allow your clients to automatically
>> encrypt
>> >> data
>> >> > prior to uploading, and to decrypt it when downloading. This app
>> could
>> >> > also
>> >> > be designed to interact with your server component to obtain signed
>> >> URLs
>> >> > to
>> >> > gain access to S3.
>> >> >
>> >> > The problem is, such an app doesn't exist as far as I know. The
>> >> > combination
>> >> > of JetS3t's CockpitLite and Gatekeeper applications comes close, but
>> >> > CockpitLite does not do any encryption.
>> >> >
>> >> > Hope this helps,
>> >> > James
>> >> >
>> >> > ---
>> >> > http://www.jamesmurty.com
>> >> >
>> >> >
>> >> > On Tue, May 26, 2009 at 9:24 PM, MarkAtHarvest
>> >> > <mark@...>wrote:
>> >> >
>> >> >>
>> >> >> My jets3t client is working fine, current I am trying to upgrade it
>> to
>> >> >> DEV
>> >> >> Pay account. I would like to ask if I can get little bit guidance
>> on
>> >> >> following
>> >> >>
>> >> >> I am using Amazon Dev Pay, then so as that an evil administrator is
>> >> not
>> >> >> able
>> >> >> to see the S3 files of customers, what exactly do I need to do
>> >> >> 1. can be to encrypt all the files stored in S3.
>> >> >> The problem I see with that approach is,
>> >> >> *. I cannot use Amazon HTTP Post to directly upload files on S3,
>> >> >> without
>> >> >> going through my server.
>> >> >> *. I cannot use createSignedGetUrl(), to create signed URLs to
>> >> expose
>> >> >> links for a temporary time, as the files need to be brought to my
>> >> sever
>> >> >> before user can download it.
>> >> >>
>> >> >> Is there a way I can solve the above two problems..
>> >> >>
>> >> >> --
>> >> >> View this message in context:
>> >> >>
>> >>
>> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23735783.html
>> >> >> Sent from the JetS3t Users mailing list archive at Nabble.com.
>> >> >>
>> >> >>
>> >> >>
>> ---------------------------------------------------------------------
>> >> >> To unsubscribe, e-mail: users-unsubscribe@...
>> >> >> For additional commands, e-mail: users-help@...
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23741804.html
>> >> Sent from the JetS3t Users mailing list archive at Nabble.com.
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscribe@...
>> >> For additional commands, e-mail: users-help@...
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23747665.html
>> Sent from the JetS3t Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@...
>> For additional commands, e-mail: users-help@...
>>
>>
>
>

--
View this message in context: http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23750391.html

Sent from the JetS3t Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Questions about security and confidentiality

2009-05-27T13:40:31Z

Thank you Jonathan , James.
I think I need to work more on IT Standards which acceptable. Will comeback on it

About dual access, I need to use S3, I am very comfortable with it, rather than going for a new solution. I think with Subcloud I can use Jets3t (need to confirm on it though)

Jonathan Harlap wrote:

Hi Mark,

First off, I'm leaving aside what I think is an obvious issue you'll need to
resolve about the two access methods conflicting (ie, users manipulating the
same data via your web app and smb simultaneously). James already mentioned
it and it's a property of your apparent design not of the AWS services.

What strikes me as particularly interesting your statement that S3 will be
mounted in an EC2 instance to provide smb access. If you're doing that,
couldn't you just as well put your data in an EBS volume, mount that to your
EC2 instance for smb access and *also* have your web app interact with the
data which now appears to exist on a local mount point? Then S3 (as an API)
is out of the picture entirely and you have what sounds like a simpler
problem to solve. I put that aside now as well, as it's more a question of
your design than anything S3 specific, which was the purpose of the
question, I believe.

S3 signed urls are not related to jets3t, although jets3t does facilitate
their creation. However, I interpret your question to say that you lose the
ability to point user's web browsers directly at a signed url, and this
would be correct if you encrypt the data in S3. If you encrypt the data you
store, then you will need to provide a client (whether it be a web app or a
thick client) that will perform the encryption/decryption for your clients.

Unfortunately, I don't think anyone outside your business can really answer
the questions you're posing beyond what has already been said. We don't
know what your application does. We can't tell you whether to encrypt your
data and, if so, how, nor how to adapt your app design accordingly.

Good luck with your project.

Cheers,
Jonathan

On Wed, May 27, 2009 at 1:58 PM, MarkAtHarvest <mark@harvestinfotech.com>wrote:

>
> Thanks Jonathan for the interesting links.
>
> My requirement is very simple, just want to build a application based on S3
> where I can store customers data, and give them a guarantee that your data
> is safe and secure, while at move and while at rest.
> 1. My JetS3t Web application will access the data for the customer as well
> as
> 2. S3 will get mounted on EC2 for a SAMBA access.
>
> Data while at move can be protected with SSL, I am more concerned with data
> while at rest, also about accessKey, SecretKey, TokenIDs.
>
> Questions which arise are
> 1. Do i need to encrypt the S3 data for such type of situations (If I
> encrypt, then i loose the JetS3t features like getSignedUrl, which creates
> a
> link to open the file directly from S3 instead of coming to our servers)
> 2. what are must must things I need to do.
>
> Thanks again for your inputs
>
>
>
>
>
>
>
> Jonathan Harlap wrote:
> >
> > Mark,
> >
> > I suspect one source of confusion for you is the claim of HIPAA
> > compliance.
> > Firstly, AWS itself is not HIPAA compliant nor does it try to be. The
> > whitepaper in question discusses some of the strategies that might be
> used
> > by AWS customers to build HIPAA compliant systems that run on the AWS
> > cloud.
> >
> > As to crypto, high security, and so on - I don't find your need very
> > clear,
> > so I can't offer any specific advice other than to say that crypto is a
> > very
> > tricky thing to do right, and a very easy thing to do wrong, so for your
> > clients' sake, do it carefully. For a fun example of how the little
> > details
> > make a difference, see
> > http://www.codinghorror.com/blog/archives/001267.htmland
> > http://www.codinghorror.com/blog/archives/001268.html
> >
> > Cheers,
> > J
> >
> > On Wed, May 27, 2009 at 8:47 AM, MarkAtHarvest
> > <mark@harvestinfotech.com>wrote:
> >
> >>
> >> Thank you for clearing my doubt!
> >> But I see white papers saying Amazon AWS is HIPAA compliant, which is a
> >> stringent security standard.
> >> The confusion I have is
> >> 1. I have a webcient build on JetS3 application,soon be adding a DevPay
> >> support.
> >> 2. I also want to sync S3 files on EC2 using something like Subcloud and
> >> provide access to the data
> >> on clients windows explorer using SAMBA.
> >>
> >> Now if I do not encrypt, and use ACL to control access on the S3, so
> that
> >> only the respective client has an access to it.
> >> Subcloud will not be able to import the data , if it does not have
> >> permission, or it will be able to import the data using secret key.
> >>
> >> As per what you say, I need to keep that secretKey really secret and
> only
> >> one Admin can know it and there should be policies to access that key.
> >>
> >> So all the S3 products out there in market who use S3, administrator do
> >> have
> >> a access to data, which might be controlled by some measures. So how can
> >> be
> >> claim high security compliance
> >>
> >> Thanks for addressing this question, I am not able to get a clean head
> >> path
> >> on it
> >>
> >>
> >>
> >> Now is that acceptable in terms of Secu
> >>
> >>
> >>
> >> James Murty-3 wrote:
> >> >
> >> > The only way to allow clients to store encrypted data in S3 in such a
> >> way
> >> > that administrators cannot read it, is to provide some kind of
> >> application
> >> > the client can run on his/her own machine to do this work.
> >> >
> >> > If you don't trust your admins, there is little point encrypting a
> >> user's
> >> > files on your own server because admins will have simply be able to
> >> access
> >> > to the data there, rather than from S3 directly. Encrypting the data
> on
> >> > your
> >> > server would protect it from the Amazon admins who maintain S3, but
> not
> >> > from
> >> > your own server admins.
> >> >
> >> > A custom client app would allow your clients to automatically encrypt
> >> data
> >> > prior to uploading, and to decrypt it when downloading. This app could
> >> > also
> >> > be designed to interact with your server component to obtain signed
> >> URLs
> >> > to
> >> > gain access to S3.
> >> >
> >> > The problem is, such an app doesn't exist as far as I know. The
> >> > combination
> >> > of JetS3t's CockpitLite and Gatekeeper applications comes close, but
> >> > CockpitLite does not do any encryption.
> >> >
> >> > Hope this helps,
> >> > James
> >> >
> >> > ---
> >> > http://www.jamesmurty.com
> >> >
> >> >
> >> > On Tue, May 26, 2009 at 9:24 PM, MarkAtHarvest
> >> > <mark@harvestinfotech.com>wrote:
> >> >
> >> >>
> >> >> My jets3t client is working fine, current I am trying to upgrade it
> to
> >> >> DEV
> >> >> Pay account. I would like to ask if I can get little bit guidance on
> >> >> following
> >> >>
> >> >> I am using Amazon Dev Pay, then so as that an evil administrator is
> >> not
> >> >> able
> >> >> to see the S3 files of customers, what exactly do I need to do
> >> >> 1. can be to encrypt all the files stored in S3.
> >> >> The problem I see with that approach is,
> >> >> *. I cannot use Amazon HTTP Post to directly upload files on S3,
> >> >> without
> >> >> going through my server.
> >> >> *. I cannot use createSignedGetUrl(), to create signed URLs to
> >> expose
> >> >> links for a temporary time, as the files need to be brought to my
> >> sever
> >> >> before user can download it.
> >> >>
> >> >> Is there a way I can solve the above two problems..
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23735783.html
> >> >> Sent from the JetS3t Users mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: users-unsubscribe@jets3t.dev.java.net
> >> >> For additional commands, e-mail: users-help@jets3t.dev.java.net
> >> >>
> >> >>
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23741804.html
> >> Sent from the JetS3t Users mailing list archive at Nabble.com.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@jets3t.dev.java.net
> >> For additional commands, e-mail: users-help@jets3t.dev.java.net
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23747665.html
> Sent from the JetS3t Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@jets3t.dev.java.net
> For additional commands, e-mail: users-help@jets3t.dev.java.net
>
>

Re: Questions about security and confidentiality

2009-05-27T11:49:27Z

Hi Mark,

First off, I'm leaving aside what I think is an obvious issue you'll need to resolve about the two access methods conflicting (ie, users manipulating the same data via your web app and smb simultaneously). James already mentioned it and it's a property of your apparent design not of the AWS services.

What strikes me as particularly interesting your statement that S3 will be mounted in an EC2 instance to provide smb access. If you're doing that, couldn't you just as well put your data in an EBS volume, mount that to your EC2 instance for smb access and *also* have your web app interact with the data which now appears to exist on a local mount point? Then S3 (as an API) is out of the picture entirely and you have what sounds like a simpler problem to solve. I put that aside now as well, as it's more a question of your design than anything S3 specific, which was the purpose of the question, I believe.

S3 signed urls are not related to jets3t, although jets3t does facilitate their creation. However, I interpret your question to say that you lose the ability to point user's web browsers directly at a signed url, and this would be correct if you encrypt the data in S3. If you encrypt the data you store, then you will need to provide a client (whether it be a web app or a thick client) that will perform the encryption/decryption for your clients.

Unfortunately, I don't think anyone outside your business can really answer the questions you're posing beyond what has already been said. We don't know what your application does. We can't tell you whether to encrypt your data and, if so, how, nor how to adapt your app design accordingly.

Good luck with your project.

Cheers,
Jonathan

On Wed, May 27, 2009 at 1:58 PM, MarkAtHarvest <mark@...> wrote:

Thanks Jonathan for the interesting links.

My requirement is very simple, just want to build a application based on S3
where I can store customers data, and give them a guarantee that your data
is safe and secure, while at move and while at rest.
1. My JetS3t Web application will access the data for the customer as well
as
2. S3 will get mounted on EC2 for a SAMBA access.

Data while at move can be protected with SSL, I am more concerned with data
while at rest, also about accessKey, SecretKey, TokenIDs.

Questions which arise are
1. Do i need to encrypt the S3 data for such type of situations (If I
encrypt, then i loose the JetS3t features like getSignedUrl, which creates a
link to open the file directly from S3 instead of coming to our servers)
2. what are must must things I need to do.

Thanks again for your inputs

Jonathan Harlap wrote:
>
> Mark,
>
> I suspect one source of confusion for you is the claim of HIPAA
> compliance.
> Firstly, AWS itself is not HIPAA compliant nor does it try to be. The
> whitepaper in question discusses some of the strategies that might be used
> by AWS customers to build HIPAA compliant systems that run on the AWS
> cloud.
>
> As to crypto, high security, and so on - I don't find your need very
> clear,
> so I can't offer any specific advice other than to say that crypto is a
> very
> tricky thing to do right, and a very easy thing to do wrong, so for your
> clients' sake, do it carefully. For a fun example of how the little
> details
> make a difference, see
> http://www.codinghorror.com/blog/archives/001267.htmland
> http://www.codinghorror.com/blog/archives/001268.html
>
> Cheers,
> J
>
> On Wed, May 27, 2009 at 8:47 AM, MarkAtHarvest
> <mark@...>wrote:
>
>>
>> Thank you for clearing my doubt!
>> But I see white papers saying Amazon AWS is HIPAA compliant, which is a
>> stringent security standard.
>> The confusion I have is
>> 1. I have a webcient build on JetS3 application,soon be adding a DevPay
>> support.
>> 2. I also want to sync S3 files on EC2 using something like Subcloud and
>> provide access to the data
>> on clients windows explorer using SAMBA.
>>
>> Now if I do not encrypt, and use ACL to control access on the S3, so that
>> only the respective client has an access to it.
>> Subcloud will not be able to import the data , if it does not have
>> permission, or it will be able to import the data using secret key.
>>
>> As per what you say, I need to keep that secretKey really secret and only
>> one Admin can know it and there should be policies to access that key.
>>
>> So all the S3 products out there in market who use S3, administrator do
>> have
>> a access to data, which might be controlled by some measures. So how can
>> be
>> claim high security compliance
>>
>> Thanks for addressing this question, I am not able to get a clean head
>> path
>> on it
>>
>>
>>
>> Now is that acceptable in terms of Secu
>>
>>
>>
>> James Murty-3 wrote:
>> >
>> > The only way to allow clients to store encrypted data in S3 in such a
>> way
>> > that administrators cannot read it, is to provide some kind of
>> application
>> > the client can run on his/her own machine to do this work.
>> >
>> > If you don't trust your admins, there is little point encrypting a
>> user's
>> > files on your own server because admins will have simply be able to
>> access
>> > to the data there, rather than from S3 directly. Encrypting the data on
>> > your
>> > server would protect it from the Amazon admins who maintain S3, but not
>> > from
>> > your own server admins.
>> >
>> > A custom client app would allow your clients to automatically encrypt
>> data
>> > prior to uploading, and to decrypt it when downloading. This app could
>> > also
>> > be designed to interact with your server component to obtain signed
>> URLs
>> > to
>> > gain access to S3.
>> >
>> > The problem is, such an app doesn't exist as far as I know. The
>> > combination
>> > of JetS3t's CockpitLite and Gatekeeper applications comes close, but
>> > CockpitLite does not do any encryption.
>> >
>> > Hope this helps,
>> > James
>> >
>> > ---
>> > http://www.jamesmurty.com
>> >
>> >
>> > On Tue, May 26, 2009 at 9:24 PM, MarkAtHarvest
>> > <mark@...>wrote:
>> >
>> >>
>> >> My jets3t client is working fine, current I am trying to upgrade it to
>> >> DEV
>> >> Pay account. I would like to ask if I can get little bit guidance on
>> >> following
>> >>
>> >> I am using Amazon Dev Pay, then so as that an evil administrator is
>> not
>> >> able
>> >> to see the S3 files of customers, what exactly do I need to do
>> >> 1. can be to encrypt all the files stored in S3.
>> >> The problem I see with that approach is,
>> >> *. I cannot use Amazon HTTP Post to directly upload files on S3,
>> >> without
>> >> going through my server.
>> >> *. I cannot use createSignedGetUrl(), to create signed URLs to
>> expose
>> >> links for a temporary time, as the files need to be brought to my
>> sever
>> >> before user can download it.
>> >>
>> >> Is there a way I can solve the above two problems..
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23735783.html
>> >> Sent from the JetS3t Users mailing list archive at Nabble.com.
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscribe@...
>> >> For additional commands, e-mail: users-help@...
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23741804.html
>> Sent from the JetS3t Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@...
>> For additional commands, e-mail: users-help@...
>>
>>
>
>

--
View this message in context: http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23747665.html

Sent from the JetS3t Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Questions about security and confidentiality

2009-05-27T10:57:59Z

Thanks Jonathan for the interesting links.

My requirement is very simple, just want to build a application based on S3 where I can store customers data, and give them a guarantee that your data is safe and secure, while at move and while at rest.
1. My JetS3t Web application will access the data for the customer as well as
2. S3 will get mounted on EC2 for a SAMBA access.

Data while at move can be protected with SSL, I am more concerned with data while at rest, also about accessKey, SecretKey, TokenIDs.

Questions which arise are
1. Do i need to encrypt the S3 data for such type of situations (If I encrypt, then i loose the JetS3t features like getSignedUrl, which creates a link to open the file directly from S3 instead of coming to our servers)
2. what are must must things I need to do.

Thanks again for your inputs

Jonathan Harlap wrote:

Mark,

I suspect one source of confusion for you is the claim of HIPAA compliance.
Firstly, AWS itself is not HIPAA compliant nor does it try to be. The
whitepaper in question discusses some of the strategies that might be used
by AWS customers to build HIPAA compliant systems that run on the AWS cloud.

As to crypto, high security, and so on - I don't find your need very clear,
so I can't offer any specific advice other than to say that crypto is a very
tricky thing to do right, and a very easy thing to do wrong, so for your
clients' sake, do it carefully. For a fun example of how the little details
make a difference, see http://www.codinghorror.com/blog/archives/001267.htmland
http://www.codinghorror.com/blog/archives/001268.html

Cheers,
J

On Wed, May 27, 2009 at 8:47 AM, MarkAtHarvest <mark@harvestinfotech.com>wrote:

>
> Thank you for clearing my doubt!
> But I see white papers saying Amazon AWS is HIPAA compliant, which is a
> stringent security standard.
> The confusion I have is
> 1. I have a webcient build on JetS3 application,soon be adding a DevPay
> support.
> 2. I also want to sync S3 files on EC2 using something like Subcloud and
> provide access to the data
> on clients windows explorer using SAMBA.
>
> Now if I do not encrypt, and use ACL to control access on the S3, so that
> only the respective client has an access to it.
> Subcloud will not be able to import the data , if it does not have
> permission, or it will be able to import the data using secret key.
>
> As per what you say, I need to keep that secretKey really secret and only
> one Admin can know it and there should be policies to access that key.
>
> So all the S3 products out there in market who use S3, administrator do
> have
> a access to data, which might be controlled by some measures. So how can be
> claim high security compliance
>
> Thanks for addressing this question, I am not able to get a clean head path
> on it
>
>
>
> Now is that acceptable in terms of Secu
>
>
>
> James Murty-3 wrote:
> >
> > The only way to allow clients to store encrypted data in S3 in such a way
> > that administrators cannot read it, is to provide some kind of
> application
> > the client can run on his/her own machine to do this work.
> >
> > If you don't trust your admins, there is little point encrypting a user's
> > files on your own server because admins will have simply be able to
> access
> > to the data there, rather than from S3 directly. Encrypting the data on
> > your
> > server would protect it from the Amazon admins who maintain S3, but not
> > from
> > your own server admins.
> >
> > A custom client app would allow your clients to automatically encrypt
> data
> > prior to uploading, and to decrypt it when downloading. This app could
> > also
> > be designed to interact with your server component to obtain signed URLs
> > to
> > gain access to S3.
> >
> > The problem is, such an app doesn't exist as far as I know. The
> > combination
> > of JetS3t's CockpitLite and Gatekeeper applications comes close, but
> > CockpitLite does not do any encryption.
> >
> > Hope this helps,
> > James
> >
> > ---
> > http://www.jamesmurty.com
> >
> >
> > On Tue, May 26, 2009 at 9:24 PM, MarkAtHarvest
> > <mark@harvestinfotech.com>wrote:
> >
> >>
> >> My jets3t client is working fine, current I am trying to upgrade it to
> >> DEV
> >> Pay account. I would like to ask if I can get little bit guidance on
> >> following
> >>
> >> I am using Amazon Dev Pay, then so as that an evil administrator is not
> >> able
> >> to see the S3 files of customers, what exactly do I need to do
> >> 1. can be to encrypt all the files stored in S3.
> >> The problem I see with that approach is,
> >> *. I cannot use Amazon HTTP Post to directly upload files on S3,
> >> without
> >> going through my server.
> >> *. I cannot use createSignedGetUrl(), to create signed URLs to expose
> >> links for a temporary time, as the files need to be brought to my sever
> >> before user can download it.
> >>
> >> Is there a way I can solve the above two problems..
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23735783.html
> >> Sent from the JetS3t Users mailing list archive at Nabble.com.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@jets3t.dev.java.net
> >> For additional commands, e-mail: users-help@jets3t.dev.java.net
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23741804.html
> Sent from the JetS3t Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@jets3t.dev.java.net
> For additional commands, e-mail: users-help@jets3t.dev.java.net
>
>

Re: Questions about security and confidentiality

2009-05-27T09:28:32Z

As John points out, the level of HIPAA compliance you can achieve using Amazon's services depends very much on how you use the services. You should read Amazon's whitepaper carefully to understand the recommended techniques.

Obviously, for a service to be useful someone must have access to the data somewhere. In many cases that will mean that admins have access to data. Whether or not this situation meets "high security compliance" depends on the kind of compliance being claimed, and the policies that control the admins access.

If you want to provide a certain level of security for your customers, you need to have a very clear policy of who can access data, when and why. Compliance is simply a measure of whether you keep to the stated policy.

I am not sure what you are trying to achieve using both ACLs and Subcloud. Do your clients have direct access to their data in S3? If not, then there is no need for ACL controls as the Subcloud server should be the only entity accessing the storage. If your clients do have direct access to S3, I'm not sure how well this will work with Subcloud because multiple access points will be "fighting" over shared storage space. You will need to talk this through with the Subcloud vendor.

James

On Wed, May 27, 2009 at 6:17 AM, Jon Harlap <jharlap@...> wrote:

Mark,

I suspect one source of confusion for you is the claim of HIPAA compliance. Firstly, AWS itself is not HIPAA compliant nor does it try to be. The whitepaper in question discusses some of the strategies that might be used by AWS customers to build HIPAA compliant systems that run on the AWS cloud.

As to crypto, high security, and so on - I don't find your need very clear, so I can't offer any specific advice other than to say that crypto is a very tricky thing to do right, and a very easy thing to do wrong, so for your clients' sake, do it carefully. For a fun example of how the little details make a difference, see http://www.codinghorror.com/blog/archives/001267.html and http://www.codinghorror.com/blog/archives/001268.html

Cheers,
J

On Wed, May 27, 2009 at 8:47 AM, MarkAtHarvest <mark@...> wrote:

Thank you for clearing my doubt!
But I see white papers saying Amazon AWS is HIPAA compliant, which is a
stringent security standard.
The confusion I have is
1. I have a webcient build on JetS3 application,soon be adding a DevPay
support.
2. I also want to sync S3 files on EC2 using something like Subcloud and
provide access to the data
on clients windows explorer using SAMBA.

Now if I do not encrypt, and use ACL to control access on the S3, so that
only the respective client has an access to it.
Subcloud will not be able to import the data , if it does not have
permission, or it will be able to import the data using secret key.

As per what you say, I need to keep that secretKey really secret and only
one Admin can know it and there should be policies to access that key.

So all the S3 products out there in market who use S3, administrator do have
a access to data, which might be controlled by some measures. So how can be
claim high security compliance

Thanks for addressing this question, I am not able to get a clean head path
on it

Now is that acceptable in terms of Secu

James Murty-3 wrote:
>
> The only way to allow clients to store encrypted data in S3 in such a way
> that administrators cannot read it, is to provide some kind of application
> the client can run on his/her own machine to do this work.
>
> If you don't trust your admins, there is little point encrypting a user's
> files on your own server because admins will have simply be able to access
> to the data there, rather than from S3 directly. Encrypting the data on
> your
> server would protect it from the Amazon admins who maintain S3, but not
> from
> your own server admins.
>
> A custom client app would allow your clients to automatically encrypt data
> prior to uploading, and to decrypt it when downloading. This app could
> also
> be designed to interact with your server component to obtain signed URLs
> to
> gain access to S3.
>
> The problem is, such an app doesn't exist as far as I know. The
> combination
> of JetS3t's CockpitLite and Gatekeeper applications comes close, but
> CockpitLite does not do any encryption.
>
> Hope this helps,
> James
>
> ---
> http://www.jamesmurty.com
>
>
> On Tue, May 26, 2009 at 9:24 PM, MarkAtHarvest
> <mark@...>wrote:
>
>>
>> My jets3t client is working fine, current I am trying to upgrade it to
>> DEV
>> Pay account. I would like to ask if I can get little bit guidance on
>> following
>>
>> I am using Amazon Dev Pay, then so as that an evil administrator is not
>> able
>> to see the S3 files of customers, what exactly do I need to do
>> 1. can be to encrypt all the files stored in S3.
>> The problem I see with that approach is,
>> *. I cannot use Amazon HTTP Post to directly upload files on S3,
>> without
>> going through my server.
>> *. I cannot use createSignedGetUrl(), to create signed URLs to expose
>> links for a temporary time, as the files need to be brought to my sever
>> before user can download it.
>>
>> Is there a way I can solve the above two problems..
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23735783.html
>> Sent from the JetS3t Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@...
>> For additional commands, e-mail: users-help@...
>>
>>
>
>

--
View this message in context: http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23741804.html

Sent from the JetS3t Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Questions about security and confidentiality

2009-05-27T06:17:27Z

Mark,

I suspect one source of confusion for you is the claim of HIPAA compliance. Firstly, AWS itself is not HIPAA compliant nor does it try to be. The whitepaper in question discusses some of the strategies that might be used by AWS customers to build HIPAA compliant systems that run on the AWS cloud.

As to crypto, high security, and so on - I don't find your need very clear, so I can't offer any specific advice other than to say that crypto is a very tricky thing to do right, and a very easy thing to do wrong, so for your clients' sake, do it carefully. For a fun example of how the little details make a difference, see http://www.codinghorror.com/blog/archives/001267.html and http://www.codinghorror.com/blog/archives/001268.html

Cheers,
J

On Wed, May 27, 2009 at 8:47 AM, MarkAtHarvest <mark@...> wrote:

Thank you for clearing my doubt!
But I see white papers saying Amazon AWS is HIPAA compliant, which is a
stringent security standard.
The confusion I have is
1. I have a webcient build on JetS3 application,soon be adding a DevPay
support.
2. I also want to sync S3 files on EC2 using something like Subcloud and
provide access to the data
on clients windows explorer using SAMBA.

Now if I do not encrypt, and use ACL to control access on the S3, so that
only the respective client has an access to it.
Subcloud will not be able to import the data , if it does not have
permission, or it will be able to import the data using secret key.

As per what you say, I need to keep that secretKey really secret and only
one Admin can know it and there should be policies to access that key.

So all the S3 products out there in market who use S3, administrator do have
a access to data, which might be controlled by some measures. So how can be
claim high security compliance

Thanks for addressing this question, I am not able to get a clean head path
on it

Now is that acceptable in terms of Secu

James Murty-3 wrote:
>
> The only way to allow clients to store encrypted data in S3 in such a way
> that administrators cannot read it, is to provide some kind of application
> the client can run on his/her own machine to do this work.
>
> If you don't trust your admins, there is little point encrypting a user's
> files on your own server because admins will have simply be able to access
> to the data there, rather than from S3 directly. Encrypting the data on
> your
> server would protect it from the Amazon admins who maintain S3, but not
> from
> your own server admins.
>
> A custom client app would allow your clients to automatically encrypt data
> prior to uploading, and to decrypt it when downloading. This app could
> also
> be designed to interact with your server component to obtain signed URLs
> to
> gain access to S3.
>
> The problem is, such an app doesn't exist as far as I know. The
> combination
> of JetS3t's CockpitLite and Gatekeeper applications comes close, but
> CockpitLite does not do any encryption.
>
> Hope this helps,
> James
>
> ---
> http://www.jamesmurty.com
>
>
> On Tue, May 26, 2009 at 9:24 PM, MarkAtHarvest
> <mark@...>wrote:
>
>>
>> My jets3t client is working fine, current I am trying to upgrade it to
>> DEV
>> Pay account. I would like to ask if I can get little bit guidance on
>> following
>>
>> I am using Amazon Dev Pay, then so as that an evil administrator is not
>> able
>> to see the S3 files of customers, what exactly do I need to do
>> 1. can be to encrypt all the files stored in S3.
>> The problem I see with that approach is,
>> *. I cannot use Amazon HTTP Post to directly upload files on S3,
>> without
>> going through my server.
>> *. I cannot use createSignedGetUrl(), to create signed URLs to expose
>> links for a temporary time, as the files need to be brought to my sever
>> before user can download it.
>>
>> Is there a way I can solve the above two problems..
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23735783.html
>> Sent from the JetS3t Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@...
>> For additional commands, e-mail: users-help@...
>>
>>
>
>

--
View this message in context: http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23741804.html

Sent from the JetS3t Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: Questions about security and confidentiality

2009-05-27T05:46:58Z

Thank you for clearing my doubt!
But I see white papers saying Amazon AWS is HIPAA compliant, which is a stringent security standard.
The confusion I have is
1. I have a webcient build on JetS3 application,soon be adding a DevPay support.
2. I also want to sync S3 files on EC2 using something like Subcloud and provide access to the data
on clients windows explorer using SAMBA.

Now if I do not encrypt, and use ACL to control access on the S3, so that only the respective client has an access to it.
Subcloud will not be able to import the data , if it does not have permission, or it will be able to import the data using secret key.

As per what you say, I need to keep that secretKey really secret and only one Admin can know it and there should be policies to access that key.

So all the S3 products out there in market who use S3, administrator do have a access to data, which might be controlled by some measures. So how can be claim high security compliance

Thanks for addressing this question, I am not able to get a clean head path on it

Now is that acceptable in terms of Secu

James Murty-3 wrote:

The only way to allow clients to store encrypted data in S3 in such a way
that administrators cannot read it, is to provide some kind of application
the client can run on his/her own machine to do this work.

If you don't trust your admins, there is little point encrypting a user's
files on your own server because admins will have simply be able to access
to the data there, rather than from S3 directly. Encrypting the data on your
server would protect it from the Amazon admins who maintain S3, but not from
your own server admins.

A custom client app would allow your clients to automatically encrypt data
prior to uploading, and to decrypt it when downloading. This app could also
be designed to interact with your server component to obtain signed URLs to
gain access to S3.

The problem is, such an app doesn't exist as far as I know. The combination
of JetS3t's CockpitLite and Gatekeeper applications comes close, but
CockpitLite does not do any encryption.

Hope this helps,
James

---
http://www.jamesmurty.com

On Tue, May 26, 2009 at 9:24 PM, MarkAtHarvest <mark@harvestinfotech.com>wrote:

>
> My jets3t client is working fine, current I am trying to upgrade it to DEV
> Pay account. I would like to ask if I can get little bit guidance on
> following
>
> I am using Amazon Dev Pay, then so as that an evil administrator is not
> able
> to see the S3 files of customers, what exactly do I need to do
> 1. can be to encrypt all the files stored in S3.
> The problem I see with that approach is,
> *. I cannot use Amazon HTTP Post to directly upload files on S3, without
> going through my server.
> *. I cannot use createSignedGetUrl(), to create signed URLs to expose
> links for a temporary time, as the files need to be brought to my sever
> before user can download it.
>
> Is there a way I can solve the above two problems..
>
> --
> View this message in context:
> http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23735783.html
> Sent from the JetS3t Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@jets3t.dev.java.net
> For additional commands, e-mail: users-help@jets3t.dev.java.net
>
>

Re: Questions about security and confidentiality

2009-05-26T22:21:39Z

The only way to allow clients to store encrypted data in S3 in such a way that administrators cannot read it, is to provide some kind of application the client can run on his/her own machine to do this work.

If you don't trust your admins, there is little point encrypting a user's files on your own server because admins will have simply be able to access to the data there, rather than from S3 directly. Encrypting the data on your server would protect it from the Amazon admins who maintain S3, but not from your own server admins.

A custom client app would allow your clients to automatically encrypt data prior to uploading, and to decrypt it when downloading. This app could also be designed to interact with your server component to obtain signed URLs to gain access to S3.

The problem is, such an app doesn't exist as far as I know. The combination of JetS3t's CockpitLite and Gatekeeper applications comes close, but CockpitLite does not do any encryption.

Hope this helps,
James

---
http://www.jamesmurty.com

On Tue, May 26, 2009 at 9:24 PM, MarkAtHarvest <mark@...> wrote:

My jets3t client is working fine, current I am trying to upgrade it to DEV
Pay account. I would like to ask if I can get little bit guidance on
following

I am using Amazon Dev Pay, then so as that an evil administrator is not able
to see the S3 files of customers, what exactly do I need to do
1. can be to encrypt all the files stored in S3.
The problem I see with that approach is,
*. I cannot use Amazon HTTP Post to directly upload files on S3, without
going through my server.
*. I cannot use createSignedGetUrl(), to create signed URLs to expose
links for a temporary time, as the files need to be brought to my sever
before user can download it.

Is there a way I can solve the above two problems..

--
View this message in context: http://www.nabble.com/Questions-about-security-and-confidentiality-tp23735783p23735783.html
Sent from the JetS3t Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Questions about security and confidentiality

2009-05-26T21:24:51Z

My jets3t client is working fine, current I am trying to upgrade it to DEV Pay account. I would like to ask if I can get little bit guidance on following

I am using Amazon Dev Pay, then so as that an evil administrator is not able to see the S3 files of customers, what exactly do I need to do
1. can be to encrypt all the files stored in S3.
The problem I see with that approach is,
*. I cannot use Amazon HTTP Post to directly upload files on S3, without going through my server.
*. I cannot use createSignedGetUrl(), to create signed URLs to expose links for a temporary time, as the files need to be brought to my sever before user can download it.

Is there a way I can solve the above two problems..

Re: setting object ACL

2009-04-07T10:03:59Z

See my response in the JetS3t Users Google group here:

http://groups.google.com/group/jets3t-users/browse_thread/thread/3a97e3ca34724223?hl=en

James

On Tue, Apr 7, 2009 at 9:33 AM, Lia Lotz <lia.lotz@...> wrote:

Hello,

Can anyone describe how to handle in the following situation:

During the creation of a S3Object,
how do I set a S3Object's ACL to public read for all users?
(the object has to be uploaded to an existing bucket)

Thanks, in advance

Casimir

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

setting object ACL

2009-04-07T09:33:16Z

Hello,

Can anyone describe how to handle in the following situation:

During the creation of a S3Object,
how do I set a S3Object's ACL to public read for all users?
(the object has to be uploaded to an existing bucket)

Thanks, in advance

Casimir

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

Re: java.lang.ExceptionInInitializerError in creating AWSCredentials on Ubuntu

2009-03-18T08:23:31Z

There is a problem in your Java installation.

How are you running this JetS3t code? Are you running it stand-alone, or in an application server like Tomcat? And what was the initial problem that prompted you to install sunpkcs11.jar?

From a quick Google search it looks like you may have downloaded and installed the wrong version of the security libraries. With any new version of Java you should not need to add security libraries.

Look at the replies to this post:
http://www.mail-archive.com/users@.../msg42161.html

For this person, the issue was caused by:
1) Adding the wrong version of JSSE (which caused the Normalizer error)
2) Launching Tomcat from within the Eclispe IDE, which caused the original error for which installing the new JSSE library was the wrong solution.

Hope this helps,
James

---
http://www.jamesmurty.com

On Wed, Mar 18, 2009 at 6:26 AM, Kenji Imasaki <kimasaki@...> wrote:

Hi,

I am trying to run simple Jets3t upload example on Utuntu
8.1/java-6-sun-1.6.0.10 like the followings:

public class jetS3tTest {
@Test
public void upload() {
String awsAccessKey = "XXXX";
String awsSecretKey = "YYYY";
AWSCredentials awsCredentials =
new AWSCredentials(awsAccessKey, awsSecretKey); << ERRROR here

....
}
}

Then, I got the following error at creating of AWSCredentials. I did
some google search but I found I have to install sunpkcs. So, I
installed sunpkcs11.jar. But it did not fix the problem.

Could you let me know how to fix it?

Thanks.

FAILED: upload
java.lang.ExceptionInInitializerError
at sun.text.normalizer.NormalizerBase.decompose(NormalizerBase.java:707)
at sun.text.normalizer.NormalizerBase$NFKDMode.normalize(NormalizerBase.java:348)
at sun.text.normalizer.NormalizerBase.normalize(NormalizerBase.java:1592)
at sun.text.normalizer.NormalizerBase.normalize(NormalizerBase.java:1573)
at java.text.Normalizer.normalize(Normalizer.java:146)
at sun.security.x509.AVA.toRFC2253CanonicalString(AVA.java:986)
at sun.security.x509.RDN.toRFC2253StringInternal(RDN.java:430)
at sun.security.x509.RDN.toRFC2253String(RDN.java:409)
at sun.security.x509.X500Name.getRFC2253CanonicalName(X500Name.java:714)
at sun.security.x509.X500Name.equals(X500Name.java:400)
at sun.security.pkcs.PKCS7.getCertificate(PKCS7.java:609)
at sun.security.pkcs.SignerInfo.getCertificate(SignerInfo.java:202)
at sun.security.pkcs.SignerInfo.verify(SignerInfo.java:328)
at sun.security.pkcs.PKCS7.verify(PKCS7.java:494)
at sun.security.pkcs.PKCS7.verify(PKCS7.java:511)
at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:199)
at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:176)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:277)
at java.util.jar.JarVerifier.update(JarVerifier.java:188)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:321)
at java.util.jar.JarFile.getInputStream(JarFile.java:386)
at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:689)
at sun.misc.Resource.cachedInputStream(Resource.java:59)
at sun.misc.Resource.getByteBuffer(Resource.java:154)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:249)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Caused by: java.lang.RuntimeException: could not locate data
at sun.text.normalizer.NormalizerImpl.<clinit>(NormalizerImpl.java:44)
... 56 more
... Removed 22 stack frames

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

java.lang.ExceptionInInitializerError in creating AWSCredentials on Ubuntu

2009-03-18T06:26:21Z

Hi,

I am trying to run simple Jets3t upload example on Utuntu
8.1/java-6-sun-1.6.0.10 like the followings:

public class jetS3tTest {
@Test
public void upload() {
String awsAccessKey = "XXXX";
String awsSecretKey = "YYYY";
AWSCredentials awsCredentials =
new AWSCredentials(awsAccessKey, awsSecretKey); << ERRROR here

....
}
}

Then, I got the following error at creating of AWSCredentials. I did
some google search but I found I have to install sunpkcs. So, I
installed sunpkcs11.jar. But it did not fix the problem.

Could you let me know how to fix it?

Thanks.

FAILED: upload
java.lang.ExceptionInInitializerError
at sun.text.normalizer.NormalizerBase.decompose(NormalizerBase.java:707)
at sun.text.normalizer.NormalizerBase$NFKDMode.normalize(NormalizerBase.java:348)
at sun.text.normalizer.NormalizerBase.normalize(NormalizerBase.java:1592)
at sun.text.normalizer.NormalizerBase.normalize(NormalizerBase.java:1573)
at java.text.Normalizer.normalize(Normalizer.java:146)
at sun.security.x509.AVA.toRFC2253CanonicalString(AVA.java:986)
at sun.security.x509.RDN.toRFC2253StringInternal(RDN.java:430)
at sun.security.x509.RDN.toRFC2253String(RDN.java:409)
at sun.security.x509.X500Name.getRFC2253CanonicalName(X500Name.java:714)
at sun.security.x509.X500Name.equals(X500Name.java:400)
at sun.security.pkcs.PKCS7.getCertificate(PKCS7.java:609)
at sun.security.pkcs.SignerInfo.getCertificate(SignerInfo.java:202)
at sun.security.pkcs.SignerInfo.verify(SignerInfo.java:328)
at sun.security.pkcs.PKCS7.verify(PKCS7.java:494)
at sun.security.pkcs.PKCS7.verify(PKCS7.java:511)
at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:199)
at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:176)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:277)
at java.util.jar.JarVerifier.update(JarVerifier.java:188)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:321)
at java.util.jar.JarFile.getInputStream(JarFile.java:386)
at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:689)
at sun.misc.Resource.cachedInputStream(Resource.java:59)
at sun.misc.Resource.getByteBuffer(Resource.java:154)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:249)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Caused by: java.lang.RuntimeException: could not locate data
at sun.text.normalizer.NormalizerImpl.<clinit>(NormalizerImpl.java:44)
... 56 more
... Removed 22 stack frames

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...