|
Synchronize didn't error but created a single, empty file
Hello,
I was trying out jets3t's Synchronize program to backup a folder structure to amazon's S3. I ran the batch file, and it appeared to run correctly - it says it copied 61,544 files, it took several hours, and no errors were spit out. However, when I use the Firefox plugin "S3 Firefox" (so that I can confirm the backup did run correctly), I see it made a single file that's 0 bytes. Now I'm trying to figure out what happened.
Here's how I ran jets3t, on a Windows 2003 server:
synchronize.bat -b UP "portentint/homerbak" e:\inetpub\wwwroot
portentint being my bucket on S3, and I wanted a directory inside that called homerbak. When I look in S3 Firefox, I see homerbak folder inside my bucket (so that worked), and then inside there I see "wwwroot" which looks like a file, and has a file size of 0.
What did I do wrong, and how could it have taken several hours copying 4-5GB but I only see this empty file?
Any help/comments are appreciated :)
|
|
Re: Synchronize didn't error but created a single, empty file
Hi Branden, I'm afraid this problem is due to an incompatibility between the way JetS3t represents folders in S3 and the way the Firefox plugin does so. All your files will be present in S3 and completely accessible to JetS3t programs, but because JetS3t and Elasticfox use different methods to indicate when an S3 object really represents a folder you will need to give Elasticfox a little help.
Instead of relying on Elasticfox recognizing folders, you will need to type in the folder path into the object path field with an added slash character at the end, e.g. "myfolder/". When you add the slash character, you should get a listing of files inside that "folder".
As an alternative, you can use the JetS3t Cockpit application which will list all the objects in a bucket. However, Cockpit does not offer the same kind of folder-specific view that Elasticfox does. It is unfortunate that there is not a standard way to represent folders stored in S3 and that different tools use different techniques. The underlying issue is that there is really no such thing as a folder object in S3, so we all use different tricks when storing folder hierarchies so we can recognize which objects represent folders and which represent files. All of which causes headaches for users.
I hope this helps, James --- http://www.jamesmurty.com
On Wed, Jul 8, 2009 at 4:12 PM, Branden Makana <branden.makana@...> wrote:
Hello,
I was trying out jets3t's Synchronize program to backup a folder structure to amazon's S3. I ran the batch file, and it appeared to run correctly - it says it copied 61,544 files, it took several hours, and no errors were spit out. However, when I use the Firefox plugin "S3 Firefox" (so that I can confirm the backup did run correctly), I see it made a single file that's 0 bytes. Now I'm trying to figure out what happened.
Here's how I ran jets3t, on a Windows 2003 server:
synchronize.bat -b UP "portentint/homerbak" e:\inetpub\wwwroot
portentint being my bucket on S3, and I wanted a directory inside that called homerbak. When I look in S3 Firefox, I see homerbak folder inside my bucket (so that worked), and then inside there I see "wwwroot" which looks like a file, and has a file size of 0.
What did I do wrong, and how could it have taken several hours copying 4-5GB but I only see this empty file?
Any help/comments are appreciated :)
|
|
Re: Synchronize didn't error but created a single, empty file
Oops, substitute "S3Fox" wherever I said "Elasticfox" in my last message. Elasticfox is the EC2 Firefox add-on, not the S3 one. James On Wed, Jul 8, 2009 at 5:03 PM, James Murty <james@...> wrote:
Hi Branden,
I'm afraid this problem is due to an incompatibility between the way JetS3t represents folders in S3 and the way the Firefox plugin does so. All your files will be present in S3 and completely accessible to JetS3t programs, but because JetS3t and Elasticfox use different methods to indicate when an S3 object really represents a folder you will need to give Elasticfox a little help.
Instead of relying on Elasticfox recognizing folders, you will need to type in the folder path into the object path field with an added slash character at the end, e.g. "myfolder/". When you add the slash character, you should get a listing of files inside that "folder".
As an alternative, you can use the JetS3t Cockpit application which will list all the objects in a bucket. However, Cockpit does not offer the same kind of folder-specific view that Elasticfox does.
It is unfortunate that there is not a standard way to represent folders stored in S3 and that different tools use different techniques. The underlying issue is that there is really no such thing as a folder object in S3, so we all use different tricks when storing folder hierarchies so we can recognize which objects represent folders and which represent files. All of which causes headaches for users.
I hope this helps,
James
---
http://www.jamesmurty.com
On Wed, Jul 8, 2009 at 4:12 PM, Branden Makana <branden.makana@...> wrote:
Hello,
I was trying out jets3t's Synchronize program to backup a folder structure to amazon's S3. I ran the batch file, and it appeared to run correctly - it says it copied 61,544 files, it took several hours, and no errors were spit out. However, when I use the Firefox plugin "S3 Firefox" (so that I can confirm the backup did run correctly), I see it made a single file that's 0 bytes. Now I'm trying to figure out what happened.
Here's how I ran jets3t, on a Windows 2003 server:
synchronize.bat -b UP "portentint/homerbak" e:\inetpub\wwwroot
portentint being my bucket on S3, and I wanted a directory inside that called homerbak. When I look in S3 Firefox, I see homerbak folder inside my bucket (so that worked), and then inside there I see "wwwroot" which looks like a file, and has a file size of 0.
What did I do wrong, and how could it have taken several hours copying 4-5GB but I only see this empty file?
Any help/comments are appreciated :)
|
|
Re: Synchronize didn't error but created a single, empty file
Thanks James - that makes sense then. Now my concern is - do I need to make sure any files put anywhere in that bucket are done so by jets3t? I was planning on using one bucket for all my files, and then "folders" inside that bucket for each server I planned on backing up. It sounds like if I need to make sure to use jets3t or else various programs might step on each other in how they do folders, is that correct?
On Wed, Jul 8, 2009 at 5:03 PM, James Murty <james@...> wrote:
Hi Branden,
I'm afraid this problem is due to an incompatibility between the way JetS3t represents folders in S3 and the way the Firefox plugin does so. All your files will be present in S3 and completely accessible to JetS3t programs, but because JetS3t and Elasticfox use different methods to indicate when an S3 object really represents a folder you will need to give Elasticfox a little help.
Instead of relying on Elasticfox recognizing folders, you will need to type in the folder path into the object path field with an added slash character at the end, e.g. "myfolder/". When you add the slash character, you should get a listing of files inside that "folder".
As an alternative, you can use the JetS3t Cockpit application which will list all the objects in a bucket. However, Cockpit does not offer the same kind of folder-specific view that Elasticfox does.
It is unfortunate that there is not a standard way to represent folders stored in S3 and that different tools use different techniques. The underlying issue is that there is really no such thing as a folder object in S3, so we all use different tricks when storing folder hierarchies so we can recognize which objects represent folders and which represent files. All of which causes headaches for users.
I hope this helps,
James
---
http://www.jamesmurty.com
On Wed, Jul 8, 2009 at 4:12 PM, Branden Makana <branden.makana@...> wrote:
Hello,
I was trying out jets3t's Synchronize program to backup a folder structure to amazon's S3. I ran the batch file, and it appeared to run correctly - it says it copied 61,544 files, it took several hours, and no errors were spit out. However, when I use the Firefox plugin "S3 Firefox" (so that I can confirm the backup did run correctly), I see it made a single file that's 0 bytes. Now I'm trying to figure out what happened.
Here's how I ran jets3t, on a Windows 2003 server:
synchronize.bat -b UP "portentint/homerbak" e:\inetpub\wwwroot
portentint being my bucket on S3, and I wanted a directory inside that called homerbak. When I look in S3 Firefox, I see homerbak folder inside my bucket (so that worked), and then inside there I see "wwwroot" which looks like a file, and has a file size of 0.
What did I do wrong, and how could it have taken several hours copying 4-5GB but I only see this empty file?
Any help/comments are appreciated :)
|
|
Re: Synchronize didn't error but created a single, empty file
Brendan, You will certainly need to test how well different software interacts if you plan to use different tools to store items in subfolders of the same bucket. For the most part things should be OK, in that all your data will be stored in S3 and will be accessible using the same tool you used to upload it. However, you may face problems if you later try using program X to download files uploaded with program Y, or if you use both X and Y to upload files to the same "subdirectory" in S3.
It would be very interesting if you could follow up here with your experiences using a few different tools, although that would make you something of a guinea pig. Depending on exactly how other tools work, it may be possible to add options to JetS3t to make it more compatible with them. I have done this already for one or two other common tools, though not S3Fox.
James --- http://www.jamesmurty.com
On Wed, Jul 8, 2009 at 5:21 PM, Branden Makana <branden.makana@...> wrote:
Thanks James - that makes sense then. Now my concern is - do I need to make sure any files put anywhere in that bucket are done so by jets3t? I was planning on using one bucket for all my files, and then "folders" inside that bucket for each server I planned on backing up. It sounds like if I need to make sure to use jets3t or else various programs might step on each other in how they do folders, is that correct?
On Wed, Jul 8, 2009 at 5:03 PM, James Murty <james@...> wrote:
Hi Branden,
I'm afraid this problem is due to an incompatibility between the way JetS3t represents folders in S3 and the way the Firefox plugin does so. All your files will be present in S3 and completely accessible to JetS3t programs, but because JetS3t and Elasticfox use different methods to indicate when an S3 object really represents a folder you will need to give Elasticfox a little help.
Instead of relying on Elasticfox recognizing folders, you will need to type in the folder path into the object path field with an added slash character at the end, e.g. "myfolder/". When you add the slash character, you should get a listing of files inside that "folder".
As an alternative, you can use the JetS3t Cockpit application which will list all the objects in a bucket. However, Cockpit does not offer the same kind of folder-specific view that Elasticfox does.
It is unfortunate that there is not a standard way to represent folders stored in S3 and that different tools use different techniques. The underlying issue is that there is really no such thing as a folder object in S3, so we all use different tricks when storing folder hierarchies so we can recognize which objects represent folders and which represent files. All of which causes headaches for users.
I hope this helps,
James
---
http://www.jamesmurty.com
On Wed, Jul 8, 2009 at 4:12 PM, Branden Makana <branden.makana@...> wrote:
Hello,
I was trying out jets3t's Synchronize program to backup a folder structure to amazon's S3. I ran the batch file, and it appeared to run correctly - it says it copied 61,544 files, it took several hours, and no errors were spit out. However, when I use the Firefox plugin "S3 Firefox" (so that I can confirm the backup did run correctly), I see it made a single file that's 0 bytes. Now I'm trying to figure out what happened.
Here's how I ran jets3t, on a Windows 2003 server:
synchronize.bat -b UP "portentint/homerbak" e:\inetpub\wwwroot
portentint being my bucket on S3, and I wanted a directory inside that called homerbak. When I look in S3 Firefox, I see homerbak folder inside my bucket (so that worked), and then inside there I see "wwwroot" which looks like a file, and has a file size of 0.
What did I do wrong, and how could it have taken several hours copying 4-5GB but I only see this empty file?
Any help/comments are appreciated :)
|
