user-interface suggestions

[Richard Neill-3]

Dear All,

Firstly, may I say thank you very much for the excellent ddrescue. It
has saved a substantial amount of data.

May I suggest some UI enhancements?


I am afraid I'm guilty of not fully reading the manual, and I started to
run the program without a logfile. Clearly this is an common user-error,
as shown by the very existence of the -g option!
May I propose the following changes:


1. Make it more obvious that a logfile is strongly recommended. Change
the man page synopsis to remove the [] from logfile. Thus

SYNOPSIS
        ddrescue [options] infile outfile [logfile]

would become:
        ddrescue [options] infile outfile logfile



2. If a logfile is not proposed, then print a warning message suggesting
the user cancel and start again.

Or, perhaps ddrescue should quit with an error, unless the user
explicitly over-rides the logfile by specifying
   --really-don't-use-a-logfile



Some other thoughts:

3. The -d option could perhaps be automatic? How is the end-user
supposed to know whether the block sizes are correctly set, and whether
the system does or doesn't support direct access? Presumably it's easier
for ddrescue to follow any rules that the user might himself apply?
Also, could ddrescue automatically switch between -n and -d mode as
appropriate?


4. Given that ddrescue will usually not terminate, it would be useful if
ddrescue could print the time since it last recovered any data.
Something such as "ddrescue has now been working for 8 hours without
recovering any more data; you've probably got all you're going to get".


5. Lastly, for compatibility with regular dd, may I recommend graceful
handling (or at least ignoring) of "kill -USR1"  ?


Thanks very much,

Richard



P.S. I wrote the following summary for the linux-thinkpad mailing list
recently; if you would find it useful to share as documentation, feel free.


-----------------
Dear All,

I've just saved a friend's SATA notebook disk, and thought it might be
worth writing up a brief account.

Symptom:
   Laptop wouldn't boot at all, stalling at the first BIOS screen.
   HDD wasn't spinning.
   Unplugging HDD allowed the laptop to boot via live-CD.
   => HDD faulty.

Confirm:
   There is nothing else that can be done with this disk. Professional
    data recovery is too expensive. So OK to take a risk.

Open the disk:
   Carefully, using a torx screwdriver.
   One screw is hidden under the label.
   Clean room would be nice, but I don't have one

Discovery:
   The head appears to have cold-welded itself somewhere in the
   middle of the disk. The disk can't spin at all. This is the
   second time I've come across a 2.5" disk with this problem.
   User says it wasn't mis-treated.

Solution:
   Carefully, without touching the platter, use a fingernail to
   force the head back into the parked position. Some considerable
   force (perhaps 10 newtons?) is required.
   Then quickly re-assemble drive before too much dust gets in.

Recovery:
   Put disk into thinkpad. Boot thinkpad from a live CD. (Mandriva 2009).
   Fortunately, the "dead" disk spins up.
   In the live-CD environment, add the package repositories, and install
   ddrescue.  (Note there are 2 similar programs: ddrescue and dd_rescue.
   I've had success with both.)

   Mount an external USB HDD. Let ddrescue do its stuff:
     ddrescue /dev/sda sda.img ddrescue.log
   After about 24 hours, ddrescue has recovered all 80 GB of the disk,
   excepting about 5 MB.
   Also, save fdisk info:
     fdisk -l /dev/sda > sda.fdisk.txt

Extract partition 1.
    dd if=sda.img of=sda1.img bs=512 skip=63    #value depends on fdisk.

    Mount the partition loopback:
      mount -o loop,ro sda1.img /mnt/tmp

    Note: it's supposed to be possible to do
       mount -o loop,ro,offset=32256 sda.img  /mnt/tmp
     but I found it didn't work.

    Note 2: It would normally have be easier to directly extract
    the partitions one at a time, at the ddrescue stage,
    But in this case, the offending disk is a Windows XP
    system, and I want to be able to re-image the OS onto a new
    drive. A filesystem-level copy won't work here. [In Linux, it's
    easy: just fdisk, mkfs, rsync, then fix fstab and install grub]

Recover data:
    copy stuff from /mnt/tmp

Re-image to a new disk:
    Plug in new HDD, boot from liveCD.
    dd if=sda.img of=/dev/sda


I hope that's useful to someone...

Morals of the story:
   (1) ddrescue is really useful.
   (2) Linux can do almost anything.
   (3) Even a totally stuck HDD can be saved without needing a cleanroom.
   (4) Having a USB-SATA adapter would be useful.

Best wishes,

Richard



_______________________________________________
Bug-ddrescue mailing list
Bug-ddrescue@...
http://lists.gnu.org/mailman/listinfo/bug-ddrescue

[Antonio Diaz Diaz-2]

Hello Richard,

Richard Neill wrote:
> 1. Make it more obvious that a logfile is strongly recommended. Change
> the man page synopsis to remove the [] from logfile. Thus

Given that I use regularly ddrescue as a replacement for dd, I don't
like the idea of forcing logfile usage (or having to type an option to
override the logfile).

I can add a warning to the man page saying something like "you should
use a logfile unless you know what you are doing".


> 3. The -d option could perhaps be automatic?

No. Using -d correctly in every case is an art. It can't be (easily)
automated.


> 4. Given that ddrescue will usually not terminate, it would be useful if
> ddrescue could print the time since it last recovered any data.
> Something such as "ddrescue has now been working for 8 hours without
> recovering any more data; you've probably got all you're going to get".

Sounds well. I'll see what can be done.


> 5. Lastly, for compatibility with regular dd, may I recommend graceful
> handling (or at least ignoring) of "kill -USR1"  ?

I can easily handle it like SIGINT.


> P.S. I wrote the following summary for the linux-thinkpad mailing list
> recently; if you would find it useful to share as documentation, feel free.

Thanks. It is an interesting rescue story.


Regards,
Antonio.


_______________________________________________
Bug-ddrescue mailing list
Bug-ddrescue@...
http://lists.gnu.org/mailman/listinfo/bug-ddrescue

[Richard Neill-3]

Dear Antonio,

Thanks for your reply.

> Richard Neill wrote:
>> 1. Make it more obvious that a logfile is strongly recommended. Change
>> the man page synopsis to remove the [] from logfile. Thus
>
> Given that I use regularly ddrescue as a replacement for dd, I don't
> like the idea of forcing logfile usage (or having to type an option to
> override the logfile).
>
> I can add a warning to the man page saying something like "you should
> use a logfile unless you know what you are doing".

OK - I agree about not forcing a logfile. I'd also suggest removing the
[] brackets in the synopsis, i.e. replace "[logfile]" by "logfile", to
indicate that most of the time, logfiles are needed. That was the visual
cue that I myself was confused by.

>
>
>> 3. The -d option could perhaps be automatic?
>
> No. Using -d correctly in every case is an art. It can't be (easily)
> automated.

The manual says:

      Use direct disc access to read the input file, bypassing the kernel
      cache. Hardware block size must be correctly set for this to work.
      Not all systems support this.

May I suggest a few more sentences of documentation would help? In the
common case (a modern Linux or BSD system), is it fair to say that -d
will work? If ddrescue is invoked with -d, but direct access isn't
possible, will it fail gracefully, or will bad things happen? How can I
find out whether the hardware block size is correctly set?

I think there are two separate issues here:

* When should -d as opposed to -n be used?
   I think this is what you refer to as an art.

* Is it appropriate to use -d at all? Are the block sizes correct? Does
   my system support this?
   That's what I suggest should be automated.


>
>
>> 4. Given that ddrescue will usually not terminate, it would be useful
>> if ddrescue could print the time since it last recovered any data.
>> Something such as "ddrescue has now been working for 8 hours without
>> recovering any more data; you've probably got all you're going to get".
>
> Sounds well. I'll see what can be done.

Thanks.
>
>
>> 5. Lastly, for compatibility with regular dd, may I recommend graceful
>> handling (or at least ignoring) of "kill -USR1"  ?
>
> I can easily handle it like SIGINT.

For regular dd,
     killall -USR1 dd
will make it print out statistics of where it has got to.

That's probably unnecessary here, but I think that USR1 should be
trapped and *ignored* rather than treated the same as SIGINT.


Best wishes,

Richard


_______________________________________________
Bug-ddrescue mailing list
Bug-ddrescue@...
http://lists.gnu.org/mailman/listinfo/bug-ddrescue

[Antonio Diaz Diaz-2]

Richard Neill wrote:
> May I suggest a few more sentences of documentation would help? In the
> common case (a modern Linux or BSD system), is it fair to say that -d
> will work? If ddrescue is invoked with -d, but direct access isn't
> possible, will it fail gracefully, or will bad things happen? How can I
> find out whether the hardware block size is correctly set?

Yes. I'll try to answer these questions in the manual.


> * When should -d as opposed to -n be used?
>   I think this is what you refer to as an art.

This is not an art at all. -d and -n are totally unrelated options. You
can combine them as you like. -d determines how to access the data,
while -n makes ddrescue stop after the trimming pass.


> * Is it appropriate to use -d at all? Are the block sizes correct? Does
>   my system support this?
>   That's what I suggest should be automated.

Here is where the art resides. -d may be faster or slower than cached
reads. There may be several valid combinations of block size / read
size. All of it depending on the type of medium and OS.


> That's probably unnecessary here, but I think that USR1 should be
> trapped and *ignored* rather than treated the same as SIGINT.

OK. Ignoring it is as easy as handling it.


Regards,
Antonio.


_______________________________________________
Bug-ddrescue mailing list
Bug-ddrescue@...
http://lists.gnu.org/mailman/listinfo/bug-ddrescue

[Richard Neill-3]

Antonio Diaz Diaz wrote:
>> May I suggest a few more sentences of documentation would help?
>
> Yes. I'll try to answer these questions in the manual.

Thanks.

>
>
>> * When should -d as opposed to -n be used?
>>   I think this is what you refer to as an art.
>
> This is not an art at all. -d and -n are totally unrelated options. You
> can combine them as you like. -d determines how to access the data,
> while -n makes ddrescue stop after the trimming pass.

Sorry - my brain was quite clearly failing when I wrote that. What I
meant was whether to use -d or not to use it.
>
>
>> * Is it appropriate to use -d at all? Are the block sizes correct? Does
>>   my system support this?
>>   That's what I suggest should be automated.
>
> Here is where the art resides. -d may be faster or slower than cached
> reads. There may be several valid combinations of block size / read
> size. All of it depending on the type of medium and OS.


Here lies the problem. It's going to be hard (I think) to write a
suitable heuristic for ddrescue.
But the user certainly doesn't know. And trying to acquire the "art" at
the moment of panic when rescuing a damaged disc is not going to work.

There's no easy way out of this. I assert that the art ought to be
in the program, rather than the user (because that way, the author has
to do the work once, as opposed to each and every user learning for
himself) - but of course that represents a lot of work for you.

Is there any way to simplify the heuristic such that ddrescue can make
its own decisions about what is "nearly" the best outcome? It doesn't
have to be spot on. But at the least, ddrescue should protect the user
from any bad consequences of the misuse of -d.

Regards,

Richard


_______________________________________________
Bug-ddrescue mailing list
Bug-ddrescue@...
http://lists.gnu.org/mailman/listinfo/bug-ddrescue

[Kim Pedersen-3]

Antonio Diaz Diaz wrote:
>> * Is it appropriate to use -d at all? Are the block sizes correct? Does
>>   my system support this?
>>   That's what I suggest should be automated.
>
> Here is where the art resides. -d may be faster or slower than cached
> reads. There may be several valid combinations of block size / read
> size. All of it depending on the type of medium and OS.

 From my experience, the -d option has only been faster (Using various
Linux distributions, like Mandriva, CentOS, Trinity and RIP)

Without it I have had drive rescue operations take 4-5 days, but when
using the -d switch, the same operation would take about 8 hours.

As I understand it, the -d option affects whether ddrescue get affected
by any OS read ahead options.

So if you have OS read-ahead default set to 16384 bytes, and ddrescue
reaches an area with lots of bad sectors then ddrescue has to wait for
the drive to stutter, timeout and retry it's way through the to 32
(16384/512) bad sectors.

Whereas if you use the -d option, ddrescue won't be as fast on good
areas (Sequential throughput suffers without readhead), but when it hits
a bad sector it will quickly skip and move on to other good parts on the
drive, rather than get caught up because the OS tries to readahead a
whole cluster of bad blocks.


Anyways, it's been a few months since I last looked at all of this,
maybe what I am saying is hogwash, but after seeing a 4 day recovery
being done in about 8 hours, I now always use the -d option.


--
Regards,

Kim Pedersen






_______________________________________________
Bug-ddrescue mailing list
Bug-ddrescue@...
http://lists.gnu.org/mailman/listinfo/bug-ddrescue

[Antonio Diaz Diaz-2]

Richard Neill wrote:
> Here lies the problem. It's going to be hard (I think) to write a
> suitable heuristic for ddrescue.

The problem is that, AFAIK, there is no way for ddrescue to know the
block size of a device except by means of user-supplied arguments.


> Is there any way to simplify the heuristic such that ddrescue can make
> its own decisions about what is "nearly" the best outcome? It doesn't
> have to be spot on. But at the least, ddrescue should protect the user
> from any bad consequences of the misuse of -d.

There aren't such bad consequences. If the values supplied are correct,
ddrescue will rescue data. If not, all reads will fail fast, but you can
stop it and try other values.


Regards,
Antonio.


_______________________________________________
Bug-ddrescue mailing list
Bug-ddrescue@...
http://lists.gnu.org/mailman/listinfo/bug-ddrescue

[Richard Neill-3]

Antonio Diaz Diaz wrote:
> Richard Neill wrote:
>> Here lies the problem. It's going to be hard (I think) to write a
>> suitable heuristic for ddrescue.
>
> The problem is that, AFAIK, there is no way for ddrescue to know the
> block size of a device except by means of user-supplied arguments.

So, how does the user know this? If I sit down at a Linux machine, which
has some item of dodgy hardware (probably a regular SATA disk, but maybe
  PATA, maybe a floppy, maybe a RAID array, or maybe even a USB-disk or
a  CD-ROM), how am I supposed to find out the block size?

Is this a property of the underlying hardware, of the linux device
driver, or of the filesystem?

Googling for this has multiple answers, stating authoritatively that the
BS is one of  512, 1k, 2k, 4k or 16k. The only thing I've learned is
that "block size" can mean more than one thing.

>
>> Is there any way to simplify the heuristic such that ddrescue can make
>> its own decisions about what is "nearly" the best outcome? It doesn't
>> have to be spot on. But at the least, ddrescue should protect the user
>> from any bad consequences of the misuse of -d.
>
> There aren't such bad consequences. If the values supplied are correct,
> ddrescue will rescue data. If not, all reads will fail fast, but you can
> stop it and try other values.

Is that obvious to the user? Is it clear from the error message that the
error is due to the wrong blocksize specification, and not just because
the disk itself is failing? (After all, the user probably *expects*
reads to fail because the hardware is broken).

Best wishes,

Richard


_______________________________________________
Bug-ddrescue mailing list
Bug-ddrescue@...
http://lists.gnu.org/mailman/listinfo/bug-ddrescue

[Ian Oliver-2]

In article <4A070A2D.7010002@...>, Richard Neill wrote:
> (4) Having a USB-SATA adapter would be useful.

Maybe it depends on adaptors but I have never had good results running
ddrescue on a drive connected via USB. The drive/adaptor/whatever seems
to get into a state where reading always returns junk data (copies of
previous stuff?) and no error. So, ddrescue does the job but the output
file is useless.

Connecting the drive via IDE/SATA works massively better IME.

YMMV.

Ian





_______________________________________________
Bug-ddrescue mailing list
Bug-ddrescue@...
http://lists.gnu.org/mailman/listinfo/bug-ddrescue