Oddities when tab-completing filenames containing backticks

[bash-bugs]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gnu' -DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H   -I.  -I../bash -I../bash/include -I../bash/lib   -g -O2 -Wall
uname output: Linux atu 2.6.27-14-generic #1 SMP Mon Aug 31 12:58:38 UTC 2009 x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-gnu

Bash Version: 4.0
Patch Level: 39
Release Status: release

Description:
   
   Also affects Bash 4.0.33.
   
   When a filename begins with a backtick ` then tab-completion works
   as expected if you go \`<tab>, however if you try to tab-complete
   the filename inside single quotes, several strange behaviours appear.

Repeat-By:

   1) '`<tab>      This causes all commands available on the system to be
                   listed such as `awk `bash etc., even though it's inside
                   single quotes.

   2) '`text<tab>  This completes the filename, but it does not insert a
                   trailing single quote as tab-completion would usually do.

   3) '`dir<tab>   This completes the directory name and adds the trailing
                   forward slash, but not the trailing single quote.

   4) 'text`<tab>  This completes the filename and closes the single quote
                   as expected.

   5) '`text<tab>' When using tab-completion inside single quotes on a
                   filename beginning with a backtick, then it completes
                   the filename and adds an additional closing single quote,
                   e.g. '`filename.txt''

   6) '`dir<tab>'  When using tab-completion inside single quotes on a
                   directory name beginning with a backtick, then it
                   completes the directory name and adds an additional
                   closing single quote after the directory name and before
                   the forward slash, e.g. '`directory'/'

[Chet Ramey]

bash-bugs@... wrote:

> Bash Version: 4.0
> Patch Level: 39
> Release Status: release
>
> Description:
>    
>    Also affects Bash 4.0.33.
>    
>    When a filename begins with a backtick ` then tab-completion works
>    as expected if you go \`<tab>, however if you try to tab-complete
>    the filename inside single quotes, several strange behaviours appear.

You need to remember that readline understands the characters in
rl_completer_quote_characters as those which, in pairs, delimit quoted
substrings in the line.  It performs completion (allowing the application
to take first crack, of course) on the substring, using the text after
the open quote.

The other thing to keep in mind is that readline doesn't look past point
when performing completion -- it only considers characters from the
beginning of the line or a quote character up to point.

My answers below are for the built-in bash completion, not any programmable
completions you might have defined.

> Repeat-By:
>
>    1) '`<tab>      This causes all commands available on the system to be
>                    listed such as `awk `bash etc., even though it's inside
>                    single quotes.

Readline does completion on substrings; it passes ` to the bash completion
code, which performs command completion.

>
>    2) '`text<tab>  This completes the filename, but it does not insert a
>                    trailing single quote as tab-completion would usually do.

Bash receives "`text", does command completion on `text' and suppresses
the append of the close quote, since it's usually wrong in this case.

>    3) '`dir<tab>   This completes the directory name and adds the trailing
>                    forward slash, but not the trailing single quote.

Again, appending the trailing single quote without the trailing ` is
generally going to be wrong.

>    4) 'text`<tab>  This completes the filename and closes the single quote
>                    as expected.

Interesting.  I don't get this behavior.

>    5) '`text<tab>' When using tab-completion inside single quotes on a
>                    filename beginning with a backtick, then it completes
>                    the filename and adds an additional closing single quote,
>                    e.g. '`filename.txt''

Also interesting.  I get normal command completion without any closing
quote appended.

>    6) '`dir<tab>'  When using tab-completion inside single quotes on a
>                    directory name beginning with a backtick, then it
>                    completes the directory name and adds an additional
>                    closing single quote after the directory name and before
>                    the forward slash, e.g. '`directory'/'

I don't see this behavior either.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    chet@...    http://cnswww.cns.cwru.edu/~chet/

[bash-bugs]

2009/10/16 Chet Ramey <chet.ramey@...>:
> You need to remember that readline understands the characters in
> rl_completer_quote_characters as those which, in pairs, delimit quoted
> substrings in the line.  It performs completion (allowing the application
> to take first crack, of course) on the substring, using the text after
> the open quote.

Then it is a bug in readline that it is incapable of
properly handling nested quotes.

> The other thing to keep in mind is that readline doesn't look past point
> when performing completion -- it only considers characters from the
> beginning of the line or a quote character up to point.

If it considered characters from the beginning of the line,
then there would be no issue.  The backtick is inside an
open single quote, and the backtick is *not* a special
character under that circumstance -- it's no more special
to Bash than the letter "j" inside an open single quote.

> My answers below are for the built-in bash completion, not
> any programmable completions you might have defined.

I don't use any programmable completion, these are all stock,
and confirmed by other users of irc://irc.freenode.net/#bash

>> Repeat-By:
>>
>>    1) '`<tab>      This causes all commands available on the system to be
>>                    listed such as `awk `bash etc., even though it's inside
>>                    single quotes.
>
> Readline does completion on substrings; it passes ` to the bash completion
> code, which performs command completion.

Again, a backtick is no more of a special character than the
letter "a" inside an open single quote.  This is incorrect
behaviour.

>>    2) '`text<tab>  This completes the filename, but it does not insert a
>>                    trailing single quote as tab-completion would usually do.
>
> Bash receives "`text"

What do you mean by that?  Didn't you say Bash is sending
that to readline?

> does command completion on `text'

How would it do completion on something that isn't there?
I did not fill in a closing single quote.

> and suppresses the append of the close quote,
> since it's usually wrong in this case.

What appended closing quote, there is none?

If you go:

   # ls '/usr/bin/mpg3<tab>

You get the following:

   # ls '/usr/bin/mpg321'

But if you have a file called /tmp/`rubble.txt and go:

   # ls '/tmp/`rub<tab>

Then you get:

   # ls '/tmp/`rubble.txt

No closing single quote.  Incorrect and inconsistent
behaviour.  Inside single quotes, a backtick is no
different from any other character allowed in filenames,
including control characters, tabs, newlines, accented
letters, line-drawing characters, dashes, exclamation
points, etc.

Another example:

   # ls '/tmp/!rub<tab>

Completes as expected with the closing single quote for:

   # ls '/tmp/!rubble.txt'

The same goes for these filenames:

   # ls '/tmp/~test.txt'
   # ls '/tmp/"test.txt'
   # ls '/tmp/$test.txt'
   # ls '/tmp/-test.txt'
   # ls '/tmp/(test.txt'
   # ls '/tmp/&test.txt'

They all close the single quote.  The only one that
messes up is the filename beginning with a backtick.

>>    3) '`dir<tab>   This completes the directory name and adds the trailing
>>                    forward slash, but not the trailing single quote.
>
> Again, appending the trailing single quote without the trailing ` is
> generally going to be wrong.

Please explain yourself.  Backticks *are not special* inside
single quotes.  Do you need a closing "h" if you have an
opening "h" in "hello world"?  No?  Then why would you need a
closing backtick which is just another normal character?  This
is obviously incorrect behaviour.

>>    4) 'text`<tab>  This completes the filename and closes the single quote
>>                    as expected.
>
> Interesting.  I don't get this behavior.

This behaviour makes snese, because it is the correct
behaviour.  The other behaviour above is the incorrect
behaviour.  A backtick is *not* special to Bash from
inside single quotes.

>>    5) '`text<tab>' When using tab-completion inside single quotes on a
>>                    filename beginning with a backtick, then it completes
>>                    the filename and adds an additional closing single quote,
>>                    e.g. '`filename.txt''
>
> Also interesting.  I get normal command completion without any closing
> quote appended.

Are you following my example exactly?  I'll provide
exact steps to reproduce, below.

Step 1:

   touch '/tmp/`example.txt'<enter>

Step 2:

   ls '/tmp/`exam'

Step 3:

   Place your cursor on top of the closing single quote,
   after the "m", and press <tab>

Result:

   touch '/tmp/`example.txt''

Compare this to:

   touch '/tmp/example.txt'
   touch '/tmp/exam<tab>'
   touch '/tmp/example.txt'

Works as expected, no double-closing-single-quote.

>>    6) '`dir<tab>'  When using tab-completion inside single quotes on a
>>                    directory name beginning with a backtick, then it
>>                    completes the directory name and adds an additional
>>                    closing single quote after the directory name and before
>>                    the forward slash, e.g. '`directory'/'
>
> I don't see this behavior either.

Steps to reproduce:

Step 1:

   mkdir '/tmp/`dir-example'

Step 2:

   ls '/tmp/`dir'

Step 3:

   Place your cursor on top of the closing single quote,
   after the "r", and press <tab>

Result:

   ls '/tmp/`dir-example'/'

Again, when using other normal characters than the
backtick, the behaviour is as expected:

   mkdir '/tmp/dir-example'
   mkdir '/tmp/dir<tab>'
   mkdir '/tmp/dir-example/'

[Chet Ramey]

bash-bugs@... wrote:
> 2009/10/16 Chet Ramey <chet.ramey@...>:
>> You need to remember that readline understands the characters in
>> rl_completer_quote_characters as those which, in pairs, delimit quoted
>> substrings in the line.  It performs completion (allowing the application
>> to take first crack, of course) on the substring, using the text after
>> the open quote.
>
> Then it is a bug in readline that it is incapable of
> properly handling nested quotes.

Not really, but I can fix it in bash, and I will change bash-4.1 so it
doesn't try to perform command substitution completion when completing
text beginning with a backquote inside a single-quoted string.

>
>> The other thing to keep in mind is that readline doesn't look past point
>> when performing completion -- it only considers characters from the
>> beginning of the line or a quote character up to point.
>
> If it considered characters from the beginning of the line,
> then there would be no issue.  The backtick is inside an
> open single quote, and the backtick is *not* a special
> character under that circumstance -- it's no more special
> to Bash than the letter "j" inside an open single quote.

Thank you, I'm familiar with the syntax.  I think you missed the point,
though:  Readline doesn't check characters after point when deciding
the word to complete.

>>> Repeat-By:
>>>
>>>    1) '`<tab>      This causes all commands available on the system to be
>>>                    listed such as `awk `bash etc., even though it's inside
>>>                    single quotes.
>> Readline does completion on substrings; it passes ` to the bash completion
>> code, which performs command completion.
>
> Again, a backtick is no more of a special character than the
> letter "a" inside an open single quote.  This is incorrect
> behaviour.

This one I can fix inside bash.  Readline makes the appropriate information
available to the calling application.

>>>    2) '`text<tab>  This completes the filename, but it does not insert a
>>>                    trailing single quote as tab-completion would usually do.
>> Bash receives "`text"
>
> What do you mean by that?  Didn't you say Bash is sending
> that to readline?

No.  Readline allows the calling application to perform application-
specific completion before it performs filename completion.  When it
does so, in this case, it considers the text after the single quote
and sends it to the application for completion.

>
>> does command completion on `text'
>
> How would it do completion on something that isn't there?

I'm not sure you understood when I meant.  When bash-4.0 receives "`text"
for possible completion from readline, it decides (however incorrectly)
that it should perform command completion, and does so on "text".


>> and suppresses the append of the close quote,
>> since it's usually wrong in this case.
>
> What appended closing quote, there is none?

Ummm...correct.  I just said bash suppressed the closing quote when it
thinks it's performing command completion inside backquotes.  There's
usually more of the command to come, and most of the time the user will
just erase any closing backquote or single quote.

> If you go:
>
>    # ls '/usr/bin/mpg3<tab>
>
> You get the following:
>
>    # ls '/usr/bin/mpg321'
>
> But if you have a file called /tmp/`rubble.txt and go:
>
>    # ls '/tmp/`rub<tab>
>
> Then you get:
>
>    # ls '/tmp/`rubble.txt
>
> No closing single quote.

Right again.  Since bash decided that it could perform command completion,
it suppressed the closing quote.  When bash is fixed to not attempt
command completion when inside a single-quoted string, it will insert the
single quote after performing filename completion.

>
>>>    3) '`dir<tab>   This completes the directory name and adds the trailing
>>>                    forward slash, but not the trailing single quote.
>> Again, appending the trailing single quote without the trailing ` is
>> generally going to be wrong.
>
> Please explain yourself.  

I did above.  When bash performs command completion inside a command
substitution, it suppresses the appending of the closing quote.

>
>>>    4) 'text`<tab>  This completes the filename and closes the single quote
>>>                    as expected.
>> Interesting.  I don't get this behavior.
>
> This behaviour makes snese, because it is the correct
> behaviour.  The other behaviour above is the incorrect
> behaviour.  A backtick is *not* special to Bash from
> inside single quotes.

I get the behavior you do when there is an existing file whose name
starts with "text`".  There wasn't any indication of the existing file
in your original message.  If there isn't such a file, bash doesn't
attempt command completion, but doesn't successfully complete anything,
either.

>
>>>    5) '`text<tab>' When using tab-completion inside single quotes on a
>>>                    filename beginning with a backtick, then it completes
>>>                    the filename and adds an additional closing single quote,
>>>                    e.g. '`filename.txt''
>> Also interesting.  I get normal command completion without any closing
>> quote appended.
>
> Are you following my example exactly?

The steps you provide are not the same as the example above.  If the
substring to be completed begins with a backquote, as above (but not as
in your following example, where it begins with "/tmp"), bash attempts
command completion, even when within single quotes.

When I use a filename that doesn't begin with a backquote, I get the
doubled single quote you see.  I will have to see if I can fix it in
readline.  The problem is that, as I stated above, readline doesn't
look at the characters beyond point when deciding how to complete the
filename.  In this case, that means that readline decides it needs to
insert "'/tmp/`example.txt'", but notices that the existing text in
rl_line_buffer already has a single quote, so it skips over the single
quote at the beginning of the replacement text.  It doesn't pay
attention to the corresponding single quote after point.  Maybe it will
be enough to check the character at point: if it's the same quote
character, readline can skip over the quote at the end of the replacement
text.


> Step 1:
>
>    touch '/tmp/`example.txt'<enter>
>

>>>    6) '`dir<tab>'  When using tab-completion inside single quotes on a
>>>                    directory name beginning with a backtick, then it
>>>                    completes the directory name and adds an additional
>>>                    closing single quote after the directory name and before
>>>                    the forward slash, e.g. '`directory'/'
>> I don't see this behavior either.
>
> Steps to reproduce:

Again, the text above doesn't match the text used in your example.  I
think the same change as for item 5 will work for this, though, since
the decision on the closing quote is made before the decision whether
or not to append the `/', you will probably get "'/tmp/`directory'/",
which may be a minor annoyance.

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    chet@...    http://cnswww.cns.cwru.edu/~chet/