highlighting "weird" characters...

[Mitch Wiedemann-2]

Hi all,

I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007 22:17:23)
to write mainly XHTML/PHP and I sometimes have to get content from word
processed documents and paste it into Vim for HTML markup.  This usually
results in having non-visually detectable characters (which I assume are
high ASCII) which display strangely on the Web.

Is there a way I can have my Vim highlight these characters so I can see
them and replace them with their HTML counterparts?

I've searched Google, the Vim e-mail archive, and I've helped Ugandan
children :),  but I'm no closer to the answer.

Any hints?

Thanks,
Mitch
Freeville, NY

[Cyril Slobin-2]

On 3/19/07, Mitch Wiedemann <mitch@...> wrote:

> Is there a way I can have my Vim highlight these characters so I can see
> them and replace them with their HTML counterparts?

No idea about highlighting, but exactly to replace strange characters
with HTML or LaTeX or anything else or to do it in backward direction
I've write this plugin:

http://www.vim.org/scripts/script.php?script_id=1761

Documentation inside. I hope this helps.

--
Cyril Slobin <slobin@...> `When I use a word,' Humpty Dumpty said,
<http://45.free.net/~slobin> `it means just what I choose it to mean'

[Gary Johnson]

On 2007-03-19, Mitch Wiedemann <mitch@...> wrote:

> Hi all,
>
> I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007 22:17:23)
> to write mainly XHTML/PHP and I sometimes have to get content from word
> processed documents and paste it into Vim for HTML markup.  This usually
> results in having non-visually detectable characters (which I assume are
> high ASCII) which display strangely on the Web.
>
> Is there a way I can have my Vim highlight these characters so I can see
> them and replace them with their HTML counterparts?
>
> I've searched Google, the Vim e-mail archive, and I've helped Ugandan
> children :),  but I'm no closer to the answer.
>
> Any hints?

One way to do this would be to

    :set isprint=

which will tell vim that only the characters in the range 32 - 126
are "printable".  Vim will then highlight all the other characters
as SpecialKey.  You can then search for these "non-printable"
characters with

    /[^[:print:]]

Another, probably better, way would be to simply search for

    /[^Vx80-^Vxff]

where ^V means a literal Ctrl-V.  That will search for any character
in the range 0x80 - 0xff and will highlight them all with the Search
highlight if 'hlsearch' is set.  I think this way is better because
it preserves vim's rendering of the non-ASCII characters, which may
make it easier for you to choose their replacements.

HTH,
Gary

--
Gary Johnson                 | Agilent Technologies
garyjohn@...     | Mobile Broadband Division
                             | Spokane, Washington, USA

[Gary Johnson]

On 2007-03-19, Gary Johnson <garyjohn@...> wrote:
> On 2007-03-19, Mitch Wiedemann <mitch@...> wrote:
> > Hi all,
> >
> > I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007 22:17:23)
> > to write mainly XHTML/PHP and I sometimes have to get content from word
> > processed documents and paste it into Vim for HTML markup.  This usually
> > results in having non-visually detectable characters (which I assume are
> > high ASCII) which display strangely on the Web.

P.S.

See

    :help ga

HTH,
Gary

--
Gary Johnson                 | Agilent Technologies
garyjohn@...     | Mobile Broadband Division
                             | Spokane, Washington, USA

[Tim Chase-2]

> I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007 22:17:23)
> to write mainly XHTML/PHP and I sometimes have to get content from word
> processed documents and paste it into Vim for HTML markup.  This usually
> results in having non-visually detectable characters (which I assume are
> high ASCII) which display strangely on the Web.
>
> Is there a way I can have my Vim highlight these characters so I can see
> them and replace them with their HTML counterparts?
>
> I've searched Google, the Vim e-mail archive, and I've helped Ugandan
> children :),  but I'm no closer to the answer.

Well, if you want to highlight them, you can do something like

        :match Error /[\x7f-\xff]/

which should catch most of the offenders.  Or you can search for
them with the same regexp.  And if you have ":set hls", it will
highlight them too. :)

It does require certain knobs to be set properly in your
'cpoptions' setting, but if you run vim (rather than "vi"), they
should automatically be set unless you had reason to bung with them.

Once you're on a particular one, you can use

        ga

to determine the ASCII value of the character in question, and
can use that value for search&replace.

Hope this helps,

-tim

[iler61]

On 3/19/07, Mitch Wiedemann <mitch@...> wrote:

> Hi all,
>
> I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007 22:17:23)
> to write mainly XHTML/PHP and I sometimes have to get content from word
> processed documents and paste it into Vim for HTML markup.  This usually
> results in having non-visually detectable characters (which I assume are
> high ASCII) which display strangely on the Web.
>
> Is there a way I can have my Vim highlight these characters so I can see
> them and replace them with their HTML counterparts?

Try this:

:hi clear SpecialKey
:hi link SpecialKey Error

All uprintable chars will be highlighter red
(or whaterver your Error color is defined).

Yakov

[Tobia Conforto]

Mitch Wiedemann wrote:
> non-visually detectable characters (which I assume are high ASCII)

If I may nitpick, "high ASCII" is not the right terminology here.  
The ASCII table only contains 128 characters, with codes 0…127 (where
only codes 32…126 have a visual representation.)

What you call "high ASCII" are Unicode characters that are autside
ASCII, or Latin-1 characters that are autside ASCII.  In both cases they
have codes greater than 127.


> Is there a way I can have my Vim highlight these characters so I can
> see them and replace them with their HTML counterparts?

I usually do a :set hls (highlight search results) followed by /[^ -~]
(search for any character that is not between space and tilde, ie. that
is not a printable character from the ASCII set.)  You can add the tab
character (with Ctrl-V Tab) inside the square brackets if you use it in
your documents.  It will show as /[^ -~^I] with a blue ^I.

This has the added benefit of hitting n to get to the next one.


Tobia

[Mitch Wiedemann-2]

Thanks everyone for your suggestions!  I'm now looking around for a
typical document that I can paste in to use a a guinea pig.

Viva la Vim!

[Mitch Wiedemann-2]

Mitch Wiedemann wrote:

> Hi all,
>
> I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007 22:17:23)
> to write mainly XHTML/PHP and I sometimes have to get content from word
> processed documents and paste it into Vim for HTML markup.  This usually
> results in having non-visually detectable characters (which I assume are
> high ASCII) which display strangely on the Web.
>
> Is there a way I can have my Vim highlight these characters so I can see
> them and replace them with their HTML counterparts?
>
> I've searched Google, the Vim e-mail archive, and I've helped Ugandan
> children :),  but I'm no closer to the answer.
>
> Any hints?
>  

I've added two new maps to my ~/.vimrc

" maps to quickly find Unicode characters within the document
map ,uni :match Error /[\x7f-\xff]/<CR>
map ,uni2 /[^ -~]<CR>

I've found both methods useful, one just highlights (allowing me to
/find other stuff) and one highlights via /find.

Neat. Thanks again for the help.

[Tobia Conforto]

Mitch Wiedemann wrote:
> " maps to quickly find Unicode characters within the document
> map ,uni :match Error /[\x7f-\xff]/<CR>
> map ,uni2 /[^ -~]<CR>

FYI [\x7f-\xff] doesn't find non-ASCII characters that are also outside
Latin-1, such as em-dash —, typographic quotes “”, ellipsis …, and the
hundred thousand others defined by Unicode :-)  At least on my system.

[^ -~] on the other hand finds all non-ASCII characters.

You can use either one in both :match Error and /


Tobia

[Mitch Wiedemann-2]

Tobia wrote:

> Mitch Wiedemann wrote:
>  
>> " maps to quickly find Unicode characters within the document
>> map ,uni :match Error /[\x7f-\xff]/<CR>
>> map ,uni2 /[^ -~]<CR>
>>    
>
> FYI [\x7f-\xff] doesn't find non-ASCII characters that are also outside
> Latin-1, such as em-dash —, typographic quotes “”, ellipsis …, and the
> hundred thousand others defined by Unicode :-)  At least on my system.
>
> [^ -~] on the other hand finds all non-ASCII characters.
>
> You can use either one in both :match Error and /
>  

Ah! Good to know.  The characters you mention are some of the most
frequent offenders.

~/.vimrc updated...

" maps to quickly find unicode characters within the document
map ,uni :match Error /[^ -~]/<CR>
map ,uni2 /[^ -~]<CR>

[Spencer Collyer]

On Mon, 19 Mar 2007 16:38:23 -0400, Mitch Wiedemann wrote:

> Hi all,
>
> I use VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 23 2007
> 22:17:23) to write mainly XHTML/PHP and I sometimes have to get
> content from word processed documents and paste it into Vim for HTML
> markup.  This usually results in having non-visually detectable
> characters (which I assume are high ASCII) which display strangely on
> the Web.
>
> Is there a way I can have my Vim highlight these characters so I can
> see them and replace them with their HTML counterparts?
>
> I've searched Google, the Vim e-mail archive, and I've helped Ugandan
> children :),  but I'm no closer to the answer.
>
> Any hints?
>
> Thanks,
> Mitch
> Freeville, NY

Not a VIM-based method, but there is a Perl script called Demoroniser
(http://www.fourmilab.ch/webtools/demoroniser/) which is designed to
handle a lot of these characters and convert them into ASCII characters.

Spencer

--
<<< Eagles may soar, but weasels don't get sucked into jet engines >>>
8:47am up 25 days 15:29, 18 users, load average: 0.08, 0.14, 0.18
Registered Linux User #232457 | LFS ID 11703

[Lionel Flandin-2]

Hi everyone,

I'm trying to find the command that would find the next error in a file.
For instance the following code contains a typo, how can one find it ?

-------------------------------------------------
#include <stdio.h>
int main(int argc, char **argv)
{
     printf("Hello world!\n");
        ]  // <- Here it is :-), vim detects it and makes it red
     return 0;
}
-------------------------------------------------

Thx in advance,

Lionel

[iler61]

On 4/4/07, Lionel Flandin <Lionel.Flandin@...> wrote:

> Hi everyone,
>
> I'm trying to find the command that would find the next error in a file.
> For instance the following code contains a typo, how can one find it ?
>
> -------------------------------------------------
> #include <stdio.h>
> int main(int argc, char **argv)
> {
>      printf("Hello world!\n");
>         ]  // <- Here it is :-), vim detects it and makes it red
>      return 0;
> }
> -------------------------------------------------

This seems to be a recurring request. In the past, I
I posted to this list a vimscript function that does it. I made it into
small plugin:
         http://www.vim.org/scripts/script.php?script_id=1851
         SearchSyntaxError : search for the next syntax error detected
by the vim highlighting mechanism
The usage is very simple:
command :ERR searhes for the next syntax error, or <Ctrl-K><Ctrl-E>
That's it.

Yakov

[Lionel Flandin-2]

GREAT !

thank you it works just fine.

Lionel

@ 14:58 04/04/2007, >>>Yakov Lerner<<< Wrote ------------------------:wq!

>On 4/4/07, Lionel Flandin <Lionel.Flandin@...> wrote:
>>Hi everyone,
>>
>>I'm trying to find the command that would find the next error in a file.
>>For instance the following code contains a typo, how can one find it ?
>>
>>-------------------------------------------------
>>#include <stdio.h>
>>int main(int argc, char **argv)
>>{
>>      printf("Hello world!\n");
>>         ]  // <- Here it is :-), vim detects it and makes it red
>>      return 0;
>>}
>>-------------------------------------------------
>
>This seems to be a recurring request. In the past, I
>I posted to this list a vimscript function that does it. I made it into
>small plugin:
>         http://www.vim.org/scripts/script.php?script_id=1851
>         SearchSyntaxError : search for the next syntax error detected
>by the vim highlighting mechanism
>The usage is very simple:
>command :ERR searhes for the next syntax error, or <Ctrl-K><Ctrl-E>
>That's it.
>
>Yakov