VM

Unicode display

Asked by Johan Vromans on 2011-07-16

I use VM (currently 8.1.1 with GNU/Emacs 23.1 on Fedora14. I have been using Emacs and VM for 25 years or so.

I think my major problem is history. In 25 years many things have
evolved (changed) in the world of Mail and News, and my setups contain
many quirks to work around limitations that may not exists anymore. I
was an early adopter of MIME, non-ASCII display, GUI Emacs (Epoch) and
Unicode (Mule). The setups may have reached the point of becoming
inconsistent.

Some of the problems that I experience during daily use:

- No proper display of Unicode characters (Emacs and Gnus do this right,
  VM doesn't).

- Mysterious handling of message attachments (some are displayed inline,
  some are displatched to external 'viewers', but most are offered to be
  saved to disk). This may be a problem external to Emacs.

- Conflicts btween VM and BBDB and mail aliases.

Let's start with the first item. I have Emacs set up for Unicode and this works flallessly, except with VM.

When I receive a message with a From: line in iso-8859-1, it is stored in the VM data as iso-8859-1, which cannot be shown correctly in an UTF-8 summary buffer. So

From: =?iso-8859-1?Q?Ren=E9e?= <***@***.***>

Becomes "Ren\351e" instead of "Renée".

Similar problems occur when the message contains UTF-8 data. E.g. I see (vm-scroll-forward):

 tr[\316\261-\317\211][\316\221-\316\251]

instead of:

 tr[α-ω][Α-Ω]

When I edit the buffer (vm-edit-message) I get the message in an UTF8 buffer (type 'U') but it is still displayed as shown above.

When I reply to the message (vm-reply-include-text) I get the reply message in a 't' buffer and the UTF-8 data is now readable again.

Any idea where to start looking to fix this?

TIA,
-- Johan

Question information

Language:
English Edit question
Status:
Answered
For:
VM Edit question
Assignee:
No assignee Edit question
Last query:
2011-07-19
Last reply:
2011-07-19
Uday Reddy (reddyuday) said : #1

The first thing to do would be to save your .vm file somewhere, and start a
new .vm file that has just barebones VM configuration settings. (By
"configuration" settings, I mean path names, external programs etc., but
none of the VM options.) Then you can see how VM behaves with the default
settings and slowly add in the other option settings you need. Before you
add in any of your old settings, please check the VM manual to see whether
you need it and what its current behaviour is.

I am not sure what you mean by "UTF-8 summary buffer". Emacs uses its own
internal character set. The summary buffer is displayed in Emacs internal
character set.

My hunch is that you might have set vm-mime-default-face-charsets, which
might be blocking proper decoding. You should remove the setting and see
what you get.

Cheers,
Uday

Johan Vromans (jvromans) said : #2

[Quoting Uday Reddy, on July 17 2011, 11:26, in "Re: [Question #16504"]
> The first thing to do would be to save your .vm file somewhere, and
> start a new .vm file that has just barebones VM configuration
> settings.

Yes, this was one of the things I had in mind too.

> My hunch is that you might have set vm-mime-default-face-charsets,
> which might be blocking proper decoding. You should remove the
> setting and see what you get.

I had several lines setting vm-mime-default-face-charsets. I've
commented them out. Likewise I commented out the lines setting
vm-mime-mule-charset-to-coding-alist.

Now Summary display (after vm-fix-my-summary!!!) looks okay, and UTF-8
encoded messages look okay, too.

That's a great step forward already!

-- Johan

Uday Reddy (reddyuday) said : #3

That is good.

Regarding the handling of attachments, there is nothing "mysterious" about
it. The manual explains quite clearly how the attachments are handled. I
suggest that you read the section 3.3 "Reading MIME Messages", and redo your
options settings. You have to follow the sequence described in the manual
because there will be interdependencies between the option settings. If
there is something particular that is anomalous then please let us know.

As for "conflicts" between VM and BBDB mail aliases, what exactly is the
problem? VM doesn't have mail aliases anyway. You must be talking about
Emacs mail aliases. Do you mean that you have conflicting definitions and
you expect them to get sorted out somehow?

Cheers,
Uday

Johan Vromans (jvromans) said : #4

[Quoting Uday Reddy, on July 17 2011, 13:26, in "Re: [Question #16504"]
> Your question #165045 on VM changed:
> https://answers.launchpad.net/vm/+question/165045
>
> Status: Open => Answered
>
> Uday Reddy proposed the following answer:
> That is good.

I cheered just a little bit too early...

Upon restarting VM the summary display is wrong again.

New messages that arrive with non-ASCII sender names are displayed
correctly.

Here are X-VM headers of an old and a new message.

New message:

X-VM-v5-Data: ([nil nil nil nil t nil nil nil nil]
 [nil "Friday" "1" "July" "2011" "08:08:46" "" "=?iso-8859-1?Q?Ren=E9e?=" "<email address hidden>" nil "8" "Some subject" "^From:" nil nil "7" nil nil (number " " mark " R =?iso-8859-1?Q?Ren=E9e?= Jul 1 8 " thread-indent "Some subject\n") nil nil nil nil nil nil nil]
 nil)

Old message:

X-VM-v5-Data: ([nil nil nil nil t nil nil nil nil]
 [nil "Friday" "15" "July" "2011" "21:07:28" "+0200" "=?iso-8859-1?Q?Ren=E9e?=" "<email address hidden>" nil "21" "Re: Suid africa" "^From:" nil nil "7" nil nil (number " " mark " R Ren\351e Jul 15 21 " thread-indent "Re: Suid africa\n") nil nil nil nil nil nil nil]
 nil)

(Note that \351 is in reality a literal \351 -- ISO 8859.1 for é)

Apparently, there's a problem with the old header.

vm-fix-my-summary!!! fixes it visually, but does not change the X-VM
header.

Any idea?

-- Johan

Uday Reddy (reddyuday) said : #5

Johan Vromans writes:

> Apparently, there's a problem with the old header.
>
> vm-fix-my-summary!!! fixes it visually, but does not change the X-VM
> header.
>
> Any idea?

VM 8.1.1 is probably not doing a good job of saving the cached summary lines
in the headers. If you upgrade to VM 8.2.0, the problem should be solved.
Or, for the time being, you can force the cached summary lines to be saved
by doing something like this:

- mark all messages
- delete and undelete all marked messages

That wold force the header lines to be updated.

Cheers,
Uday

Johan Vromans (jvromans) said : #6

[Quoting Uday Reddy, on July 18 2011, 13:16, in "Re: [Question #16504"]
> Uday Reddy proposed the following answer:

Thanks for your kind and patient answers.

> VM 8.1.1 is probably not doing a good job of saving the cached summary lines
> in the headers. If you upgrade to VM 8.2.0, the problem should be solved.
> Or, for the time being, you can force the cached summary lines to be saved
> by doing something like this:
>
> - mark all messages
> - delete and undelete all marked messages
>
> That wold force the header lines to be updated.

Neither the delete/undelete trick, nor 8.2.0a fixes it.
I even tried the delete/undelete trick with 8.2.0a, to no avail.

Delete/undelete says that the summary is updated, the summary display
gets updated and is okay, the mail buffers are written to disk, but
upon restart it's all wrong again.

I tried this with some other mailboxes, and there it works as you
described. So something strange is going on.

I'll try to isolate the problem.

-- Johan

Uday Reddy (reddyuday) said : #7

Johan Vromans writes:

> Delete/undelete says that the summary is updated, the summary display
> gets updated and is okay, the mail buffers are written to disk, but
> upon restart it's all wrong again.

Are you sure that the mail buffers were written to disk? You can check the
*Messages* buffer to see what was actually done. Set the variable
`vm-verbosity' to 10 to make sure that you don't miss any messages.

It might also help to wait for 90 seconds after you fix the summary to make
sure that the cached-data is written into headers. (It shouldn't be
necessary in 8.2.0a, but just to make sure.)

After that, visit the mail folder as a plain text file and check if the
cached-data headers have been changed.

If it still doesn't work, then you are facing a bug. Please try to produce
a small folder that reproduces the problem, and do M-x vm-submit-bug-report.

Good luck!

Uday

Johan Vromans (jvromans) said : #8

[Quoting Johan Vromans, on July 19 2011, 00:20, in "Re: [Question #16504"]
> I tried this with some other mailboxes, and there it works as you
> described. So something strange is going on.
>
> I'll try to isolate the problem.

I haven't been able to reproduce this problem with a small mailbox. In
my operational mailbox (6MB, 400 messages) it leads to all kinds of
surprises, including double-encoded UTF-8 characters.

So I decided to leave it, hoping that new messages will be dealt with
okay.

A quick upgrade (8.1.0 -> 8.2.0a) question: How can I stop VM from
automatically raise the mail window when new mail arrives?

-- Johan

Uday Reddy (reddyuday) said : #9

Johan Vromans writes:

> I haven't been able to reproduce this problem with a small mailbox. In
> my operational mailbox (6MB, 400 messages) it leads to all kinds of
> surprises, including double-encoded UTF-8 characters.
>
> So I decided to leave it, hoping that new messages will be dealt with
> okay.

Another idea that occurs to me is that you could save all the old messages
in a newly created mail folder. If everything works in that folder, then
you could delete the old copies in the main folder and move the copied
messages back to the main folder.

> A quick upgrade (8.1.0 -> 8.2.0a) question: How can I stop VM from
> automatically raise the mail window when new mail arrives?

I have no idea. I have never seen it happen.

What I recommend is that you move all your preferences settings into a
separate .vm.preferences file, and keep only the configuration settings in
the .vm file. Then you can do C-u M-x vm-load-init-file to load just the
.vm file, and try everything with the default settings.

Cheers,
Uday

Can you help with this problem?

Provide an answer of your own, or ask Johan Vromans for more information if necessary.

To post a message you must log in.