Tom Horsley writes:
Some of the messages show the international characters,
In this thread, those are Ed's.
and some do not.
It would be interesting to examine the detailed headers of the
message where the characters disappeared,
I looked in my local folder, and there's nothing interesting the
headers. Not surprising; the mail header is not very expressive.
Archiving happens *after* message transformations, so I'm seeing the
same headers you would in HyperKitty. We'd probably need to see the
pre-Mailman header to identify any problems with Mailman via the
header, but Mailman has never kept those. Perhaps if Tim sent a mail
both to the list and to himself, comparing those headers might tell us
something, but his messages are simple text/plain; charset=UTF-8, so I
doubt it. I'm prety sure it has something to do with the Unicode
Ed sent his message in Unicode with NFC normalization (the é is
pre-composed) and UTF-8. Tim's message contains two ?, indicating a
pair of unknown characters. One possibility is that Tim's MUA
(Evolution) converts that to NFD normalization and something in
between chokes on that, and produces the doubled ?? instead of a
single é. Renormalization is perfectly conformant to both Unicode and
mail standards, but lots of software has issues with NFD. Mailman
should not, since it doesn't need to interpret anything other than
ASCII text, and passes anything else along (or deletes/quarantines
whole MIME parts), and I've not heard of such problems (but Mailman 3
is a completely new code base, so it's possible a new issues has been
It's also possible that Tim's MUA double-UTF-8-encodes the é, which
results in an illegal code point sequence which might also be
represented as ??. Of course double-encoding is a bug, and if Mailman
receives such email, it's quite likely that it would replace the
broken text with ??. This seems highly unlikely, as Tim would be
seeing issues all over the place, including in mail directly to
himself, which he has tried without problem.
So most likely something between Tim's MUA and Mailman (the list
manager, not HyperKitty) is mishandling the text, in one of the ways
described above. I can't exclude either endpoint, but in both cases
somebody should be seeing a lot of similar mojibake. Tim reports
some, but not in direct to self, and I've never seen Mailman cause
anything like this. More objectively, the fact that the ??s are in
HyperKitty rather than some 8-bit mojibake strongly suggests that even
if Mailman is directly responsible for the ??s, it was replacing
existing mojibake with ??.
but that seems to be impossible with hyperkitty.
I believe there's work being done on more detailed archiving at GNU
Mailman that might help diagnosing this kind of issue (not done yet
though). I don't know if lists.fedoraproject is tracking us, though.
Associate Professor Division of Policy and Planning Science
Faculty of Systems and Information
Email: turnbull(a)sk.tsukuba.ac.jp University of Tsukuba
Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN