Encoding Problem


Messages with unusual encoding (eg Chinese) may have some characters turned into question marks ("?")


TehPenguin wrote Oct 20, 2009 at 2:49 AM

I haven't been able to recreate this error. What is the encoding of the emails, and what characters are being turned into a "?"?

chineseleper wrote Oct 20, 2009 at 12:32 PM

I have once received the mail directly with exchange, where the encoding worked perfectly and once through GMX (freemailer in germany), which got downloaded by p3ss and then sent to exchange (??? only here).
I need to add that the title showed up correctly, only the content was wrong.

This is the header from the correct message. I can forward the complete message to you if you like.
Received: from cg-p00-ob.rzone.de ( by server.qianqin.de
( with Microsoft SMTP Server id 8.1.393.1; Wed, 14 Oct 2009
19:46:18 +0200
Received: from bruce.store ([]) by scum-cg-03.store (RZmta 22.1)
with ESMTP id v067a3l9EGBRlh for xxxx@qianqin.de; Wed, 14 Oct 2009 19:44:20
+0200 (MEST)
Received: (from Unknown UID 115767@localhost) by post.webmailer.de
(8.13.7/8.13.7) id n9EHiJni027302; Wed, 14 Oct 2009 17:44:19 GMT
Date: Wed, 14 Oct 2009 17:44:19 +0000
Message-ID: 200910141744.n9EHiJni027302@post.webmailer.de
To: xxxx@qianqin.de
Subject: =?UTF-8?B?6aaW5bGK4oCc5a6d546u5p2v4oCd5YWo5b635Y2O5Lq65LmS5LmT55CD57K+6Iux6LWb?=
From: tub@berlin-chinese.de
X-Priority: 1 (Highest)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="--b4e8df7cda70d7ed5747c5024f440109"
Content-Transfer-Encoding: 7bit
X-Mailer: Php/libMailv1.3
X-RZG-SCRIPT: :dT5KJxL9Na66sT5hK5niAiCdqRuMnl2kI1KGGvqLVsprMVUIT/yqC1ysqg5hWXGkaAECAmag6DeBZrRYoGyG/762eB1wBVfqZFu1a4JstHKESDEs+hJGxpO811H3bA==
Return-Path: postmaster+115767@post.webmailer.de X-MS-Exchange-Organization-PRD: berlin-chinese.de
X-MS-Exchange-Organization-SenderIdResult: None
Received-SPF: None (PURE.qianqin.local: tub@berlin-chinese.de does not
designate permitted sender hosts)
X-MS-Exchange-Organization-SCL: 0
X-MS-Exchange-Organization-PCL: 2
X-MS-Exchange-Organization-Antispam-Report: DV:3.3.8121.600;SID:SenderIDStatus None;OrigIP:

TehPenguin wrote Oct 20, 2009 at 12:56 PM

I can see that you are using "7bit" encoding for the Content (which answers one question)

I'd appreciate seeing the content - but what I really need to know is what characters aren't working (from there I can create a 'dummy' test email)

Please also be aware that it may take a couple of weeks before I can look into this issue (due to exams approaching for University)

TehPenguin wrote Oct 29, 2009 at 11:28 AM

The problematic message has been provided to me from chineseleper. The error appears to be due to various Chinese (Traditional or Simplified?) characters - however they are stored in the email in base 64 encoding. I haven't done any testing with the provided email, but will see if I can reproduce, and fix, the bug over the next couple of weeks

TehPenguin wrote Nov 7, 2009 at 10:20 AM

Looks like this has something to do with Work Item 3099 (http://p3ss.codeplex.com/WorkItem/View.aspx?WorkItemId=3099)

TehPenguin wrote Nov 8, 2009 at 6:05 AM

chineseleper wrote Nov 11, 2009 at 12:24 PM

still not working, now I am getting = throughout the mails. Like this:
------- Forward=d message -------
From: "xxx" xxx@urania-berlin.d=
To: xxx@gmx.de Cc: "Qian Qin" mail@qianqin.de=BR>Subject: Darstellung von Partnerlogos und Links
Date: Thu, 05 Nov=009 17:08:08 +0100
=allo =Jonas,
=ie Darstellung von =Partnerlogos klappt noch nicht immer. So habe ich unter http://www.urania.de/programm/2009/=121 Logobild =und Link eingefügt, die Ansicht aber ist fehlerhaft.
=weite Frage: =Wie kann =man ein 2. Partnerlogo einfügen mit Verlinkungsmöglichkeit? Wäre d=ingend =erforderlich! Grüße Uli


bugmenot2 wrote Oct 12, 2010 at 1:41 PM

i've got the same problem- using outlook, p3ss and ms exchange server.
most problems got passed by changing some registry entries (described here http://support.microsoft.com/kb/916299/en)
still tho, all emails come by as utf-8 and just a few emails (mostly .ru/.cn - where subject has valid cyrrillic letters) have body text full of '?'
p3ss has encoding set to utf-8.

Might there be some way to log the email body access?

bugmenot2 wrote Oct 12, 2010 at 2:07 PM

i've kinda got the same problem
using ms exchange,outlook and p3ss.
emails etc. are coming with utf-8 encoding.
but with few (mostly .ru/.cn) where the "From" is displayed correctly
the body text is full of '?'

forwarding the same email from my email web interface to p3ss/exchange/outlook the bodytext doesnt have the problem.
is there a way to catch all input/output going text of this email or log in/output informations?

bugmenot2 wrote Oct 20, 2010 at 10:55 AM

maybe more detailed infos:
using latest p3ss, with ms exchange server and outlook 2007

info of an email totally full of ??s

Microsoft Mail Internet Headers Version 2.0
Received: from smile ([]) by ........ with Microsoft SMTPSVC(6.0.3790.3959);
 Wed, 13 Oct 2010 10:25:55 +0200
Return-Path: web@billing.domishko.ru
Delivery-Date: Wed, 13 Oct 2010 10:24:51 +0200
Received: from billing.domishko.ru (billing.domishko.ru [])
by .... (node=mxbap0) with ESMTP (Nemesis)
id 0Lnznk-1OUh0l22hs-00fteS for .......; Wed, 13 Oct 2010 10:24:51 +0200
Received: from web by billing.domishko.ru with local (Exim 4.63)
(envelope-from <web@billing.domishko.ru>)
id 1P5wjj-0007qd-2H; Wed, 13 Oct 2010 12:30:59 +0400
To: ........
Subject: =?UTF-8?B?0JTQsNC90L3Ri9C1INC90L7QstC+0LPQviDRhdC+0YHRgtC40L3Qs9C+0LLQ?= =?UTF-8?B?vtCz0L4g0LDQutC60LDRg9C90YLQsA==?=
Date: Wed, 13 Oct 2010 12:30:59 +0400
From: =?UTF-8?B?0JTQvtC80LjRiNC60L4=?= support@domishko.ru
Message-ID: b542880e416d1bbce3caa9df2df20e2d@billing.domishko.ru
X-Priority: 3
X-Mailer: PHPMailer 5.1 (phpmailer.sourceforge.net)
MIME-Version: 1.0
Content-Type: multipart/alternative;
Sender: web@billing.domishko.ru
X-UI-Junk: AutoMaybeJunk +30 (SPA);
Envelope-To: .......
X-OriginalArrivalTime: 13 Oct 2010 08:25:55.0390 (UTC) FILETIME=[4161ADE0:01CB6AB0]

Content-Type: text/plain; charset = "UTF-8"
Content-Transfer-Encoding: 8bit

Content-Type: text/html; charset = "UTF-8"
Content-Transfer-Encoding: 8bit

and an email partly broken:
Microsoft Mail Internet Headers Version 2.0
Received: from .... ([]) by .... with Microsoft SMTPSVC(6.0.3790.3959);
 Wed, 13 Oct 2010 02:10:54 +0200
Return-Path: info@zoonar.com
Delivery-Date: Wed, 13 Oct 2010 02:14:51 +0200
Received: from worker-MAIL-server.irz42.net (Worker-MAIL-Server.irz42.net [])
by ..... (node=mxbap0) with ESMTP (Nemesis)
id 0LgG2e-1OI8gp28uK-00nVl6 for .....; Wed, 13 Oct 2010 02:14:51 +0200
Received: from localhost (localhost [])
by worker.irz42.net (Postfix) with ESMTP id 1E268CC1BA
for <...........>; Wed, 13 Oct 2010 02:14:51 +0200 (CEST)
Received: from worker-MAIL-server.irz42.net ([])
by localhost (worker []) (amavisd-new, port 10024) with ESMTP
id 16771-02-4 for <......>; Wed, 13 Oct 2010 02:14:07 +0200 (CEST)
Received: from localhost (fwgw.zoonar.de [])
by Worker-MAIL-Server.irz42.net (Postfix) with ESMTP id 60D58CC1FB
for <............>; Wed, 13 Oct 2010 01:52:59 +0200 (CEST)
To: ..........
From: Zoonar GmbH info@zoonar.com
Subject: Ihr Bild-Upload bei Zoonar
Content-type: text/plain; charset=utf-8
Message-Id: 20101012235259.60D58CC1FB@worker.irz42.net
Date: Wed, 13 Oct 2010 01:52:59 +0200 (CEST)
X-Virus-Scanned: by amavisd-new at irz42.net
X-UI-Junk: AutoMaybeJunk -96 (BAY);
Envelope-To: .......
X-OriginalArrivalTime: 13 Oct 2010 00:10:54.0265 (UTC) FILETIME=[1A21AA90:01CB6A6B]

i hope this might help to analyse+fix the problem