Wrongly encoded Umlaute in CalDAV

[Nurgotora]

Hi,

in the course of testing EssentialPIM before purchase, I seem to have discovered a bug in the CalDAV synchronisation. Using EssentialPIM Pro 5.54 (30 day trial)) on Windows XP SP3 32bit with a OwnCloud 5.0.6, there's a problem that German Umlaute and the German sz are not transmitted correctly. In this post it was speculated that it might be an OwnCloud-Problem, but digging further it appears that it is EssentialPIM that does not correctly encodes these characters ("ä Ä ö Ö ü Ü ß") in utf-8.

Here's an example (see attached logfile) with an event whose summary/subject is just the German Umlaut "ä".

essentialPim sends: "0x3F,0xA4"
OwnCloud receives: "0x3F,0xA4"
OwnCloud converts to: "0x3F,0xC2,0xA4"
essentialPIM receives: "0x3F,0xC2,0xA4"

The reason for this is that the sequence "0x3F,0xA4" is not valid UTF-8 for the German Umlaut "ä" (should be 0xC3,0xA4, see below), so the php function mb_detect_encoding() which is used by OwnCloud/SabreDAV falsly detects 'ISO-8859-1' and in consequence re-encodes the data with utf8_encode(), leaving the "0x3F" in place an encoding the 0xA4 to "0xC2,0xA4", which it then sends back to essentialPIM.

The same applies to the characters "ö" and "Ö", which are send as "0x3F,0xB6" and "0x3F,0x96" respectivly.
The characters "ü", "Ü", and "ß" are treated differently: they all become become just "0x3F,0x3F" each (which prevents me to patch OwnCloud - no way to differentiate between them).

I hope this meets the standards required for bug reports.

_______________________________________________
German Umlaut "ä", Unicode code point "U+00E4"

Code: Select all

              1110 0100  | 0x4E
          000 1110 0100  | + 3 leading 0-bits to make 11
    110x xxxx 10xx xxxx  | Definiton for 2 bytes, 11 bits
    1100 0011 1010 0100 
         0xC3      0xA4

[Max]

Hello!

Thank you for such a detailed report, which is way above all other reports we have received.
However, the quickest way for us to fix it is to reproduce it. So can you list steps for me to reproduce it with owncloud 5.06 and empty EPIM database?

[Max]

The problem is, we have a similar report, but I can not reproduce it!
In this topic there are my screenshots showing how data flows from EPIM to OwnCloud:
viewtopic.php?f=2&p=59319

[Nurgotora]

Ok, fine Here are steps and Screenshots:

For testing, I am using fresh and clean virtual machines (VmWare) with no other software installed, to minimize side effects.

1. Setup Debian 6.0 as LAMP, no other software.
2. Install OwnCloud 5.0.6
3. Create a new Calendar "TestUmlaut"


1. Install Windows XP Professional 32bit, German, Service Pack 3, get all updates. Install no other software. No screenshots for this, ok? )
2. Install EssentialPIM 5.54, Choose "English" as program language (easier for you, I guess).
3. Configure "Tools" => "Options" => "Synchronisation" => "Enable extensive logging"
4. "File" => "New..." => "EPIM Data File"
5. Save it as "TestUmlaut"
6. Add a synchronisation with

http://Server/owncloud/remote.php/calda ... testumlaut

AddSync.jpg (29.45 KiB) Viewed 8207 times

7. Add an Event with "äÄöÖüÜß" as Subject

AddEvent.jpg (294.26 KiB) Viewed 8207 times

8. Sync (Ctrl+Alt+D)
9. Look at the log: EssentialPIM clearly has sent already wrongly encoded UTF-8 (complete log attached below)

EssentialPIM_log.jpg (36.92 KiB) Viewed 8207 times

You can see that the äÄöÖ sequence is encoded, albeit each character incorrectly with a 0x3F instead of 0xC3 as leading byte, while üÜß has been turned into six "?" (0x3F).

Optional: 10. Look at OwnCloud (I am quite sure that Debian/OwnCloud is not part of the issue as the wrong encoding already appears in EssentialPIM's own log, but I might be mistaken.)

OwnCloud.jpg (27.9 KiB) Viewed 8207 times

If you need more info, let me know. I am more than very interested to get this solved.

[Nurgotora]

Checked with WinVista and Win7 (same OwnCloud server as above): Same issue (see attached logfiles).

Two things: If I can reproduce it at will (on different OSs and different machines) and you cannot at all, (a) may I have your logfile? I'd like to look at it. (b) please have the developers take a closeer look at the server-responses in the your and mine logfiles (especially *before* the iCalendar-data is sent). It's a long shot, but maybe different server responses trigger different encoding-methods? Wouldn't explain ü, Ü, and ß being encoded identically as "??", though...

[Nurgotora]

Interestingly, in the CardDav synchronization EssentialPIM does (with the same server) encode all Umlaute and sz (äÄöÖüÜß) correctly (they show as correct 0xC3-sequences in the logfile). There's hope I'd say

Have a nice weekend!

[Max]

Thank you! In my attempt to reproduce it, can you please tell what do you have setup in Windows 7 for following setting (Language for non-Unicode programs, screenshot attached)

Clipboard01.png (20.11 KiB) Viewed 8185 times

[Nurgotora]

As you wish
Windows 7

Win7.png (24.43 KiB) Viewed 8176 times

Windows Vista:

WinVista.jpg (242.54 KiB) Viewed 8176 times

Windows Xp:

WindowsXp.jpg (151.02 KiB) Viewed 8176 times

[Max]

Thank you very much for so detailed reports! It happens very rarely (unfortunately).
We have fixed the problem and you will see it in the next build.

[Nurgotora]

I am amazed and you've just got a new customer! Thank you very much for the lightening quick responses. Where can I see when the next build is out (and then grab it )?

[Max]

EPIM will give you a notice about update, we also send emails to newsletter subscribers http://www.essentialpim.com/news/newsletter and post on Facebook: https://www.facebook.com/essentialpim

[Nurgotora]

Pity, I had high hopes: Not fixed in 5.55. Actually no change whatsoever. Tested with EssentialPIM Pro Network 5.55 again on Windows XP SP3 German 32bit with OwnCloud 5.0.7. The Umlaute ÄäÖö are still wrongly encoded and Üüß still as "??" each.

(CardDav still encodes everything correctly)

[Nurgotora]

Details:
The epim database

Test_Baker.epim.zip

(209.91 KiB) Downloaded 386 times

contains one event on June 17th, 16:00 which contains the usual suspects:

Code: Select all

Event with Umlaute: ÄäÖöÜüß

As you can see in the synchronizaion logs, the Umlaute are still not encoded correctly.

Test_Baker_CalDav.log

(2.68 KiB) Downloaded 282 times

Please note again, that CardDav does this perfectly correct (in the same installation).

Cheers!

[Max]

HI,

sorry for this. We have just now made a silent update to address this issue. Please download from http://www.essentialpim.com/get-epim and check. The issue should be fixed.

[Nurgotora]

Sorry, but no.

Downloaded EssentialPIMProNet5.exe (Size: 19.485.512, MD5 DFE237D5B78DBD8DA68946C1C708C71F), tested on WindowsXP Professional 32bit, SP3, German as virtual machine in Vmware 9.0. Issue unchanged. The only change I could detect so far, is that the CalDav sync doesn't write logs anymore (and yes, extensive logging is enabled).

Made a double check with EssentialPIMPro5.exe (Size 12.404.904, MD5 4CA76C8DCE75544060F45AE64FA919F7) on WindowsXP Professional 32bit, SP3, German as virtual machine in Vmware 9.0. Same Umlaut issue, no log written.

Made a triple check with EssentialPIMPro5.exe (Size 12.404.904, MD5 4CA76C8DCE75544060F45AE64FA919F7) on a real Windows 7 Professional 64bit, German. Same Umlaut issue, no log written.

You want my VM?