It seems I found part of the problem. In the log I can see some phone numbers are really "creative" and do contain a lot more than numbers and grouping symbols like spaces or brackets. While many phones are ignoring these, the Passat phone might stop at these characters rather than skipping. In this special case a bunch of my numbers do contain hey C2A0 characters. These are displayed as blank on some screens (like Google web contact display) but ignored on the Passat.
I have no idea where these come from, but I found them all in the log and thus can repair that manually.
Is there a way to modify the number during transfer to simply not transfer any character other than the plus symbol, the space, brackets, dashes and numbers on the fly? And, even better, write an error log to display the phone book they come from plus the changes being done?