Strange characters appearing into the words

[contedavid]

When I write a message or a comment and use an apostrophe or accented vowels, everything seems fine at the time,
but after I send it, strange characters appear in their place when I read the message.
For example, instead of the apostrophe appears ’
Does anyone know why?
Thanks.

[TheOldNick]

Does anyone know why?

Yes. I know it. It is a problem that occurred about five years ago. As a German I had the dame problem with äöüß and so on. And of course I know how to solve the problem. But I'm not the Admin and I don't have access to the database.

1.) There are damaged HTML-Entities in the database. That leads to character decoding like "Bühl" for "Bühl" in the messaging system.
2.) The problem you show (and the main problem with, for example, the profile pages) is a bad transposition from Latin-1 (I guess it's ISO-8859-1) to UTF-8 from the database.

A pssible solution could be (as I already said I don't know the database) to use a script like this on the database tables. Of course in a backup copy....

Code: Select all

function universal_entity_repair($text_from_db) {
    if (empty($text_from_db)) return $text_from_db;

    // STEP 1: Catch doubly encoded leftovers (e.g., if à became à over time)
    $text_from_db = str_replace('Ã', 'Ã', $text_from_db);
    $text_from_db = str_replace('Â', 'Â', $text_from_db);
    $text_from_db = str_replace('&Etilde;', '&Etilde;', $text_from_db);

    // STEP 2: Unpack the mutated HTML Entity pairs ("The Packaging")
    // This regex looks for TWO HTML entities directly touching each other.
    // It captures pairs like ü or ³ while leaving single, valid entities like " alone.
    $text = preg_replace_callback('/(&#[0-9]+;|&[a-zA-Z0-9]+;)(&#[0-9]+;|&[a-zA-Z0-9]+;)/', function($matches) {
       
        // Force-decode the specific broken pair back into raw ISO-8859-1 bytes
        // This converts e.g. "ü" into the byte-salad "ü"
        return html_entity_decode($matches[0], ENT_QUOTES, 'ISO-8859-1');
       
    }, $text_from_db);

    // STEP 3: Heal the remaining UTF-8 byte-salad ("The Core")
    // This converts the byte-salad (like "ü" or "’") back into proper UTF-8 ("ü" or "’")
    $final_text = mb_convert_encoding($text, 'UTF-8', 'ISO-8859-1');

    return $final_text;
}

// --- HOW TO RUN IT ON THE BACKUP DB ---
// 1. Loop through your corrupted table (e.g., comments, messages)
// 2. Run the function: $clean_text = universal_entity_repair($row['comment_text']);
// 3. Update the row in the backup database.



1.) Run the script above on a fresh database backup. It will reverse the encoding chain reaction of the last 5 years flawlessly without touching old, intact posts.

2.) Switch the Live Connection: Before restoring the clean backup to the live site, the live server's DB connection script must be updated to enforce UTF-8 (e.g., using utf8mb4 in the PDO DSN connection string or running SET NAMES 'utf8mb4';

HTH

@Longhairfish If you encounter problems with my codes or want to discuss the solution you know where you find my PM box.

[longhairfish]

A few things.

Nothing on the site will have the attention of the IT guys or web page design guys. That is the word I got.

So - that's how it goes. I would suggest only using a-z , A-Z and 0-9. I do it. Annoying. But at least I don't see dumb characters.

[TheOldNick]

Sorry, I thought you were authorized to do that.

Nothing on the site will have the attention of the IT guys or web page design guys. That is the word I got.

Then tell them, it's ok.

But if they shred the database into a mess (yes it was them who did it) and cannot repair it within five years with a few simple lines of code (which took me 30 minutes to develop, even though I could only analyze the bug from the outside) they should rather do a job they are actually capable of. Obviously, they missed their calling as garbage collectors.

I don’t call that a bug, that’s professional incompetence. And a reason to be fired immediately.

No wonder they have those job alerts plastered at the top of the page. They are desperately "seeking Data Scientists, AI Engineers, and DevOps talent to flex their coding chops with cutting-edge tech." But honestly? Before their tech department tries to build artificial intelligence, they should hire someone who understands basic character encoding first. You can’t build a high-tech rocket ship if your foundation is a five-year-old pile of digital trash.

Anyway, I'll stop complaining now. But you just can't help but worry about a platform you've been hanging around on for years.