Solved: Mark and The Mystery of The Overly Oriental Text

Today, I received my second Chinese-character text message.  I don't speak Chinese.  These texts are from my mom.  She doesn't speak Chinese either.

  

Now, my mother is awesome for a multitude of reasons, and her texts tend to be as well.  Normally, I wouldn't think twice about a barrage of Chinese characters.  But these Chinese characters weren't just obfuscating some boring old, plain-jane, run-of-the-mill FYI text.  They were obfuscating a mother's love!  And, I think we can all agree. That's just not okay.

Coupling these likely love-laden texts with a history of way too much computer troubleshooting and you have no choice, but to get to the bottom of the issue.  I wasn't about to wait idly by whilst the technological top dogs argued about who's fault it was, let alone muster up a fix.  Besides, whatever fix they would eventually muster probably wouldn't be applied retroactively anyways, meaning while future texts would be saved, this current treasure-trove-of-a-text would be forever lost in translation.  

I wasn't about to allow such a travesty.  Not on my watch.  Errr phone.  Yes, phone.  Which also kind of serves as my watch.

So, here's the crux of the issue:

1)  You can be minding your own business when suddenly someone you know sends you a somewhat lengthy text.  To the sender this text message looks exactly like the text message they (or auto-correct) typed.  To the receiver, said text message appears like a Chinese version of alphabet soup stirred by a kitchen-aid on a rampage.  It is not legible.

2)  It's a recent issue, appearing to have started on the early-side of mid-October, around abouts October 10th.

3)  The issue is widespread and phone-agnostic, meaning it's affecting phones of all types regardless of brand, not just iPhone or Android.

4)  Nearest I can tell, the issue seems to only be happening on AT&T's network.  I have yet to see anyone on another network report this issue.  The only people reporting it seem to be on the AT&T network.  Please, correct me if you see otherwise.

So in summary:  A phone-agnostic, network-specific issue is turning texts from a sender into oriental gibberish for the receiver.  Fun.

After extensive research and troubleshooting.  I have found zee issue!  And not just an issue, but a solution!  It is...get ready for it...an encoding issue!

What does all this mean?  Well, "encoding" is basically the technological equivalent of translation.  All those lovely little characters we type in our texts are actually stored as unique number values under the hood.  These unique number values are then sent across a network, and translated back to their alphabetic, eye-friendly form.  At least, that's how it's supposed to work anyways.

The problem is, in the same way there are a lot of languages in the world, there are also a lot of encoding formats.  How are you storing the number-representation of those characters?  Thankfully, some encoding formats are more common than others.  Two of the most common are UTF-8 and UTF-16 (UTF stands for Universal Transformation Format).  Fun fact: About 90% of websites use UTF-8 encoding, probably including these characters you are reading right now (that's kind of deep...character-encoding-ception?).  

Hmmmm.  So, what would happen if we used UTF-16 when we were supposed to be using UTF-8?  Brace yourself for scary (yet informative) barrages of numbers and characters.

Take this barrage of less than helpful text I recently received:

䁈䁩䀠䁍䁡䁲䁫䀡䀊䁓䁯䁲䁲䁹䀠䁉䀠䁭䁩䁳䁳䁥䁤䀠䁹䁯䁵䁲䀠䁣䁡䁬䁬䀮䀠䁄䁡䁤䀠䁳䁡䁹䁳䀠䁨䁥䀠䁷䁯䁵䁬䁤䀠䁬䁥䁴䀠䁴䁨䁥䀠䁤䁥䁡䁬䁥䁲䀠䁴䁡䁫䁥䀠䁴䁨䁥䀠䁯䁬䁤䀠䁳䁮䁯䁷䀠䁴䁩䁲䁥䁳䀮䀠䁈䁥䀠䁤䁯䁥䁳䁮䀧䁴䀠䁨䁡䁮䁧䀠䁯䁮䀠䁴䁯䀠䁯䁬䁤䀠䁳䁮䁯䁷䀠䁴䁩䁲䁥䁳䀮䀠䀊堼ꜻ䀠䁌䁯䁶䁥䀠䁡䁮䁤䀠䁈䁵䁧䁳䀬䀊䀠䀠䀠䀠䀠䀠䁍䁯䁭

Converted to UTF-16 we get these lovely blocks of encoded goodness:

4048 4069 4020 404D 4061 4072 406B 4021 400A 4053 406F 4072 4072 4079 4020 4049 4020 406D 4069 4073 4073 4065 4064 4020 4079 406F 4075 4072 4020 4063 4061 406C 406C 402E 4020 4044 4061 4064 4020 4073 4061 4079 4073 4020 4068 4065 4020 4077 406F 4075 406C 4064 4020 406C 4065 4074 4020 4074 4068 4065 4020 4064 4065 4061 406C 4065 4072 4020 4074 4061 406B 4065 4020 4074 4068 4065 4020 406F 406C 4064 4020 4073 406E 406F 4077 4020 4074 4069 4072 4065 4073 402E 4020 4048 4065 4020 4064 406F 4065 4073 406E 4027 4074 4020 4068 4061 406E 4067 4020 406F 406E 4020 4074 406F 4020 406F 406C 4064 4020 4073 406E 406F 4077 4020 4074 4069 4072 4065 4073 402E 4020 400A 583C A73B 4020 404C 406F 4076 4065 4020 4061 406E 4064 4020 4048 4075 4067 4073 402C 400A 4020 4020 4020 4020 4020 4020 404D 406F 406D

Now, you'll notice, each of these blocks starts with a "40."  Well, that's a little conspicuous.  Let's nix all those 40s, leaving us with the following:

48 69 20 4D 61 72 6B 21 0A 53 6F 72 72 79 20 49 20 6D 69 73 73 65 64 20 79 6F 75 72 20 63 61 6C 6C 2E 20 44 61 64 20 73 61 79 73 20 68 65 20 77 6F 75 6C 64 20 6C 65 74 20 74 68 65 20 64 65 61 6C 65 72 20 74 61 6B 65 20 74 68 65 20 6F 6C 64 20 73 6E 6F 77 20 74 69 72 65 73 2E 20 48 65 20 64 6F 65 73 6E 27 74 20 68 61 6E 67 20 6F 6E 20 74 6F 20 6F 6C 64 20 73 6E 6F 77 20 74 69 72 65 73 2E 20 0A 0x583C 0xA73B 20 4C 6F 76 65 20 61 6E 64 20 48 75 67 73 2C 0A 20 20 20 20 20 20 4D 6F 6D

Now, the above looks a lot like what could be considered UTF-8 (which remember is the most common encoding format on the web).  Treating this as UTF-8 and translating it back into alphabet world we get something slightly less suspicious.  Not only that, we get this highly-important, world-changing text message!

Hi Mark!

Sorry I missed your call. Dad says he would let the dealer take the old snow tires. He doesn't hang on to old snow tires. 

convertUTF82Char: error1 583C! convertUTF82Char: error1 A73B!  Love and Hugs,

      Mom

(Note:  I'm assuming the "errors" are emojis.)

Hey! That almost looks like a text message!

Now, let's really break it down.  You'll recall what was really mucking everything up was a "40" at the beginning of each character block.  This 40 is actually hexadecimal.  If you convert 40 from hexadecimal to binary (which is what all these numbers are ultimately represented as) what do you get?  Something absurdly simple, a 1 followed by a bunch of 0 (six zeroes to be exact).  That's 1 0 0 0 0 0 0 or 0 1 0 0 0 0 0 0 (to include the full eight bits for convention sake).  

40 (hexadecimal) = 0 1 0 0 0 0 0 0 (binary)

If we change that 1 to a 0, that 40 that was originally mucking everything up just becomes plain old zero.  And zeroes preceding numbers don't muck anything up. 

So what's going on in this big, nasty issue?  Somewhere along the line in AT&T's network an extra 1 is getting tacked on.  That's one bit getting flipped from 0 to 1.  Off to on.

To anyone out there who thinks one person can't make a difference, just remember:  Flipping a lone bit from 0 to 1 is the difference between legible text messages and gobbly-gook.  You matter.  You make a difference.

Now, unfortunately, I cannot change that 1 back to a 0.  That's AT&T's job.  But I can at least show you how to unscramble the gobby-gook into something meaningful until AT&T does fix it.

So what should you do if you receive crazy weird Chinese texts like this from people you know don't speak Chinese?  With the following, you too can save the texting-world with relative ease:

1)  Send the barrage of Chinese characters to yourself as an email (so you can do this on a computer rather than a phone).  On your computer, copy the characters from your email.

2)  Go to this helpful website (that will do all the fancy UTF conversion for you.) and paste the Chinese characters from your email into the big green box.  Hit the big button "Convert."

3)  Scroll down to the box labeled UTF-16, and copy all the characters.

4)  Open a word-processor and paste the UTF-16 characters.  Do a "Find and Replace" operation, finding all the "40"s and replacing them with nothing (essentially deleting them).  Copy the now, 40-less text.

5)  Go back to the super helpful website from earlier and paste the text into the UTF-8 box.  Hit the button "Convert."

6)  Scroll to the top and behold.

Alternatively, you can just pester the sender to shoot you a picture of their text.  That's probably easier.  A whole lot easier.  But, then you wouldn't feel like a cryptic, code-cracking, cyber-junky, which everyone should experience briefly at least once in their lifetime, even if just to read their mom's text message.

Phew!  After all the tech-shenanigans you've been through, you deserve nothing short of a vacation, or at least a good stress-relieving, endorphin-releasing laugh.  Head over to this chuckle-inducing post.  Enjoy!