Ancestors

Written by Rob Napier on 2024-10-19 at 12:28

Yesterday I finally solved the third hardest discrete bug I’ve worked on in my career. Took about 3.5 weeks of work. At the end, I spent 10 consecutive days doing nothing but investigating this bug, all day, every day. I then took a day off to pick up my folks in Pennsylvania because they wrecked their car (everyone is fine, just car damage). Came back, and yesterday I finally found it.

1/

=> More informations about this toot | More toots from cocoaphony@mastodon.social

Written by Rob Napier on 2024-10-19 at 12:32

The bug was in Android code that worked fine on arm64 and x86, but crashed on x86_64. Heisenbug: adding logs would change its behavior. All the relevant bits were deep in C++ code, 3 layers of modules from the Java, so a debugger is basically impossible. Just logs.

Different modules have to be built on different machines with different build systems and assembled by hacking .so files into the APK by hand. And the results had to be tested on a Windows box for x86_64.

2/

=> More informations about this toot | More toots from cocoaphony@mastodon.social

Written by Rob Napier on 2024-10-19 at 12:36

Logs indicated that on x86_64, even incredibly simple function calls would corrupt their parameters. Local, constant strings would log as garbage, and sometimes just logging them would crash the app.

I figured it was a compiler setting mismatch. Maybe something like Microsoft parameter passing conventions, though clearly it wasn’t that. Checked for weird 64/32-bit mismatches. Maybe an NDK mismatch. Found a mismatch on the version of Android targeted, but that wasn’t it.

3/

=> More informations about this toot | More toots from cocoaphony@mastodon.social

Written by Rob Napier on 2024-10-19 at 12:44

Finally realized the Heisenbug nature. Code that worked stopped working when I added logs.

Tried calling __android_log_print directly rather than using the LOGE macro and the corruption went away.

It was the logger the entire time.

The default Log() function that came with the 3rdparty code passed a va_list to printf(). It has been modified to also pass the va_list to android_log_print. Reusing a va_list is not legal. On x64 it led to crashes.

Quick fix.

/fin

=> More informations about this toot | More toots from cocoaphony@mastodon.social

Toot

Written by Craig Hockenberry on 2024-10-19 at 15:23

@cocoaphony DID YOU TRY UNPLUGGING IT AND PLUGGING IT AGAIN CONGRATS

=> More informations about this toot | More toots from chockenberry@mastodon.social

Descendants