My Favorite Hack

As a longtime Windows developer on a major software project, SuperStatistics (not it's real name, but you can figure it out), sometimes you must do something to work around a "feature" or dare I say "bug" in the operating system. My favorite hack that I am proud of is faking out Windows property sheet activations.

In SuperStatistics, all the analytic dialogs were scriptable and recordable. This allowed someone to perform some actions within SuperStatistics and then after he was done, he could click a button to generate a macro that matched all the actions he had taken. For instance, you could open a Spreadsheet, run a multiple regression with certain outputs, and then click a button to generate a macro that opens a Spreadsheet and runs a multiple regression. You could save this macro. Or, you could create an analytic node in an analytic workflow (called a Workspace), and when running the workflow, the macro would run. It was a beautiful sight to behold and many of our customers found it extremely useful.

However, we found a problem...or, maybe I better say, a customer found a problem. You could create Workspaces of macros that executed that might create thousands of outputs and a macro or workflow might run for minutes or half an hour. What does a user do when he is waiting for something to finish? Well, he browses the web or works on a report or does some other task or browses the web or browses the web. No, he does not browse the web, we'll say he works on a report. So he works on a report in Word. And, as he goes to type in words, all of a sudden SuperStatistics is in the foreground and flashing and doing things. Damn! Did I hit a wrong key? So, the user clicks on Word to bring it to the foreground, and then it goes to the background because SuperStatistics put it in front.

What the %$*@?

How come the "geniuses" in Tulsa didn't see this? Were the developers and QA sleeping? You might think so, but as a developer, I was watching SuperStatistics. I wasn't watching Word. Same for QA.

However, the customer wasn't happy.

Houston, er, Tulsa, we have a problem...

So, how do you fix a low level Windows problem? Well, you give it to Joe. That's what you do. Maybe he'll pull the rabbit out of the hat?

So, I was tasked to fix the problem. It took me a while to diagnose, and I also had help from StackOverflow--that great font of programming wisdom. What I discovered was that these analytic dialogs being created invisibly and invisible through scripting, were nevertheless stealing the focus away from Word or the browser or whatever application was in the foreground. In reality, most of the analytic dialogs weren't really "dialogs" in the truest sense, but they were property sheets with many property pages on each sheet. So, these invisible property sheets were stealing the focus from the foreground application.

After debugging to a very low level, I determined that actually, it was Windows that was doing it. When creating an invisible property sheet, during the creation, Windows would call the SetForegroundWindow() function. What did SetForegroundWindow() do? Well, it brought SuperStatistics to the foreground to show an invisible property sheet. Why would you want to do that? Well, you wouldn't. We didn't want to. The user certainly didn't want to do that. He wanted to browse the web... err.. I mean, he wanted to work on his Word report for management.

So, this "feature" (don't call it a "bug", we are woke here in Tulsa)... so this "feature" gets triggered by calling the PropertySheet() function in Wndows whereby the PropertySheet() function calls SetForegroundWindow() and there is not any parameter that you can possibly change in the call to PropertySheet() that will cause it not to call SetForegroundWindow(). No matter how hard you try, no matter how much positive thinking you employ, or how hard you pray, or how many goats you sacrifice, PropertySheet() is going to call SetForegroundWindow() and bring SuperStatistics to the foreground.

"So, basically, you're telling me we're screwed?"

"Yeah, but that's why you pay me the big bucks..."

This article is called "My Favorite Hack" is it not?

Well, there is a solution out of this and it involves a clever hack...clever, if I do say so myself, and so did our director of engineering.

What allowed me to work around this? Import tables and the fact that PropertySheet() was in one Windows system DLL and SetForegroundWindow() was in another Windows system DLL.

What are import tables and how do the functions being in different DLLs help?

Let me explain...

Windows executables (DLLs and EXEs) have import tables and export tables. Import tables are a list of functions that a Windows executable uses from another DLL. And, an export table is a list of functions that other DLLs can call. What happens when a process starts up, a library like common controls (comctl32.dll) has an import table entry for the SetForegroundWindow() which is in the USER32 library (user32.dll). The USER32 lib has an export entry for SetForegroundWindow(). When the system loads the common controls library, it has to replace the function addresses in the import library with the addresses for functions in the export table of the other library. So, when comctl32.dll is loaded, the system loader looks at the import table, sees an entry for SetForegroundWindow() and in the that entry puts the address of SetForegroundWindow() that it reads from the export table of comctl32.dll. When code within comctl32.dll tries to call SetForegroundWindow(), it reads the address of the function from the import table in comctl32.dll that the system loader set when comctl32.dll was loaded. This address will actually be in the address space of user32.dll.

That's confusing. How does it help?

Well, the import table is just an area in memory that has a bunch of function pointers. If it's just an area in memory, then that area of memory can be changed.

Still not following...

Well, if you're a little clever, in your own code you know when you are going to call PropertySheet() which will call SetForegroundWindow(). What you can do, if you're a little clever and sneaky, is to change the address in the import table to point to some benign function that does nothing. So, when PropertySheet() calls "SetForegroundWindow()" (so it thinks), it really ends up calling "BigDoNothing()" because the function pointer in the import table has been changed from pointing to SetForegroundWindow() to pointing to BigDoNothing(). If you call BigDoNothing(), then SetForegroundWindow() will not be called and the focus won't change. Problem solved.... OH, BUT DON"T FORGET TO RESTORE THE IMPORT TABLE RIGHT AFTER YOU CALL PropertySheet(). Yeah, that would be bad. You don't want to change the function of your whole program... just these invisible property sheets being called through scripting.

And, that is what I did in SuperStatistics. We changed the import table right before creating an invisible property sheet, and restored it back immediately. If the property sheet was going to be made visible, then we didn't change the import table. We didn't change it because in the case of a visible property sheet we wanted it to come to the foreground. We wanted the default Windows behavior. I can't for the life of me understand why Microsoft thought it was necessary to call SetForegroundWindow() on an invisible window, but yet they did. My favorite hack was to get around that default functionality.

So, that is my favorite hack. The StackOverflow thread where I got some answers is here: winapi - Creating invisible and modeless property sheet causes focus change? - Stack Overflow

What's your favorite hack?