Be careful who you blame

Don’t you just hate it when people mistakenly blame Windows and Microsoft for operating system crashes? I do. Most people don’t realize that it’s often the fault of a third-party component such as a kernel-mode driver. It must be rough for Microsoft — or any operating system developer, really — to deal with being blamed for others’ faults.

The other day I caught myself making the exact same mistake, although in a slightly different setting. For quite some time, I’ve been having issues with Outlook 2003/2007. The application would hang for 30-60 seconds at startup and every time I received more than one e-mail. Outlook would always become responsive after a while, though, so I just ignored the problem and did other things during the wait.

This unresponsive behavior was particularly annoying if I was just opening Outlook to look up a particular e-mail and instead had to wait around for the application to load. I would see the same behavior if I received multiple e-mails and tried viewing the first one while the others were still downloading.

I had the great idea to blame Outlook — in specific, the IMAP portions. After all, Microsoft Office is a huge suite and has shown to be buggy in the past. I probably blamed the IMAP portion because the aforementioned behavior first started when I added an IMAP account to Outlook. (This turned out to be a coincidence, though.)

Being fed up at this waste of time, however, I finally decided to take a closer look at what was really doing on. I attached a debugger and after a cursory look at the running threads, I noticed the following thread’s call stack (I’ve highlighted the interesting part in red):

0  Id: e98.f6c Suspend: 1 Teb: 7ffdf000 Unfrozen
ChildEBP RetAddr  Args to Child
002ef868 774d06a0 776177d4 00000508 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
002ef86c 776177d4 00000508 00000000 00000000 ntdll!NtWaitForSingleObject+0xc (FPO: [3,0,0])
002ef8dc 77617742 00000508 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xbe (FPO: [Non-Fpo])
002ef8f0 1190ad11 00000508 ffffffff 1192226c kernel32!WaitForSingleObject+0x12 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
002ef910 119096c8 011992a4 00000000 002ef940 Scanotlk!ExchEntryPoint+0xc18
002ef920 1190a864 02b707c8 011992a4 0119b6d8 Scanotlk+0x96c8
002ef940 3010b112 02b707d4 011992a4 00000001 Scanotlk!ExchEntryPoint+0x76b
002ef964 2feb0bef 0119b60c 0098a3ac 02cc02c4 OUTLOOK!GetCurrentDate+0xac028
002ef984 30131c0c 0119460c 0098a3ac 02cc02c4 OUTLOOK!HrDisplayFolderPickerForOutlookToday+0x4ff65
002ef9b4 30131b47 00000001 083b7f40 00000001 OUTLOOK!GetCurrentDate+0xd2b22
002ef9e8 2faa2b74 00000001 00000000 083b7f40 OUTLOOK!GetCurrentDate+0xd2a5d
002ef9fc 67af64a2 01ca46a4 00000001 083b7f40 OUTLOOK!SmoothScroll+0x2591
002efa1c 76661a10 000d03be 0000040d 00000000 olmapi32!HrThisThreadAdviseSinkEx+0x328
002efa48 76661ae8 67af6443 000d03be 0000040d USER32!InternalCallWinProc+0x23
002efac0 76662a47 003e40ac 67af6443 000d03be USER32!UserCallWinProcCheckWow+0x14b (FPO: [Non-Fpo])
002efb24 76653c8a 67af6443 00000001 002efb44 USER32!DispatchMessageWorker+0x322 (FPO: [Non-Fpo])
002efb34 6945d8c7 002efb5c 002efb5c 002efb7c USER32!DispatchMessageA+0xf (FPO: [Non-Fpo])
002efb44 2fa62ec2 002efb5c 00000000 3041aa00 mso!Ordinal4661+0x399
002efb7c 2fa62dab 00000000 00000000 00000000 OUTLOOK!GetCentralObject+0x16ed4
002efba4 2f9a51cd 2f9a0000 00000000 003a23ac OUTLOOK!GetCentralObject+0x16dbd

As the output above shows, Outlook calls into the Scanotlk module and then waits for something. Astute readers might notice that the thread being blocked on the wait is actually the main thread of execution, i.e. the UI thread. Ah a! That explained why the UI was hanging.

The Scanotlk turns out to be a part of McAfee VirusScan Enterprise:

0:033> lm mScanotlk
start    end        module name
11900000 11942000   Scanotlk   (export symbols)       C:\Program Files\McAfee\VirusScan Enterprise\Scanotlk.dll

When I disabled the On-Delivery E-mail Scanner and relaunched Outlook, the application stopped hanging on the receipt of e-mail. Problem solved.

This story has an important morale to it, though. I mentioned in the beginning that we software users should really be more careful who we blame when something doesn’t work. Often, it’s not what’s actually right in front of us. In this case, I instinctively blamed Outlook. Who should’ve known that the hang was actually caused by a third-party add-in.

Resources:

• ExchEntryPoint: http://support.microsoft.com/kb/285999