Dealing with mach_kernel in Shark

Sometimes when profiling a bunch of time ends up in mach_kernel. Figuring out why isn't always easy but here are two tips that should help a bit:

• You can get better symbols for mach_kernel by downloading a KernelDebugKit
  This can help a bit when trying to figure out what's happening in the kernel. For example, _dtrace_get_cpu_int_stack_t becomes _mach_call_munger.
  


• Shark has a System Trace profiling mode. This can show you what code is causing the kernel to do work. It can break down time by system call or by vm fault which should account for most things.
  
  While trying this out I noticed we were spending a fair amount of time in ChildViewMouseTracker::WindowForEvent(NSEvent*). This gave me the idea that the reason that Firefox causes the WindowServer process to start using a huge amount of CPU is because we tell the WindowServer to give us all of the mouse events instead of the ones only targeted at our window. Presumably this causes the WindowServer to build up a very large queue of events when the Firefox process is stopped and thus use lots of CPU. This turns out to be the case. nsToolkit::RegisterForAllProcessMouseEvents causes us to listen to all mouse events and disabling the code there fixes the problem. Bug 611068 tracks the problem.