With regards to CoolBook, did you go through the process of finding the minimum voltage pairs and such? If you just run it without any configuration it won't do anything for the heat.

If you have done this and it's still much hotter than in Windows, I don't really understand what's going on. As I understand it, by lowering the voltage through CoolBook you will necessarily create less heat than the default configuration in Windows. Intel sets their base voltages higher than they could, just to be safe since the lower limit for any given processor is not consistent. By performing stability tests you can find the limit for your processor, which is usually significantly lower. So perhaps CoolBook isn't working properly or maybe there is something to do with your OS X installation that is pushing the processor harder?

Replacing thermal paste is messy business, and it still won't account for the difference between Windows and OS X... so hopefully it won't get there.