Without Quartz Extreme supported you will find as you mentioned apps not loading and poor video quality. I have not seen the 6150 SE onboard graphics chipset supported by OSX. You do however have an expansion slot and so could add a compatible graphics card. This would make for a much more reasonable experience.

We can still bring your system up to 10.5.8, Is there a certain application you want to run that requires 10.5.8?

---

MOBO: Z77MX-QUO-AOS CPU: Core i7 3770K GPU: MSI N760 TF 2GD5/OC Case: Modded MacPro2,1
Memory: 32GB Corsair Vengeance (CMY32GX3M4A1600C9) Wifi: Airport Extreme bcm94321MCA BIOS: HermitCrab Labs H3A.816M
Monitor: AOC Q2963Pm 29" WFHD 2560x1080 21:9
Interests: KDE on Apple Darwin, Keeping it real with the command line, Helping those that help themselves