I've tested all thoes kernels and they're all fine, surely I got kernel panics once in a while but not related to the network.

I guess the next step is to try with some realtek card, borrow one from a friend or just buy it, and if you get kernel panics with that then you can be sure it's not the card/kext.