Turns it out it was my router. When I put my desktop in the DMZ, it works flawlessly now.
Weird that on my macbook pro it works all the time regardless but with the desktop it needs to be placed in the DMZ to work or else the router blocks the connection somehow. Both are using the same router to connect to the internet, the macbook wirelessly and the desktop through ethernet.
Prior to placing the desktop in the DMZ, no computer was in it.