For any future readers, I found the problem. The solution was in this thread: You must specify "level=3" in the advanced x264 options. I have no idea what that does, but it now works with the PSP.
Wait, so the PSP works on level 3? I'm going to suggest that MeGUI's PSP profile be changed to that then.