Ok... I was messing with it for a while, and I found the problem. It's partially my fault for not testing things on default settings. I had had "Head Related Transfer Function (virtual surround for headphones)" (hrtf) enabled, with that enabled, none of the AVIs I tried worked.
I did some googleing and found that apparantly hrtf only works for 48 kHz audio. According to
http://www.mplayerhq.hu/DOCS/HTML/en/advaudio.html if the audio source is something other than 48 kHz it needs to be resampled.
I managed to dig up an avi file that had 48 kHz audio, and it played fine. Of course all AVIs played fine if I had hrtf disabled.
So basically some condition needs to be written to take care of that. Maybe enabling that setting could always resamples the audio even if the source is already 48 kHz (easiest thing I can think of).
With that problem to be taken care of, another unrelated issue remains, where when using core video, it is possible to enter full screen only once per session.