A couple of months ago, I was doing some re-encodes for Extra Curricular Lesson with Hideaki Anno and I used mkvtoolnix to wrap the video, audio and subtitles streams into an mkv. The first thing that I do afterwards is to check the file by playing it in mplayer, pretty much my video player/encoder of choice. I notice that mplayer just pukes, I just get stuff like this over and over again:
Eventually, mplayer gives up. This worried me a little, having used mkvtoolnix hundreds of times over the years to wrap my own rips, encodes, and re-encodes, I felt that I was doing everything right, but now I'm left poking random options that I've never needed to use before to see why mplayer won't play an mkv created by mkvmerge when it played one made by the same program maybe a month before. Randomly, I noticed that when I use the file command on the mkv, it just says "data" instead of the usual "Matroska data". I also note that CCCP's Media Player Classic seemed to be able to play it fine, along with VLC but I really wanted it to work under mplayer. The version of mplayer that I have compiled is 1.0 rc4 p20091026-r1, which is kind of old, I guess, so I went into /etc/portage/package.keywords and unmasked p20100506 and p20100612 and tried each of those. Both of the newer versions gave me an unrelated (maybe?) error, "double free or corruption (!prev)" which seemed to have something to do with the stylized subtitles. I also tried unmasking newer versions of mkvtoolnix (I had 2.9.9 installed, which was the oldest version in the portage tree) but none of that helped either. I was running out of ideas.
This is around the time that I stumbled on libMatroska. I noted that in my emerge.log, libMatroska was updated from version 0.8.1 to 0.9 in July. I added a mask in /etc/portage/package.mask for media-libs/libmatroska-0.9.0 and recompiled libMatroska. Incidentally, libebml (an extensible binary format library) also got downgraded from 0.8 to 0.7.8. So now that libMatroska was back at 0.8.1 and libebml at 0.7.8, I needed to rebuild mkvtoolnix or else it would complain about the size of symbols in shared objects not jiving. I reran mkvmerge and the resulting mkv played in mplayer (as well as in MPC and VLC). Mission accomplished.
It may be that some USE flags that I had turned on (or off) caused this weird thing to happen between libMatroska and mplayer. At some point I may have to face the music and fuss with it some more because I'm sure at some point, I'm going to need to update libMatroska and libebml, and I'm going to want the mkv's that I make using mkvtoolnix to play in mplayer. It may even be the case that other people using these tools and mplayer will have no problems. But eventhough this problem cropped up last July and I found the fix for it, I decided to make a post about it now almost 3 months later because I was re-wrapping something into an mkv because I didn't like the a52 audio stream and I ran into mplayer puking on me again, and I was reminded that a few months ago, I fucked around with some libraries to get this to work. Also, recently, I noticed a few other people complain about this issue when an anime fansub group released a couple of episodes of an anime that wouldn't play in mplayer. It was the same problem, mplayer puking about the nal size and how it can't decode frames. Re-wrapping it using the older versions of libMatroska and libebml fixed it. A lot of people thought that it was the actual encoding of the video, when it was really just the matroska wrapper.
Let's hope this just gets magically fixed soon so I can not think about it, ever again.
Update (12/25/2010): Looks like it magically got fixed. I randomly decided to try out the SVN-r32624-4.3.4 (November 2010) build of mplayer and it seems to be able to play these mkv's now.