Decrypting Blu-Ray discs is a really confusing process ... I'm still not even sure of all the steps that are involved.  Everything I understand has been cobbled together from posts on the doom9 forums.  While the forums are a great resource, it's not a comprehensive one at times.

I was playing around with aacskeys (from doom9 forums, available in portage), and it managed to decrypt / find the keys / whatever it's doing / work successfully on most of my movies.  I'm not sure how to get them off after that, though, or why that's important yet, but I do know it's a good sign.

For now I'm taking the simple route of using shareware to access my movies.  There's two programs I've used so far to rip my Blu-Rays, AnyDVDHD and MakeMKV.  They are both nice programs with some good features, but MakeMKV is the only one that has a Linux port.

The last time I tried MakeMKV, it couldn't decrypt all my discs, so I had to use my PS3 to rip the ISOs, and then use AnyDVDHD.  This time, though, using the most recent version (1.5.6), it managed to decrypt all of my discs.  I was going through my Blu-Rays to see if it could handle all of them, but I gave up after the 15th one, since it was working on every single one.

While AnyDVDHD will extract the original, unencrypted files to your harddrive, MakeMKV will additionally mux them at the same time into Matroska.  I kinda wish I could still have the originals, but I'm not going to be picky. (Edit: you can, see comments)

So, no real plans after this except to play around and post my results.  I really don't have that much interest in playing with Blu-Rays on Linux other than curiosity.  I don't wanna rip them and stream them to my HTPC just yet since I don't have the storage space, and because my frontend isn't quite as HD-ready as I'd like it to be (I still need to update some software and tweak settings ... lots of testing, meh).

I am going to be looking at some other tools and see if I can get them in portage or our multimedia overlay, which reminds me, I just added MakeMKV to there this morning if someone else wants to try it out.

I found a few bugs in my ripper this weekend, one of them was that I was only storing one possible subtitle type in my Matroska rips.  If a DVD had both VobSub and Closed Captioning, it'd only mux the first one I added.  Fixing it was fun, since it was one of those moments where you open up the code trying to find the reason for it, and you find a big comment labeled "FIXME: Add this feature here."  Heh.  So, now it muxes both, if available.  Woots.

There is still one DVD subtitle format that I am having absolutely zero luck in finding anything about -- English SDH (Subtitled for the Deaf and Hard of hearing).  According to Wikipedia, it's basically closed captioning with color.  I can play / watch / rip closed captioning just fine (watching: mplayer -subcc dvd://, ripping: ccextractor), but not SDH.  And I haven't seen anything that can even play them yet, although in fairness I've only been playing with Linux applications.  And everytime I try to explain to someone what I'm trying to do, they think I'm talking about VobSub subtitles.  Usually I get tired of trying to explain the difference and give up searching.  I could try finding some Windows apps to rip / play them, but if I can't get something in Linux that's scriptable to access them, then it doesn't matter anyway.  So, if someone knows of something ... plz to drop me a line, kthx.

Speaking of subtitles and MPlayer, I've come to the conclusion that MPlayer's support for them is just plain sub-par.  The options to play them back (or force them off) are buggy and inconsistent across the bar.  For example, here's a small roundup:

- Flagging a subtitle track as "default" when muxing a Matroska stream means that, if you turn on subtitles in the viewer, that should be the first one to show up.  It does not mean "these are forced subtitles, so display them automatically."  That's why Matroska has a "forced" tag.  default != forced.  If you're still lost, look at the original audio and video tracks, and you'll see they are also muxed with the "default" flag fipped on.  It's purpose makes more sense with video with multiple audio tracks -- if there's more than one, which one do you play by default?  The one with the "default" flag!  Same principle should apply with subtitles when you turn them on.

- MPlayer can't load Matroska subtitles externally.  You can, if you wish, mux just subtitle streams into a Matroska wrapper (ex: mkvmerge subtitles.{idx,srt} -o subtitles.mks).  But using "mplayer -sub subtitles.mks" won't work.  Bummer.   I understand that in this case, the Matroska stream could contain more than one subtitle stream (VobSubs and CC in my example), and it generally expects just one (-sub subtitles.idx, fex), but still, it'd be a fancy feature.

- MPlayer can't dump CC to SRT, even though it can play them (mplayer -subcc).  Bummer.

- Random rant about -noforcedsub and -nosub and -sub are conflicting / confusing, but too lazy to put together data about it, and it's mostly related to the Matroska one above.

I just had to get that stuff off my chest.   I have faith in MPlayer eventually improving in said areas, and filing bugs would probably be good on my part.  I generally don't deal with subtitles much anyway, so for me it's kind of a "would be nice to have" set of features.  Meaning, I've already worked around the bugs and they don't bother me as much anymore.  I would be curious to get SDH read support though.

I'm starting to notice a general trend here -- I complain a lot about certain issues and bugs in detail, but never go out of my way to report them.  I'm becoming the kind of user that as a developer I totally hate!  Oh noes!

In reality, I like being able to be on both sides of the coin, and I'd have to agree with the assessment of most user complaints I see, that are: the barrier to entry to reporting bugs is too hard.  I could go into detail about that, but I don't really want to, as I don't wanna focus on the negative.  But generally speaking, sometimes it's too much of a hassle to easily report a bug.  If it means me creating yet another user account on a bug tracker or subscribing to a mailing list, I weigh that against the strain of just ignoring or working around the bug.

I am, of course, to blame for my laziness, and I completely understand that developers (such as myself) need a detailed report with contact information along with the ability to quickly index reports.  I wonder if there's some magical middle ground, though, where users who aren't regular bug reporters can just easily report their issues and be on their way.  I know in Gentoo, we tend to use the forums as a poor-man's bugzilla sometimes, and maybe that's one way to do it.  Interesting stuff to think about.  Drive-by bug reporters, kinda thing.  They'll come by once or twice, but not regularly.

Anyway, I can't think of any other interesting DVD stuff I ran into this weekend.  Other than I bought season three of Taxi and it wasn't as entertaining as I remembered it to be.  Oh well.  You win some, you lose some.

The coolest thing is that two of my patches got submitted upstream, one for mplayer, and one for ffmpeg.  In both cases, they needed to be changed a bit, but I'm still happy with the results.  The ffmpeg one was the patch I wrote about previously, to have the LAVF demuxer pull out all the metadata that's in the Matroska container.  That's in there as of revision 19184.  Thanks, aurel.

There's no way currently of cleanly pulling it out of MPlayer for display, though my hack works just fine.  The demuxer for mplayer needs a new function to iterate through all the metadata that's available, and add it to the demuxer info.  Currently, it's only pulling out a few named keys specifically.

Here's a screenshot of how it looks where I'm pulling it out, in this case I'm just using it as an OSD menu screen display.

The actual documentation on how to access the OSD menus and work them is pretty non-existent.  I'll try and write some up and get it submitted when I get a chance.  In the meantime, if you want to see what my menu configuration looks like, have at it.  I haven't cleaned it up at all.

I'm too tired to lay this down in story format, so I'm just gonna dump it out best I can.

The other week, I noticed that a new version of mkvtoolnix had come out (the tool to mux audio/video files into the Matroska container), and it totally flew under my radar. I started playing with it, and noticed some real improvements in speed, with regards to parsing MPEG-2 video. After playing around a bit, I started reading some more of the documentation, and found out about this excellent tagging system that the specification declares.

You can read all the gory details about it here, but basically, when building a Matroska file, you can create an XML file that has global tags that can store pretty much every metadata tag I could ever dream of possibly wanting.

I never really had the itch to pack much metadata into the container up until this time, when I realized just how much factual data I could stuff in there and not depend on the database for. Pretty much the only thing I really cared about was the title. In fact, all I wanted originally was to be able to get MPlayer to display the metadata title that was in the file.

Going off on a tangent here, I poked an open bug I have on MPlayer's bugzilla, and Reimar, an mplayer dev, was kind enough to oblige me once more and updated the code so that I could pull it out. If you're using a recent snapshot (for Gentoo, the 20090530 one has it), you can pull it out using "get_property metadata/title" in slave mode. If you wanted to display it on-screen, you would map a keypress event or LIRC event to this: osd_show_property_text "${metadata/title}". Quotes and all.

Anyway, Reimar added that in for me (thanks, man), so I started poking around with mplayer's features to see what else it could pull out for me. Now that I was going to be storing lots more data in the container, I wanted to be able to pull it out, too.

Jump forward a bit to today, where I woke up this morning and was determined to get it out somehow. My original plan that I had decided on was, since I can't really hack on C code, to just work around the limitation by using a fifo for mplayer. I'll spare you the ugly details, but basically I was going to have an event send a command to an external script that would query the .mkv file for the metadata tag I requested, and send a command back through the pipe to print that out to the screen. Quite a run around.

Well, I've tried before to grep the code of mplayer a bit to see if I could wrap my head around the stuff and see if I could figure it out for myself, but it hasn't worked out real well. I decided to give it another whirl today, though. This time, however, my approach was a little bit different. Normally I would just search for keywords where I *think* mplayer would be doing what I would think it was doing, tinker with the code, recompile it, run it, and see what it does. A really slow process, but sometimes it works. And I really don't mind spending the time on it, either.

This time, I did things a little bit differently. I found a file where I was sure that it was accessing matroska metadata, and I read the entire thing, and took copious notes, explaining to myself the whole time, basically what I thought the purpose of each major element was, trying to figure out the pattern to this. Now, bear in mind, that I'm still learning some C++ myself, and the C syntax is pretty similar (in fact ... I still can hardly tell the difference, myself), so a lot of times I have a vague understanding of what it *might* be doing, but never enough to be sure ... so there is still a lot of guesswork involved.

Anyway, after about 10 hours of going back and forth, making notes, testing code, printing out functions and variables and metadata, I got it figured out. And the final patch is something like 2 lines long, heh. All I did was add one if statement. But, that was enough to get me going, and it solved a nagging issue for me. But, what is far more valuable, is the fact that I've learned how I can go into this code and figure out how to fix things myself. That's gonna really come in handy. I'm sure I won't be submitting patches upstream anytime soon, but if I can get what I want hacked in there, and working, I'll be happy as a clam.

For the record, the problem with the metadata was this: MPlayer has a single key=value pair that it assigns to metadata values with it is parsing it with the libavformat demuxer. That is normally well and good, except in the case of Matroska, the tags can be nested with similar names.

So, for example, say you have two target tags in your matroska container: Collection and Episode. If it were a TV show, let's say it's CHiPs. Great show, btw. Now, in the tagging specification, both of them can have a title. The title for the collection would be CHiPs. If you had a Matroska A/V file that was just one episode, then the title for that would be "Ponch Delivers A Baby on the Disco Floor" or whatever (which really does happen, I kid you not). They key for both of those would be "title", but the values would be different. The LAVF demuxer just overwrites the old value and assigns it to whatever comes last. Kind of a problem.

So all I did was told the demuxer to prefix the key names with whatever the name of the target tag was (Collection, Season, Episode, etc.). That way you can have distinct key value pairs, but they are just more verbose. The metadata property names now are metadata/collection/title instead of metadata/title. Pretty simple, really.

That was the easy part. The second part, I haven't figured out yet -- how to get it out. The metadata is all in a separate object created by the LAVF demuxer, which I don't know enough C to figure out how to access that outside of that class. So, I just hacked it to add it to the metadata myself in a rather ugly, but working fashion. Upstream probably wouldn't be interested in that patch.

Another hard day's work, and I'm still not done. And I've got a lot more to write about it, so I'll just stop here for now. I'm gonna go port the patches to my frontends.

Edit: For reference, the clean version of the patch, a sample XML file of what I would mux in with an episode, and the ugly hack I personally am using to get it all out where my lack of C knowledge is very much publicly exposed.  Note that you have to use -demuxer lavf with mplayer for it to work.

It turns out that the issue with the DVDs from Universal Studios weren't completely related to disc quality after all. I sat down today determined to figure out why everything was out of sync so much, and the answer was that it has been happening in recent versions of mkvmerge. So, it was the Matroska muxer all along. Although, I'd gather, I imagine it's probably the MPEG demuxer when it comes to TV DVDs. Everything with v2.4.x of mkvtoolnix would throw off the sync by -.400 to -.500 ms. I honestly wouldn't mind if everything else wasn't already "standard" at a smaller drift, only -.100.

It turns out, after much poking around, that I haven't even done any ripping since September of last year. And that was the real key. I kept asking myself "what's changed in the past few weeks that suddenly everything is out of sync." Well, I forgot one big item: I added a second mythfrontend. And when I did that, I started watching shows from the ripped library more, and once I started watching things more, I wanted to rip some more shows to keep the recent titles stocked.

Believe it or not, after all this work I've gone through to setup this whole thing, I don't actually *watch* stuff on there very often at all. In fact, it wasn't until I got my second one that I even started using it regularly. About the only times I'd normally watch stuff on my original frontend was I usually watch a cartoon in the morning right before going to work, and I'll sometimes watch an episode while I'm doing something else, or just get tired of watching movies that I'll poke around the episodes. It's not very common, though, which is what explains why I haven't been ripping very often. In fact, one episode I just watched the other day, I originally ripped over a year ago. If it hadn't been for the fact that mkvinfo prints out the exact versions of the Matroska libraries and binaries that it was created with, I probably never would have nailed down that it was a problem with mkvmerge.

It wasn't just the Universal discs that were having problems, it was everything that I'd ripped recently. I did have one show that for some reason was a real anomaly (Perfect Strangers) and it would have the same drift regardless of the version of mkmerge ... and the only other new stuff I had ripped was from Universal, so considering my past issues with them, I figured it was the cause.

So, the simple solution was to go back to the same versions that I was using 5 months ago to rip stuff. For the record, that'd be mkvtoolnix 2.1.0 and libebml 0.7.7. And, everything's back to normal. I'm glad.

And I'm pretty sure about the bug being in mkvmerge, since if I re-encode the video with ffmpeg just by copying it (ffmpeg -i movie.vob -acodec copy -vcodec copy movie.vob) and then mux it, the sync will be just fine, regardless of the version. So, it looks like that ffmpeg does a bit better job than mkvmerge when it comes to poorly authored MPEG streams, which really isn't quite a surprise. Honestly, I'm more impressed actually that ffmpeg can even handle and correct them so well.

The eventual, inevitable future though is that I'll be encoding all my video someday. I'm just not up for it right now, for a couple of reasons. For one, my disk space is too small, so I have to rotate content on a regular basis. Since things come and go so often (relatively speaking), I don't want to spend time encoding stuff if I'm not going to be keeping it long term. The second issue is that I just don't want to wait for it to encode. Both issues will go away, with time, as I get a better computer that can encode much faster, and as I expand my harddrive space.

Speaking of encoding, one thing I finally did this morning was to take a systematic approach at what encoding options work well for me for ripping my DVDs. Since those files, unlike the TV shows, will not get removed anytime soon, they are a prime candidate for encoding to save some space. I know I keep saying that there's no magic bullet for every media source out there, but at least for DVDs, I think I found something that'd work for me. It's not as advanced as most people would use ... in fact, one solution to my many problems was to use old codecs and formats so as to avoid any bugs that may still be present in newer codecs. I settled on using LAVC MPEG4, with no extra options, 2400k for the video, MP2 at 192k for the audio, and AVI. I know, not exciting at all. But, it works great so far, and I've tested it on DVDs from four studios (Fox, Disney, Warner, Paramount) and they all turned out perfect. The MPEG4s only get down to about 42% on average of the original size, which again isn't all that impressive, but for a light set of options, it works great, and most importantly ... the video quality is nice, and I have no complaints. That's a first.

For the record, here's the current command I'm using: ffmpeg -y -i movie.vob -r 30000/1001 -vcodec mpeg4 -ab 192k -b 2400k -acodec mp2 -map 0:0 -map 0.1:0.1 -ac 2 movie.avi

You gotta watch out for those audio streams when using ffmpeg -- the ordering is sometimes flipped, so it would be (for example) starting at 0x83 and ending at 0x80, which means you'd probably get the wrong audio track by default. So I just throw in the default -map commands anyway, so it's easy and quick to edit. I'll eventually just throw in a way to check for the right one in my scripts (pretty simple really -- check for your language and the highest number of channels from the VOB) and that'll be that.

Anyway, I'm pooped, I've been hacking on this all day, but made some real progress. I'm glad I got my old setup back up and working. I tell you what. It was a pain to figure out, but I'm glad I managed to pinpoint the source.

In my setup, I'm once again skipping a few steps to just plain keep this simple, mostly at the cost of space, though even that is hardly anything. As an example, on a 1.5 GB MPEG2, the resulting VobSub file is 1 MB. I can live with that.

Anyway, here's what I have so far ... and I promise to update the Gentoo wiki as soon as I get time to cover this more in detail. The first step is to rip the subtitles from the DVD. To do that you have to calculate the subtitle index, which I won't go into right now. On the DVD I used, "It's the Great Pumpkin, Charlie Brown" it was the first (and only) subtitle track for the movie, so it was pretty easy for me.

In fact, here's the command I used to rip both the movie and the subtitles at once:

$ mencoder dvd://1 -ovc copy -oac copy -vobsubout subtitles -vobsuboutindex 0 -sid 0 -o pumpkin.avi

This created three files: pumpkin.avi (the unencoded MPEG2), subtitles.idx and subtitles.sub.

After that, and this is where I never realized how easy it is, to dump it all into Matroska, you just add the .idx file along with the others you are going to mux.

$ mkvmerge -o pumpkin.mkv pumpkin.avi subtitles.idx

When watching the movie with MPlayer, you can toggle through the possible subtitles with the 'j' key.

I can't believe it was as simple as that.