A couple of weeks ago, I decided that I was going to figure out, once and for all, how best to take MKV formatted fansubs files with soft subtitles and convert them into files with burned-in subtitles so I could play them back on the AppleTV and/or PS3.

By all rights, this should have been easy.  I use mplayer to play back mkv files on my Mac, so I should be able to use mencoder to convert them, right?

There’s one problem with that, though.  Well, two problems.  The first is that mencoder really doesn’t write h.264 files very well.  It’s GREAT if you want to convert video to divx-encoded AVI files, mind you, but I wanted to make .m4v container files.

The second, and larger, problem is this:  Most fansubs use “ass/ssa” style subtitles.  This lets the subtitler do all kinds of neat positioning and graphical effects with the text.  Here’s a VERY simple example.

Note that Mplayer will correctly interpret ass/ssa style subtitles and display them:

Mencoder won’t.  The same frame after running the file through mencoder looks like:

…you see the problem.

I did eventually hit on a solution.  You can run mplayer in a mode that directs its output to a file instead of to the screen.  This makes a huge file, but it gives you what you need to then make proper looking .m4v files that play back on either an Apple TV OR a PS3… and, hell, while I was at it, I decided I’d figure out how to make PSP and iPhone compatible files.

This took a little time, and it’s not for the faint of heart or for anyone who fears the OSX Terminal window.

Here’s how I did it.

First: I needed mplayer and ffmpeg.  There’s an excellent guide available on how to get these on to your mac.

Fair warning:  You need to be comfortable compiling programs from source code.

Then, I spent a couple of weeks trying out different encoding settings, and finally hit on some that seem to be both compatible and fairly high quality.  I wasn’t optimizing for file size, mind you.  I owe Robert Swain a great deal for his tutorials, I would not have been able to figure things out without them.

After that, I wrote a shell script that I could run from inside a directory of .mkv files.  This script iterates through every .mkv file in the directory and encodes two new versions of each:  One for the Apple TV and PS3, and one for the PSP and iPhone.

If you want the script, here it is.  Copy this and paste it into an “mkvconvert.sh”, chmod it to executable, and you’re good to go.

Some warnings:

1. You need to be doing this on a partition that supports files over 4GB.  So it needs to be an HFS+ partition, not FAT32.
2. The PSP conversion assumes that your source has an aspect ratio of 16:9.  4:3 files will be stretched and look terrible.
3. I don’t like spaces in file names, so this converts all spaces in file names to underscores.
4. I have no idea whether this will work on anyone’s mac but my own.
5. This rather heavily relies on the input file’s resolution and frame rates being compatible with the Apple TV, since I don’t monkey around with them.  If you throw a 1920×1080 mkv at this, you’re going to get a 1920×1080 m4v out of it and the AppleTV won’t know what to do with it.
6. I’ve had problems with files that throw a “invalid and inefficient vfw-avi” error, because mplayer drops frame like crazy if it has problems.  If you see these messages, you’re going to have out-of-sync video and audio.  I’m trying to figure out a way to remux files to fix this.

With that said, here’s the script.  If you choose to go down this path, I wish you the best of luck.

#!/bin/bash

# fullfilename = full name including mkv extension

# filename = file name with mkv extension stripped

# m4vfile = file name with new m4v extension

# pspfile = file name with _psp.m4v added

for videofile in *.mkv

do

fullfilename=${videofile// /_}

mv "$videofile" $fullfilename

filename=${fullfilename//.mkv/}

m4vfile=${fullfilename//.mkv/.m4v}

pspfile=${fullfilename//.mkv/_psp.m4v}

echo Input file $fullfilename Output files $m4vfile and $pspfile

mplayer -ass -ao pcm -vo yuv4mpeg $fullfilename

ffmpeg -i stream.yuv -i audiodump.wav -vcodec libx264 -vpre hq -vpre baseline -crf 18 -threads 0 -b 3000k -acodec libfaac -ab 160k -y $m4vfile

ffmpeg -i stream.yuv -i audiodump.wav -vcodec libx264 -vpre hq -vpre main -level 21 -refs 2 -threads 0 -s 480x272 -b 500k -maxrate 768k -bufsize 2000k -ac 2 -ar 48000 -acodec libfaac -ab 128k -y $pspfile

done