Importing MovableType into WordPress without comments

Warning: Near-fatal geek levels in the post below.

One of the big problems that made me follow Will in abandoning MovableType in favour of WordPress was the constant onslaught of comment spam. The problem with moving, however, is that if I want to keep continuity, I have to import the content of the old blog into the new blog. That's easy enough -- MT very nicely provides export, and WordPress is elegantly set up to import directly from WordPress -- but the problem is that it comes with all the spam. In the case of the now-defunct, that was 22MB of spam tacked onto the end of a mere 400k of actual content. Now, doubtless there were some real comments in there too, but 99% was spam.

So I wanted to import just the content, not the spam. Unfortunately, there's no nice nice way to do this in either MT or WordPress. So I wrote one, and in case you have this problem yourself, here it is below. Loading a 22MB+ text file into memory and then spitting it back out again tends to cause the average web server to complain, so this solution is deliberately low-memory: it uses an absolute maximum of 30k at a time, usually significantly less, and it spits output directly as it goes.

/** MoveableType export file comment-stripper
  This is a PHP CLI script, i.e. you're supposed to run it like so:
     php.exe this_script_name.php mt_export_file.txt
  It will then output everything to the command line -- this lets you 
  see if it's working. A good idea, once it's working, is to instead
  pipe all the output to a file, e.g. mt_no_comments.txt. Like so:
     php.exe this_script_name.php mt_export_file.txt > mt_no_comments.txt
  You can then upload the resulting file into whatever your 
  new blog software is. In our case, WordPress, which handled it perfectly.

// first argument is file, open it
$filename = $_SERVER['argv'][1];
$handle = fopen($filename, "r");

read every line
  if not incomment
    add prevline2 to filtered
  shift prevline1 to prevline2
  shift current line to prevline1
  store current line
  if currentline is "COMMENT:" then
     set incomment = true
  if currentline is "TITLE:" then
     set incomment = false

// initialize vars
$filtered = "";
$prevline2 = "";
$prevline1 = "";
$currentline = "";
$inComment = false;

// don't do anything if the file didn't open
if ($handle)
	// cycle through the whole file
	while (!feof($handle))
		if ( ! $inComment )
			$filtered .= $prevline2;

		$prevline2 = $prevline1;
		$prevline1 = $currentline;
		$currentline = fgets($handle, 10000);
		if ( strpos($currentline, "COMMENT:" ) !== false )
			$inComment = true;	
		} else if ( strpos($currentline, "TITLE:") !== false ) 
			$inComment = false;			
		echo $filtered;
		$filtered = "";
} else {

	echo "Couldn't open $filename !n";


Having sorted out this problem also means it will be easier to migrate all the remaining MT blogs (dammit, I keep forgetting there are so many of you...).