git filter-branch --tree-filter 'rm filename' HEAD
From 0f7b5a99cef87c294a3c3591b4990ca4077b0d0a Mon Sep 17 00:00:00 2001
From: Junio C Hamano
git filter-branch --tree-filter 'rm filename' HEAD
However, if the file is absent from the tree of some commit, +a simple rm filename will fail for that tree and commit. +Thus you may instead want to use rm -f filename as the script.
A significantly faster version: