Taxpayer-Teacher Protection Act of 2004 - Amends the Higher Education Act of 1965 to: (1) reduce certain special allowance payments to holders of student loans; and (2) provide for additional amounts of student loan forgiveness for certain elementary or secondary school teachers of mathematics, science, or special education.