Consolidated Student Loan Reduction Act of 2004 - Amends the Higher Education Act of 1965 to allow borrowers to refinance Federal consolidation loans and Federal direct consolidation loans at the same interest rate as Federal Stafford loans, whenever the interest rate of the respective consolidation loan exceeds one percent of the Stafford loan rate. Prohibits charging a fee to any student for such refinancing.