Textbook Affordability Act of 2004 - Amends the Internal Revenue Code to include the cost of books required for enrollment or attendance at an institution of higher education in the tax deduction for qualified tuition and related expenses. Limits the amount of such deduction to $1,000 with a prorated reduction for taxpayers with adjusted gross incomes over $65,000 (adjusted for inflation). Makes the tax deduction for such books permanent.