Veterans Housing Affordability Act of 2004 - Increases the maximum amount of Department of Veterans Affairs-guaranteed home loans to veterans for the purchase or construction of a home from $60,000 to the dollar amount equal to 22.5 percent of the Freddie Mac conforming loan limitation determined under the Federal Home Loan Mortgage Corporation Act, as adjusted for the year involved.