Geriatricians Loan Forgiveness Act of 2004 - Amends the Public Health Service Act to include each year of fellowship training in geriatric medicine or geriatric psychiatry required in order to obtain a certificate of added qualification in those fields as a year of obligated service under the National Health Service Corps Loan Repayment Program. Authorizes the payment of up to $20,000 on behalf of the individual for each year of obligated service for the repayment of loans.