Higher Education Sustainability Act of 2004 - Amends the Higher Education Act of 1965 to direct the Secretary of Education to make grants to eligible institutions of higher education to establish university sustainability centers to develop and implement integrated environmental, economic, and social sustainability programs through administrative and operational practices as well as multidisciplinary research, education, and outreach.