Special Olympics Sport and Empowerment Act of 2004 - Authorizes the Secretaries of Education, of State, and of Health and Human Services to award grants to, or enter into contracts or cooperative agreements with, Special Olympics for specified education, international, and health activities, including ones promoting Special Olympics and a greater understanding of contributions to society by individuals with intellectual disabilities both within and outside of the United States.