Grey Towers National Historic Site Act of 2004 - Designates all lands and improvements formerly encompassed within Grey Towers National Historic Landmark in Milford, Pennsylvania, as Grey Towers National Historic Site.
Directs that the Site be administered for specified purposes, including: (1) education, public demonstration projects, and research related to natural resource conservation, management, and use; (2) leadership development within the natural resource professions and the Federal civil service; (3) study and interpretation of the life and works of Gifford Pinchot who was the first Chief of the Forest Service and a major influence in formulating and implementing forest conservation policies in the early 20th century; and (4) protection and enjoyment of the scenic and natural environs.
Requires the Secretary of Agriculture (the Secretary) to administer federally owned lands and interests at the Site and associated lands and improvements outside of the Grey Towers National Historic Landmark within Pike County, Pennsylvania, as components of the National Forest System.
Authorizes the Secretary and the Secretary of the Interior to cooperate in promoting public use of Grey Towers and Delaware Water Gap National Recreation Area and in furthering the administration and purposes for which both areas were designated.
Allows the Secretary to impose fees and charges for admission to and use of facilities on Grey Towers. Requires any monies received by the Forest Service in administering Grey Towers to be deposited into the Grey Towers National Historic Site Fund for support of programs at Grey Towers and any other expenses.