State Accountability and Identity Fraud Elimination Act of 2004 - Directs the Secretary of Transportation to withhold specified Federal highway funds in increasing percentages starting in FY 2006 from any State that has enacted a law that allows the issuance of an identification card or a driver's license to an alien who is not legally authorized to be in the United States. Provides that funds withheld from apportionment to any State: (1) on or before September 30, 2005, shall remain available until the end of the third fiscal year following the fiscal year for which the funds are authorized to be appropriated; and (2) after September 30, 2007, shall not be available for apportionment to the State.