Protect American Voters Act of 2004 - Amends the National Voter Registration Act of 1993 to require States to provide notice and an opportunity for review before removing any individual from the official list of eligible voters for elections for Federal office by reason of criminal conviction or mental incapacity.