Voter Protection Act of 2004 - Amends the Federal criminal code to prohibit willfully providing misleading or fraudulent information to an individual and thereby interfering with that individual's voting, qualifying to vote, qualifying or campaigning as a candidate, or qualifying or acting as any legally authorized official in any primary, special, or general election.