Criminal Gang Activity Act of 2004 - Amends the Brady Handgun Violence Prevention Act to prohibit anyone who participates in a criminal street gang from possessing firearms or ammunition. Defines such participation as committing, or conspiring or attempting to commit, two or more predicate gang crimes, or inducing or facilitating the commission of a predicate gang crime, in furtherance of gang activities or for the purpose of gang membership or position.