Small Investment Tax Relief Act of 2004 - Amends the Internal Revenue Code to allow a noncorporate taxpayer a tax exclusion for up to $100,000 of the gain from the sale of exchange of certain publicly traded small business stock held for more than one year.