Agricultural Business Security Tax Credit Act of 2004 - Amends the Internal Revenue Code to allow a retailer of agricultural products or a manufacturer, formulator, or distributor of food-use pesticides a business tax credit for up to 50 percent of the cost of protecting certain hazardous chemicals or food-use pesticides from unauthorized access. Sets an annual limit on such credit of $2 million and a per facility limitation of $50,000 (reduced by credits received for the five prior taxable years).