Farming Flexibility Act of 2004 - Amends the Farm Security and Rural Investment Act of 2002, with respect to direct and counter-cyclical payments, to: (1) permit a producer to reduce soybean base acreage by electing to plant such acres with fruits or vegetables for processing; (2) require such election to be made annually; and (3) provide that if a farm's base acreage is recalculated, such fruit or vegetable production shall be considered the same as the planning, prevented planting, or production of soybeans.