No Child Left Behind Improvement and Flexibility Act of 2004 - Amends the Elementary and Secondary Education Act of 1965 (ESEA) to revise certain requirements which were added by the No Child Left Behind Act of 2001, including ones relating to: (1) academic assessments and adequate yearly progress; (2) teacher qualifications; (3) special education students; and (4) rural schools.