Deficit Accountability Act of 2004 - Amends the Legislative Reorganization Act of 1946 to provide that pay for Members of Congress may not be increased by any adjustment scheduled to take effect in a calendar year if the Office of Management and Budget determines there was a deficit in the preceding fiscal year.