Small Disaster Assistance Act of 2004 - Amends the Robert T. Stafford Disaster Relief and Emergency Assistance Act to authorize the President, after an event that results in major damage to or destruction of 100 homes or fewer, to declare, at the request of a Governor, that a major disaster exists and provide Federal assistance upon determining that such assistance is necessary due to the existence of one or more of the following factors in the affected area: (1) a large population of historically and economically disadvantaged individuals; (2) a median family income level below the national, State, or local average; (3) high unemployment; (4) a large population of uninsured homeowners or renters; (5) a large elderly population; (6) a high concentration of damage; and (7) a high level of trauma, including injuries, deaths, disruption of community services, and emergency needs.