Community Restoration and Revitalization Act of 2004 - Amends the Internal Revenue Code to: (1) allow an increased rehabilitation tax credit for certain low-income buildings; (2) allow a basis reduction adjustment for property eligible for the credit; (3) increase the credit for certain smaller buildings; (4) allow property eligible for the credit to be used for lodging purposes; (5) modify placed-in-service rules for credit property; (6) modify qualification rules for credit property that is tax-exempt use property; and (7) increase the credit for buildings in high cost areas.