Small Business Intermediary Lending Pilot Program Act of 2005 - Establishes a three-year small business intermediary lending pilot program under which the Small Business Administration (SBA) may make direct loans to eligible nonprofit lending intermediaries for the purpose of making fixed interest rate and mid-size loans ($35,000 to $200,000) to startup, newly-established, and growing small businesses. Authorizes the SBA, under the program, to make one percent, 20-year loans of up to $1 million, on a competitive basis, to up to 20 nonprofit lending intermediaries. Requires geographic diversity and representation of urban and rural communities under the program.