Peremptory Challenge Act of 2006 - Amends the federal judicial code to require: (1) reassignment to another appropriate judicial officer of any civil or criminal case to be tried in a federal district court, the United States Court of Federal Claims, or a bankruptcy court if all parties on one side file a reassignment request within 20 days (with certain exceptions) after initial assignment of a judicial officer; and (2) the chief judge of the court of appeals for the circuit in which the case is to be tried, or another judge designated by the chief judge, to determine any question arising as to which parties should be grouped together as a side for such purpose.