The Daily Beast today proclaimed The Cooper Union as the second hardest college in the US, based on several quantitative measures.

http://www.thedailybeast.com/galleries/2014/11/06/20-most-rigorous-colleges-photos.html#c39cab18-a9c8-4bc1-9f7e-0fa8cedd97aa

From the article:

Cooper Union
New York, New York
Smartest Professors Score: 8.8
Percent of Applicants Admitted: 8%
Student to Faculty Ratio: 8:1
Workload Manageability Score (lower=harder): 2.4