A monumental painting of the American West by Emanuel Leutze breaks records while works by Rockwell and Hopper flop