From Cauleen Smith’s feminist films at the Whitney to Rodney McMillian’s conceptual landscapes at Petzel Gallery