I suggest that we retire the term "affirmative action" due to its extreme amorphousness. Does "affirmative action" mean a strict quota system for Supreme Court justices, or hiring preferences for members of historically disadvantaged groups, or aggressive recruiting by law schools at historically black colleges? I guarantee that each of these (among others) is the meaning of "affirmative action" to at least some. A favorite tactic of dishonest argument is to let the meaning slide back and forth in the course of a single discussion.
The test scores and certificates that proponents of strict meritocracy would have us base decisions on give an illusion of precise measurement. Does anyone seriously think we should choose Supreme Court justices based on the highest SAT score? SAT scores are a crude predictor for some things but an exact predictor of nothing, not even an individual's performance on the next SAT.
My personal view on the issue: All the measurements and data in the world provide only a rough estimate of an individual's likely performance in any job or academic program. In reality there is no "best candidate", only a pool of candidates who are all the "best qualified" given the imperfect state of our knowledge. History shows that predictions of future performance for Supreme Court justices are especially imprecise. Acknowledging the level of our ignorance in such matters yields freedom. Given that there is no uniquely determined "best candidate" why should we not choose from among the best with some secondary goal (such as diversity) in mind? This does not unfairly disadvantage anyone, and I think an honest assessment of our level of ignorance leaves lots of room for encouraging diversity and remedying past injustice. If this is what is meant by "affirmative action", then I'm all in favor.