A Human’s guide to testing AI

What is ‘AI’?

Validate your hypothesis

Algorithm impact on user behavior

A Human’s guide to Machine Intelligence

What is testing AI?

  • Facebook researchers have a hypothesis that social media posts influence user behavior.
  • A software developer notices that businesses are not getting good SEO rankings.
  • A user complains that his email suddenly flags messages as spam.

Unanticipated consequences

Real World Data

The problem with testing AI

  • The start of a test is the question we ask. A question may come up from exploration, experiments or from a problem faced by the user.
  • In general terms, you will need to be creative, as opposed to following a process, to ask the right questions.
  • you ask the question is much more important than who asks or when you ask.
  • Asking the question is more important than what led you to ask the question — whether it is the use of tools or a thought experiment.
  • The examples I have described in this article are definitive, i.e., there is a clear hypothesis followed by an experiment. The actual process of testing (and thinking) is much more fluid and situational. The result of experiments lead to more questions and more experiments. You may also investigate multiple questions. You may use the software, like a user would, to discover more problems. The overarching purpose of testing is to keep learning about the software in order to find unknown risks.

Notes

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store