Our eyes tend to take a lot of shortcuts. This is why, for example, in the Müller-Lyer illusion below, the upper horizontal bar looks longer than the lower one (they’re actually the same length). These kinds of illusions can be thought of as “adversarial examples” for our visual system. They exploit the shortcuts we take to measure distances, perceive depth or infer colors (among others), often revealing something about how our vision works.

What would an adversarial example look like for a data visualization? Could there be configurations of bars in a bar chart that could distort our perception of the mean or range?

If we could find such examples, they could show us what kinds of shortcuts we take when reading charts. In the field of Data Visualization, we call these shortcuts “perceptual proxies,” and there is a lot of interest in figuring out which ones we use. Aside from learning more about what makes charts so powerful, this knowledge could also lead to chart designs that are more effective and more robust to misleading edge cases.

### Creating adversarial examples

How do we find such adversarial examples for visualizations? In our first
approach, we start with a list of educated guesses for what proxies our visual
system might use, and use those to try to create them. Below is an example for convex-hull-area as a proxy for the mean of
a bar chart. This is like the area filling a rubber band wrapped around the
bars. We use simulated annealing to create pairs of charts where the chart
with the *larger* mean has a *smaller* convex-hull-area.

Now we can ask people which one has the larger mean and see if they get it wrong, potentially catching the visual system in the act of taking its shortcut. In our experiments the most adversarial charts were based on a “centroid” proxy, like the charts below. The bars in the yellow charts below have centroids further to the right than the blue ones, though the blue charts have higher means.

### Human-guided optimization

In those experiments, we had to guess at proxies that people might be
using. But what if we could just let the visual system show us what an
adversarial chart looks like? We can do this by treating how big a chart’s
mean *appears* as a black-box function that we are optimizing. There are
many methods to optimize unknown functions, but here we use Dueling Bandit
Gradient Descent, which has the added
benefit of needing only ranked comparisons.

The algorithm begins with an *exploration* phase, in which we take a random
step from the previous iteration (with the first iteration initialized randomly)
and project to the constraint space, which in this case returns the mean to
the same value.

Next we need to rank the previous point against this new point
with our “function,” which, remember, is how large the mean appears to the human
visual system. Since both charts actually have the same mean, we get this
ranking by inserting the pair into a list of other pairs that *do* have different
means. We show all of these pairs to participants, asking them to pick the chart
with the larger mean for each, and leaving them (hopefully) none the wiser to
the misdirection.

For the final, *exploitation* phase, we take a step towards the exploratory
point if and only if was ranked higher, letting us ascend the hypothetical
gradient of our function (if the new point was not ranked higher, we can do
nothing, although in our implementation, we actually take a small step in the
opposite direction). The algorithm then repeats for a number of iterations or
until convergence.

Below are some (cherry-picked) results. These are charts that started as random data and were optimized (in theory, at least) purely by human judgment.

Many of them look a lot like the adversarial charts with high centroid proxies from the previous experiment: mixtures of very short and very long bars. Although more investigation of the results of both experiments is needed, they provide elegantly complementary evidence that the centroid proxy is at least on the right track.

#### Face to Face: Evaluating Visual Comparison

**Brian Ondov**, Nicole Jardine, Niklas Elmqvist, Steven Franconeri

pdf • doi

#### The Perceptual Proxies of Visual Comparison [Hon. Mention for Best Paper]

Nicole Jardine,

**Brian Ondov**, Niklas Elmqvist, Steven Franconeri

pdf • doi