Sampler Chatter: Meager Data & Big Declarations in the Art World

Earlier today, specialist insurer Hiscox released a report on the state of the online art market (downloadable in full here). Its headline findings look monumental at first. Hiscox estimates that internet-based fine art sales eclipsed the $1B mark for the first time in 2013, cresting at $1.57B; and that based on this result, the industry’s online sector could reach $3.76B in sales by 2018. Surely, happy times are here for art e-tailers… right? 

The answer is a resounding, chest-thumping “Uh… probably, I guess.” As with so much else in the art world, it all depends on which telescope you’re using to view the landscape. 

The uncertainty revolves less around the general outlook of online sales than their specific magnitude of growth. As Coline Milliard of artnet pointed out here, Hiscox’s results vary dramatically from those of March’s more comprehensive TEFAF Art Market Report. How dramatically? In its online sales portion (available here), lead author Clare McAndrew estimates that online art sales accounted for $3.57B last year, i.e. 227% of Hiscox’s reported 2013 total. I’m not a statistician, but my antennae usually go up when the delta between two findings - in this case, +/-$2B - is larger than one of the two figures being compared in the first place. Also worrisome is the fact that TEFAF’s 2013 estimated online sales total is just $200,000 shy of Hiscox’s projection for where the online market will ascend in five years.

This begs a couple of obvious questions: How can two reports end in a discrepancy this mammoth? And what does it mean for how we should judge the data overall? 

The answer to the first question comes down to sample size. In the appendix to their report, Hiscox notes that…

“The survey findings are based on responses from 506 art buyers surveyed through ArtTactic’s client mailing list, Twitter, and Facebook." 

ArtTactic is, according to their own website, a subscription-based "progressive art market analysis firm” that pulls its data from “all aspects of the art ecosystem,” i.e. more than just auction records. Today is the first I’d heard of them. So apparently they granted Hiscox access to their mailing list, and afterward Hiscox took to social media to round out their results. Later in the appendix, it’s revealed that those 506 clients frequented a total of only 23 online art retailers. That maths out to 22 clients from each individual source. 

If those numbers didn’t start to make you squirm in your seat a little, then there’s this:

“Although the central focus is around fine art, this survey also explored online buyers’ buying habits of other collectibles.”

So in summary, Hiscox’s findings in a study titled “The Hiscox Online Art Trade Report 2014” (emphasis added by me) are based on survey responses from a grand total of 506 clients who frequented only 23 online dealers… and some unspecified number of those 506 people did not even acquire objects that qualify as fine art. From this paltry sample size, the firm is extrapolating about a multi-billion dollar sub-industry. I don’t think I’m being unfair if I suggest that methodology is less than bombproof. (Credit to Melanie Gerlis of The Art Newspaper, the only Hiscox recapper I saw go out of her way to mention some of the caveats here.)

Walead Beshty, “FedEx Large Box” (2005). Image credit: Tiki Chris via Flickr.

With all those flaws in mind, it seems like a given that the TEFAF report must be a more accurate view of the online art market - especially if I mention that its generally regarded as the most rigorous of the annual reports that materialize in the art world. The simplest way to determine if it’s indeed the case would be to compare sample sizes.

So how does the TEFAF report stack up? The answer appears to be “favorably,” though the comparison is not necessarily 1:1, either. For starters, McAndrew and her team surveyed dealers, not clients. Since each of those dealers are selling to multiple clients, retrieving data from one dealer should be a vastly more worthwhile proposition than from one collector. The TEFAF report approached 5,500 art dealers to compile their report. Sounds pretty good, right? 

But a closer look at the TEFAF report’s methodology begins to reveal a few hair line fractures of its own. It turns out that all of those 5,500 dealers were in the same fiscal tier: annual sales greater than 2M EUR. More disappointing, the response rate from those 5,500 dealers was only about 12%, meaning the report’s findings were actually generated from only about 660 dealers. That number still dwarfs the 23 online dealers and 506 total clients represented in the Hiscox report. Yet the big picture precision of the TEFAF report starts to crumble even further in light of their estimate that in 2013, the industry consisted of a staggering 295,000 art dealers worldwide. 

Now, it’s important to keep mind that there is a ton of variance among those estimated 295,000 dealers. Theoretically, it includes literally anyone who sold any type of fine art or antiquities at any price point anywhere in the world. Still, if that estimate is accurate, then the TEFAF report only attempted to survey about 2% of the industry (5,500 of 295,000), and only about 0.2% of the overall market responded. From that sample size, the report drew conclusions about an estimated $66B industry. 

I don’t put these facts in halogen high beams to scold McAndrew or her results. Trying to quantify a (mostly) closed market is a nearly impossible task, and the TEFAF report is, as far as I can tell, the best summary we have for sales data in the private sectors of the art world year after year. It’s remarkable to me that she and her team were able to accumulate as much data as they did. If asked to choose, I would favor the TEFAF report’s results on the online art industry over Hiscox’s as quickly as my fast-twitch muscle fiber would allow.

But when looking at the projected art sales numbers blazing across media headlines like neon comets week after week, it’s crucial to remember just how little hard data is propelling them. The most rigorous reports likely do reflect general trends in the market, but relying on the specific numbers they tout is a dubious proposition. Ironically, when viewed with an objective eye, even the best findings succeed at nothing so well as reminding us just how little we know for sure about this industry’s past, present, or future.