The Most Valuable Dataset the Government Doesn't Care About
Program evaluation meets modern service delivery
In 2014, the FDA launched the ‘Real Cost’ campaign. An abundance of research had recently shown that preventing youth smoking was the best way to curb the public health costs of tobacco overall. The famous tobacco settlement cases of the late 90s provided consistent funding through taxation from cigarette sales, and combined with federal funding the FDA had launched its Center for Tobacco Products, essentially with the goal of helping America quit tobacco altogether. Fifteen years later, the research that sprung up around that litigation had come to fruition and the facts were clear – nearly 9 out of 10 adults who smoke cigarettes daily first try smoking by age 18, and 99% first try smoking by age ~26. If the FDA could impact youth smoking uptake, then a whole generation could avoid the cataclysmic health effects of long term tobacco use.
The initial campaigns were effective. The high amount of funding, especially from the direct tobacco tax, gave the FDA CTP’s team the ability to build a world class marketing operation from scratch. Most of the work of public health is essentially persuasion marketing, and the same tools and systems that work for Nike work for the FDA, under the banner of a $900 million contract doled out over a few years. When I came into Google Ads to help lead the technical relationship with the FDA in 2015, they were just starting to see good results from the Real Cost campaign. Data at the time and research in general showed those early years campaigns to have contributed to preventing almost 600,000 youth ages 11-19 from starting to smoke nationwide, saving the economy almost $53 billion in smoking-related externalities as well.
The campaign's structure was common for public health. There were control and exposed geographic locations, a variety of creative message testing, in-person and digital advertising mixes and tests, and quite a few longitudinal experiments going on. It takes a lot of time to see whether your gigantic, nationwide public health campaign is having material impact, so the researchers who were contracted to study the data from the campaign had set a roughly five to seven year window for analysis. A year or two to set up the proper experiments with the right set of parties from a representative sample around the country, a couple years to run those studies, and a final year and change for the analysis. Researchers would fly into schools, festivals, and concerts. They would show the ads, ask questions about ad recall, tobacco product consideration, influencers that the FDA was using, and a million other things. Major government studies are quite meticulous, and their goals are well defined from the outset. In this case, the goal was mostly around tobacco use by the popular products at the time – dip tobacco, swisher sweets and other cigarillos knockoffs from convenience stores, and generally cheap cigarette brands. The research about the media the FDA was running, starting in 2014, would take until almost 2020 to complete.
Late in 2015, I was sitting at my desk compiling data reports for the advertising agency I was working with on the campaigns. The data was starting to look strange. There was a new type of product and query set appearing in our campaigns. Juul, vape, e-cigarette, these terms hadn’t appeared before. I started a report that showed the relationship of the advertising to these terms, and it was clear. Regardless of what the message was about, who was in it, or where it was I would see data about this new set of products come back. Surveys, search queries, video comments, all started mentioning these products. We were engaged with the FDAs team directly at that time, so I did the natural thing and started sharing this info directly, in its raw format so the far more qualified researchers could draw their own conclusions.
A couple weeks into doing this, I got two phone calls. One I had predicted, it was one of the many ad agency vendors between myself at Google and the FDA team, upset that we were sharing data without their involvement. Happens. The second I did not predict. It was from a representative in the FDAs Office of the Inspector General (OIG). There is no conceivable moment where getting a call from the OIG is a good sign, and it was not in this case.
The person on the other end of the line explained very patiently to me that I was, in fact, potentially running afoul of two regulations in my earnestness to give the FDA their own data from technical tools they were already paying for. First was that I was technically under the media buying contract, not the parallel analytics contract, so OIG felt that it was not legal for me to be contributing to or influencing the FDAs research vendors in any way that might favor Google. No one else at the FDA looked at this data without our or another ad vendors involvement. But if we had valuable data to contribute, and other media vendors like radio or billboard companies did not, and the FDA or the marketing firms between them and us valued it, it could lead them to buy more of it for the sake of the data rather than media itself, giving us an uncompetitive advantage. Second was more technically straightforward – they had nowhere to put the data. Millions of dollars of ad spend at any given moment creates a lot of data, often in semi-structured formats that require data pipelines to manage. I had been sharing back large amounts of data and the research teams, set up to do very specific, previously-crafted experiments, had no idea what to do with the data I was sending. They literally had no place to store it even if they wanted to. I stopped sharing the reports.
The FDAs campaigns also started to get less effective. While they were running around touting the drop off in youth smoking, vaping had not only overtaken tobacco, it was ripping through the youth population faster than smoking had ever done. As the FDA slowly started new studies for this trend, the slow-rolling of the gears of government led them to pause at critical moments that needed legislating. Early entrants into the market were explicitly marketing at kids, with fruity flavors and poppy packaging. By 2020 when the FDA got around to regulating these products, it was too late. By then, the American Lung Association had already branded the product youth popularity as an epidemic, and started a new foundation to combat the trend. Today, 2.5 million of today's youth under the age of 19 regularly use e-cigarettes of some kind, nearly 10% overall, and over 14% of all high school kids. In 2015 when I joined FDAs efforts and just before the youth epidemic took off, that number was under 9%.
Scott Gottlieb, the FDA commissioner from 2017 to 2019, blamed the rise in digital marketing and companies like Juul having the ability to exploit easy access to teens online. Only me and a few other people knew the bitter irony of that statement, and a couple years later I would move to the Cloud team to pursue this problem directly – namely that by ignoring the real time experiences of the constituents that the FDA was supposedly helping, processes that make sense on paper can have disastrous consequences when no one is given room to budge when new facts arise.
This process by which long-term government thinking has prevented redirecting obvious failures when they become apparent is a feature, not a bug, of our civil service. The amount of work each individual actor in the bureaucracy needs to do to change course once set is astronomical. Even if the entire FDA team agreed with the data and instantly decided to change course, there would have been about a hundred different contracts and job functions disrupted. All it takes is one or two high level actors to reject that pivot, and the entire effort fails.
I wrote earlier about the need for a CDP in the public sector as a way to get to that data, but the cultural change that the civil service has to go through to tackle any systemic issue is quite extreme. In fairness, going from fixed, rule-based systems to dynamic, personalized ones is the same sort of change that every industry is grappling with, it is by no means just a public sector problem. The public health apparatus of the United States, the most well-funded part of the government after Defense, has failed their mission three epidemics in a row – youth vaping, opioids, and COVID-19, and shows no sign of improving. Our lack of keeping up with public needs is restricted to public health either, this same static program rollout & evaluation process impacts all forms of government. This dynamic created issues with the Census, who rejected including digital data as a part of their official counting processes and faced data quality issues in the wake of COVID-19. It impacts the Department of Defense, who always saw digital as a way to share information with youth rather than as valuable data to inform a better recruiting strategy, in favor of nearly decades-long contract cycles. It impacts social services and labor agencies, who’s systems couldn’t handle individual-sized recovery payments during COVID-19, so had to settle for one-size-fits-all Pandemic Unemployment Assistance (PUA) checks. PUA was an amazing program with huge benefits, but also may have lost 20-50% of its total funding to fraud.
Too often, government agencies see digital platforms as nothing more than service delivery tools and places to store press releases. As I argued in the post on a CDP, the rest of the world uses these tools as the critical feedback mechanisms that they are. Scaled feedback, even if far messier than perfectly catered research studies, have the value of time and instant results, allowing our public systems to iterate towards success. We will have to do so anyways, as policies that work one day may not work the next as constituent needs evolve with the modern pace of technological and social progress.
There should absolutely be formal, longitudinal research that guides government and policy action, but there needs to be ‘release valves’ or other mechanisms to change course if the population does. The 500M+ users that our Federal agency websites engage with each month alone provide arguably the most powerful dataset in human history to get feedback on policy programs of any kind. Today, simply keeping the dataset alive is the work of an unheralded team inside GSA ‘Digital Analytics Program’ that deserves to be funded a hundred times over their current rate.
It’s time we saw these decisions to forego using the platforms driving the rest of the world's economy as a policy choice, rather than an unfortunate incident of bygone policymaking processes. Despite what many inside (and outside) government agencies believe, the law is very flexible in how these platforms and data can be used, as it is actually far less intrusive from a PII perspective than traditional research methods. HHS’s TPWA process, like this one for CMS’s use of Facebook for example, is a simple way to account for commercially available data that mirrors other agency processes, a far cry from OIRA reviews.
Agencies have already paid for these platforms, and typically have frameworks for how to manage them. Often the levers of control sit either in central IT organizations directly managing the data systems, or farmed out to vendors and their resellers managing it on behalf of government agencies. In any constituent-facing program, these can serve as valuable ways to check the internal logic of public programs, and help back up agency decisions or redirect them towards impact with nearly instant feedback. It’s time to put down longitudinal survey techniques and turn to managing modern bureaucracy like every other industry does. It’s been too long already.