How Experimentation Helps Us Meet Our Clients’ Needs

When people think about applying for government services, they might imagine a paper-based form—like those you fill out at the DMV or to get a passport. But government is more online than ever before. There are over 5,300 websites hosted by the federal government and thousands more by state, county, tribal, and local governments. Millions of people across the country turn to these pages to find critical information, prepare for a weather emergency, start a business, or access a wide swath of other services. Perhaps most critically, these websites offer an accessible channel to apply for and renew essential benefits. 

Which means: a website can make the difference in putting food on the table, accessing medical treatments, or finding shelter. And every design choice on these sites has the chance to help or hurt clients’ chances of getting benefits. Government administrators, ranging from IT to program departments, ought to consider running pilots or experiments to test the impact of website changes  (vendors who build many of these websites should do this too!). Just like pilot projects are a common practice to test the potential impact of changes in program operations, experiments can illuminate what changes will actually solve problems and assess ways to deliver a more human-centered government service. 

Our test-and-learn approach in the real world

Ten years ago, we launched GetCalFresh, our SNAP application digital assister in California. From the beginning, we had a clear goal: make it as easy as possible to apply for food benefits online. Experimentation and data-driven iteration have been critical tools for realizing this goal. Since GetCalFresh assists with about 80% of online SNAP applications in California, every design, experience, and product choice we make has a significant impact on clients seeking food assistance. With this in mind, we test and learn about client experience and outcomes in order to continuously improve our service and help more people get and keep their benefits. This model has helped us accelerate our understanding of client needs and build evidence for tactics that eliminate or reduce obstacles across the client’s SNAP benefit journey.

Various experiments have helped us get more clients to renew their SNAP benefits. We have long leveraged messaging—via SMS and email—and send clients up to three reminders to submit their SAR 7 (a renewal form clients must submit to keep food benefits). These invitations include a link to quickly and easily submit this form online which in turn increases renewal rates. 60% of clients invited to submit their SAR 7 via our message reminders do so. But we also wanted to know how to encourage the other 40% of people to renew their benefits. Completing the SAR 7 form is critical; nearly 15% of new applications for CalFresh come from people who just lost benefits because they missed the deadline to submit their renewal form. This ultimately creates more work for both applicants and government workers to begin a whole new enrollment process. 

When faced with this challenge, our team started by asking why our current system of reminders made it hard to renew. We conducted interviews with clients and reviewed messages they sent us to learn more. This led us to a key insight: with many other obligations, from caring for loved ones to keeping up with work obligations, not to mention a swath of mismatched reminders from us, the county, and the state, families had a hard time knowing the deadline for this task.

To make this easier, we decided to test a last-minute reminder sent right before the deadline that made it clear that this form was required to continue receiving CalFresh benefits. This final reminder was very effective—it led to a nearly eight-fold increase in submission rates among clients who had not yet submitted their SAR 7 at the time of the experiment (from 1.5% to ~12%). We’ve now scaled this last-minute reminder to every client who needs to renew, leading to an estimated 315 people renewing their benefits each month who would not have otherwise. That’s thousands more families and tens of thousands of dollars in food benefits each year that we helped deliver through a practice of continuous experimentation. 

Want to learn more about some of our texting experiments? Our Texting Playbook includes learnings and best practices of sending reminder messages to CalFresh clients about required steps in the enrollment process.

Knowing what to test

Experimentation can help improve a digital benefits experience across a variety of outcomes: it can help more people successfully submit an application, renew benefits, upload documents, report income accurately, make their interview time, and much more. But how do we know what to test? To identify fruitful avenues for experimentation, we start by looking across the client application and renewal journey for inspiration. This means asking questions like:

• How does the benefits funnel function? How many people start, complete, get approved for, and then renew benefits? Are there any places where many clients drop-off between one step and the next? Does success vary by device type, language preference, geographical location, household size, or income level?
• What common problems do clients report to call centers? Are there particular aspects of the process where clients need extra support (for example: rescheduling interviews)?
• What are clients and eligibility workers telling us directly? By asking for and using feedback people provide us, we can yield interesting insights into parts of the benefits journey that tend to be trickiest for them to navigate.

Ensuring a test is worth the potential risk

When designing a public service, unlike in the private sector, there are no “edge cases” which means no one should be left behind by a design change to a government service. In a space where reliability and consistency are top concerns for meeting critical needs, change can be difficult and governments can be understandably hesitant to experiment. 

We often hear from partners that experimentation is risky; changes might make a process worse, cause a headache for staff to adjust to a new process, or cost valuable resources. These fears are valid, and the best way to de-risk them is through small-scale experiments. Rolling out something to a small group of clients and comparing the results to those who didn’t see the intervention can help validate the idea before the problem is too entrenched. It can also help collect valuable data to assess the cost and benefit and make a strong case to invest more resources. After all, the status quo is a choice, too, and might leave opportunities unrealized to get more people the benefits they need, reduce operational costs, or create a better workflow for staff. 

We  often partner with government to mitigate the risks of experimentation, using tactics like:

• Setting a “stopping condition.” Our first rule of thumb is to do no harm. Generally, experiments should run until we hit the necessary sample size to draw conclusions from the data. But in a government context dealing with critical benefits, an experiment not working might cause real harm for clients. During the planning phase, we decide on a metric that points to client harm. We then monitor the experiment data regularly and stop the intervention if it’s leading to a harmful outcome. 
• Picking a clear success metric up front. Whenever we change something, we have a clear and shared understanding of why we’re doing it in the first place and what outcome we’d like to achieve. Then we make sure we have a way to know if the desired change took place. For example, if our goal is to reduce churn in a program, we might look at the number of people who submit a renewal form or the number of people who still receive benefits six months after application. We would look at the same data points again after the change is implemented, comparing the treatment group and control group, to assess whether or not our test achieved the desired outcome.
• Measuring secondary metrics to understand the holistic impact of the intervention. Occasionally, changes may improve the outcome of interest, but cause harm in another area. For example, if we adjust the way we ask about income in a benefit application to get more people to submit an application, we might examine conversion as our primary metric, but might also look into the likelihood that clients upload income documents. If application submission improves, but fewer clients upload documents, stakeholders may have to make a tradeoff of what matters most.

Experiments can lead to long-term change

The end goal of our experimentation work is not only to help clients who are utilizing our services, but also support long-term improvements to government digital products and services. Experimentation is a powerful tool to deliver more effective and compassionate government services that keep improving over time. A test-and-learn approach builds in feedback loops that help government agencies and service providers know they’re on the right track, putting data at the forefront of all important decision making processes. Embracing small-scale experiments, with clear success metrics, drives an overall climate of innovation—leading to a more responsive, client-centered experience that connects everyone to the critical government services they need.

Are you a state government official interested in partnering with us to strengthen the safety net? Get in touch.