Getting Your Data Ready for AI

Government agencies are increasingly exploring how to integrate AI into their systems—and the options for use are growing. From using large language models (LLMs) to assist with research experiments to experimenting with chatbots to help answer questions about benefits, there’s a lot of potential for reducing administrative burden, simplifying processes for agencies, and improving the customer experience. 

But the transition from traditionally rule-based systems to AI-powered ones isn’t always straightforward. For AI to work efficiently and responsibly in government applications, the data systems that feed these models need to be properly prepared. We’ve put together four steps to ensure that government data systems are AI-ready before deployment.

Ensure data quality and consistency

AI is only as good as the data it works with. These models rely heavily on data quality to deliver accurate results. For government agencies, the data used in AI models comes from various sources, ranging from public records to internal reports. Ensuring this data is consistent, clean, and reliable is essential for AI to function correctly. Here’s how to get started:

• Audit data sources: Agencies can begin by identifying and documenting all the data sources that an AI model will pull from. This could include public datasets, legacy databases, and real-time feeds. Make sure that the data is complete, accurate, and up-to-date and include documentation of who produces and owns the data. This information will come in handy if there are questions about the data throughout the AI modeling process.
• Standardize formats: Government data often comes in a variety of formats—spreadsheets, PDFs, or outdated legacy systems. Decide on a standard data format to help ensure consistency and determine a process to convert data—common standards like JSON or CSV allow for easier processing.
• Clean data: This step can include resolving incomplete or incorrect data—such as removing duplicates or determining a solution for missing values and incorrect data. Establish a routine to maintain clean data as it flows into the system.

If AI systems are fed inconsistent or incomplete data, they’re more likely to produce inaccurate or biased results. Ensuring high-quality data is a foundational step in building trust and ensuring the integrity of government AI applications.

Ensuring high-quality data is a foundational step in building trust and ensuring the integrity of government AI applications.

Establish data privacy and security protocols

When considering AI implementation, data privacy and security should be top of mind. AI systems require vast amounts of data, some of which may contain people’s sensitive personal information. So before deploying AI, agencies have to do the work to ensure their data systems are secure.

• Conduct a data privacy assessment: Start by reviewing the types of data the AI system will access, and identify sources of sensitive or confidential data. This could include financial information, or personally identifiable information (PII) like addresses, Social Security numbers, dates of birth, and biometric data like fingerprints. Create documentation around these fields so they can comply with regulations.
• Ensure compliance with legal standards: Ensure that all data storage and processing practices comply with regulations, such as California Consumer Privacy Act or the EU’s General Data Protection Regulation. 
• Implement data encryption: Use encryption both for data at rest and in transit. Choose the appropriate encryption protocol for each stage of the pipeline, as that method that works for data storage isn’t the same as the protocol used for data transmission. This ensures that data is protected from unauthorized access during collection, processing, and storage.
• Limit access with role-based permissions: Ensure only authorized personnel have access to sensitive data by implementing role-based access controls (RBAC). We suggest strict access controls based on the principle of least privilege, meaning that users only have the absolute minimum level of access needed to perform their job functions. Consider implementing multifactor authentication for additional security.

AI systems that handle government data must be secure in order to maintain public trust. Some best practices around AI and data privacy can be found in the NIST AI Risk Management Framework Playbook.

Want to brush up on some common terms we use when talking about AI? Check out our cheat sheet for AI in government.

Build AI-ready infrastructure

For AI to produce meaningful outputs, the underlying infrastructure—both hardware and software—must be capable of supporting its computational demands. AI systems often require significant processing power and storage, which could be a challenge for some legacy systems that states are using. It’s worth evaluating the current state of technical infrastructure to see if AI can function well with reasonable processing times in your context. 

• Evaluate current infrastructure: This can start with asking some basic questions about the existing IT infrastructure. Are your current servers and data storage solutions capable of handling large AI models and datasets? Is your data in the right format to be ingested? What are the terms of usage of your data for use in an AI model? 
• Consider cloud providers: Services such as Amazon Web Services, Google Cloud, and Microsoft Azure could be good options for governments looking to scale their data storage capacities and leverage their infrastructure to get started on using AI through pre-installed models, APIs and platforms.

Without the proper infrastructure, AI systems may suffer from delays or fail to work in the ways you want them to. Thinking ahead about an infrastructure plan to support AI is one of the best ways to ensure that any new technology implemented will have a chance of succeeding.

Thinking ahead about an infrastructure plan to support AI is one of the best ways to ensure that any new technology implemented will have a chance of succeeding.

Implement ethical guidelines and transparency measures

There are a lot of justifiable concerns about the use of AI in government. There’s potential for algorithmic bias and lack of transparency or opaque decision-making processes that can lead to unfair outcomes. Government should proactively put ethical guidelines in place to ensure AI is used responsibly and ethically.

• • Establish ethical standards: Create a set of ethical guidelines that will govern your AI use. These should include taking a human-centered approach, prioritizing safety, and employing a deep sense of responsibility. (Check out our principles for the responsible use of AI to see how we’re thinking about this problem.) 
  • Make sure there’s human oversight: This means establishing “human-in-the-loop” processes that ensure there’s human oversight for critical decisions. 
  • Provide transparency to the public: Government should indicate when it is using AI in service delivery and provide clear explanations for how and why AI is being implemented. It may help to foster open discussions about AI deployment through public engagement forums and feedback channels. 

• Continually monitor model output: Use automated monitoring tools to flag anomalies or unexpected results. Set up dashboards to track metrics of interest, such as accuracy, false positives, or demographic differences. 

• Create a process for regular system audits: To make sure AI systems are functioning as intended, audits should focus on detecting biases, verifying outputs, and ensuring compliance with ethical standards. These can be conducted by third-party groups. 

Implementing AI responsibly can help tailor government services to the specific needs of the people government serves, and ethical AI practices are crucial to avoiding harm and ensuring that AI systems serve people fairly. 

Getting ready for AI is possible

Getting government data systems ready for AI deployment doesn’t have to be an insurmountable task. By following these steps, government agencies can set themselves up to be ready for AI implementation in a responsible way. AI has strong potential to improve the efficiency and effectiveness of government services, but it requires a solid foundation, proper planning, and regular auditing to ensure that it works as intended.

Want to work together to build your city or state’s AI readiness? We’ve partnered with governments across the country to implement AI in safe and responsible ways. Contact us to learn more.