Evaluating use cases

When evaluating potential GenAI use cases, consider the following questions in order to explore the benefits and risks of deploying this technology:

  • What is the problem statement?
  • What are potential inequities in problem formulation?
  • What are the data inputs?
  • How and when will the solution be implemented and integrated into existing and future processes and delivery of services?
  • What is the return on investment? What are the alternatives to solving the problem?
  • Who will be the GenAI team responsible within the program area to monitor, validate, and evaluate the GenAI tool?
  • How does using the GenAI tool build trust with the end users like State staff or Californians?
  • Is the GenAI tool accessible and culturally appropriate?

Before initiating a GenAI project, it is also critical to consider the communities and users that may be impacted by the use case, whether they are the primary service population or groups that may be unexpectedly affected. A few criteria to begin thinking about the potential impacts of GenAI on communities include severity, scale, duration, and types of potential impacts of GenAI tools across a range of community group sizes.

Severity of risk

The degree and type of impact should be considered in determining an overall assessment of risk level for a GenAI use case.

Scale of risk

The scale and duration of the impact should be a consideration in determining the risk level for a GenAI use case. This should include an impact analysis of various affected groups – including, but not limited to, individuals, workers, communities, businesses, and local governments.

Important considerations in understanding GenAI risks include:

  • The risk of biased outputs from GenAI tools that amplify existing societal biases, particularly in use cases that may impact the delivery of public services.
  • The risk of hallucinated outputs from GenAI tools that can impact the validity, accuracy, or performance stability of State services.
  • The risk of harmful or inappropriate materials generated by GenAI, such as deepfakes that could spread misinformation.
  • The risk of GenAI tools lacking a human-in-the-loop reviewer who can validate the outputs of the system.
  • The risk of automation bias (an over-reliance on automated GenAI systems to make decisions), given the ability of GenAI tools to produce answers that “sound right” despite having no factual accuracy.
  • The risk of security vulnerabilities in GenAI tools exposed through new, natural language interfaces or source datasets.
  • The risk of black box GenAI applications that are unable to explain the rationale behind its recommendations for services that require this capability.
  • The risk of privacy re-identification issues for datasets including vulnerable communities that depend on anonymity for safe data analysis.

Source: State of California Benefits and Risks of Generative Artificial Intelligence Report, November 2023.

The Government Operations Agency (GovOps), Department of Technology (CDT), and Office of Data and Innovation (ODI) are working to develop guidelines for evaluating the impact of GenAI use cases on historically vulnerable and marginalized communities. These guidelines will be added to this toolkit once available.