Participant satisfaction surveys can be used to assess participants’ experience. Questions may be open-ended to gather qualitative insights on participant experience or can use scales to provide quantifiable data that can be tracked across different engagements. Surveys can be distributed at the end of an engagement event or after an engagement process has concluded. Understand that the more controversial a project is, the more likely it is that public perceptions of the engagement event or process are impacted by bias or dissatisfaction with the project itself, despite efforts made toward inclusive and effective engagement.

Characteristics of participants can be compared to the known demographics of the project area to understand the engagement’s representativeness.

Determining geographic scale

When deciding which geographic scale to use, consider the project’s area of impact, for example citywide, neighborhood-level or site-specific. If assessing whether participants broadly reflect the city’s demographics, use citywide data. For a community area, community plan areas or council districts may be the most appropriate. To find neighborhood area-level patterns, use census tract-level data. If insights about smaller areas around project sites are required, census block-level data can be used. In the instance that the project area does not neatly fit into its suggested geography, use the next boundary level up or down that is appropriate for the project. For example, if a project’s impact is smaller than the community-level, tract-level information may be the most appropriate.

When deciding which geographic scale to use, consider the project’s area of impact, for example citywide, neighborhood-level or site-specific. If assessing whether participants broadly reflect the city’s demographics, use citywide data. For a community area, community plan areas or council districts may be the most appropriate. To find neighborhood area-level patterns, use census tract-level data. If insights about smaller areas around project sites are required, census block-level data can be used. In the instance that the project area does not neatly fit into its suggested geography, use the next boundary level up or down that is appropriate for the project. For example, if a project’s impact is smaller than the community-level, tract-level information may be the most appropriate.

Below is a table with project types, examples, and their suggested geographies.

To determine if the people who participated in an engagement process or event are representative of the project area’s demographics, compare characteristics of the participation sample (like age, race, income level, etc.) to known demographics. Datasets from the U.S. Census and SANDAG will contain raw counts per demographic category, so when calculating demographic data to evaluate the engagement process, check that the data is being aggregated appropriately. This means:

• Demographic categories are the same between the Census or SANDAG data and the input choices from the engagement (e.g., race, gender, ethnicity categories, etc.)
• If demographic categories from the Census or SANDAG were aggregated, the ranges match the input choices from the engagement (e.g., age ranges, income bands, etc.)

While there are several different ways to assess how closely the demographics of an engagement represent the project area, there are two methods outlined below to help draw quick and insightful conclusions about participation:  

• Visual Inspection is a lightweight, descriptive statistical technique to assess data quickly and intuitively.
• Proportion Testing is a formal inferential statistical technique used to draw firm, rigorous conclusions about a set of data.

This section will briefly cover what each method is, the strengths and limitations of each, and which method to select based on what conclusions need to be drawn from the analysis.

Sign-in sheets can quantify how many people attended an engagement event.

Email notification metrics like number of opens and link clicks can be reviewed to understand digital reach. Additionally, staff can review email list sign-up form responses to quantify how many people demonstrated interest in a project.

The Department of Information Technology’s Web Services team and Communications Department can help acquire metrics related to engagement.

• Work with the Web Services Team for a report (accessible to City staff only) on webpage performance metrics such as page visits.
  • Knowing the sources of the page visits may also inform staff of engagement techniques that have acquired the most engagement.
  • Work with the Communications Department for a report on social media metrics like impressions and reach.  

Review input received for its quality through a content analysis. Identify quality by checking if sentiments and input themes are applicable to the project and specific enough to guide decision-making.

Check in with project staff and partners to:

• Assess engagement strengths and areas for improvement
• Summarize observations and collected opinions of the project and engagement activities
• Make recommendations for future projects

Look for any articles or media that have been published about the project. Assess whether the tone of press coverage is positive or has suggestions for improvement and consider these for future projects.

Other metrics to consider in evaluation include awards received, quality of partnerships formed, relationships improved and first-of-a-kind interactions, among others.