The 10 commandments for designing a company demographic survey

This post is part of a series focused on diversity and inclusion. View all posts in the series or start from the first post.

Last fall, the Clever Diversity and Inclusion Committee decided that we wanted to start measuring the demographic makeup of our team. Tracking this information would help us become more data-driven in our efforts to improve diversity at Clever, and publishing our statistics would help contribute to the industry-wide conversation about diversity.

In this post, we’ll share some lessons we learned in the process. Of course, these “commandments” aren’t written in stone (*wink*), but we do hope they can serve as best practices when designing your own demographic survey. We’re also publishing our survey for other organizations to use as a model.

1. Thou Shalt Survey

As we first started discussing how to track our team’s demographics, we faced a choice: either we could label employees’ gender and race ourselves, or we could ask them to self-report using a survey.

Labeling had some advantages: it would be quick, low effort, and, since we wouldn’t need to rely on employees’ willingness to fill out a survey, we would be guaranteed to have complete data… or would we?

As we discussed the options, we realized that while labeling employees’ gender and race seemed straightforward, we didn’t actually know every employee’s gender and race. We could assume—but we couldn’t know for sure without asking. So we might have “complete” data—but it would likely be incorrect.

We make assumptions about the gender and race of the people around us every day. Every time we meet a new person, we assign them a gender and race in our mind. It’s an unconscious process, and sometimes we get it wrong, which is awkward at best, and offensive at worst.

But when we’re talking about measuring and reporting demographic data, the stakes are higher than a social interaction. If we mislabel someone’s gender or race, we’ll do more than just offend them—we will invalidate our data! Not to mention that other demographic facets we wanted to measure like age and religion are much harder—if not impossible—to label.

Ultimately, we decided that demographic identities are up to each individual to decide for themselves. It would be unethical to label employees’ demographic identities, so letting them self-report using a survey was the only option.

2. Thou Shalt Anonymize

Once we decided to do a survey, we quickly ran into a second tricky dilemma: should the survey be anonymous? On one hand, an anonymous survey would make employees feel safer and hopefully encourage more participation. On the other hand, being able to identify specific employees’ identities would allow us to follow up with employees who had not completed the survey. Most importantly, non-anonymous data would open up much more impactful analysis. For instance, we could correlate the data with our annual employee happiness survey to understand if certain demographic groups were feeling less included at Clever, or we could compare the salaries of different groups.

Ultimately, since our goal was to use the data to make change, we decided to collect employee email addresses but anonymize the results. In addition, we ensured that only a few select people had access to the raw data, and we blacked out the email addresses in the spreadsheet containing the raw data to prevent accidental exposures.

Most importantly, we communicated all of this to employees when we administered the survey to ensure that they knew exactly how their data would be used.

3. Thou Shalt Not Require Participation

Even though we were anonymizing the survey results, we knew that some employees would resist participating in the survey. Some might feel uncomfortable revealing their identities to their employer, anonymous or not. Others might have ethical or political objections to the process as a whole.

Thus, to respect employees’ autonomy, we included an option to opt out of each survey question: “I prefer not to answer.” This allowed employees to participate as much or as little as they wanted. We then required every employee to fill out the survey to ensure that nobody was accidentally left out, and some employees just chose “I prefer not to answer” for every question.

4. Thou Shalt Include Many Identities

Next, we had to pick what demographic identities we were going to ask about in the survey. Gender and race were obvious choices, since they are the most widely discussed. But we also wanted to ensure that other underrepresented groups were given space as well. By definition, underrepresented groups such as veterans, people with disabilities, and religious minorities often have nobody speaking up on their behalf.

Of course, some people may view questions about ability and religion as intrusive. But we hoped that by taking a broad view of diversity, we would be making our survey more inclusive. For instance, imagine if you were the only Muslim at your company. As a Muslim living in the U.S. you may have faced discrimination based on stereotypes about your religion (or, if you haven’t faced discrimination, you may at least fear it). What would it feel like to not be asked about your religion in a demographic survey? Would you feel ignored? Annoyed? Alone?

After administering our survey, we heard from various employees that seeing their specific underrepresented identity in the survey made them feel seen—it made them feel more included to be counted. Simply counting someone can help demonstrate that your company cares about them.

Ultimately, we decided to survey:

• Gender
• Race
• Age
• Religion
• Sexuality
• Marital status
• Whether you were the caretaker of a minor (a more inclusive view of parenting)
• Ability
• Veteran status

We also included a question at the end of the survey asking if we had missed anything, and many people pointed out that we had left out socio-economic background and educational background. We plan to include both in our next survey.

5. Thou Shalt Offer Standard Labels

Once we decided on a set of of questions, we had to figure out what we would allow as answers. In other words, we needed to pick the set of labels we would allow respondents to pick from for each identity (e.g. for gender identity, “Man” and “Woman” are possible labels).

Offering a standard set of labels for each survey question makes it easy to analyze results. We based labels on those used in the U.S. Census. For instance, when asking respondents to identify their ability, we used the language from the American Community Survey, which is based on current, widely accepted models of disability.

However, a standard set of labels might not include the labels that all respondents use, which leads us to our next commandment…

6. Thou Shalt Allow Write-In Labels

Every question in our survey had an extra response option “Other,” which also allowed respondents to write in their own answer (this is natively supported by Google Forms, the tool we used to build our survey).

While write-in labels took some extra work to factor into our analysis of results, we thought they were important for making the survey feel inclusive to respondents. When analyzing results, we tried to aggregate any write-in labels. For example, when asked about religion, one respondent wrote in “Not sure, but I believe in Santa!” We counted this respondent under “Agnostic.”

7. Thou Shalt Not Restrict Respondents to One Label

All of the questions in our survey allowed respondents to choose multiple options. This may appear odd at first glance. For example, how can someone identify their race as both white and black at the same time?

At the end of the day, many demographic labels are not objectively defined—they are socially defined by the individuals who identify with those labels. Someone born to a white parent and a black parent might identify as white, black, both, or neither!

8. Thou Shalt Protect Anonymity in Results

After we finished administering the survey and began to analyze the results, we realized that due to the small size of our team, certain labels were only chosen by very small groups. In order to prevent people from trying to guess who belonged to those small groups, we aggregated any labels chosen by only one or two respondents into larger groups. For example, only a few people chose the labels “lesbian,” “gay,” “bisexual,” “queer,” and/or “asexual,” so we grouped them together when reporting the results.

9. Thou Shalt Resurvey Regularly

By the time we finished analyzing the results of our survey, our team had already grown! We quickly realized that we would need to resurvey our team regularly in order to keep our statistics current and to measure progress over time. In addition, respondents might change their labels over time (for example, if they have children).

For now, we’ve decided to resurvey our team once a year. We’re also exploring recording demographic identities in our HR system so that we can track demographics continuously instead of annually. We plan to ask each new employee to fill out their demographic identities in the HR system during their onboarding.

10. Thou Shalt Publish Results

When our Diversity and Inclusion Committee finished analyzing the results of the survey, we published them internally so that the entire Clever team could see where we stand. We annotated the results with parallel statistics from other tech companies (to help see how we fit into the bigger industry picture), as well as from K-12 schools (to understand how our team demographics related to those of our users).

We then published the results externally in order to raise awareness, hold ourselves accountable to increasing our diversity and improving our inclusion – and inspire other companies to do the same. We hope that our learnings will be helpful when you design your company’s demographic survey, and that you’ll publish your results as well!