The Unrealized Impact 2.0 Dataset.
Data Geeks: This Post is For You.

by Michael Corral | Dec 1, 2021

Sometimes clients or audience members ask us whether our public research findings are “statistically significant.” As a refresher for those who are more than a few years out from their last statistics, economics or business course, statistically significant findings are those that can be confidently explained as reliable facts and not simply a calculation or conclusion derived from luck or chance. 

What is motivating the question? In our work we find that for some, this sort of question can come from a feeling of defensiveness: our findings might challenge someone’s leadership or identity and draw attention to inequities they may have perpetuated. This post is not for those readers. Further, the question “are those differences statistically significant” can be motivated by an assumption that if pain is not being broadly experienced, it doesn’t merit our attention. In response to that we offer our perspective that any pain should be acknowledged and attended to – whether or not the sample size or severity meets a bar for statistical significance in a pure data sense. Instead, this post is for those with a statistics, research or evaluation background who may be curious about our methodology, data samples and results.

As a social science researcher, I run rigorous statistical analyses beyond the percentage-based findings we share with clients and what we published in our Unrealized Impact reports. Below I give deeper context on our surveys, data and results — specifically those about race and ethnicity. I also include some technical language for those seeking to understand, at a statistically significant level, what our survey data tells us.

As of October 2021, Promise54 has collected over 60,000 DEI Staff Experience Surveys. We’ve used the data from these surveys to publish our latest Unrealized Impact 2.0 report, as well as our report from 2017. While our DEI Staff Experience Survey is anonymous, participants have the option of answering questions about demographics and identities. We use these responses to cross-tabulate identities, like race/ethnicity, with various questions from the survey. To analyze the surveys and generate insights, we’ve used both qualitative and quantitative research methods, with the majority of our findings coming from Likert scale questions and responses. 

We began our latest report with over 3,000,000 data points that represented people from all 50 states, DC, and Puerto Rico, and representing over 500 different organizations – primarily across the education sector. We then ran a set of analyses starting with one simple research question: Are people of various races/ethnicities having different experiences within their organizations?

Running a Spearman rank correlation analysis across all of our survey questions, we were able to identify the questions that yielded the most efficient and holistic data in the fewest amount of questions possible. As we isolated these highly efficient and telling survey questions, a pattern became clear: they were the same questions our clients most often ask about — or worry about — when they receive our customized DEI Staff Experience Report. 

The following four isolated survey questions arose:

  • Bias: “I have been on the receiving end of bias (either overt or implicit) at our organization.”
  • Intent to Stay: “I fully expect to be working in this organization three years from now.”
  • Whole Self: “I can bring my “whole self” to work (i.e., I am able to engage at work with a free and authentic expression of who I really am).”
  • Net Promoter Score: “On a scale of 0 to 10, how likely are you to recommend a qualified friend or colleague to work here?”

After running the statistical tests, we found that there, in fact, were significant differences along the following identities and measures: 

  • Race/ethnicity and how people experience bias, H(5)=887.69, p<.001.*
  • Race/ethnicity and the Intent to Stay question (proxy for retention/turnover), H(5)=1030.353, p<.001*
  • Race/ethnicity and how strongly people feel that they can be their “whole self” at their organizations/schools, H(5)=546.795, p<.001.* 
  • Race/ethnicity and how people score on the Net Promoter Score, H(5)=690.901, p<.001*

Once we verified that statistically significant differences existed among race/ethnicities along the identified four survey questions, we performed a post-hoc pairwise comparison to test each race/ethnicity group against one another and identify where the statistically significant differences existed. Declaring that a difference exists is one thing, but we also wanted to identify where those differences specifically existed. We wanted to know how groups of people and identities were spread across the spectrum of experiences. Then, controlling for Type I errors across tests by using an adjusted Bonferroni alpha level of .003 (0.05/15), we found the following (all ps<.003):

Experiencing bias within their organization

  • Black survey participants experience bias (mean rank=24774.88) more than…
    • White staff (mean rank=20918.90)
    • AAPI staff (mean rank 22951.15)
    • Latinx staff (mean rank=24031.59)
    • There are no statistically significant differences between Black, Multiracial (mean rank=25129.25) and Native American (mean rank=25331.82) survey participants. 
  • White survey participants experience bias (mean rank=20918.90) less than every other race/ethnicity category of survey participants.

Feeling that they can bring their “whole self” to work

  • Multiracial staff feel statistically significantly less able to bring their whole self to work (mean rank=19851.33) compared to…
    • White staff (mean rank=22910.18)
    • Black staff (mean rank=21437.34)
    • AAPI staff (mean rank=22425.86)
    • Latinx staff (mean rank=24864.36)
  • Of the three largest race/ethnicity subgroups, Black staff score statistically significantly lower (mean rank=21437.34) than white staff (mean rank=22910.18) and Latinx staff (mean rank=24864.36).

Net Promoter Score

  • Multiracial staff are statistically significantly less likely to recommend their organization to a friend or colleague (mean rank=19965.86) compared to…
    • White staff (mean rank=22465.37)
    • Black staff (mean rank=21475.01)
    • AAPI staff (mean rank=22041.34)
    • Latinx staff (mean rank=25574.32)
  • Of the three largest race/ethnicity subgroups, Black staff score statistically significantly lower (mean rank=21475.01) than white staff (mean rank=22465.37) and Latinx staff (mean rank=25574.32).

Intent to Stay

  • Staff that identify as Multiracial and AAPI scored statistically significantly lower (mean rank=20960.19 and mean rank=20774.55, respectively) on the Intent to Stay question compared to…
    • White staff (mean rank=21826.99)
    • Black staff (mean rank=22408.49)
    • Native American staff (mean rank=24702.91)
    • Latinx staff (mean rank=26431.37)
  • Of the three largest race/ethnicity subgroups, white staff scored statistically significantly lower (mean rank=21826.99) than Black staff (mean rank=22408.49) and Latinx staff (mean rank=26431.37). Note: There are some sociological implications to explore on this point — but that is for another day. 

This data serves as a critical reminder that we all have a long way to go on this road towards equity and justice. I hope these findings further amplify the disparities in experience that take place for People of Color and/or those with other historically marginalized identities,  and drive action to close those gaps. 

Finally, a reminder: when evaluating survey data, if you find yourselves or your colleagues asking questions like: “Okay, but is this just one person who feels this way?” or “Does this response accurately represent the overall experiences of our collective staff?”, we urge you to make sure that concerns around “statistical significance” or a “small sample size” do not lead to overlooking the pain or trauma that can be hiding in data. At Promise54, we believe there is validity (and significance!) in understanding what data is revealing … no matter what the sample size or p-value is.  

If you have additional questions or want to break some virtual bread over data and theory, you can reach me at Michael@promise54.org.  

* Each of the above mentioned statistically significant findings were found using a Kruskal-Wallis H Test to examine significant differences based on the identity of race/ethnicity and a specific Likert scale question on our DEI Staff Experience Survey from 1 (strongly disagree) to 5 (strongly agree) and 0-10 for the single NPS question.

 

MICHAEL CORRAL

Associate Partner

He/His/Him

Michael is: A person of faith, a follower of Christ, and a question asker. He is the proud son of Mexican Immigrant parents—the two hardest working people he will ever know—and the youngest of three. He is a high school graduate, the first in his family unit. He is a husband to an amazingly brilliant woman who is much smarter than he will ever be, a reality that makes him jealous on most days. He is a lover of sports, history, music, ice cream, cookies, and his mother’s mole con pollo y arroz. Through his experiences navigating an education system that was never designed to support all children in reaching their full potential—especially Children of Color in low-income communities—he believes that we all play a part in either dismantling or upholding our inequitable, racist, and discriminatory systems throughout society. Michael has previous work and research experience in the K-12, higher education, and non-profit sectors of education. Past roles have included a middle and high school math interventionist/teacher in Phoenix, AZ, adjunct professor and research assistant at the University of Connecticut, Director of State Affairs at Teach For America, and Research Associate at Inflexion. He holds a B.S. in Business Administration from Eastern Oregon University, an M.Ed. in Educational Leadership and Administration from the American College of Education, and a Ph.D. in Learning, Leadership, and Education Policy from the University of Connecticut.

Contact Michael: michael@promise54.org

4  Back to Blog