A cautionary tale in data collection

Making changes to what data we collect can often seem deceptively simple. Shifting from free-form text answers to multiple choice, or tweaking how a question is phrased are changes that can be made so quickly that it’s easy to make a split-second decision and move on. However, it’s important to slow down and think about the broader impact, particularly when making changes to a survey that is routinely distributed.

One example of how this kind of change can go wrong comes from Statistics Canada. According to Census data, between 2011 and 2016, Canada’s Jewish population dropped in half from approx. 309,000 to 143,000 - a big enough fluctuation to flag concerns in the research and advocacy communities. What happened? In the 2016 Census, the Jewish identity was removed from the list of examples next to the question “What were the ethnic or cultural origins of this person’s ancestors?”, which is based on the top responses from the previous Census.