2021 Census Fact Sheets
2019 Census Content Test: Design and methodology

1. Introduction

Statistics Canada takes a number of key steps to ensure that each census produces relevant information for Canadians and decision makers.

Preparing for each census requires several stages of consultation, testing and test data evaluation before recommended questionnaire content for the upcoming census can be proposed to the Cabinet of Canada for approval. These steps include

• a content consultation with stakeholders and census data users
• qualitative testing of proposed modifications and additions to the content
• quantitative testing to evaluate content and respondent behaviour on a larger scale
• an evaluation of the test results guided by a content determination framework. 

Upon Cabinet approval of the census questions, the Governor in Council reviews the questions, and an order in council prescribes the questions to be asked under section 19 or 20 of the Statistics Act. Within 30 days of the signing of the order in council, the content of the census is published in the Canada Gazette.

Statistics Canada conducted extensive consultations from fall 2017 to spring 2018 using an online questionnaire and face-to-face discussions with stakeholders. More than 2,800 respondents participated in the online consultation. This unprecedented number demonstrates a high level of interest in helping to shape an important source of demographic and social information for decision making and analysis.

In addition to the online consultation, Statistics Canada met with respondents from 14 federal departments and other interested organizations in person. To understand the needs of Indigenous organizations and communities, more than 60 in-person discussions were held in 30 Indigenous and non-Indigenous communities across Canada with more than 400 contributors.

Based on the findings from consultations and discussions, Statistics Canada modified the questions asked in the census.^{Note 1} In 2018, various versions were tested qualitatively through cognitive, one-on-one interviews. These interviews were conducted across Canada in both official languages.

Based on the results of qualitative testing, the census questionnaires were further revised and tested quantitatively during the 2019 Census Test. The 2019 Census Test evaluated changes to the wording and flow of some of the questions, as well as the potential addition of new questions. This test also incorporated the evaluation of new communications material and variations to further improve collection methods, which are not discussed in this document. Testing ensures that the 2021 Census provides high-quality data to support a wide variety of programs and services in communities across the country.

After statistically analyzing the results of the 2019 Census Test and considering costs, operational factors and safeguards against quality loss, Statistics Canada made recommendations to the government, which determined the final content of the 2021 Census questionnaire.

2. Content testing in the 2019 Census Test

To ensure that the census measures important trends in society, many content changes were proposed for the 2021 Census. To quantitatively assess the impact of modifications to content, three versions of the questionnaire were tested. The control version contained the 2016 Census content, with some changes that are described below. This was the benchmark version of the questionnaire, without significant changes from the previous cycle. Two alternate versions, Version 1 and Version 2, were developed to test new and modified content. Comparisons between the various versions helped assess what questions, wording, skip patterns, verification messages and other features worked well and would provide reliable information.

The control questionnaire was, essentially, an updated version of the 2016 questionnaire. It included a question on religion, which has been asked every 10 years, and some question categories were updated with a different reference period (for example, the category “2016 to 2019” was added as a possible response to the question on the year the dwelling was built).

The short-form questionnaire (Form 2A) and the long-form questionnaire (Form 2A-L) were tested, both as online (or electronic) questionnaires (EQs) and paper questionnaires (PQs). This procedure was used because of differences in flows and skip patterns in the EQ and PQ, and because the demographic profiles of EQ and PQ respondents differ.

The content test component of the 2019 Census Test required developing different test and control questionnaires, in English and in French, distributed among nine analysis panels. The 2019 Census Test was mandatory for respondents to adequately meet the test objectives.

3. Target population and sampling

The 2019 Census Test was made up of two components: a content test to evaluate new and modified questions, and a field operations test to assess new and modified procedures and technologies for use in data collection. The following describes the design of the content test.

Taking into account the operational constraints of the test (for example, the absence of field staff to deliver questionnaires), the sample was selected from private dwellings located in mail-out areas in the 10 provinces. Mail-out areas are areas where Statistics Canada can mail census material, covering about 90% of dwellings. Collective dwellings were excluded from this test.

Only households in private dwellings selected to respond to the long-form questionnaire in 2016 were invited to respond to the 2019 Census Test to maximize comparability between 2016 and 2019 responses among matched respondents.

As mentioned, some of the proposed changes applied only to the PQ, while others applied only to the EQ. The target population, collection method and sample sizes of the analysis panels were defined to obtain enough paper responses and electronic responses to ensure that all the proposed changes were tested properly. This was necessary for obtaining an adequate level of accuracy during data analysis to detect statistically significant differences by response mode (PQ or EQ).

To meet all the test objectives, nine panels were needed:

• The first three panels were used to test the EQ content. Each panel received one of the three 2A-L versions. Given the design of the 2A-L EQ application, these panels were sufficient to test both the short-form and the long-form questionnaires. This was possible because, in the EQ, the questions from the short-form questionnaire are asked in the same format at the beginning of the long-form questionnaire. Respondents were also given the option to respond by completing a PQ, but the sample sizes were determined to ensure enough EQ responses would be received. These panels were also used to test the 2021 Census communication strategy, another objective of the 2019 Census Test.
• The six remaining panels were used strictly for testing the content changes in the PQ. The universe for the PQ panels was households that self-responded to the 2016 Census 2A-L PQ. This universe was used in an effort to have a representative sample of paper respondents. Panels 4 to 6 were used to test the 2A PQ, while panels 7 to 9 were used to test the 2A-L PQ. Short-form and long-form questionnaires needed to be tested separately because the paper forms are designed differently, and the design can influence how people respond.

Figure 1 illustrates the design of the test.

Many of the content changes for the 2019 Census Test applied to very specific, often small, subpopulations. Therefore, the sampling strategy used a stratified simple random sample, where the strata grouped specific subpopulations of interest for the test. The final sampling strategy was chosen after conducting simulations incorporating different groups of subpopulations to see which would work well for all stakeholders. These simulations led to the final sampling strategy, which included four strata: minority language rights holders, Indigenous people, Veterans of the armed forces and others.

Additional households were selected from the Indigenous and minority language rights-holders strata. This allowed for more households to be sampled to test questions specific to these subpopulations, as they are relatively rare in the Canadian population. The sample selected for Veterans was obtained from the survey frame for the Life After Service Survey. This frame was at the individual level rather than the dwelling level. Each individual in the sample who could be linked with certainty to a dwelling from the 2016 Census was included in the test. Therefore, the Veteran subpopulation was treated separately from the rest of the sample.

The strata that were created took into account different regions (provinces or groups of provinces) and the language of response (English or French). The number of households expected to answer in each official language was controlled so that an adequate number of respondents would be selected to test the French questionnaires. Two samples were selected, one for the EQ panels and another for the PQ panels.

The final sample selected for the content test component of the 2019 Census Test consisted of 134,757 households across all 10 Canadian provinces.^{Note 2} Proportional stratified sampling was used to select all but one sample—the minority language rights holders French EQ sample. This sample was allocated to ensure that 70% of responses would come from Quebec and 30% would come from all other provinces. This procedure was followed to collect more responses from other provinces to test the proposed question on rights holders with households outside Quebec (otherwise, the sample would not have contained enough French respondents outside Quebec).

4. Wave approach to the 2019 Census Test

A wave approach was used for the 2019 Census Test collection. The approach was very similar to the one used in the 2016 Census and the one planned for the 2021 Census. The wave approach increases data quality by reducing non-response and encouraging Internet self-response, and it ultimately decreases the cost of conducting a census. Under this approach, respondents are sent invitations and reminders at certain times (called waves) during the collection period. Building on the success of the previous censuses, the 2019 Content Test incorporated potential changes to all three waves. Moreover, after Wave 3, a sample of dwellings received a text message reminder, as a test for the 2021 Census. Table 1 shows the communications materials and key dates for each wave.

5. Data collection

The Census Test Day was May 14, 2019. Collection for the 2019 Census Test took place from May 6 to June 28, 2019.

The Census Help Line (CHL) was open during collection, so those who had questions or required assistance to complete their questionnaire could talk to an agent. When the purpose of a call was to assist a respondent in completing the questionnaire, the agents used the interviewer application, which is an EQ developed specifically for this purpose.

Non-response follow-up was not conducted during the content test. Therefore, most of the reported responses were self-responses. A negligible proportion of responses was obtained by the CHL.

6. Processing of returned questionnaires

For the content test, EQs completed by respondents were sent directly to the servers at Statistics Canada’s Data Operations Centre (DOC) and saved automatically upon receipt. The PQs that were mailed back were also saved at the DOC by scanning the barcode on the cover of the questionnaire. After they were recorded, the PQs were processed for data capture.

Once captured, the PQ data were combined with the EQ data in a single file. A complex integration process was required to standardize the data for each response mode. The purpose of this process was to obtain a single file for all response modes and panels. Processing rules were applied to this file to ensure that certain problems and inconsistencies were identified and corrected (e.g., a PQ returned with no responses, or both a PQ and an EQ completed for the same household). An edit also made it possible to identify questionnaires that contained no information or insufficient information to proceed to the processing and analysis stages.

Given the analysis objectives, the data were not subjected to the edit and imputation process. In addition, not all write-in responses were coded.

7. Return rates

Table 2 presents the return rates of the 2019 Content Test, with an overall return rate of 76%. This rate was calculated using the number of completed questionnaires divided by the number of dwellings in the sample. It includes saved and auto-saved electronic responses, as well as fully and partially completed questionnaires. Unoccupied dwellings (some of which were identified as such by respondents) were included in both the numerator and the denominator.

8. Conclusion

Following the 2019 Census Test, the results of the test and control versions of the questionnaires were compared to assess the impact of proposed content changes. Responses from the PQs and EQs were analyzed separately. For the new questions, the results of the test versions of the questionnaires were also compared with the results of the control version to ensure that the new questions did not have an effect on neighbouring questions or other related questions. Questions for which no changes were tested were also analyzed to ensure that the results remained accurate. Paradata^{Note 3} from EQs were analyzed to assess response burden and behaviour (e.g., time required to complete the form and whether the questionnaire was saved to be completed later or completed in one session) and EQ features such as automated messages that prompt respondents to provide an answer when a field is left blank. Lastly, the test results were analyzed and compared with data from the 2016 Census and other datasets.

The key findings and conclusions from the analyses that were conducted are summarized in a series of fact sheets that are available online. These informative, educational and easy-to-read short documents provide an overview of each census topic. These fact sheets explain why the questions are asked, describe the changes considered and the resulting approach for 2021, and mention what to expect in 2021. Three technical reports containing more in-depth information (including the findings from the 2019 Content Test) on the topics of gender, ethnic or cultural origins, and minority language rights holders are also available online.