11. Historical estimates of population coverage error

11.1 Estimates

This section presents historical estimates of population coverage error. Chart 11.1 shows the estimated population undercoverage rate ${\widehat{R}}_U$ for the 1971 Census to the 2011 Census, and the estimated population overcoverage rate ${\widehat{R}}_O$ and the estimated population net undercoverage rate ${\widehat{R}}_N$ for the 1991 Census to the 2011 Census. The series for overcoverage and net undercoverage begin in 1991 because the overcoverage rate was first estimated for the 1991 Census following an experimental study done for the 1986 Census.

Population coverage error is a growing data quality concern; undercoverage has doubled since 1981 and overcoverage is two and a half times higher than it was 1996. Changes in net undercoverage from census to census reflect changes in undercoverage and/or overcoverage, which in turn reflect changes in the demographic situation, changes in the living arrangements of Canadians, changes in census methodology, and changes in the methodology of the coverage studies. The last issue is discussed in Section 11.2.

As shown in Chart 11.1, the undercoverage rate declined slightly in 2011, and the overcoverage rate continued to rise. The undercoverage rates were similar for the 1971, 1976 and 1981 censuses (1.93%, 2.04% and 2.01% respectively). Undercoverage increased to 3.21% for the 1986 Census and to 3.43% for the 1991 Census, and then decreased to 3.18% for the 1996 Census. It rose sharply to 3.95% for 2001 and then to 4.26% for the 2006 Census, after which it fell back to 4.07% in 2011, between the rates for 2001 and 2006. Overcoverage increased from 0.74% for the 1996 Census to 0.96% for the 2001 Census. The increase between 1991 and 1996 was due to a change in the methodology of the coverage studies. The largest increase in the overcoverage rate was between 2001 and 2006, when it rose from 0.96% to 1.59%. The rate reached its highest point in 2011, at 1.85%. From 2006 to 2011, Yukon and New Brunswick had the largest increases (0.83% and 0.71% respectively). There were also some decreases, including 0.54% for the Northwest Territories and 0.27% for Nunavut.

In 2011, net undercoverage continued the decline that began in 2006, as a result of a combination of lower undercoverage and higher overcoverage. Coverage error is attributable to errors by respondents, such as incorrect application of the rules on whom to include, and to errors by census staff, such as the erroneous exclusion of recently built dwellings. It is also worth noting that the methodology used in the 2011 Census included a number of modifications, which could have resulted in changes in undercoverage and overcoverage. Even though all census operations have to meet high standards of quality, those modifications may have had an impact on population coverage error. In the 2011 Census:

• The entire process of census enumeration was based on the short questionnaire.
• Mail-out areas (areas where the questionnaires were delivered by Canada Post) were expanded.
• A wave methodology was used. There were several follow-up waves after the initial mail-out of a letter asking dwelling residents to complete their census questionnaire online.
• Online responses jumped from 17.8% in 2006 to 53.9% in 2011.
• Recruitment of field staff was easier than in 2006.
• In 2011, enumerators checked only selected mail-out areas to update the dwelling list, whereas in 2006, they checked all mail-out areas.

Looking back at undercoverage since the 1981 Census, we see that the increase in undercoverage observed in the 1986 Census led to the creation of the Address Register (AR) for the 1991 Census. The AR provided a separate list of urban dwellings that should have been enumerated. For the 1996 Census, the use of enumerators (EN) instead of self-enumeration in some central parts of large cities reduced undercoverage. In addition, moving Census Day from early June to mid-May helped to control undercoverage because people were more likely to be at home and less likely to be moving. In 2006, mailing out the questionnaires in urban areas reduced the number of employees required for collection. The introduction of online questionnaires also reduced data capture problems. In 2011, the introduction of wave methodology made it possible to target census follow-up activities more effectively, and a sharp increase in online responses further reduced data capture problems. The elimination of the long questionnaire probably resulted in a slight decrease in the census non-response rate.

Top of Page

Estimates of undercoverage are presented in Table 11.1 and Table 11.2. Note that 1971 is not included in Table 11.2 because estimates were produced for different age groups for the over-24 population.

These tables show the following:

Undercoverage is usually higher in the three territories. Among the provinces, undercoverage is generally higher in British Columbia and Ontario. However, in 2011, Alberta was the province with the highest rate. Between 1971 and 2006, British Columbia was the province with the highest undercoverage rate in every census except 1991 and 2006, when Ontario had the highest rate. Undercoverage rates for Quebec and the Atlantic provinces tend to be lower than the national rate.

Undercoverage is higher for young adults and higher for males. There are two persistent demographic trends. First, undercoverage for males is higher than undercoverage for females. Second, undercoverage is higher for young adults, especially those who have never been married, regardless of sex (Dolson 2012). As shown in Chart 11.2, undercoverage for males is higher than undercoverage for females for every census since 1971, increasing from 2.27% to 5.51% in 2006 and falling back to 5.07% in 2011 for males, and increasing from 1.59% to 3.08% for females. Chart 11.2 also shows that undercoverage for men aged 20 to 24 is higher than undercoverage for all males. This is also the case for women aged 20 to 24, but the rate for women aged 20 to 24 is lower than the rate for men in the same age group in every census except 2011, when the rates were 9.83% and 9.37% respectively. The undercoverage rate for young women was at its highest in 2011 (9.83%). In contrast, the rate for young men is at its lowest since 1991. Though somewhat lower, the undercoverage rates for never-married people aged between 25 and 34 are also high. Higher undercoverage for young adults is due in part to their less stable living arrangements. Young adults are more likely than older adults or children to change their living arrangements because they are moving away from home to work or attend a postsecondary institution or moving in with friends or spouses.

Estimates of overcoverage rates are presented in Table 11.3 and Table 11.4.

These tables show the following:

Overcoverage is consistently higher for British Columbia than for the other provinces. British Columbia has been the province with the highest rate of population overcoverage for the past three censuses.

Overcoverage is more common for school-aged children and young adults. The 5-to-17 and 18-to-24 age groups have higher overcoverage rates. For school-aged children, this situation is largely due to the fact that children whose parents do not live together are often enumerated by both parents. Overcoverage for young adults is probably attributable to the same less stable living arrangements that can also lead to undercoverage. Overcoverage was up in the majority of provinces and territories in 2011. There were large increases in Yukon (+0.83%) and New Brunswick (+0.71%). Yukon, Manitoba and the Northwest Territories had the lowest rates, at 1.17%, 1.35% and 1.44% respectively. Nationally, overcoverage rates were above 3% for young adults (18 to 24), both men and women, and for males aged 15 to 17. In some provinces, the rates for some age-sex groups even exceeded 4%.

Top of Page

11.2 Changes in the design of population coverage studies

Because of differences in the design of the coverage studies over time, the rates in Table 11.1, Table 11.2, Table 11.3 and Table 11.4 are not strictly comparable. A list of the methodological changes made since 1976 is provided below. It is worth noting that the fundamentals of the Reverse Record Check (RRC) approach to measuring undercoverage have not changed much since the 1966 Census. A sample is selected from frames covering the target population that are independent of the census. Census records are then checked (Reverse Record Check) to determine whether the sampled persons were actually enumerated. There have been more changes in the measurement of overcoverage. Multiple studies were carried out for 1991, 1996 and 2001. In 1996, the RRC was expanded to include the measurement of overcoverage. In 2006, the RRC was no longer used to estimate overcoverage, and a new study was introduced to measure all overcoverage cases on the basis of probabilistic and exact matches using name, date of birth and sex.

2011 Census coverage studies:

1. The 2011 RRC was very similar to the 2006 RRC. Some changes were introduced to make it more efficient, including improvements in the monster match program, more effective strategies for searching the Census Response Database, and the use of new births frames.
2. For the first time, the weighting of the census frame sample took into consideration the overcoverage in this frame.
3. With automated methods, it was possible to use provincial and territorial parameters instead of national parameters in developing the COS frame.

Like the 1996, 2001 and 2006 RRCs, the 2011 RRC did not estimate the number of persons missed for incompletely enumerated Indian reserves and Indian settlements. For more information on this topic, see Section 12.2.

2006 Census coverage studies:

Both the RRC and the Census Overcoverage Study (COS) made optimal use of the name field added to the 2006 Census Response Database (RDB) in their matching and searching operations. In addition,

1. The measurement of overcoverage was restricted to the COS. The methodology of the RRC was subsequently changed so that not all cases were sent for field collection. Since 2006, the RRC has had a processing step that is carried out prior to collection to determine whether collection is required. The RRC version of the Census Response Database (RRC RDB) was searched for the sampled persons using information from the sampling frame and the various update sources, such as tax data. If the search located the sampled person in the RRC RDB, collection was not required. The only exception was a sample of persons that had been found in order to collect data required for the non-response adjustment.
2. The three coverage studies conducted in 2001 to measure overcoverage were replaced by the COS in 2006. The COS used a methodology that was different from any previous overcoverage study. Essentially, it employed a matching technique based on surnames, given names, sex and date of birth and manual verification to identify overcoverage.

2001 Census coverage studies:

1. The institutional component of the Collective Dwelling Study (CDS) was dropped, and overcoverage estimates for this population were produced by the RRC.
2. The Dwelling Classification Study (DCS) replaced the Vacancy Check (VC), which was used in previous censuses to re-examine dwellings classified as unoccupied by the enumerator. The DCS is an extension of the VC that estimates the number of persons living in non-response dwellings.

Top of Page

1996 Census coverage studies:

1. The 1996 RRC did not estimate the number of persons missed on incompletely enumerated Indian reserves.
2. The Temporary Residents Study was cancelled because of concerns about the quality of the data, and because it was recognized that the RRC would measure most of this type of undercoverage appropriately.
3. First, a measure of overcoverage that was more comprehensive than the 1991 measure was produced by incorporating the Private Dwelling Study into the RRC so that each sampled person could be identified as having been enumerated more than once. This approach resulted in an increase in the number of addresses to be processed where overcoverage could have occurred. Second, the Automated Match Study (AMS) was expanded substantially compared with 1991, so that overcoverage could be measuring not only for an enumeration area (EA) but also for a large region (Atlantic, Quebec, Ontario, western Canada and the territories).

1991 Census coverage studies:

1. Non-permanent residents were included in the target population for the first time.
2. Following experimental studies in 1986, the measurement of population overcoverage commenced in 1991. The results of three studies were combined to form a comprehensive estimate: the Private Dwelling Study (PDS), the Collective Dwelling Study (CDS) and the Automated Match Study (AMS).

1986 Census coverage studies: The rates shown in Table 11.1 for the 1986 Census differ from the results published in the User's Guide to the Quality of 1986 Census Data: Coverage, as they include revisions made after the 1986 publication, when incompletely enumerated Indian reserves were included as missed. In the original 1986 publication, they were included as 'enumerated' since provincial data included an estimate of persons missed for Indian reserves.

1976 Census coverage studies: Census data did not include an estimate from the Vacancy Check (VC) of persons missed in dwellings incorrectly classified as unoccupied. The 1976 population undercoverage rate would have been 1.78% if it had included the results of the 1976 VC. There was no VC in the 1971 Census.

For more details on the history of coverage studies, see Dolson (2010).