Note: NHS coefficients of variation

This note provides context for the coefficients of variation (CVs) released for the 2011 National Household Survey (NHS). First, some highlights are provided. Then the CVs for the 2011 NHS are described and compared from a conceptual and methodological viewpoint with the corresponding CVs from the 2006 Census long form. Finally, in Appendix 1, some 2011 CVs are compared with the corresponding 2006 CVs for a few characteristics.

Description of the CV

The CV is a measure used to describe the precision of an estimate. For example, it can be used to measure the precision of an estimated total, an estimated proportion, an estimated mean, etc. The smaller the CV is, the more precise the estimate. More specifically, the CV of an estimate is the ratio of the standard error of the estimate to the estimate itself. The standard error is estimated and therefore, the CV is also an estimate.

Distinction between the CV, the response rate and the global non-response rate in the 2011 NHS

The CV does not measure bias, such as the bias due to non-response. It is important not to confuse the CV with the response rate and the global non-response rate for the NHS. The CV measures the precision of an estimate, while the response rate is an indicator of the risk associated with household non-response error, and the global non-response rate is an indicator of the risk of household non-response error plus individual question non-response error.

Conceptual and methodological differences between the 2011 NHS and 2006 Census long-form CVs

Several factors influence the values of CVs and must be considered when the 2011 and 2006 CVs are compared. First, the target population and the estimation methods for the 2011 NHS differ from those of the 2006 Census long-form sample. Moreover, the variability measured by the 2011 CVs is different from the variability measured by the 2006 CVs. In 2006, only the variance due to sampling was included in the estimate of the CV as the contribution of other sources of variability was relatively very small. In 2011, the estimate of the CV includes both the sampling variance and the variance due to household non-response. In both cases, variability was measured assuming that the estimation methods had corrected the biases. The factors that help to reduce sampling variability are: a smaller population size, a larger sample, smaller variability in the population of the characteristic being studied and more efficient estimation methods and survey design. The main factors that influence variability due to non-response are the magnitude of non-response and the estimation methods used.

Some of the conceptual and methodological differences between the 2011 NHS and the 2006 Census long-form sample are described in the following sections.

Target population

The 2011 NHS targeted the Canadian population as of May 10, 2011, while the 2006 Census long form targeted the Canadian population as of May 16, 2006. Furthermore, the target population in 2011 included only persons living in private dwellings, while that of 2006 also included persons living in non-institutional collective dwellings (approximately 1% of the population).

Survey design

The survey design of the 2006 Census long form included a single sampling phase in which approximately one dwelling in five was drawn. The survey design for the 2011 NHS is more complex and includes two sampling phases. First, a sample of an average of one dwelling in three was drawn. After several weeks of collection, i.e., on July 14, 2011, the initial sample was reduced: the respondents obtained were retained and a follow-up subsample of approximately one dwelling in three was drawn from the remaining non-respondents. Approximately one-third of the dwellings in the initial sample had not yet responded at the time the subsample was drawn.

Magnitude of non-response

The rate of non-response to the 2006 Census long form was 6%. By comparison, the rate of non-response to the 2011 NHS weighted by the sampling weight (to take account of the follow-up subsample) is 23%.

Estimation methods

Although non-response was greater in 2011, more information was used to reduce non‑response error than in 2006. In 2006, only geographic information and household size were used for the non-response adjustment, while in 2011, essentially all census variables as well as some administrative data were used.

In 2006 as in 2011, calibration was performed to produce the estimates (totals, proportions, means, etc.). In calibration, an effort was made to ensure that certain estimates of totals in the survey corresponded to known totals. In both surveys, calibration was performed using totals from the full census. In 2011, there was calibration to some totals of family and language variables, which was not the case in 2006. Also, unlike in 2006, sociodemographic totals for small domains were not used in 2011.

 Appendix 1 : CVs for the 2006 Census long form sample and the 2011 NHS for different levels of geography for estimates of totals for some variables common to 2006 and 2011

Analysis of 2006 Census long form and 2011 NHS CVs

CVs are provided for different characteristics of the 2006 Census long form and the 2011 NHS. Levels of geography provided are national level, provincial and territorial level and four census subdivisions (CSD).

It should be noted that the CVs for Nunavut were nil in 2006, but not in 2011. Since all households were included in the sample, there was no sampling variance in this territory. However, there was a small non-response variance, which was measured not in 2006 but is in 2011. The following analysis will therefore not include Nunavut.

For all characteristics and levels of geography presented in Appendix 1, the order of magnitude of the 2011 CVs is generally similar to that of the corresponding 2006 CVs. All shown 2011 and 2006 CVs are under 20%. For the national level and more populated provinces like Quebec, Ontario, Alberta and British Columbia, the 2006 and 2011 CVs for the characteristics presented in Appendix 1 are all under 1%. For Manitoba and Saskatchewan, they are all under 2.5%. For Newfoundland and Labrador, Nova Scotia and New Brunswick, all 2006 and 2011 CVs never exceed 5.5%. For Prince Edward Island, Yukon, Northwest Territories, the CSD of Charlottetown (City), Niagara Falls (City) and Port Coquitlam (City), the CVs are all under 12%. The 2006 and 2011 CVs for the smallest CSD presented in Appendix 1, Joliette (Ville), are somewhat higher but all below 20%. Larger CVs presented in Appendix 1 can also be observed for rarer characteristics like visible minority, immigrants or unemployed for all levels of geography.

About half of the 2006 CVs presented in Appendix 1 are larger than the 2011 CVs and vice versa. In Newfoundland and Labrador, Prince Edward Island, Nova Scotia, New Brunswick and Saskatchewan, the 2011 CVs tended to be higher than those for 2006. At the Canada level, in Ontario, British Columbia, Northwest Territories, Joliette (Ville), Niagara Falls (City) and Port Coquitlam (City), the 2006 CVs tended to be higher than those for 2011. In Quebec, Manitoba, Alberta, Yukon and Charlottetown (City), the differences went in both directions.

The largest differences between the 2011 and 2006 CVs tend to occur for population characteristics that are rarer or for small levels of geography. For example, this is the case for the characteristic visible minority in Prince Edward Island, Yukon, Northwest Territories, Charlottetown (City) and Joliette (Ville) where the 2006 CVs exceed the 2011 CVs by somewhat less than five percentage points. The difference can also be explained by the increase in the estimates between 2006 and 2011. Some other situations where the 2006 CVs are higher than the 2011 CVs can also be observed for the characteristic immigrants in the Northwest Territories, Charlottetown (City) and Joliette (Ville). In all the situations where the 2011 CVs are higher than the 2006 CVs, the differences are below one percentage point.

The simpler survey design, the lower non-response rate and the fact that the non-response variability was not covered by the 2006 CVs are all factors that may explain why, in some cases, 2006 CVs are smaller than the 2011 CVs. Factors that may explain why 2011 CVs are smaller than those for 2006 include characteristics that are less rare in 2011 than in 2006, a larger number of responses, and changes in calibration.

Also, calibration may either increase or reduce the variability of the estimates. On the one hand, variability may be reduced by calibration for variables that are related to calibration variables. On the other hand, when many calibration totals are used, the variability of the estimates may increase. The fact that, overall, more calibration was performed in 2006 than in 2011 could, to some extent, explain why CVs are sometimes larger in 2006.