In Debunking Weighting Misperceptions, our first post in the weighting data mini-series, we reviewed the benefits of weighting and debunked misconceptions. Now, we review how to appropriately weight and evaluate the weighting scheme.

When considering weighting it is important to consult a marketing scientist. The Lightspeed data processing team can escalate questions about weighting to the appropriate resources. The marketing scientist can:

• Determine if weighting is really necessary. In some cases, there may be skews in the data, but it is on a variable that doesn’t impact key measures so weighting won’t change the business decisions. 
• Advise on the appropriateness of weighting small sample sizes. When sample sizes are small there is less stability in the weights. 
• Raise cautions around extremely unbalanced samples. Weighting is not a good option when imbalances are large.  A general rule of thumb is that weighting should not be used to increase the proportion of a sub-group more than double or decrease it by more than half.
• Help develop the weighting scheme by identifying the appropriate variables, breaks, and target quotas to use. Just as it is important to accurately profile the target population for survey quotas the same is also true for weights.  An inaccurate profile will result in weighted data that may not be representative of the target and may lead to incorrect business decisions.     
• Examine the weighting diagnostics to make sure they meet certain criteria and there is no need for adjustments.

It is extremely important to evaluate the weighting scheme to make sure it is valid and does not violate any statistical rules.  To do this first the weighting efficiency is evaluated by comparing the effective base size to the original base size. An effective base size is used to reduce the likelihood of the statistics producing significant results simply because the weighting has made adjustments to the data. There are no hard and fast rules for weighting efficiency because it always depends on the circumstances of the weighting, however, any time the effective base size is less than 70% of the original base size the weighting should be carefully examined. If necessary, the number of weighting variables or breaks might be reduced to increase the weighting efficiency.    

In addition to weighting efficiency, it is also important to look at the actual size of the weights. There are several basic rules:

• No weights should be above 5.0. If there are just a few weights above 5.0 then the weights should be capped at 5.0.  If there are many weights above 5.0 then the weighting scheme needs to be reevaluated. 
• The percent of respondents with weights 2.0 or greater should not exceed 10% of original base and/or when weighted those with weights of 2.0 or greater should not exceed 30% of the effective base.
• The average weight for outliers (weights of 2.0 or greater) should not exceed 3.0.
• Any weights close to zero (less than .01) suggests there is something wrong with the choice of variables to weight on and the weighting scheme should be reexamined.

After the weighting scheme passes the statistical validity tests mentioned above, the weighted data needs to be examined. First, make sure that the weighting has the desired effect on demos and habits. Second, key measures should be carefully examined to understand if absolute and relative results have changed due to the weighting.

Remember, weighting is no replacement for appropriate sampling, but it can be extremely beneficial for research by making the sample more representative of the target population, adjusting for varying response rates, comparing across samples, reducing sample costs and assuring a representative sample. Following the rules outlined above will help assure a quality weighting scheme and correct business decisions.