Conclusions may be clear but entirely inaccurate because your survey items are not measuring what you intended for them to measure. This can have disastrous consequences on your business- especially if you use survey results to influence business decisions. It is in these circumstances that the accuracy of your research is essential. With Infosoft Research you have the piece of mind knowing that our measurement consultants primary concerns are validity and integrity of your data

Factors that can threaten the validity of your results fall into three broad categories:
1. Survey Construction
2. Survey Administration
3. Analysis and Interpretation

Below contains some frequently asked questions and common pitfalls regarding survey construction and analysis. Feel free to give us a call or email us if you have questions: +1 360.714.0831 email: analytics@infosoftresearch.com

HOW SHOULD I PHRASE THE SURVEY ITEM?
Survey items can be constructed as questions or statements.
Example:
Did you feel that customer phone support was attentive to your needs?
Customer phone support was attentive to your needs.
What is most important here is that biases are minimized. If the item "leads" the respondent to answer in a particular way there is probably a better way to phrase it.

DOUBLE-BARRELED ITEMS
This is a common pitfall that makes it almost impossible to interpret a respondent's answer. The problem occurs when a single survey item addresses two separate issues.
Examples:
1. I believe my benefits and retirement packages are fair.
2. The site was helpful and easy to navigate.
In example one respondents may be referring to either the benefits package or retirement package or both. The item should be restructured so that one survey item assesses the perceived fairness of the benefits package and a second survey item assesses the perceived fairness of retirement package. In the second example it is unclear whether the survey item refers to the usefulness of information provided on the site, or the overall ease of navigability of the site. Again, the solution is to address only one issue per item.

ORDER EFFECTS
These occur when one survey item effects the interpretation and response of the following survey item.
Example:
Overall, my impression of the site was positive.
It was difficult to navigate the website.
In the above example, the respondent's overall evaluation of the site will influence subsequent responses about specific aspects of the site such as navigation. One way to minimize order effects is to place specific statements earlier in your survey, and more general questions toward the end of your survey. A more traditional solution is to "counter-balance" the item or randomize the order of presentation.

RESPONSE OPTIONS
There are variety of response options that can be used such as: open-ended, yes-no, true-false, 4-point scales, and 5-point scales. You may ask yourself, "Should the survey use four response options ranging from strongly agree to strongly disagree or should there be five response options with a neutral point?" There is no definitive answer to this question because biases are introduced in both cases. For example, with 4-point response options, the respondent is forced to take a position when they may actually think or feel neutrally about the situation. With 5-point response options, the respondent may fall back on responding neutrally when they have a definite position on the matter, but do not want to express it. Ultimately, the goal is to minimize the bias that is most detrimental to your study.

ITEM DIRECTION
Should a survey item be worded positively? Negatively? Or Neutrally?
Example:
I was satisfied with the customer support I received.
I was not satisfied with the customer support I received.
How satisfied were you with the customer support you received?
Ideally, you should use neutral statements so you aren't leading an individual to respond a certain way. However positively and negatively worded items are frequently used. The most important consideration is balance. If there are negatively worded items then there should be an equal number of positively worded items as well.

Sampling Considerations
Suppose you are trying to determine people's impression of your retail mountain climbing gear Web site. It would not be appropriate to ask a group of people who are not interested in climbing mountains to complete your survey. Nor would it make sense to administer the survey to a group of people that will never shop at your site. Analyzing data and making business decisions based on opinions from either of those groups may take your business backwards.
Sampling considerations take into account meaningful questions including, but not limited to:
1. Who is your target population?
2. How will you reach your target population?
3. Will the way you reach your target population unintentionally exclude certain members of that population?
4. How will you minimize biases and over- or under- representation of various age groups, cultural groups, income groups or occupational groups?

Statistical Analysis:
It is important that you know how you want to analyze the data before constructing the survey. The kind of analysis is directly related to overall research objectives of the survey and what questions you plan to answer as a result of the survey. It is essential that statistical assumptions (properties of the data that have an important influence on the validity of a statistic) are met prior to analyzing the data. Two types of statistical analyses are Descriptive Analysis and Inferential Analysis

Inferential Analysis provides that extra information, revealing if relationships are different enough (or similar enough) to warrant impacting your business decisions. Inferential Analysis involves applying statistics to the data to draw more general conclusions about the overall population. These conclusions may be based on the relationship between two topics.

For example, a customer satisfaction survey may center around two "satisfaction factors"- which can be derived using a statistical procedure known as factor analysis. These factors may be the core of what satisfies your customers. There are variables that predict satisfaction - which can be identified with another statistical modeling procedure known as multiple regression. Understanding which factors contribute to overall customer satisfaction and identifying the variables that predict satisfaction will reveal what is and is not working in your business.
Let us analyze your survey data for you. We will compile the results and mine the data using the appropriate statistical procedures and provide you with a practical, straightforward report with tables and graphs that you can use for your presentations.
Descriptive Statistics: There are a wide range of statistics that are used to describe or summarize data. Most tables in survey reports present descriptive statistics for one or more groups or sub-groups among the respondents. For surveys, these usually are limited to either the the various percentages of respondents who select from a range of categorical alternatives (e.g., "Very Dissatisfied" to "Very Satisfied", women v. men, position in the company, etc.) or as averages (mean) or median (and percentile) scores on items. Descriptive statistics computed for sample data (survey respondents) are used as estimates for the target population (e.g., all customers or employees) and as such, they should be accompanied by estimates of the margin of error or of the confidence interval that give some indication that is useful for gauging how accurate these descriptions are.

Tests for Significant Differences: Usually, tests for significant differences are conducted to examine changes that occur over time or differences between groups or subgroups. Which specific procedures are used will depend on the type of measurement scale used for the survey items under scrutiny. Differences in categorical data, for example, may be analyzed using a procedure called Chi2 and differences in average (mean) scores may be analyzed using procedures such as the t-test or the analysis of variance. By consensus, differences that are unlikely to be the result of chance fluctuations in the data alone (defined as a less than 0.05 probability) are referred to as statistically significant. Alternatively, you can also examine margins of error or confidence intervals for two groups or subgroups; if they do not overlap, the differences are likely to be statistically significant using other tests.

Statistical Modeling: There are a variety of procedures from simple correlations between items to more elaborate statistical techniques that allow researchers to evaluate how items relate to other items, individually or jointly. Regression analysis and more complex statistical modeling procedures are based on the intercorrelations among items and can be used to identify clusters of items that fall together into single dimensions (e.g., which items appear to reflect loyalty?) and to relate these items or dimensions to desired outcomes (e.g., how do satisfaction with compensation, opportunities for advancement, and the effectiveness of organizational communications and other factors relate to employee loyalty and commitment?). If you have a large enough sample (generally a few hundred or more respondents) these techniques can help you to develop a useful model of how your business objectives can be achieved.