Online survey performance depends on more than the percentage of invitations that produce a response. The figures below show how completion, question design, device choice, survey mode and fieldwork cost can change the quality and efficiency of online research. Most of the evidence comes from Pew Research Center studies of its American Trends Panel (ATP), with additional comparisons from Survey Practice.
Contents
- Participation and completion
- Question-level nonresponse
- Devices and respondent experience
- Online versus phone survey modes
- Response rates across collection methods
- Survey timing, samples and retention
Participation and completion
Pew Research Center reports several different measures of online survey participation, and they describe different stages of the process. In its explanation of how it keeps American Trends Panel surveys from running too long, Pew says that about 80% or more of respondents selected for each online panel survey complete the questionnaire. It also says that survey-level response rates for its online panels are usually 80% or higher. These figures describe a mature panel process and should not be treated as a universal benchmark for every open-web survey.
Among respondents who log on and complete at least one item, about 99% typically finish the entire questionnaire, according to Pew Research Center. The same source puts the usual break-off rate for online panel surveys at about 1%. A separate ATP methodology summary reports a 2.7% break-off rate among panelists who logged on and completed at least one item in that particular study. The difference illustrates why the measurement period and study design matter when comparing rates.
Pew’s app-versus-web experiment produced a sharper contrast between collection modes. The experimental web group had an overall response rate of 84%, compared with 58% for the app group. Among people who agreed to participate, the web group’s completion rate was 94%, while the app group’s was 76%.
| Measure | Experimental web group | App group |
|---|---|---|
| Overall response rate | 84% | 58% |
| Completion among those agreeing to participate | 94% | 76% |
| Completed 10 or more surveys | 53% | 40% |
| Completed all 14 surveys | 15% | 9% |
The same experiment found that 53% of the web group completed 10 or more surveys, compared with 40% of the app group. Fifteen percent of the web group completed all 14 surveys, compared with 9% of the app group. Pew Research Center reports that 80% of invited panelists in the app group downloaded the app; that download rate represented 61% of everyone invited.
Question-level nonresponse
A high survey-level completion rate does not mean that every question receives an answer. Pew Research Center’s analysis of 30 open-ended questions drawn from 26 surveys examined an ATP panel of more than 10,000 U.S. adults. Across those questions, nonresponse rates ranged from 4% to 25%, with a median of 13%.
The format of the prompt mattered. Questions asking for a quick one-word response had a median nonresponse rate of 12%, and questions asking for a short sentence also had a median rate of 12%. Prompts asking for multiple sentences had a higher median nonresponse rate of 17%. Pew also reports that nonresponse rates above 4% are uncommon in its broader ATP open-ended analysis, while the 30-question study shows that some open-ended formats can be notable exceptions.
For comparison, closed-ended questions on the ATP had average nonresponse rates between 1% and 2%. Closed-ended questions made up more than 95% of questions in a typical ATP survey wave. This makes question mix important: a survey can have strong overall completion while still losing more answers in its smaller set of open-text prompts.
Respondent characteristics were associated with different open-ended nonresponse levels in the Pew analysis. Respondents with no political lean had a median nonresponse rate of 31%. Republicans and Republican-leaning independents had a median rate of 13%, while Democrats and Democratic leaners also had a median rate of 13%. These are subgroup results from the cited ATP analysis, not general rates for all online surveys.
Devices and respondent experience
The device used to answer an online survey is another measurable part of the respondent experience. In Pew Research Center’s open-ended analysis, respondents using a tablet had a median nonresponse rate of 15%, and respondents using a cellphone also had a median rate of 15%. Respondents using a desktop or laptop had a lower median nonresponse rate of 11% on the open-ended prompts studied.
The app-versus-web experiment also recorded a change in the platform mix among app-device users. In the initial survey, iOS users represented 57% of app-device users and Android users represented 42%. By survey 14, iOS had fallen to 51% and Android had risen to 48%.
Pew’s earlier ATP response-rate summary provides a longer view of mobile use. Across eight waves, the web response rate varied between 60% and 64%, while the cumulative response rate stayed near 3.5%. The share of respondents completing on a mobile device rose from 24% in wave 1 to 34% in wave 9. Smartphone completion rose from 16% in wave 1 to 26% in wave 9, while tablet completion stayed between 8% and 9% of total completions.
These figures come from different studies and periods, so they should not be combined into a single trend line. Together, they show why an online questionnaire needs to be usable on smaller screens and why device composition should be monitored when reviewing missing answers.
Online versus phone survey modes
Pew Research Center’s phone-versus-web mode experiment tested 60 questions with 3,003 respondents. Of these, 1,494 were assigned to complete the survey by phone and 1,509 were assigned to complete it on the web. Overall completion was 64% in the web group and 63% in the phone group.
The answers were not identical across modes. The mean absolute difference across the 60 questions was 5.5 percentage points, the median difference was 5 percentage points, and the differences ranged from 0 to 18 percentage points. Completion also varied across groups. Women were 6 percentage points more likely to complete on the web than on the phone. Non-Hispanic Black respondents had a 56% completion rate by phone and a 42% completion rate on the web.
| Result from the phone-versus-web experiment | Reported figure |
|---|---|
| Questions tested | 60 |
| Total respondents | 3,003 |
| Phone-assigned respondents | 1,494 |
| Web-assigned respondents | 1,509 |
| Mean absolute difference across questions | 5.5 percentage points |
| Median difference across questions | 5 percentage points |
| Range of differences | 0–18 percentage points |
Geography and political outlook also showed mode differences. Rural response was 10 percentage points higher by phone than on the web. Very conservative respondents were 11 percentage points more likely to respond by phone, while very liberal respondents were 4 percentage points more likely to respond on the web. These findings indicate that mode can affect both participation and the distribution of answers.
Response rates across collection methods
Survey Practice reported results from a New York Adult Tobacco Survey study that compared paper, online, landline, cellphone and address-based sampling (ABS) approaches. The paper survey response rate was 38.9%, the online survey response rate was 28.6%, the landline response rate was 22.2%, and the cellphone response rate was 14.7%. The overall phone response rate was 16.4%, while the overall ABS response rate was 33.7%.
The completed-case counts also differed. The online sample had 1,142 completed cases, the paper sample had 1,660 completed cases, and the phone sample had 4,186 completed cases. In the final recruitment mailing, about half of the online survey sample responded via the paper instrument, representing 567 cases. The paper survey sample had 1,633 cases that completed via the paper instrument in that final mailing.
Survey length differed by mode in the same study. The online survey contained 49 questions, the paper survey contained 41 questions, and the phone survey contained 140 questions when all skip-pattern possibilities were counted. The figures are study-specific and reflect the instruments and recruitment procedures used there.
Costs, timing, samples and retention
The New York Adult Tobacco Survey comparison also reported cost differences. The online survey protocol cost about 1.4 times as much as the paper survey protocol, while the phone survey cost about 2.8 times as much as the paper protocol. Paper surveys cost about $15 per complete. Incentive cost per complete was $21.89 for online surveys, $15.06 for paper surveys and $20.00 for phone surveys.
Pew Research Center’s ATP overview describes a different operating model. Panelists take surveys online on a computer, tablet or smartphone. Starting in mid-2024, people joining the panel were given a choice to take surveys online or over the phone. ATP surveys are typically available for 6 to 12 days, and Pew typically sends one or two ATP surveys each month. For a given wave, Pew typically samples only a subset of panelists, such as 5,000 out of more than 10,000 panelists.
One ATP methodology summary reported an overall response rate of 6.7% for its study. The ATP sample had 8,057 respondents out of 12,442 sampled, corresponding to a survey-level response rate of 65%. That sample included 5,499 ATP members and an oversample of 2,558 Black respondents from Ipsos KnowledgePanel. Four panelists were removed for extremely high refusal or straight-lining rates. The cumulative response rate, accounting for recruitment nonresponse and attrition, was 3.3%.
The same methodology summary reported a 22% survey-level response rate for the AmeriSpeak sample, a weighted AAPOR 3 recruitment response rate of 24%, a weighted household panel retention rate of 86% and a weighted cumulative response rate of 4.5%. Its address-based sampling discussion notes that 1,231 addresses in the “Other” category were never mailed because their drop-point counts were greater than four.
Taken together, these measurements separate invitation response, survey completion, panel retention and cumulative participation. Keeping those definitions alongside the measurement period and geography is essential when using online survey statistics to assess collaboration, idea gathering or other feedback programs.