Social Security Administration, Office of Policy: 2001 Customer Satisfaction Survey, Final Report

Prepared by The Gallup Organization, 901 F Street, NW, Washington, D.C. 20004
December 21, 2001

Table of Contents

Executive Summary

Background

Over the past few years, the Social Security Administration (SSA) has made a concerted effort to strengthen its research, statistical and policy analysis functions. Many of these functions have been consolidated in a reorganized and expanded Office of Policy. This report presents results from a survey of customer satisfaction among those using the products and services of the Office of Policy. Specifically, the goals for this survey were: To determine, among decisionmakers and others involved in Social Security and related issues, (1) the extent to which SSA's research, statistical, and policy analysis work was focusing on topics and issues of widespread concern, including new and emerging issues, and (2) the extent to which these research and analysis products were eliciting high levels of satisfaction from those who made use of them.

Customer Satisfaction

Eighty-six percent of respondents said they were somewhat or very satisfied with the overall quality of SSA's research, statistical, or policy information in the preceding 24 months. Thirty-seven percent were very satisfied. Current subscribers to SSA publications reported the highest levels of overall satisfaction compared to other sample groups.

In regard to specific attributes of both statistical tables and analytical articles, customers were most satisfied with accuracy (almost 90% very or somewhat satisfied), while ease of finding information was rated lowest (about 70% very or somewhat satisfied). Overall, statistical tables received slightly higher ratings than analytical articles. In terms of specific publications, respondents gave the highest ratings to the Social Security Bulletin (83% very or somewhat satisfied) and the lowest to Social Security Programs Throughout the World (64% very or somewhat satisfied).

Another attribute of customer satisfaction is SSA's performance in keeping up with new and emerging research and policy issues. Respondents did not give as high ratings on this measure as they did on overall satisfaction: 62% said they were very or somewhat satisfied; 18% said they were very satisfied. Consistent with other findings, Subscribers gave higher ratings than other sample groups on SSA's performance in keeping up with new and emerging issues.

Respondent Interest Areas

Respondents indicated somewhat more interest in Social Security issues than in SSI and somewhat more interest in disability than in retirement issues. From a list of topics of interest, the three mentioned most by respondents were current programs or proposals for change, economic well being of the aged or disabled, and work-related issues (e.g., disability and work). The three subgroups of interest mentioned most by respondents were disabled adults, women, and low-wage workers.

The top research or policy issues that respondents think SSA should be working on in the next year or two are Social Security reform and privatization, financing issues and solvency, and work incentives. In regard to administrative data, respondents are most interested in public use data files that have SSA data linked to surveys. Publications most widely used by the respondents were the Social Security Bulletin, the Annual Statistical Supplement, and Fast Facts and Figures about Social Security.

Respondent Recommendations

Only about two-fifths of respondents offered recommendations for improving Social Security's research, statistical, or policy products and services. The most common recommendation was for additional data or analysis in a particular topic area. Next most frequent were recommendations for improving the dissemination of information, including SSA's website, and for increasing the clarity of information.

Respondent Types

Nearly one-fourth of respondents worked for the federal government and another one-fifth were employed in higher education. Almost half had been interested in SSA related issues for more than 20 years. Of the respondents who used SSA information, almost half said they had received or sought SSA information more than 10 times in the preceding 24 months. Also of the respondents who used SSA information, about 90% got the information from published or hard copy materials and two-thirds from SSA web sites.

Background

Over the past few years, the Social Security Administration (SSA) has made a concerted effort to strengthen its research, statistical and policy analysis functions, recognizing that information generated through these activities is essential to the development of sound and effective policies and programs. Many of these research and policy functions have been consolidated in a reorganized and expanded Office of Policy (OP). This empirical and analytical information is provided to decisionmakers, both inside and outside of SSA, but it is also disseminated to members of the broader research/policy community who are concerned with social security and income maintenance issues. Among other venues, dissemination occurs through regular publications, postings on the Internet, briefings and other presentations, and responses to information requests.

In order to evaluate the perceived quality of SSA's research, statistical, and policy analysis work and to promote further improvements, SSA's FY 2000 Performance Plan included the development of a customer satisfaction survey to be conducted in FY 2001. The goals for the survey were as follows: To determine, among decisionmakers and others involved in Social Security and related issues, (1) the extent to which SSA's research, statistical, and policy analysis work was focusing on topics and issues of widespread concern, including new and emerging issues, and (2) the extent to which these research and analysis products were eliciting high levels of satisfaction from those who made use of them—specifically, in terms of such attributes as accuracy, comprehensiveness, and responsiveness. Information gained from the survey will be used to guide the agency's efforts for further improvements in its research, statistical, and policy analysis products and services.

Essentially, this report contains two main sections: one on customer satisfaction data and the other on respondent interests and recommendations. The first section of the report outlines customer satisfaction results in several different categories. Satisfaction was asked in terms of SSA's statistical tables and analytical reports, specific SSA publications, overall quality of information, and SSA's performance in working on new and emerging research and policy issues. In addition, some analyses are reported by sample group.

The second section of the report addresses additional respondent issues. The section begins with a discussion of respondents' research and policy interests, including particular topics and subgroups, and priority issues for SSA to address in the next year or two. The presentation then describes recommendations made by respondents for improving SSA's research, statistical, and policy analysis products and services.

A brief final section analyzes types of respondents who were very satisfied with these products and services, and appendices provide details on the survey's methodology and a facsimile of the mailed questionnaire.

Survey Results

The Sample and Respondents

The sample for this survey came from four different sources. Eighteen hundred cases were sampled from across four unduplicated lists. The first sample group (Decisionmakers) included 59 individuals in high-level positions in SSA and related Federal agencies. The second sample group (Subscribers) consisted of 885 individual subscribers to a cross section of SSA-OP publications. The third sample group (Non-Subscribers) included 512 cases from a series of lists including attendees at various research and policy-related conferences, as well as members of the National Academy of Social Insurance. The fourth sample group, from a database of stakeholders maintained by SSA's Office of Communication (OComm), included 344 individuals who had indicated an interest in receiving SSA's statistical publications but were not current subscribers.

The table below shows the total number in the sample and the corresponding number of respondents from each of the four sample sources who completed the survey and who identified themselves as actively interested in research, statistics, or policy issues relevant to SSA or SSI. There were also three active completes whose sample group could not be identified. Further information about sampling and response rates can be found in Appendix A: Survey Methodology.

Sample Source Number in
Sample
Number of
Active
Completes
Decisionmakers 59 45
Subscribers 885 483
Non-Subscribers 512 211
OComm 344 171
Unknown NA 3
Total 1,800 913
 

In addition to the 913 active completed questionnaires, 130 respondents indicated that they were inactive in this research or policy area and were therefore ineligible for most of the survey's questions. The final response rate for this survey, including active and inactive completes, was 58%.

Included in the 913 active completes are 73 respondents who said they were not users of SSA information in the two years preceding the survey. These non-users were included in questions about interests and recommendations, but were not asked the questions about customer satisfaction. The table below shows that the non-users were distributed fairly evenly across three of the sample sources.

Sample Source Number of Non-Users
among the Active Completes
Subscribers 21
Non-Subscribers 26
OComm 26
Total 73
 

Customer Satisfaction

Satisfaction With Statistical Tables And Analytical Articles Or Reports. Over three-quarters of respondents reported that they received information from SSA in the form of either statistical tables or analytical articles and reports.

In addition to these two forms specified in the questionnaire, some respondents said they received the information in another format. Of those who did receive information in another format, the most commonly mentioned was verbal reports, presentations, and discussions.

Q15: Was this information in the form of statistical tables, analytical articles or reports, or something else?
Bar chart showing that 82.9 percent of customers had received information in the form of analytical articles and 76.4 percent got information from statistical tables. Some respondents also checked the other-specify category. Several forms of information were coded from these responses: 6.4 percent of respondents got information through verbal reports, presentations and discussions; 3.5 percent got it through direct mail; and 2.1 percent got information from data files. An additional 5.3 percent of the responses were too diverse to code in separate categories and are shown here as a residual category, other.
 

The respondents who answered that they received information in the form of statistical tables or analytical articles and reports were asked several follow up questions to measure their satisfaction with various attributes of the information. These attributes included accuracy, clarity, comprehensiveness, objectivity, how up to date the information was, how useful it was, and how easy it was to find the information the respondent was looking for. The charts below summarize satisfaction with these attributes. Accuracy had the highest ratings for both statistical tables and analytical articles while ease of finding information had the lowest ratings.

Q16: Now thinking about the statistical tables that you received from SSA in the past 24 months, how satisfied were you with the following aspects of the information?
Bar chart linked to text description.
 
Q17: Now thinking about the analytical articles or reports that you received from SSA in the past 24 months, how satisfied were you with the following aspects of the information?
Bar chart linked to text description.
 

From these measures we created three index scores that collapse the answers for individual items. The first index score is the average of the seven "very satisfied" ratings for statistical tables; the second index is a comparable average for analytical articles and reports; and the third index score is a simple average of the first two scores, indicating the average percent very satisfied with both types of products combined (it should be noted that adjusting these calculations for slight differences in the numbers of product and attribute ratings had no meaningful impact on the averages). As shown in the chart below, respondents were more likely to be very satisfied with the statistical tables than with the analytical articles.

Index Scores: Average Percent "Very Satisfied" for Statistical Tables, Analytical Articles, and Both Combined
Bar chart. For statistical tables, an average of 43.3 percent of customers were very satisfied across the seven aspects, and for analytical articles an average of 36.9 percent were very satisfied. For the two forms combined, the average number of very satisfied customers was 40.1 percent.
 

We created another index score that averages the "very satisfied" ratings for each of the seven attributes across the two product types (analytical articles and statistical tables). The index scores are shown below. The highest index scores were for accuracy (59.3%) and the lowest for ease in locating the information (25.4%).

Index Scores: Average Percent "Very Satisfied" with Aspects of Statistical Tables and Analytical Articles
Bar chart. Presented in order, the average percent very satisfied were as follows: for accuracy, 59.3 percent; objectivity, 55.6 percent; clarity, 41.5 percent; usefulness, 34.4 percent; comprehensiveness, 34.3 percent; up to date, 30.3 percent; and easy to find, 25.4 percent.
 

Satisfaction With Specific SSA Publications. Overall, a majority of respondents were satisfied with the publications they were asked to evaluate. The publications are listed below along with the number of people who rated each publication. Respondents gave the highest ratings to the Social Security Bulletin; 83.3% were very or somewhat satisfied. The lowest ratings were for Social Security Programs Throughout the World; 64.0% were very or somewhat satisfied. Although Fast Facts and Figures About Social Security was number three in terms of the number of respondents who were either very or somewhat satisfied, it had the highest number of very satisfied respondents (44.1%).

Q18: Thinking now about specific SSA publications, please indicate how satisfied you were, overall, with each of the following publications that you used during the past 24 months?
Bar chart linked to text description.
 

We also created an index score to collapse the ratings of the various publications in Question 18. It is a weighted average of the "very satisfied" ratings for the ten publications (based on the number of responses received for each publication). The average score was 36.6%, slightly lower than the average for statistical tables (43.3%) but on par with analytical articles (36.9%).

Satisfaction With Overall Quality. Eighty-six percent of respondents said they were either very or somewhat satisfied with the overall quality of SSA's research, statistical, or policy information in the previous 24 months. However, more respondents said they were somewhat satisfied (48.8%) than said they were very satisfied (37.2%).

Q19: How satisfied were you with the OVERALL QUALITY of SSA's research, statistical, or policy information in the past 24 months?
Bar chart showing the percentage distribution for all five response categories to Question 19: 37.2 percent of customers were very satisfied; 48.8 percent were somewhat satisfied; 10.1 percent were neither satisfied nor dissatisfied; 3.7 percent were somewhat dissatisfied; and 0.2 percent were very dissatisfied.
 

Satisfaction With Overall Quality By Sample Group. There were some differences in overall satisfaction by sample group. Group differences were small for the combined ratings of very and somewhat satisfied together, but somewhat larger when restricted to very satisfied. Compared to the other sample groups, more Subscribers said they were very satisfied (42.0%) and slightly more said they were either very or somewhat satisfied (88.2%). Non-Subscribers had the lowest rating on the combined measure (82.4%), although respondents in the OComm sample had the fewest number of respondents saying they were very satisfied (23.6%). Decision makers landed in the middle of the satisfaction pack, coming in second for the combined overall score and third for the very satisfied rating.

Q19: How satisfied were you with the OVERALL QUALITY of SSA's research, statistical, or policy information in the past 24 months?
Bar chart showing the two satisfied ratings (very and somewhat satisfied) for each of the four groups who were sampled for the survey. Among Subscribers, 42.0 percent were very satisfied and 46.2 percent were somewhat satisfied with the overall quality of the information they had received. Comparable ratings for the other three groups are as follows: for Decisionmakers, 33.3 percent and 51.1 percent; for the OComm group, 23.6 percent and 60.0 percent; and for the Nonsubscriber group, 36.5 percent and 45.9 percent.
 

Satisfaction With SSA's Performance On New Issues. Respondents were also asked how satisfied they were with SSA's performance in identifying and working on new and emerging research and policy issues. Among those who expressed an opinion, respondents were not as satisfied as they were with the overall quality of SSA's current research. While almost two-thirds of respondents said they were very or somewhat satisfied with SSA's performance in this area, less than 20% said they were very satisfied.

Q8: One of SSA's goals is to identify and work on new and emerging research and policy issues. How satisfied were you with SSA's performance in this area over the past two years?
Bar chart showing the percentage distribution for all five response categories to Question 8: 17.7 percent of respondents were very satisfied; 44.3 percent were somewhat satisfied; 23.7 percent were neither satisfied nor dissatisfied; 10.8 percent were somewhat dissatisfied; and 3.5 percent were very dissatisfied.
 

Satisfaction With SSA's Performance On New Issues By Sample Group. There were modest differences in satisfaction levels among the four sample groups on this question as well. Subscribers reported the highest levels of satisfaction; 66.2% said they were very or somewhat satisfied. Respondents from the OComm list reported the lowest levels; 52.7% said they were very or somewhat satisfied.

Q8: One of SSA's goals is to identify and work on new and emerging research and policy issues. How satisfied were you with SSA's performance in this area over the past two years?
Bar chart showing the two satisfied ratings (very and somewhat satisfied) for each of the four groups who were sampled for the survey. Among Subscribers, 21.5 percent were very satisfied and 44.7 percent were somewhat satisfied with SSA's performance on new and emerging issues. Comparable ratings for the other three groups are as follows: for Decisionmakers, 18.2 percent and 47.7 percent; for the OComm group, 11.2 percent and 41.5 percent; and for the Nonsubscriber group, 13.3 percent and 45.6 percent.
 

Respondent Interests

Broad Interest Areas. A set of four questions asked respondents about their interest areas. The first set asked about interests in the broad topic areas of Social Security and SSI, retirement and disability. Respondents were somewhat more likely to be interested in Social Security issues than in SSI, and somewhat more interested in disability than in retirement issues. However, larger groups were interested in both Social Security and SSI (54.5%) and both retirement and disability issues (39.0%).

Q4: Are your professional interests more in issues relevant to Social Security, to SSI, or to both?
Bar chart showing the percentage distribution for all three response categories to Question 4: 29.0 percent indicated that they were more interested in Social Security related issues, 16.5 percent were more interested in SSI related issues, and 54.5 percent were interested in both.
 
Q5: Are your professional interests more in retirement issues, disability issues, or both?
Bar chart showing the percentage distribution for all three response categories to Question 5: 24.6 percent were more interested in retirement issues, 36.4 percent were more interested in disability issues, and 39.0 percent were interested in both.
 

Interest In Topics And Subgroups. Respondents were then asked more specifically about their interest in lists of topic areas and subgroups. More than half of respondents indicated that they were interested in current programs or proposals for change, economic well-being of the aged or disabled, work-related issues (e.g., disability and work, the retirement process), economic impact of Social Security or SSI programs, Social Security financing issues, and other government income security programs in the U.S. The top write-in response was Medicare/Medicaid and health-related issues.

Q6: Do you have a professional interest in any of the topics below?
Bar chart linked to text description.
 

The top five topic areas for each of the four sample groups were no different than the top topic areas for the population as a whole—they were even in the same order of importance with the exception of Decisionmakers. For Decisionmakers, current programs or proposals for change was the most commonly mentioned response to Question 6 (97.8%) but work-related issues and the economic impact of SS or SSI programs were tied for second place (88.9%), while Social Security financing issues were more important to Decisionmakers (86.7%) than the topic area economic well-being of the aged or disabled (80.0%).

Question 7 asked respondents about their interests in a list of specific population subgroups. Over one-half of respondents indicated that they were interested in disabled adults, women, low wage workers, racial or ethnic minorities, and disabled children.

Q7: Are particularly interested in any of the subgroups below?
Bar chart linked to text description.
 

The top five subgroups of interest were the same across all four sample groups.

Respondents' Priority Research And Policy Issues. Respondents were then asked what were the most important research or policy issues they thought SSA should be working on in the next year or two. The question was open-ended, and 23% of respondents didn't offer an opinion. Among those who did, the top response was Social Security reform/proposals, which was mentioned by 20.2% of respondents. More than 10% of respondents also mentioned financing/solvency/trust fund issues (14.7%), work incentive issues (14.4%), and issues concerning specific policies or provisions (12.8%).

Q9: Please list the most important research or policy issues that you think SSA should be working on in the next year or two (multiple responses allowed)
Bar chart linked to text description.
 

The responses to Question 9 did not differ greatly across the four sample groups. The top five most important research and policy areas were the same in the different sample groups as in the total sample with two exceptions. For Non-Subscribers, aged and retirement issues were more frequently mentioned than SS reform/proposals. For Decisionmakers, issues related to disabled children and youth were the fourth most common response.

Interest In Public-Use Data Files. Respondents were also asked if they would have any interest in having access to public-use data files about various topics. The highest number of respondents (72.9%) were interested in SSA data that has been linked to surveys. Over half of all respondents also expressed an interest in data on characteristics of Social Security beneficiaries (68.4%) and SSI recipients (62.4%).

Q10: In your work, would you be interested in having access to public-use data files in any of the following areas?
Bar chart. 72.9 percent of respondents would be interested in SSA data that has been linked to surveys; 68.4 percent would be interested in data on characteristics of Social Security beneficiaries; 62.4 percent would be interested in data on characteristics of SSI recipients; 49.8 percent would be interested in earnings history data for covered workers; and 16.0 percent would be interested in other data files.
 

Use Of Specific SSA Publications. Another way to measure the interest areas of respondents is to see which publications they said they used over the past 24 months in Question 18. Most of the publications listed were used by over half of the respondents.

Q18: Use of publications in the past 24 months.
Bar chart. 83.8 percent of respondents used the Social Security Bulletin; 76.1 percent used the Annual Statistical Supplement; 72.6 percent used Fast Facts and Figures About Social Security; 59.5 percent used the SSI Annual Statistical Report; 57.3 percent used Social Security Programs in the U.S.; 53.5 percent used Income of the Population 55 or Older; 51.0 percent used SSI Recipients by State and County; 49.3 percent used Income of the Aged Chartbook; 45.0 percent used Social Security Programs Throughout the World; and 43.2 percent used OASDI Beneficiaries by State and County.
 

Use of the publications did not differ much across the four sample groups. All groups listed the same publications as their top five.

Respondents' Recommendations for Improvement

Recommendations For Improvement. All respondents were asked an open-ended question about recommendations they had for improving Social Security's products and services. Forty-three percent did not respond, while an additional 16% said they had no recommendations. The most common recommendation was for additional data or analysis in particular topic areas (12%).

Q20: Please list any recommendations you have for improving Social Security's research, statistical, or policy products and services. (multiple responses allowed)
Bar chart linked to text description.
 

Among the small group of Decisionmakers, only 38% offered recommendations for improvements; four of the top five recommendations were similar to those from respondents as a whole although the rank order is somewhat different. The recommendation made most frequently by Decisionmakers (13%) was to improve the timeliness of research, statistical or policy products.

Q20: (Decisionmakers Only) Please list any recommendations you have for improving Social Security's research, statistical or policy products and services. (multiple responses allowed)
Bar chart linked to text description.
 

Reasons For Not Using SSA's Information. Another question that can be used to look for recommendations is Question 12. This question was asked of respondents who indicated active interest in the field but said they did not receive any research, statistical, or policy information from SSA publications, SSA web sites, or SSA staff in the preceding two years. The question asked these respondents if there was any particular reason they had not used SSA's information. Forty-one percent did not respond, and an additional 14% said there was no particular reason. The reason mentioned most frequently (18%) was that SSA's information was not related to the respondent's particular needs.

Q12: Is there any particular reason why you have not used SSA's information? (multiple responses allowed)
Bar chart showing the percentage responses to Question 12. The majority gave no reason -- either not responding (41.1 percent) or specifying no particular reason (13.7 percent). Reasons that were grouped and coded included: 17.8 percent indicated that they didn't need the information; 8.2 percent indicated that they didn't know it was available; and 6.8 percent said that they simply hadn't gotten the information. An additional 15.1 percent of responses were too diverse to be coded into meaningful categories and are listed here as other.
 

Respondent Types

Respondents were also asked several demographic or usage questions such as where they worked, how long they had been interested in Social Security related issues, how frequently they received or sought SSA information, and how they received that information.

Work Affiliation. The first question asked respondents to answer in what general category of organization they worked. Almost one-fourth (23.3%) of respondents worked in the executive branch of the federal government. Another 20.9% worked at a university or college.

Q1: Currently, which of the categories below best describes where you work?
Bar chart. 23.3 percent of respondents worked for the Federal government; 20.9 percent were at universities or colleges; 13.6 percent with nonprofit service organizations; 13.3 percent in state or local governments; 7.6 percent with private research or policy organizations; 4.3 percent in business or industry; 4.0 percent with advocacy or lobbying groups; 3.7 percent with professional associations; and 2.9 percent in a Congressional agency. An additional 1.4 percent indicated some other work affiliation, and 5.2 percent were self-employed, retired, or otherwise not affiliated in a work relationship.
 

Respondents with no work affiliation (for example, retirees or consultants) were most likely to say they were "very satisfied" with the overall quality of SSA's information in Question 19. Respondents who worked at nonprofit service organizations gave the lowest ratings for overall quality.

Percent "Very Satisfied" With the Overall Quality of SSA's Research, Statistical, or Policy Information by Employment Category
Bar chart. 68.4 percent of those with no work affiliation were very satisfied; 48.4 percent of those in private research or policy organizations; 42.9 percent of those in business or industry; 40.6 percent of those with advocacy groups; 40.0 percent of those with professional associations; 39.1 percent of those in universities or colleges; 35.2 percent of those with the Federal government; 32.4 percent of those with State or local governments; 30.8 percent of those in Congressional agencies; and 22.2 percent of those in nonprofit service organizations.
 

Respondents with no work affiliation (39.0%) were also most likely to be "very satisfied" with SSA's identification and work on new and emerging research and policy issues (Q8). Respondents in Congressional agencies (8.3%) were least likely to respond that they were "very satisfied."

Percent "Very Satisfied" With SSA's Performance Identifying and Working on New and Emerging Research and Policy Issues by Employment Category
Bar chart. 39.0 percent of those with no work affiliation were very satisfied; 27.1 percent of those in private research organizations; 22.9 percent of those in business; 17.7 percent of those with the Federal government; 16.5 percent of those in universities or colleges; 14.3 percent of those in nonprofit service organizations; 14.0 percent of those with state or local governments; 13.8 percent of those with professional associations; 8.8 percent of those with advocacy groups; and 8.3 percent of those in Congressional agencies.
 

Satisfaction By SSA Employment. SSA employees had only slightly higher satisfaction levels with overall quality than did non-SSA employees (80.4% vs. 78.8% very or somewhat satisfied).

Q19: How satisfied were you with the OVERALL QUALITY of SSA's research, statistical, or policy information in the past 24 months?
Bar chart showing the satisfaction ratings on Question 19 for two groups: among SSA employees, 33.3 percent were very satisfied with overall quality and 47.1 percent were somewhat satisfied; among non-SSA respondents, comparable ratings were 34.2 percent and 44.6 percent.
 

SSA employees were also slightly more likely to be satisfied with SSA's performance in identifying and working on new and emerging research and policy issues than were non-SSA employees (59.2% vs. 54.0% very or somewhat satisfied).

Q8: One of SSA's goals is to identify and work on new and emerging research and policy issues. How satisfied were you with SSA's performance in this area over the past two years?
Bar chart showing the satisfaction ratings on Question 8 for two groups: among SSA employees, 16.5 percent were very satisfied and 42.7 percent were somewhat satisfied; among non-SSA respondents, comparable ratings were 15.5 percent and 38.5 percent.
 

Years of Professional Interest. Respondents generally had quite a long interest in SSA related issues. Almost half (45.5%) had an interest for more than twenty years and fewer than 10% had an interest for less than five years.

Q3: During the course of your career, approximately how many years have you had some interest in issues relevant to Social Security or SSI?
Bar chart showing the percentage distribution for all four response categories to Question 3: 8.6 percent had been interested for less than 5 years; 16.0 percent for 5 to 9 years; 30.0 percent for 10 to 20 years; and 45.5 percent interested for more than 20 years.
 

Overall satisfaction was somewhat higher among respondents who had been interested in SSA related issues for more than twenty years (40.6% very satisfied).

Percent "Very Satisfied" With the Overall Quality of SSA's Research, Statistical, or Policy Information by Length of Time Interested in SSA Related Issues
Bar chart. 33.9 percent of those with less than 5 years interest were very satisfied; 32.8 percent of those with 5 to 9 years interest; 35.6 percent of those with 10 to 20 years interest; and 40.6 percent of those with more than 20 years interest were very satisfied.
 

There was no clear relationship between the length of time the respondent had been interested in SSA related issues and satisfaction with SSA's identification and work on new and emerging research and policy issues. Respondents who said they were interested in SSA related issues for five to nine years and more than twenty years were the most likely to be very satisfied with overall quality (around 19.5%); respondents who were interested in SSA related issues for between ten and twenty years were the least likely to be very satisfied (14.8%).

Percent "Very Satisfied" With SSA's Performance Identifying and Working on New and Emerging Research and Policy Issues by Length of Time Interested in SSA Related Issues
Bar chart. 16.7 percent of those with less than 5 years interest were very satisfied; 19.5 percent of those with 5 to 9 years interest; 14.8 percent of those with 10 to 20 years interest; and 19.4 percent of those with more than 20 years interest were very satisfied.
 

Frequency of Use. Of the respondents who identified themselves as users of SSA's information, a slight majority (53.2%) received or sought the information fewer than ten times in the preceding twenty-four months. About one-quarter reported use more than 20 times, including about 10 percent using more than 50 times.

Q13: During the past 24 months, about how many times have you received or sought this kind of information from SSA?
Bar chart showing the category responses to Question 13: 16.0 percent indicated less than 3 times; 37.2 percent, 3 to 9 times; 21.1 percent, 10 to 19 times; 15.9 percent, 20 to 50 times; and 9.9 percent, more than 50 times.
 

Not surprisingly, respondents who had received or sought information from SSA more than 50 times in the past 24 months were the most likely to say they were very satisfied with the overall quality of SSA information.

Percent "Very Satisfied" With the Overall Quality of SSA's Research, Statistical, or Policy Information by Number of Times Received or Sought SSA Information
Bar chart. Of those respondents who had received information less than 3 times, 39.3 percent were very satisfied; of those who got information 3 to 9 times, 34.7 very satisfied; of those who got information 10 to 19 times, 35.3 percent very satisfied; of those who got information 20 to 50 times, 40.8 percent very satisfied; and of those who got information more than 50 times, 42.7 percent very satisfied.
 

Frequency of use also seems to be related to respondent satisfaction with SSA's performance in identifying and working on new and emerging research and policy issues. Respondents who had received or sought the information twenty times or more were more likely to say they were very satisfied than less frequent users.

Percent "Very Satisfied" With SSA's Performance Identifying and Working on New and Emerging Research and Policy Issues by Number of Times Received or Sought SSA Information
Bar chart. Of those respondents who had received the information less than 3 times, 15.3 percent were very satisfied; of those who got information 3 to 9 times, 17.0 percent very satisfied; of those who got information 10 to 19 times, 17.6 percent very satisfied; of those who got information 20 to 50 times, 23.3 percent very satisfied; and of those who got information more than 50 times, 21.8 percent very satisfied.
 

Source of Information. Almost 90% of respondents said they got information from SSA through published or hard copy materials. Two-thirds got information from SSA's web sites.

Q14: How did you get information from SSA?
Bar chart. 89.3 percent of respondents got the information from published or hard copy materials; 66.7 percent from SSA's Web sites; 43.8 percent from SSA staff by telephone or fax; 36.9 percent through briefings or presentations by SSA staff; 25.3 percent from SSA's E-news newsletter; and 6.5 percent got information in other ways.
 

Appendix A: Survey Methodology

Survey Design

Questionnaire Design And Pretests. The questionnaire was developed from an initial draft by SSA-OP. It was designed to focus on two critical components of customer satisfaction. The first part of the questionnaire was designed to measure whether individuals active in this field thought that SSA was producing research and policy information on topics of interest and importance to them. The second part of the questionnaire focused on users of SSA's research, statistical, or policy information within the two years preceding the survey, and their level of satisfaction with that information.

The questionnaire was pretested three different ways. First, Gallup conducted a focus group of nine users of OP's products to talk about the concepts in the questionnaire and to discover how they evaluated OP's products and services. Second, after the questionnaire was drafted, Gallup researchers interviewed four SSA employees to get their feedback on the questionnaire. Finally, Gallup interviewers conducted separate telephone interviews with nine OP customers, alternating between two slightly different versions of the questionnaire.

Sample Frame And Selection. Given the subject matter of this survey—research, statistical, and policy information from SSA—it was impossible to operationally define the population of customers or intended customers in a comprehensive manner. Clearly, the population would include subscribers to SSA-OP's publications. But the population would also include non-subscribers who are active in this area of research/policy and who make use of SSA materials in libraries and over the internet, who contact SSA with requests for information, or who receive information through SSA presentations and briefings. This population could not be fully identified. In addition, there was no obvious list for another segment of the target population, "Decisionmakers," as mentioned in SSA's Performance Plan.

After exploring the availability and utility of various lists and sources of names, OP's survey team, in consultation with Gallup and others, developed four lists that seemed a reasonable approach to a sampling frame for this first OP Customer Satisfaction Survey.

The first of these lists was composed of a small group of Decisionmakers, individuals in high-level positions in SSA and related federal agencies who are presumed to have enhanced potential for influencing policy decisions. This list of 59 names was prepared by OP's executive staff.

A second list consisted of individual Subscribers to a cross-section of SSA-OP publications. Specifically, those publications included: 1) Social Security Bulletin, 2) ORES Working Papers, 3) Social Security Programs Throughout the World, and 4) SSI reports, including SSI Disabled Workers and WIN Provisions, Children Receiving SSI, and SSI Annual Statistical Report. There were 885 non-duplicative names on this list.

The third list, Non-Subscribers who were known to have some active interest in this research/policy area, was more difficult to construct. Various possibilities were explored before the list was compiled, primarily from the following sources: members of the National Academy of Social Insurance, attendees at recent conferences of the Retirement Research Consortium (a consortium of researchers whose work is funded by SSA), and participants at recent conferences sponsored by several other research or policy organizations. For the latter, an effort was made to select organizations reflecting a broad spectrum of political philosophies. Many of these individuals were on more than one of these lists, and others were already included in the Decisionmakers or Subscribers lists. A non-duplicative count of this segment of the target population yielded 1,192 names.

Finally, SSA-OP obtained additional non-subscribers from a list recently developed by SSA's Office of Communications (OComm). Based on input from SSA's field offices throughout the country, this OComm "stakeholder" list identifies individuals at the community level who have expressed professional interest in SSA's policies and programs. (Many of these individuals are, for example, state or local government workers in social services or health programs.) Among specific areas of interest, a subset of these persons had identified themselves as interested in "statistical publications" from SSA. The non-duplicative count of these individuals was 802.

Certain individuals were excluded from these lists: employees in SSA-OP, persons living outside the U.S., and individuals who had participated in the focus group or pretest, described above. Members of Congress and congressional staffs were also excluded from the sample for the 2001 survey. In the early stages of planning this survey, it was learned that interviews and focus group sessions with key congressional staffers were being conducted by SSA-OComm during 2000-2001 in an effort to evaluate SSA's services to Congress, including the provision of statistical and policy analysis information. To avoid duplication of effort and undue respondent burden, SSA decided not to request Congressional participation in OP's 2001 customer satisfaction survey.

The sequence of the four sampling lists described above and the exclusion of duplicate listings in each successive list reflected our decisions about different levels of importance and choices for a sample design.

First, since we intended to reach some conclusions about how well SSA is serving Decisionmakers (as suggested in the Performance Plan), a decision was made to include 100% of this small group in the survey sample. The second group (Subscribers) was also considered of primary importance to the survey's objectives. In order to develop a reliable baseline measure of customer satisfaction from the survey, it was essential to obtain a reasonably large number of responses from recent "users" of SSA's research and policy information. Of the four sampling lists, it seemed that the Subscriber group would likely include the highest proportion of these users. Thus, we sampled 100% of this group.

Among the third and fourth groups (Non-Subscribers who are actively interested in this research/policy area and the OComm list), we expected to find a number of users, but were also concerned about the interests and needs of intended customers who may not have made recent use of SSA's products and services. This important but secondary objective of the survey, along with the combined size of these two groups (roughly twice the size of the Subscriber group), suggested a lower sampling ratio for them.

There were two additional considerations in choosing the sampling ratios for these two groups and completing the sample design. First, while we did not plan to calculate statistical estimates (e.g., variances, confidence intervals) based on this imperfect specification of the population, we considered it important to achieve a completed sample of at least 900 total cases to enable meaningful subgroup analyses. Second, while Gallup planned to make every effort to achieve an 80% response rate, we also thought it was prudent to consider the possibility of a somewhat lower rate.

With these considerations in mind and with 944 respondents already included from the first two groups, a sampling ratio of .4293 was required for the third and fourth groups. The following table presents total numbers and numbers selected for the sample from each of the four sampling lists.

Stratum/Sample Group Number on
Sample Frame
Number in
Sample
1. Decisionmakers 59 59
2. Subscribers 885 885
3. Non-Subscribers 1,192 512
4. OComm database 802 344
Total 2,938 1,800
 

Data Collection

The first step in data collection was to complete missing contact information for the selected sample. This was done in several stages. The first was for a Gallup contractor to match cases with a missing address or telephone number against a computerized database. The second was to have a Gallup staff member look up any cases that were still missing contact information. After the initial mailing of the questionnaire, Gallup and SSA-OP staff looked for contact information for any cases that were incomplete at that point and still missing telephone numbers.

The data collection schedule is listed in the table below. The first step was to mail the prenotification letter. This was followed by the survey packet, which included a cover letter, the survey questionnaire booklet, and a postage-paid return envelope. A reminder postcard was mailed a week after the survey packet. Telephone interviewers attempted to contact remaining non-respondents beginning two weeks after the reminder postcard was mailed and continuing for four more weeks.

Data Collection Schedule
Item Date
Prenotification Letter June 25, 2001
Survey Packet Mailed July 2, 2001
Reminder postcard Mailed July 9, 2001
Telephone Interviewing July 23 - August 24, 2001
 

Once the questionnaires were collected, the results from the three modes of data collection were combined into one data file. The file was examined for duplicates and the duplicates were removed by taking the most complete case and filling missing items with complete information from the duplicate case if possible. Then the least complete cases were eliminated from the file. This second step removed three cases that had 12 or more of the 20 questionnaire items missing.

The table below shows the data collection results. There were 913 respondents who said they were active in the field of Social Security related research and policy, and another 130 respondents who said they were inactive in the field. Over half of the number of active completes were collected with the mail questionnaire.

Status Frequency Percent
Incomplete 757 42.1
Complete - Mail 514 28.6
Complete - Web 104 5.8
Complete - Telephone 295 16.4
Complete - Inactive 130 7.2
Total 1,800 100.0
 

The table below shows the final response rates for the total sample and by sample group. There were three cases that were returned with no questionnaire ID. Therefore it is unknown which sample group these cases came from.

Sample Group Complete -
Active
Complete -
Inactive
Number in
Sample
Response
Rate
(percent)
Decisionmakers 45 0 59 76.3
Subscriber 483 53 885 60.6
Non-Subscriber 211 38 512 48.6
OComm 171 39 344 61.0
Unknown 3 0 NA NA
Total 913 130 1800 57.9
 

The final status of the 757 incompletes is listed below. "Other" includes respondents who say they are ill or had a death in the family, or say they already completed the survey through the mail or web site. A "non-target" number is when the respondent is not at the number given. There were 226 cases where contact had been made, but the interviewer needed to call back the respondent to complete the interview; among these callback numbers, all numbers had been called at least once, with an average number of calls being 8 and the maximum 18. The table below shows the final outcome data for the 757 people who did not participate:

Final Sample Status Frequency Percent
Unlocatable - Mail 57 7.50
Unlocatable - Telephone 41 5.39
Call back 226 29.74
Other 241 31.71
Refused 47 6.18
Deafness/Language problem 1 0.13
Non-target 13 1.71
Busy 6 0.79
Answering Machine 37 4.87
No answer 22 2.89
Non-working or disconnected number 66 8.68
Total 757 100.00
 

Data File Preparation and Analysis

After the initial steps in the data file preparation, the sample variables were added to the data file including: the sample source, whether or not the respondent is an SSA employee, which publication type the respondent subscribes to, and the ICS code variable (a code that identifies a small subset of cases that will be analyzed by SSA as part of a separate project on internal customer satisfaction).

The open-ended questions were coded after data collection. For questions where the verbatim response was an "other-specify" question, the response was examined to see if it fit into any of the existing response categories. If it did, it was added to the existing response category. Proper nouns, place names, expletives, and unintelligible words were removed from the verbatim response and replaced with a code such as (undecipherable). Up to three responses were coded for each open-ended response. The coding was completed in three stages: first by a Gallup coder, then a Gallup researcher reviewed the codes, and finally all verbatims and codes were sent to SSA-OP for review.

The next step in the data file preparation was to "clean" the data to make sure there were no inconsistent responses. For the data cleaning Gallup did the following:

Item nonresponse was generally below 5%. Exceptions are listed in the table below. Questions 9 and 20 were open-ended questions, which typically have higher rates of nonresponse. No imputation was done for any missing values.

Nonresponse greater than 5%
  Nonresponse
rate (percent)
Question 7 7
Question 9 23
Question 18 6–12
Question 20 43
 

No weighting was done on the sample because the population of users is not fully defined. SSA-OP continues to try to identify their target audience. However, with the public nature of their publications, there may never be a finite population list of those using the SSA materials.

Appendix B: Survey Questionnaire

OMB No. 0960-0526
Expires: 12-31-02
SOCIAL SECURITY ADMINISTRATION - OFFICE OF POLICY
CUSTOMER SATISFACTION SURVEY

The Social Security Administration (SSA) has asked The Gallup Organization to find out what you think about its research, statistical, and policy analysis efforts.

SSA produces a variety of statistical and analytical information about its programs and about related issues. The information is disseminated in various ways, including the Internet, through SSA staff, and in publications and reports (e.g., the Social Security Bulletin, the Annual Statistical Supplement).

The purpose of this survey is to help SSA improve these products and services. As you will see, some of the survey questions are directed to anyone with professional interests in this area, while other questions are restricted to those who have actually used some of the SSA's research, statistical, or policy information in recent years. Your opinions are still important to SSA and Gallup, even if you are a "non-user".

Please take a few minutes to fill out this questionnaire and send it back to us in the enclosed postage-paid envelope. We estimate that the survey will take about ten minutes to complete. Your answers will be kept strictly confidential, and will be merged with the responses of others like yourself.

If you wish to take this survey online, you may do so by logging into http://ssaop.gallup.com The code (PIN) you should use to access the survey is on the cover letter attached to this questionnaire.

Thank you again for your participation. We greatly appreciate your time and your help.

The PRIVACY ACT requires us to notify you that we are authorized to collect this information by Section 702 of the Social Security Act. You do NOT have to provide the information requested. However, the data you provide will help the Social Security Administration - Office of Policy to evaluate and improve its products and services. The Gallup Organization guarantees the confidentiality of every respondent.

The PAPERWORK REDUCTION ACT OF 1995 requires us to notify you that this information collection is in accordance with the clearance requirements of 44 U.S.C. § 3507, as amended by section 2 of the Paperwork Reduction Act of 1995. We may not conduct or sponsor, and you are not required to respond to, a collection of information unless it displays a valid OMB control number. We estimate that it will take you about 10 minutes to complete this questionnaire. This includes the time it will take you to read the instructions and fill out the questionnaire.

  1. Currently, which of the categories below best describes where you work? (Mark only one)
    U.S. Congress or Congressional agency
    Federal government, Executive branch
    State or local government
    University or college
    Private research or policy organization
    Business or industry
    Nonprofit service organization
    Professional or trade association
    Advocacy or lobbying group
    The press or other media
    No work affiliation (for example, retired)
    Other (please specify)
  2. At any time in the past 24 months, have you had any professional interest in research, statistics, or policy issues relevant to Social Security or to Supplemental Security Income (SSI - the means tested public assistance program administered by SSA)?
    Yes → Continue with #3
    No → Stop - Please return your survey in the enclosed envelope.
  3. During the course of your career, approximately how many years have you had some interest in issues relevant to Social Security or SSI?
    Less than 5 years
    5 to 9 years
    10 to 20 years
    More than 20 years
  4. Are your professional interests more in issues relevant to Social Security, to SSI, or to both?
    Social Security
    SSI
    Both
  5. Are your professional interests more in retirement issues, disability issues, or both?
    Retirement issues
    Disability issues
    Both
  6. Do you have a professional interest in any of the topics below? (Mark all that apply)
    Current programs or proposals for change (Social Security or SSI)
    Social Security financing issues, including status of the trust funds
    Economic impact of the Social Security or SSI programs
    Work-related issues (for example, employment and earnings, the retirement process, facilitating work among the disabled)
    Other government income security programs in the U.S.
    Private sources of income security (for example, individual savings, employer pensions)
    Economic well-being of the aged or disabled
    Income security systems in other countries
    Other SSA-related topics (please specify)
  7. Are you particularly interested in any of the subgroups below? (Mark all that apply) Women
    Racial or ethnic minorities
    Low wage workers
    Early retirees
    Older workers
    Disabled children
    Disabled adults
    The oldest old (age 85+)
    Widowed, divorced, or never married persons
    Immigrants, non-citizens
    Other (please specify)
  8. One of SSA's goals is to identify and work on new and emerging research and policy issues. How satisfied are you with SSA's performance in this area over the past two years?
    Very satisfied
    Somewhat satisfied
    Neither satisfied nor dissatisfied
    Somewhat dissatisfied
    Very dissatisfied
    No opinion
  9. Please list the most important research or policy issues that you think SSA should be working on in the next year or two.
  10. SSA has various administrative data files with restricted access to protect confidentiality. In your work, would you be interested in having access to public-use data files in any of the following areas? (Mark one box for each item)

    Interested in data files Not interested in data files
    Earning histories of covered workers
    Characteristics of Social Security beneficiaries (including benefit types and amounts)
    Characteristics of SSI recipients
    SSA data that has been linked to surveys (for example, Census Bureau surveys)
    Other data files of interest (please specify)
  11. During the past 24 months, have you received any research, statistical, or policy information from SSA publications, SSA web sites, or SSA staff?
    Yes → Continue with #13
    No → Continue with #12
  12. Is there any particular reason why you have not used SSA's information?

    (If you answered #12) → Skip to #20
  13. During the past 24 months, about how many times have you received or sought this kind of information from SSA?
    Less than 3 times
    3 to 9 times
    10 to 19 times
    20 to 50 times
    More than 50 times
  14. How did you get the information from SSA? (Mark all that apply)
    Published or hard copy materials
    SSA's "E-news" newsletter
    SSA's web sites
    By telephone or fax from SSA staff
    Briefings or other presentations by SSA staff
    Other (please specify)
  15. Was this information in the form of statistical tables, analytical articles or reports, or something else? (Mark all that apply)
    Statistical tables
    Analytical articles or reports
    Other (please specify)
  16. (If "Statistical tables" in #15) Now thinking about the statistical tables that you received from SSA in the past 24 months, how satisfied were you with the following aspects of the information? (Mark one box for each item)

    Very satisfied Somewhat satisfied Neither satisfied nor dissatisfied Somewhat dissatisfied Very dissatisfied No opinion
    The accuracy of the information
    The clarity of the information
    The comprehensiveness of the information
    The objectivity of the information
    How up to date the information was
    How useful the information was to you
    How easy it was to find the information you were looking for
  17. (If "Analytical articles or reports" in #15) Now thinking about the analytical articles or reports that you received from SSA in the past 24 months, how satisfied were you with the following aspects of the information? (Mark one box for each item)

    Very satisfied Somewhat satisfied Neither satisfied nor dissatisfied Somewhat dissatisfied Very dissatisfied No opinion
    The accuracy of the information
    The clarity of the information
    The comprehensiveness of the information
    The objectivity of the information
    How up to date the information was
    How useful the information was to you
    How easy it was to find the information you were looking for
  18. Thinking now about specific SSA publications, please indicate how satisfied you were, overall, with each of the following publications that you used during the past 24 months. (Mark one box for each item)

    Very satisfied Somewhat satisfied Neither satisfied nor dissatisfied Somewhat dissatisfied Very dissatisfied Did not use
    Social Security Bulletin
    Annual Statistical Supplement
    Fast Facts and Figures About Social Security
    Social Security Programs in the U.S.
    Social Security Programs Throughout the World
    SSI Annual Statistical Report
    SSI Recipients by State and County
    OASDI Beneficiaries by State and County
    Income of the Population 55 or Older
    Income of the Aged Chartbook
  19. How satisfied were you with the OVERALL QUALITY of SSA's research, statistical, or policy information in the past 24 months?
    Very satisfied
    Somewhat satisfied
    Neither satisfied nor dissatisfied
    Somewhat dissatisfied
    Very dissatisfied
  20. Please list any recommendations you have for improving Social Security's research, statistical, or policy products and services.

THANK YOU FOR PARTICIPATING IN THIS SURVEY.
PLEASE RETURN YOUR COMPLETED QUESTIONNAIRE IN THE ENCLOSED ENVELOPE.