Ticket to Work Evaluation (January 2006)

APPENDIX D
Qualitative Data Collection Methodology

For this report, we collected qualitative data, through in-person and telephone interviews, from a variety of sources, between February and October 2004. The interviews served to highlight the operations, experiences, and perspectives of organizations involved in the program, including both service providers and agencies overseeing the program. This appendix describes these data collection activities.

We conducted in-person or telephone interviews with staff of 47 organizations that were operating, or had operated in the past, as TTW providers. These included eight SVRAs, 29 active ENs, and 10 former ENs (TTW providers that had withdrawn from the program). The interviews took place between February and April 2004. Table D.1 provides an overview of selected characteristics of the TTW providers we interviewed. Subsequent sections describe the criteria used to select TTW providers for interviews. 1. Participating Providers Our goal was to obtain a broad representation of the experiences and views of the many types of providers participating in TTW. To achieve this, participating providers were randomly selected based on two primary criteria:

• Cumulative Number of Ticket Assignments as of November 2003. Providers were classified into three strata based on the number of Tickets ever assigned to the organization as of November 2003: large = more than 30 Tickets ever assigned; moderate = 5 to 30 Tickets ever assigned; and few or none = fewer than 5 Tickets ever assigned. We drew 40 percent of the sample from among those with a large number of Ticket assignments, 35 percent from among those with a moderate number of assignments, and 25 percent from among those with few or no Ticket assignments. In addition, those with a large number of assignments were further subdivided by SVRA/EN status. One-half of the providers with a large number of Ticket assignments were selected from among SVRAs, and one-half were selected from among ENs with more than 30 Ticket assignments.

Sources: Phases(s) served and EN/former EN/SVRA status based on data provided by MAXIMUS. Other characteristics based on information collected during the provider interviews.

• Phase(s) of the State(s) Served by the Provider. We selected 40 percent of the sample from among providers serving Phase 1 states, and 60 percent from among providers serving Phase 2 states.

We used two additional criteria to select a set of participating TTW providers for in-person interviews:

• Nature of SVRA Agreements with ENs. We selected three states in which to conduct in-person interviews with the SVRA and two ENs, based on our assessment of differences in the states’ SVRA-EN agreements. We classified states as having cooperative, neutral, or competitive agreements, and selected one state from each group.
  
  
• Cooperative agreements were defined as those which allowed ENs to accept Ticket assignments, receive support from the SVRA for guidance and counseling services, reimburse SVRAs in an equitable manner for any services provided, and allow ENs to receive supplemental or bonus payments in instances in which the SVRA holds the Ticket.
  
  
• Neutral agreements were defined as those that allow ENs to accept Tickets, refer beneficiaries to the SVRA for needed services, and do not require ENs to reimburse the SVRA over and above the cost of direct employment services. Some of these agreements may allow the SVRA to refuse to pay for services the EN has indicated it can provide in its application to the Program Manager and/or on a beneficiary’s IWP, or require the EN to reimburse the SVRA for basic counseling and guidance services.
  
  
• Competitive agreements were defined as those that require the EN to provide the counseling and guidance services normally provided by the SVRA or reimburse the SVRA over and above the cost of direct services provided.
  
  
• Categorized as a Non-traditional Provider by Program Manager. We selected five providers that had been categorized by the Program Manager as non-traditional ENs and that also had Ticket assignments. The Program Manager defines non-traditional ENs as entities whose target populations and services are not solely for people with disabilities. Of the 318 non-traditional ENs identified by the Program Manager, 110 had had at least one Ticket assignment as of November 2003.

The Program Manager further categorizes non-traditional ENs by type of organization. To ensure broad representation, we randomly selected one EN from each of the following groups:

• Workforce Investment Boards/One-Stop centers
• Employers
• Placement/staffing agencies
• Education/training institutions
• Other non-traditional providers not categorized in one of the four previous groups

Of the 37 participating providers initially selected for interviews, eight had to be replaced by randomly selected alternates for the following reasons: one refused interview, stating a lack of time as the reason; two declined interview, stating that they had new TTW directors who lacked sufficient knowledge to provide useful information; and five could not be reached after numerous attempts by fax and phone.

2. Former ENs

We conducted telephone interviews with 10 providers that had formally terminated participation in TTW. We refer to these as former ENs. We targeted providers from among the subset of former ENs who (1) had at least one Ticket assignment when they were active in the program and (2) had terminated participation in TTW in September 2003 or later. Thirteen former ENs met both these criteria. We were able to complete interviews with eight of them. Four of the 13 appeared to have dissolved their businesses, and a fifth could not be contacted after multiple attempts. In addition to these eight interviews, we used information from two pretest interviews with former ENs that had terminated participation prior to September 2003. These former ENs, however, both had no Ticket assignments.

We conducted telephone interviews with 14 providers who had attended an EN Opportunity Conference sponsored by the Program Manager during 2003, but who did not subsequently submit an application to become an EN. To ensure a diverse sample, we selected seven entities operating as SVRA contractors, and seven that were not SVRA contractors; this served as an indicator of traditional versus nontraditional providers.

We used the following process to select respondents. First, we randomly selected 75 providers from the list of over 2,500 entities that had attended the conferences in 2003. Second, we began calling the 75 potential respondents and asked a brief series of screening questions to ascertain their appropriateness for our sample.

• How interested had the organization been in becoming an EN when they attended the conference? If no interest—for example, they attended for information purposes only—we terminated the interview.
  
  
• Had the organization submitted an application to become an Employment Network? If yes, we terminated the interview.
  
  
• Was the organization a contractor to the SVRA?
  

The first seven entities that answered “yes” to the final question, and the first seven to answer “no” were interviewed in-depth about the program and their decisions not to become ENs.

We conducted telephone or on-site interviews with a number of SSA Central Office staff members in Baltimore during August 2004. Interviewees represented the Office of Employment Support Programs, the Office of Systems, and the Office of Operations. We also submitted questions and received written responses from the Office of Program Development and Research. These efforts focused upon recent program implementation activities including marketing, guidance to ENs and SVRAs, management information systems enhancements, TTW implementation by regional and field offices, EN and SVRA payments, and other SSA initiatives.

We also conducted in-person interviews with officials from the TTW Program Manager, MAXIMUS, Inc., in October 2004. These interviews focused on the Phase 3 implementation of TTW, and both new and ongoing administrative issues associated with TTW. We discussed ongoing marketing and training, Ticket mailings, program administration, data systems, EN enrollment and dropouts, and activities planned for 2005.