European Commission Directorate-General for Employment,
Social Affairs and Equal Opportunity
and
University of Maryland School of Public Policy
Saturday, November 7, 2009
9:00 am–5:30 pm

Note to participants: This is a somewhat unusual conference, as one of its key objectives is to provide staff of the European Commission with information about the implementation of the U.S. Workforce Investment Act that will be of relevance to their thinking about the next round of activities supported by the European Social Fund (ESF), which is funded in seven-year cycles. The European Commission has roughly the same relationship to EU member states that the U.S. Department of Labor (DoL) has to U.S. states. Hence, the EC staff has asked that, to the extent feasible, you focus your papers and remarks on how the DoL or the U.S. Congress might learn from and respond to your findings. One way that you could think of this is to ask, when relevant, what your comments suggest concerning WIA’s reauthorization. We understand that this will not always be a perfect fit, but we appreciate any efforts you can make to provide a framework for the EC’s deliberations.

Of course, the secondary purpose of our meeting is to share information about WIA issues for a U.S. audience. Hence, we hope to publish your papers in a suitable fashion after the meeting.

8:30	Breakfast
9:00	Welcoming remarks Douglas Besharov, University of Maryland School of Public Policy  Ines Hartwig, European Commission Directorate-General for Employment, Social Affairs and Equal Opportunity Phoebe Cottingham

1. The Workforce Investment Act (WIA)
This opening session will provide an overview of the Workforce Investment Act and of efforts to assess its effectiveness over its ten-year history.

• Dianne Blank, U.S. General Accounting Office, will summarize the focus of the WIA legislation, outline the program’s implementation, and recommendations for improvement.
  AbstractPaperPresentation
• Paul Decker, Mathematica Policy Research, Inc., will summarize the history of research and evaluations accompanying WIA implementation in the context of over thirty years of prior studies of job training policies and programs.
  PaperPresentation

2. Management and implementation
This two-part session, divided at midpoint for comments and questions, will first explore some of the ways that WIA agencies seek to bring employers closer to the job training system. Much attention has been given to strategies for streamlining who is doing what, avoiding duplication of effort, helping employers more quickly find new workers, and orienting the job training providers to employer requirements.

• Jason Turner, Heritage Foundation, will describe the greater efficiencies that can be gained from service integration, drawing from examples of efforts to integrate WIA services with the training provided welfare recipients under TANF, with the latter program applying program participation requirements in return for the training and access to placement.
  AbstractPaperPresentation
• Lee Munnich, University of Minnesota, and Burke Murphy, the Minnesota Department of Employment and Economic Development, will describe Minnesota's efforts to integrate and better coordinate WIA providers with those responsible for business development.
  AbstractPaperPresentation

Discussant: Wade Horn, Deloitte Consulting

EC comments and questions, followed by general discussion.

The second part of this continuing discussion will explore the use of information and provider access systems (voucher-like mechanisms) to match employers with job seekers. The organizational entities established in WIA workforce areas are called “One-Stop” centers with services for both employers and job seekers. At the centers, training providers offer their services, job seekers with Individual Training Account credit choose their training provider, and employers look for job seekers. The speakers will focus on different aspects of this service delivery system, drawing on examples that illustrates how implementation of this central objective for WIA has varied and identifying the key challenges.

• David Long, Abt Associates, will summarize the research on job training programs in the United States that used voucher-like mechanisms as a component of the programs and will distill the cross-site lessons from these programs.
  AbstractPaperPresentation
• Carl Van Horn and Aaron Fichtner, Rutgers University, will report on case studies in four states describing how state listing of training providers that show the past performance records of training providers influenced job seeker utilization of training credits.
  AbstractPaperPresentation
• David Heaney, MAXIMUS, will describe the loss of for-profit center operators that had emerged as new actors early under WIA, and the return to dominance of non-profit or government providers, with possible weakening of performance management and overall success.
  AbstractPaperPresentation

EC comments and questions, followed by general discussion.

Lunch

Gerri Fiala, Deputy Assistant Secretary, Employment & Training Administration (ETA), U.S. Department of Labor, will discuss the Obama Administration's approach to management of the current WIA program and the principles that will be used in reauthorizing WIA legislation.

3. Impact evaluation

This two-part session, divided at midpoint for comments and questions, will first explore some of the advantages and challenges of using administrative data to evaluate WIA. It will provide an overview of the main methodological issue encountered in job training evaluations: How to control for selection bias. This will be followed by descriptions of key issues confronting evaluators of WIA, including how the program’s complexity may force deployment of both experimental designs and nonexperimental designs to arrive at good estimates of average effects pinned to particular components, results for particular subgroups, and in differing economic conditions.

• Maureen Pirog and Haeil Jung, Indiana University, will provide an overview of the selection bias issue, including comments about how close the impact estimates from the counterfactuals possible within a nonexperimental design “fit” the estimates derived from experimental designs.
  AbstractPaperPresentation
• Kevin Hollenbeck, Upjohn Institute, will illustrate the ways in which performance or administrative data can be used to assess employment and earnings impacts under WIA  and the WIA programs’ rates of return for individuals, state taxpayers, and society.
  AbstractPaperPresentation
• Carolyn Heinrich, University of Wisconsin-Madison, Peter Mueser, University of Missouri, Kenneth R. Troske, University of Kentucky, Kyung-Seong Jeon, University of Missouri, and Daver C. Kahvecioglu, IMPAQ International, will report on the largest nonexperimental evaluation of WIA so far. It drew the comparison groups from other administrative records in twelve states to examine participant gains for Adult and Dislocated Worker participants.
  AbstractPaperPresentation

Discussant: Robinson Hollister, Swarthmore College

EC comments and questions, followed by general discussion.

This second part of this continuing discussion will explore the advantages and limitations of randomized experiments. Past evaluations of U.S. job training programs using randomized experiments had significant impacts in the past on policy decisions and on development of new initiatives. WIA grew out of the demise of the prior program in job training, the Job Training Partnership Act (JTPA) that had a 16-year life until the 1998 installation of WIA. Through an experimental study, the federal evaluation of JTPA produced a basically “no-effects found” result, and in the case of dropout youth, found a significantly reduced rate of job success and earnings among program participants. Some experts, however, have criticized the JTPA study.

• Stephen Bell, Jacob Klerman, and Larry Orr, Abt Associates, will summarize major evaluation lessons from over forty years, and will consider whether administrative data is sufficient for establishing program performance estimates that are reasonable and controlled, to some extent, for selection bias.
  AbstractPaperPresentation
• Sheena McConnell, Peter Schochet, and Alberto Martini, Mathematica Policy Research, Inc., will describe alternative evaluation designs that may use nonexperimental and/or experimental methods to examine the effectiveness of WIA program components, thereby capturing possible differences between efforts that emphasize particular components.
  AbstractPaperPresentation
• Jeffrey Smith, University of Michigan, will describe lessons to be drawn from European experiences and how U.S. experiences might be applicable to European evaluations of job training programs (noting the parallels and differences with the U.S. government evaluation sector).
  AbstractPaperPresentation

Discussant: Rebecca Maynard, University of Pennsylvania

EC comments and questions, followed by general discussion.

4. Performance management and program evaluation

This two-part session, divided at midpoint for comments and questions, will first explore “data-driven decision-making” at the DoL level, focusing on some of the past weaknesses of WIA performance management and how it might be improved through the use of rigorous program evaluation techniques. The key assumption is that by establishing and maintaining accurate and regularly updated records that track individuals for an extended period of time, with post-program information added regarding outcomes, job training managers can identify which program components work best, and can look at how job seeker characteristics correlate with program performance. The central question, however, is whether the DoL and others can judge whether the state or local program is producing “value-added” over competing units.

• Cynthia Fagnoni, GAO, will summarize how the WIA accountability system was intended to function, and the difficulties the Department of Labor has faced in developing, implementing and evaluating the system.
  AbstractPaperPresentation
• Burt Barnow, Johns Hopkins University, will describe the evidence that program performance estimates do not correlate with program impacts as determined by rigorous evaluations, and the possible distortions that result from this poor correlation.
  AbstractPaperPresentation
• Stephen Wandner, U.S. Department of Labor, and Michael Wiseman, George Washington University, will review the past experiences of performance bonuses in WIA, the Temporary Assistance for Needy Families (TANF) program, and the Supplemental Nutrition Assistance Program (SNAP) to examine the viability of a performance bonus system for WIA, specifically addressing if the bonuses provided a “value-added” to the programs and if they influenced program operations.
  AbstractPaperPresentation

Discussant: Rebecca Maynard, University of Pennsylvania
                   Larry Orr, Abt Associates
                   Robinson Hollister, Swarthmore College

EC comments and questions, followed by general discussion.

This second part of this continuing discussion will explore what the federal government might try to improve its performance management systems. It will pull together the major themes from old and new research efforts, and consider where research and evaluation resources might best be deployed at a time when performance measuring is a popular direction. New federal and state government investments in performance measurement may require large start-up investments in collecting more extensive data files. There is also the question of whether there are “good enough” strategies to adopt off the shelf, or whether pilot tests would assure that deeper performance investments have payoff and are worthy of consideration.

• Randal Eberts, Upjohn Institute, will suggest possible techniques to adjustment performance measures to take account of local labor market conditions and applicant background, highlighting the importance of local WIA standard setting.
  AbstractPaperPresentation
• Christopher King, University of Texas at Austin, and Burt Barnow, Johns Hopkins University, will discuss the implications of a recent implementation study of WIA in eight states, noting the complications for performance measurement because of the different ways government authority operates at the federal, state, and local levels; the competition with employer-provided training outside of WIA; and the efforts by various education and human service competitors in both the public and private sectors.
  AbstractPaperPresentation
• William Borden, Mathematica Policy Research, Inc., will outline the various ways the WIA accountability system for measuring performance has difficulties due to the multiple units, incentives, measures across and within states, and the data collection delays, all of which are challenges to implementing a fully-functional accountability system in all jurisdictions.
  AbstractPaperPresentation

Discussant: Robinson Hollister, Swarthmore College
                   Rebecca Maynard, University of Pennsylvania
                   Robert Walker, Oxford University

EC comments and questions, followed by general discussion.

Closing remarks

Douglas Besharov, UMD
Ines Hartwig, EC