Case Studies

Centers for Medicare & Medicaid Services (CMS), July 2012 through December 2016

Objective

Evaluate the extent to which the Center for Medicare & Medicaid Innovation’s (CMMI’s) Pioneer and Advance Payment Medicare Shared Savings Program (AP MSSP) ACOs improved population health, improved the quality of care, and controlled the cost of care and determined which organizational characteristics, strategic decisions, and operational practices contributed to achieving program goals.

Challenges

The L&M team and our partner organizations were tasked with determining the effect that CMMI-supported ACOs had on population health, cost and quality. Given that participation in the program was voluntary and non-random, the team was challenged to devise comparison groups to determine the effect of the ACO initiative. This required building and working with a multi-year data set from numerous sources to create comparison groups that shared key attributes with each of the ACO (intervention) market. In addition, another challenge this analysis presented was the intervention was not specified; meaning organizations could design their own approach to meeting the three goals of the program (improve population health, quality of care and control cost of care). In order to understand the many approaches to responding to the intervention, the team collected and synthesized information about these complex organizations across a wide range of topics, e.g., leadership and governance, provider network development and engagement, marketplace/environment, organizational performance evaluation, financial planning and management, care management, information management/ health information technology (HIT) physician engagement / performance evaluation and access to care.

Approach

The L&M team employed a mixed methods approach that included collecting and analyzing primary data collection, as well as secondary data analyses. In the final analysis, these activities were complementary. The team collected primary data through quarterly telephone assessments and site visits with a cohort of 80 Medicare ACOs comprising both intervention and comparison ACOs. These data were recorded in Dedoose (a qualitative analysis software) and coded to facilitate analyses of the interviews. Secondary data were drawn from the CMS Chronic Condition Warehouse (CCW), which includes administrative data, such as the Master Beneficiary Summary Files (MBSF) as well as claims data. The team prepared a master analytic file that linked MBSF, Hierarchical Condition Categories (HCC), and claims files. Using this file, the team identified a baseline and performance year treatment group of beneficiaries assigned/aligned to the CMMI ACOs as well as baseline and performance year comparison groups of Medicare beneficiaries.

Results

Our Team:

• Collected and analyzed a vast amount of data from over 700 telephone interviews in addition to nearly 80 site visits with ACOs
• Adapted analytical approaches to accommodate the organization-designed interventions the 80 ACOs used to meet program goals
• Helped CMS better understand the Pioneer and AP MSSP ACO models and how their organizational characteristics, strategic decisions, and operational practices contribute to performance

Reports:

• Evaluation of CMMI Accountable Care Organization Initiatives—Evaluation of Skilled Nursing Facility 3-Day Pioneer ACO Waiver Final Report; December 5, 2016
• Evaluation of CMMI Accountable Care Organization Initiatives—Pioneer ACO Final Report; December 2, 2016.
• Evaluation of CMMI Accountable Care Organization Initiatives—Advance Payment ACO Final Report; November 25, 2016.
• Evaluation of CMMI Accountable Care Organization Initiatives—Pioneer ACO Evaluation Findings from Performance Years One and Two; March 10, 2015.
• Evaluation of CMMI Accountable Care Organization Initiatives—Effect of Pioneer ACOs on Medicare Spending in the First Year; November 3, 2013.