Senior Manager, Evaluations, Examinations and Assessment
You understand the critical importance of program evaluation and its role in the field of education. You recognize the complex enterprise of evaluations, examinations and assessment. You utilize various methodologies and mixed-methods research systems to continually push the boundaries of the status quo to enhance a portfolio of products. You are fluent in licensure standards, Accreditation Council for Continuing Medical Education (ACCME) standards, instructional design standards, important national assessment trends and technology solutions. You remains up-to-date with the newest teaching technologies and instructional theories on the use of technology in assessment. You are an innovative, positive and selfless leader, who empowers those around them.
If this sounds like you, please read on!
Manages examination and assessment portfolio, staff, and committee activities. Establishes product budgets, strategies, roadmaps, and release objectives that build on and extends the current portfolio of assessment products for continuing medical education and recertification. Plans and organizes examination development and implementation of all AAOS examination programs. Designs and implements scoring processes and reporting mechanisms for members and committees. Manages development databases and item bank(s) and implements new technology as business needs evolve. Works closely with Orthopaedic Residents, Residency Program Directors and AAOS Education teams to guide development of curricula. Continually pushes the boundaries of existing systems and processes to enhance the portfolio of products.
Responsible for managing the AAOS’ self-assessment product portfolio. Lead and drive the overall product lifecycle, strategies, schedules, budgets, deliverables, and capabilities for our products. Partner with Marketing and Communications to identify customer and market needs and prioritize areas of focus for product management – including competitive analyses, product positioning, and SWOT analyses. Set pricing to meet revenue and profitability goals. Deliver a monthly revenue forecast. Propose an overall portfolio budget to ensure success. Work with Chief Education Strategist, Director, Online Learning & Assessment and cross-functional teams to implement an agile examination development lifecycle process to manage the ongoing development, optimization and timely commercialization of products. Create a framework to define prioritization, roadmap planning and success metrics. Stay current with American Board of Orthopaedic Surgery (ABOS) licensure standards, Accreditation Council for Continuing Medical Education (ACCME) standards, instructional design standards, important national assessment trends and technology solutions.
Leads the EEA team in the design, development, administration and maintenance of AAOS’ examination and assessment products. Oversees item and test analyses, test equating, standard setting, score reporting, and develops technical reports. Leads the construction of the AAOS’s assessment plan, including working with outside stakeholders. Directs reliability and validity studies and other psychometric activities - in collaboration with the American Board of Orthopaedic Surgery (ABOS) as required. Manages psychometric contractors on specialized projects. Collaborates with subject matter experts to improve current assessments. Develops, implements, and manages methods for linking examination performance with physician practice improvement and patient outcomes. Contributes to and delivers Member-facing proposals and presentations. As point of contact for all internal/external customers, defends validity of self-assessment products. Remains up-to-date with the newest teaching technologies and instructional theories on the use of technology in assessment – (i.e. Adaptive Learning). Oversees the implementation of course and program-level assessment and evaluation to improve educational and learning outcomes.
Drives the creation of a positive and cooperative team by developing Direct Reports and fostering a team approach focused on Academy values, competencies, and behaviors. Works within cross functional teams to achieve desired results for the examination program. Builds strong working relationships within the Academy; provides leadership within the Academy for the Examination team through establishing performance objectives and regular performance evaluations and professional development plans for direct reports. Provides operational leadership in hiring new staff.
Staff liaison to Evaluation Committees - Assessments and Examinations Committee; Subcommittees on Evaluation (11). Manages activities for the Evaluation Committees, develops and implements training and workshop materials for members, creates and participates in new projects including special evaluative studies related to the committee charges. Organizes and staffs committee meetings. Advises committees; establishes, implements, and disseminates Academy’s policies and procedures as they apply to the evaluation programs maintaining strict confidentially. Establishes procedures for staff and committees including committee manuals, new committee member orientation and committee meetings. Supports recruitment of new Committee members.
Approximately 10 - 15 days per year
- Experience in development of examination programs.
- Advanced degree, with statistical/research experience, computer-based testing
- Working knowledge of statistical theory, CTT and IRT, research methodologies, and applied measurement
- Minimum 5 years of experience assessment management including the design, development, standard setting, processing, analysis (including scoring, scaling, equating and norming) and reporting of assessments in the certification markets.
- Must include progressive supervisory experience, 3-5 years committee support and representation of the organization
- Excellent written and verbal communication skills, including the ability to provide a supportive environment for people to embrace assessment, yet also the ability to plainly and clearly describe complicated analyses to a diverse audience.
- Excellent project planning, organizational skills, including the ability to coordinate a variety of different projects and individuals/direct reports and to adapt to unforeseen and unexpected requests or changes in priorities.
- Computer skills including strong skills in set up and use of database programs (e.g. SAS) statistical experience.
- Knowledge in registries, performance improvement and patient outcomes
- Medical association or health care organization experience
- Empowerment: The authority, information, and skills to make decisions and drive results
- Accountability: Ownership of results that drives decisions and ensures implementation
- Continuous Learning: Investment in staff self-development to become more adaptable and makes the Academy more agile, innovative, and sustainable
- Teamwork: Effective and selfless collaboration to solve complex problems and drive innovation that is beyond the scope of any one individual
If this describes YOU, please apply by sharing the following:
-Clearly communicate why you are the ideal candidate for this role, providing specific examples and experiences as proof points.
-Attach your resume, cover letter, salary requirements and any additional materials that support your application.