College Board – Learning and Assessment – Psychometrics 

100% Remote (anticipated 25% travel, usually 2-3 days at a time)

About the Team

The Psychometrics Team supports College Board in serving more than seven million students a year as they prepare for a successful transition to college through programs and services in college readiness and college success. We provide our partners with measurement solutions that set the standard for quality, efficiency, and innovation through operational excellence and research leadership that advances the field of psychometrics and supports College Board’s mission of access and equity in education. Our team of Psychometricians and Research Statisticians collaborates deeply with other departments and divisions including, but not limited to Assessment Design & Development, College Readiness Assessments, Advanced Placement & Instruction, State & District Partnerships, and Technology.

About the Opportunity

As our Executive Director, Natural Language Measurement, you are a strategic measurement leader at the forefront of responsible AI in education. Energized by our mission to enhance educational and career opportunities for millions of students, you will shape the future of automated scoring and use of AI-powered feedback in large-scale assessments. You will set the vision and multi-year roadmap for natural language measurement across SAT, AP, CLEP, Accuplacer and future products, ensuring all applications are grounded in validity, fairness, and reliability. You will expand the College Board’s innovation agenda in AI-enabled approaches that power personalized learning and feedback, reinforcing our leadership in responsible and equitable assessment. You bring experience shaping national or international best practices in responsible AI/NL for large-scale assessment and have a proven record of scaling applications from research to operational use in high-stakes contexts, making a transformative impact on both students and the field.

In this role, you will:

Strategic Leadership and Roadmap (35%)

• Shape and implement the multi-year strategy and roadmap for automated scoring and intelligent learning approaches across current flagship programs, such as SAT and AP, and emerging future products.
• Advance College Board’s innovation agenda by expanding from automated scoring to broader AI-enabled solutions that support personalized learning and actionable feedback.
• Establish model-readiness criteria and guide release decisions for senior leadership to ensure high-quality, scalable deployment of natural language-driven solutions.
• Partner closely with Technology to shape development of guided scoring products and other intelligent learning innovations, advancing next-generation assessment capabilities.

Operational and Implementation Work (35%)

• Lead scientific design, implementation, and evaluation of automated scoring and AI/NL applications, ensuring alignment with program and organizational goals.
• Apply psychometric and measurement principles to ensure all AI/NL-driven applications uphold validity, fairness, reliability, and equity.
• Guide implementation of initial automated scoring programs beginning in 2026.
• Establish and oversee operational processes that ensure quality, fairness, and reliability, including monitoring, human-in-the-loop procedures, and refinement of evaluation criteria.
• Build organizational readiness for adoption of new technologies, including criteria for model evaluation, quality assurance and responsible release decisions.

Lead and Manage the Natural Language Measurement Team (20%)

• Build and lead a team of psychometricians, language model experts, and related specialists, combining internal talent with new hires to deliver AI/NL priorities.
• Set vision and priorities for the team, track and manage progress to goals, and provide coaching and support to ensure team members meet and exceed goals, remain engaged, and contribute meaningfully to our mission.
• Cultivate an inclusive and high-achieving culture that enables all team members to live out College Board’s Operating Principles effectively.
• Bring proven leadership experience and consistently embody College Board’s Manager Expectations in your work.

Internal and External Collaboration and Communication (10%)

• Serve as College Board’s lead subject-matter expert on AI powered natural language models: represent the organization in the measurement community, with states and external partners, at conferences and in publications; advance and communicate College Board’s responsible use guidance.
• Collaborate with other College Board divisions, including close partnership with various program leaders, to ensure alignment on natural language-based initiatives and future assessment development.
• Represent College Board nationally as a thought leader in responsible AI/NL and measurement innovation, shaping best practices across the field.

About You

To qualify for this role, you must have:

• At least 10 years of experience in psychometrics, assessment, or measurement-related fields, with a strong record of applying measurement science in large-scale educational contexts and leading interdisciplinary innovation across psychometrics, AI, and product development.
• Deep expertise in measurement science and familiarity with AI/NL approaches, with the ability to apply principles of validity, reliability, fairness, and equity in large-scale assessments.
• Proven ability to move AI powered natural language systems from research into operational use, including shaping automated scoring strategies and collaborating with technology and operations partners to ensure quality, fairness, efficiency and integrity.
• Demonstrated success setting strategy and guiding innovation through cross-functional collaboration, leading teams across psychometricians, data scientists, engineers, and program staff to execute complex initiatives at scale.
• At least 5 years of experience leading and managing direct teams with diverse expertise in measurement, technology, and product, with a strong track record of hiring, developing, and retaining high-performing talent to deliver innovative solutions from ideation to implementation.
• A strategic and inclusive leadership style: you set clear priorities, build effective team structures, plan for future needs, and foster a culture of belonging.
• A proven ability to drive performance and growth: you set high expectations, deliver real-time, evidence-based feedback, and coach team members to take smart risks, stretch their skills, and achieve meaningful impact.
• A reputation as a thought leader at the intersection of psychometrics and AI/NL, with the ability to influence national and international best practices in large-scale, responsible AI-based assessment.
• Exceptional communication and organizational influence skills, with the ability to represent College Board and shape strategy across internal and external stakeholder groups.
• Deep commitment to responsible use of artificial intelligence and to advancing equity in educational measurement.
• Doctorate in psychometrics, educational measurement, statistics, psychology, or a related field.
• The ability to travel 10-12 times a year to College Board offices or on behalf of College Board business.

All roles at College Board require:

• A passion for expanding educational and career opportunities and mission-driven work
• Authorization to work in the United States for any employer
• Curiosity and enthusiasm for emerging technologies, with a willingness to experiment with and adopt new AI-driven solutions and a comfort learning and applying new digital tools independently and proactively.
• Clear and concise communication skills, written and verbal
• A learner’s mindset and a commitment to growth: welcoming diverse perspectives, giving and receiving timely, respectful feedback, and continuously improving through iterative learning and user input.
• A drive for impact and excellence: solving complex problems, making data-informed decisions, prioritizing what matters most, and continuously improving through learning, user input, and external benchmarking.
• A collaborative and empathetic approach: working across differences, fostering trust, and contributing to a culture of shared success.

About Our Process 

• Application review will begin immediately and will continue until the position is filled. This role is expected to accept applications for a minimum of 5 business days.
• While the hiring process may vary, it generally includes: resume and application submission, recruiter phone/video screen, hiring manager interview, performance exercise such as live coding, a panel interview, a conversation with leadership and reference checks.

What We Offer

At College Board, we offer more than just a paycheck—we provide a meaningful career, a supportive team, and a comprehensive package designed to help you thrive. We’re a self-sustaining nonprofit that believes in fair and competitive compensation, grounded in your qualifications, experience, impact, and the market.

A Thoughtful Approach to Compensation

• The hiring range for this role is $148,000 – $225,000.
• Your exact salary will depend on your location, experience, and how your background compares to others in similar roles at the College Board.
• We aim to make our best offer upfront—rooted in fairness, transparency, and market data.
• We adjust salaries by location to ensure fairness, no matter where you live.

You’ll have open, transparent conversations about compensation, benefits, and what it’s like to work at College Board throughout your hiring process. Check out our careers page for more.

#LI-MS1

#LI-REMOTE