Arena Intelligence
Machine Learning Scientist
About this role
Arena Intelligence seeks a Machine Learning Scientist to design experiments and develop novel evaluation methodologies for understanding AI model behavior through human preference data. You'll conduct rigorous research on model performance across multiple dimensions while collaborating with engineers and product teams to translate findings into production systems.
What you'll do
- Design and conduct experiments evaluating AI model behavior across reasoning, style, robustness, and user preference dimensions
- Develop new metrics, methodologies, and evaluation protocols beyond traditional benchmarks
- Analyze large-scale human voting and interaction data to uncover model performance insights
- Collaborate with engineers to implement and scale research findings into production
- Author internal reports and external publications contributing to ML research community
- Partner with model providers on evaluation questions and responsible model testing
What they're looking for
- Training large-scale models including reward models and RLHF/DPO fine-tuning
- Statistical rigor in experiment design and analysis
- Python proficiency with PyTorch, JAX, or TensorFlow
- LLM and modern deep learning architecture expertise
- Real-world data analysis and custom metric design
- Research publication and open-source contribution experience
- Cross-functional collaboration with engineering and product teams
- Rapid prototyping balanced with methodological rigor
Benefits
- Competitive compensation and equity
- Remote-friendly work in Bay Area
- Opportunity to contribute to open science and reproducibility
- Collaboration with researchers from UC Berkeley, Google, Stanford, and DeepMind
- Work on problems with real-world AI impact
Opens the application — the Jobs AI extension fills it for you. Set up autofill
Opens the official application on the employer’s site. No login required.
Arena Intelligence
Arena Intelligence builds a platform for evaluating AI model performance through real-world testing and human preference data, powering insights for AI labs and enterprises. The company is hiring data engineers, security engineers, machine learning scientists, and customer-facing technical roles to scale its evaluation infrastructure, protect against misuse, and deliver custom solutions to customers.
View all jobs at Arena IntelligenceLikely interview questions
- Walk us through a project where you designed and trained a reward model or preference model. What challenges did you encounter with data quality or annotation consistency, and how did you address them?
- Describe your experience with RLHF, DPO, or other fine-tuning methods. How have you evaluated whether these approaches actually improved model behavior in practice?