Software Engineer, Model Evaluation

Save to Kiter
What Openai is looking for in applicants

About the Team:

OpenAI puts particular emphasis on large-scale deep learning. Our approach results in a small number of extremely capable models. Towards that end, we need to make a lot of difficult decisions based on incomplete data. The Science of Deep Learning team tries to fundamentally understand what is happening as our models scale and how to evaluate their performance.

About the Role:

Evaluation metrics allow us to understand what is happening under-the-hood and the progress of our models. We are looking for an engineer who wants to create the requisite systems/frameworks for making sense of these evaluations.

As a software engineer on the team, you will be responsible for:

  • Creating the API specific to evaluations that will stand the test of time
  • Designing the tooling and frameworks needed for cataloging all present/future evaluations
  • Designing tooling for evaluation visualizations and analysis

OpenAI is a very small team. You will work directly with our best researchers and engineers to help make key decisions pushing our state of the art models towards previously unexplored regimes of AI capabilities. You’ll closely collaborate with the Science of Deep Learning team to refine our evaluation tools and methodology. This is a 0-1 effort and a huge area of investment for us. The ability to understand/analyze evals systematically will allow us to answer questions like, "How close are our models to AGI?"

About yourself:

  • Good taste in API design, thinking through long-term success vs short-term happiness
  • Excited by building tooling that accelerates the efficiency of our technical teams.
  • Track record of excellent engineering skills across multiple domains
  • Interested in understanding what happens under-the-hood of AI models
  • Open to technical leadership down the road

Bonus Points if You:

  • Have experience in statistical analysis, to draw correct conclusions from the sea of data we swim in
  • Have experience in database design 
  • Have data visualization experience

About OpenAI

OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity. We push the boundaries of the capabilities of AI systems and seek to safely deploy them to the world through our products. AI is an extremely powerful tool that must be created with safety and human needs at its core, and to achieve our mission, we must encompass and value the many different perspectives, voices, and experiences that form the full spectrum of humanity. 

At OpenAI, we believe artificial intelligence has the potential to help people solve immense global challenges, and we want the upside of AI to be widely shared. Join us in shaping the future of technology.

Benefits and Perks
  • Medical, dental, and vision insurance for you and your family
  • Mental health and wellness support
  • 401(k) plan with 4% matching 
  • Unlimited time off and 18+ company holidays per year 
  • Paid parental leave (20 weeks) and family-planning support
  • Annual learning & development stipend ($1,500 per year)
We are an equal opportunity employer and do not discriminate on the basis of race, religion, national origin, gender, sexual orientation, age, veteran status, disability or other legally protected statuses. Pursuant to the San Francisco Fair Chance Ordinance, we will consider qualified applicants with arrest and conviction records. We are committed to providing reasonable accommodations to applicants with disabilities, and requests can be made via

Want some tips on how to get an interview at Openai?

What is Openai looking for?
If this role looks interesting to you, a great first step is to understand what excites you about the team, product or mission. Take your time thinking about this and then tell the team! Get in touch and communicate that passion.
What are interviews for Software Engineer like?
Interview processes vary by company, role and team. The best plan is to see what others have experienced and then plan accordingly.
How to land an interview at Software Engineer?
A great first step is organizing your path to an offer. Check out Kiter for tools to get started!