Sr Software Engineer – Compute Infrastructure for Personalization, by our hiring partner, Netflix.
Location: Los Gatos, CA
Today, we think of the Netflix service less as a monolithic product and more as hundreds of millions of different products, serving uniquely personalized experiences to each of our member profiles. One of the teams powering this effort is the Personalization Infrastructure (PI) team, which builds scalable Machine Learning infrastructure for accelerating innovation for Netflix recommendations, content promotion, and search algorithms.
We are looking for a star Senior Software Engineer with experience scaling Apache Spark and other compute engines for large scale production use cases. One of the focus areas for the team is scaling the compute infrastructure for critical stages of ML training pipelines for hundreds of simultaneous A/B Tests and production workflows. This involves understanding how Spark execution plans are laid out when dealing with large data sets and optimizing them to achieve compute efficiency within acceptable latency limits.
You will be writing production grade Spark applications, consulting with researchers to scale their applications, and contributing to our own in-house ML Infra libraries that are used by these applications. This aligns with the team’s charter to accelerate ML innovation by building infrastructure and tools that help ML Engineers and Researchers rapidly run offline experimentation and online A/B Tests. You will also partner closely with the Data Platform team to identify, investigate and resolve Spark issues, including submitting pull requests for upstream open-source Spark.
To be successful in this role, you must have a passion for performance tuning and discovering the low-level details of how subsystems work. You should also possess outstanding software development skills, ability to learn fast, and work cross-functionally with several teams. You will need to exhibit strong leadership and communication skills to successfully execute on the engineering and release priorities. We value work-life balance and believe that some of the best work happens in dynamic low-process environments.
This is a great opportunity for someone with a solid compute infrastructure background to use and grow skills in the exciting ML Infra domain. This role will also allow you to gain intimate knowledge of Netflix Personalization, while working for a unique and pioneering company that is redefining how video content is consumed globally.
- Strong bias towards action and great curiosity
- Experience building large-scale applications on Apache Spark with a good understanding of Spark internals
- Experience working with large-scale distributed systems like Mesos, Zookeeper, and HDFS
- 4+ years of relevant work experience post BS/MS in Computer Science, Electrical Engineering or related field
- Production experience with Scala
- Excellent communication and people engagement skills
- Experience building systems and solutions for ML Engineers/Researchers
- Experience in Personalization/Recommendation domain
- Experience with Cloud Computing platforms like Amazon AWS
To learn more, here are some talks and blog posts from the PI Team:
ML Orchestration for Netflix Recommendations
Scala DSL for ML Training Set Stratification
Time Travel for Feature Generation
Netflix is going places (A personal insight into our culture)
You can learn more about Netflix’s unique culture of Freedom and Responsibility, that presents an opportunity to work with some of the best and the brightest, allowing you to make a difference to the business in a meaningful way.
we will review with hiring manager from Netflix.