Lead Scala Engineer - Data Platform team
New York, New York, United StatesApply NowApply Later
Job ID 778585BR Location New York, New York, United States Business Disney Streaming Services Date posted Mar. 20, 2021
Job Summary:At Disney Streaming Services you'll have the opportunity to create highly scalable and reliable services that power Disney+. You'll get to understand the challenges of facilitating a high-quality, flawless media streaming experience across mobile, connected devices, and web, all while using the latest technologies available from AWS and beyond. You'll be tasked with raising the bar by innovating and adopting new patterns and technologies to keep your teams on the bleeding edge of development. At Disney Streaming Services we feel strongly that teams should own their own processes, decide their own technologies, and design solutions for the long term. If you're interested in working in a highly collaborative team environment like this, please get in touch - we'd love to hear from you!
Disney Streaming Services has already achieved a number of key landmarks for our partners including launching Eurosport Player, ESPN+ and of course the highly successful Disney+ service. We are looking for an experienced Lead Software Engineer to join us.
The Services & Data Engineering Team is hiring Software Engineers in NYC, San Francisco, Manchester, London and Amsterdam. As a member of the team you will have the opportunity to work on the platform that powers our streaming products.
At Disney Streaming Services data is central to measuring all aspects of the business, and critical to its operations and growth. The data engineering team is responsible for data acquisition, real-time and stream processing, analysis, and distribution of data for further use using public cloud and open source technologies and offers transparency into customer behavior and business performance.
- Work within a cross functional team of engineers to build software for a large-scale data processing ecosystem supporting real-time and batch data that underpins all analytic and operational data flows.
- Collaborate with product teams, engineering teams, data analysts and data scientists to design and build data-forward solutions
- Build tools and services to support data discovery, lineage, governance, and privacy compliance across the data platform
- Build and maintain a CICD process for infrastructure components to ensure high availability while remaining nimble.
- Leadership and responsibility for technical architecture, roadmap, improvements, reliability, resilience, scalability, and cost of the Disney Streaming data platform
- Influence data architecture and organizational capabilities, enable data-driven concepts throughout the organization
- Participate in the full lifecycle of software development (requirements gathering, designing, building, testing, maintenance, and support)
- Be part of the development for some of the highest scale distributed systems in the world, solving challenging and varied engineering problems
- Leadership and responsibility for the design, building, deployment, and operations of a data platform and software solutions capable of processing and storing petabytes of data quickly, correctly, and reliably
- Drive and maintain a culture of quality, innovation and experimentation
- Work with other teams, evangelize the platform, best-practices, data driven decisions, identify new use cases and features and drive adoption
- Coach engineers best practices and technical concepts of building large scale data platforms
- Influence and drive software engineering and architecture best practices and standards within the team and wider organization
- 6+ years of software engineering, ideally Python and Scala
- Knowledge and experience with distributed systems, their development, operations, observability
- 3-5 years’ experience deploying and running cloud-based data solutions (AWS, GCP, Azure) and familiar with tools such as CloudFormation, Kinesis, ECS, S3 (or equivalent)
- 3-5 years’ experience engineering big-data solutions using technologies like Databricks, EMR, and Spark
- Experience with streaming data pipelines using tools such as Kinesis, Kafka, Spark, Storm, or Flink
- In-depth understanding of data partitioning and sharding techniques
- Experience loading and querying cloud-hosted databases such as Redshift and Snowflake
- Experience with RDBMS, key/value stores, distributed database solutions
- Knowledge of modern data solutions and architectures
- Cross-functional approach to software development, DevOps experience
- Proficient in designing and implementing large scale distributed systems
- Previous experience taking ownership of features/functionality and delivering projects end-to-end with high quality and robust architecture.
- Ability to dive deep into any technical component as well as understand and drive the overall systems architecture.
- You're a self-learner, independent, and have excellent problem-solving skills.
- You care deeply about craftsmanship in your software, and can work backwards from the customer experience.
- You have excellent written and verbal communication skills.
- Experience building distributed systems with microservices and/or service-oriented architectures
- Familiarity with binary data serialization formats such as Protobuf, Parquet, Avro, or Thrift
- Familiarity with workflow scheduler tools such as Airflow
- Familiarity with common ML tooling such as MLFlow, Tensorflow, scikit-learn
- Experience building streaming data platforms, solving problems including event definition, evolution, processing, patterns and best practices, scalability, reliability, observability, validation etc.
- Familiarity with metadata management, data lineage, and principles of data governance
About Disney Streaming Services:
Disney Streaming Services is responsible for developing and operating The Walt Disney Company’s direct-to-consumer video businesses globally, including the ESPN+ premium sports streaming service; Disney+, the dedicated streaming home for entertainment from Disney, Pixar, Marvel, Star Wars, National Geographic and BAMTECH Media, a global leader in direct-to-consumer video streaming products and solutions. Our core mission is to deliver global audiences the freedom to access content on their terms across any connected device, time or location. We serve consumers by bringing the world’s most beloved characters, timeless stories, legendary athletes, and epic sporting events to global audiences through best-in-class direct-to-consumer video services. We strive daily to imaginatively challenge convention with innovative technology that gives consumers the freedom to access content on their terms across any connected device, time or location.
About The Walt Disney Company:
The Walt Disney Company, together with its subsidiaries and affiliates, is a leading diversified international family entertainment and media enterprise with the following business segments: media networks, parks and resorts, studio entertainment, consumer products and interactive media. From humble beginnings as a cartoon studio in the 1920s to its preeminent name in the entertainment industry today, Disney proudly continues its legacy of creating world-class stories and experiences for every member of the family. Disney’s stories, characters and experiences reach consumers and guests from every corner of the globe. With operations in more than 40 countries, our employees and cast members work together to create entertainment experiences that are both universally and locally cherished.
This position is with Disney Streaming Technology LLC, which is part of a business segment we call Disney Streaming Services.
Disney Streaming Technology LLC is an equal opportunity employer. Applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability, or protected veteran status or any other basis prohibited by federal, state or local law. Disney fosters a business culture where ideas and decisions from all people help us grow, innovate, create the best stories and be relevant in a rapidly changing world.