Lead Data Engineer (Data Platform team)
New York, New York, United StatesApply NowApply Later
Job ID 774118BR Location New York, New York, United States Business Disney Streaming Services Date posted Feb. 12, 2021
Job Summary:Data Engineering's mission at Disney Streaming is to provide a robust, self-service data ecosystem that enables data producers and data consumers to provide operational and analytic transparency for our rapidly growing business. To that end, the Data Platform team is responsible for building the tools and services that empower our users to make data-driven technical and business decisions and democratize data across the organization.
- Collaborate with product teams, software engineers and data scientists to gather requirements and design self-service data solutions for our stakeholders’ needs
- Build and maintain tools and services to support operating on data in the platform, including ETL, ingress, and egress patterns.
- Build and maintain tools and services to support data discovery, lineage, governance, and privacy compliance across the data platform
- Develop data catalogs and data validations to ensure clarity and correctness of key business metrics
- Drive and maintain a culture of quality, innovation, and experimentation
- Coach data engineers on best practices and technical concepts behind large-scale data platforms
- Provide technical vision for DSS data platform, leading all technology and system design/architecture decisions
- Lead implementation/realization of aforementioned technical vision
- Work in an Agile environment that focuses on collaboration and teamwork
- Evangelize Data Platform throughout the Disney Streaming Services organization
- 6-10 years of experience developing in one or more of the following: Python, C++, or any JVM language
- 3-5 years of experience deploying and running cloud-based data solutions (AWS, GCP, Azure) and familiar with tools such as CloudFormation, Kinesis, ECS, S3 (or equivalent)
- 3-5 years of experience engineering big-data solutions using technologies like Databricks, EMR, and Spark
- In-depth understanding of data partitioning and sharding techniques
- Familiarity with metadata management, data lineage, and principles of data governance
- Experience loading and querying cloud-hosted databases such as Redshift and Snowflake
- Experience building streaming data pipelines using Kafka, Spark, Flink, or Samza
- Experience with API design
- Experience with functional programming in Scala
- Experience building distributed systems with microservices and/or service-oriented architectures
- Familiarity with containerization/virtualization, e.g., Docker, Kubernetes
- Familiarity with workflow scheduler tools, e.g. Airflow, Argo
- Familiarity with common ML tooling, e.g. MLFlow, Tensorflow, scikit-learn
- Knowledge of CI/CD best practices
- AWS experience
- Familiarity with binary data serialization formats such as Parquet, Avro, and Thrift
- Experience deploying data notebook and analytic environments such as Jupyter and Databricks
Bachelor’s degree in Computer Science or related field or equivalent work experience
About Disney Streaming Services:
Disney Streaming Services is responsible for developing and operating The Walt Disney Company’s direct-to-consumer video businesses globally, including the ESPN+ premium sports streaming service; Disney+, the dedicated streaming home for entertainment from Disney, Pixar, Marvel, Star Wars, National Geographic and BAMTECH Media, a global leader in direct-to-consumer video streaming products and solutions. Our core mission is to deliver global audiences the freedom to access content on their terms across any connected device, time or location. We serve consumers by bringing the world’s most beloved characters, timeless stories, legendary athletes, and epic sporting events to global audiences through best-in-class direct-to-consumer video services. We strive daily to imaginatively challenge convention with innovative technology that gives consumers the freedom to access content on their terms across any connected device, time or location.
About The Walt Disney Company:
The Walt Disney Company, together with its subsidiaries and affiliates, is a leading diversified international family entertainment and media enterprise with the following business segments: media networks, parks and resorts, studio entertainment, consumer products and interactive media. From humble beginnings as a cartoon studio in the 1920s to its preeminent name in the entertainment industry today, Disney proudly continues its legacy of creating world-class stories and experiences for every member of the family. Disney’s stories, characters and experiences reach consumers and guests from every corner of the globe. With operations in more than 40 countries, our employees and cast members work together to create entertainment experiences that are both universally and locally cherished.
This position is with Disney Streaming Technology LLC, which is part of a business segment we call Disney Streaming Services.
Disney Streaming Technology LLC is an equal opportunity employer. Applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability, or protected veteran status or any other basis prohibited by federal, state or local law. Disney fosters a business culture where ideas and decisions from all people help us grow, innovate, create the best stories and be relevant in a rapidly changing world.