Please scroll down, To apply

Lead Data Engineer

hiring now
New job

The Prudential Insurance Company of America

2024-11-05 18:45:47

Job location Newark, New Jersey, United States

Job type: all

Job industry: Engineering

Job description

Lead the design and development of highly scalable and efficient data architectures that meet business requirements and support analytical use-cases. Provide technical leadership to the data engineering team, guiding them in best practices, coding standards, and architectural designs. Design and implement robust ETL/ELT pipelines to ingest, transform and load data from diverse sources into the data ecosystem by leveraging AWS Cloud and Big Data technologies using Spark & Python. Demonstrate expertise in AWS data services including AWS S3, Glue, Redshift, EMR, Kinesis, and Lambda. Create automations and CI/CD pipelines using the tools available like BitBucket, Jenkins, and Cloud Formation. Optimize the data storage, query performance, costs on AWS resources for operation efficiency. Ensure data security best practices, encryption and access controls are in place to meet industry standards and regulations. Collaborate with data scientists, analysts and other stakeholders to understand the business requirements and deliver actionable insights. Maintain comprehensive documentation of data engineering processes, data models, and infrastructure configurations. Telecommuting permitted up to 100%. Candidate may reside and work anywhere in the U.S. Full time employment, Monday Friday, 40 hours per week. MINIMUM REQUIREMENTS: Bachelors degree in Computer Science or a related field and 5 years of progressive, post-baccalaureate related work experience; OR a Masters degree in Computer Science or a related field, and 3 years of related work experience. Of the required experience, must have 3 years of experience in each of the following: Extract-Transform-Load; Python; AWS Services, including S3, Glue, Lambda and IAM; Spark, Hadoop, Hive, Impala, Sentry, and PySpark Job Orchestration: Step Function, Airflow, or Autosys; Utilizing Version Control Tools: Bitbucket, GitHub, or GitLab; CI/CD Pipelines: Jenkins, TeamCity, or GitLab pipelines; Infrastructure as Code: CloudFormation, CDK, or Terraform; and Working with RDS and SQL Databases. Telecommuting permitted up to 100%. .

Inform a friend!

<!– job description page –>
Top