Sr Data Engineer

Job Type

Full Time Regular Opportunity

General Description

As a Senior Data Engineer, you'll hold a pivotal senior position within the Data Engineering function of Enterprise Data & Analytics division.

In this critical role, you will orchestrate engineering activities across AWS, Informatica IDMC, and Snowflake platforms, ensuring robust and scalable data solutions that drive our data ecosystem forward.

You'll lead a talented team of engineers and consultants, mentoring them to foster innovation and excellence while promoting best practices.

Collaborating with multiple pillars across the division, you will align engineering efforts with organizational goals.

This role will lay the modern foundation of the engineering unit, creating a strong data infrastructure, enabling advanced analytics and data-driven decision-making.

By leveraging cloud-native technologies and data integration frameworks, you'll transform how we ingest, process, and deliver data assets across the enterprise.

The ideal candidate brings deep technical knowledge alongside the leadership acumen to influence cross-functional teams and executives, positioning our data engineering capabilities as a competitive advantage in the financial services landscape.

You will collaborate with multifaceted teams of specialists spread across various locations to offer a broad spectrum of data and analytics solutions.

You will address complicated challenges and propel advancement within the Enterprise Data & Analytics function

Essential Duties and Responsibilities

• Design and implement comprehensive AWS data pipeline leveraging S3 data lake architecture, Glue ETL workflows, and serverless data processing solutions using Lambda and Step Functions for event-driven architecture.
• Establish robust security, monitoring, and automation through implementation of IAM roles and policies for secure access controls, CloudWatch monitoring and alerting for pipeline health and performance metrics.
• Build and optimize Snowflake data pipelines utilizing Streams and Tasks for incremental processing, Snowpipe for continuous data loading, and stored procedures for complex transformations.
• Configure and maintain Snowflake security features including role-based access control, Dynamic Data Masking, Row Access Policies, query optimization to manage compute costs effectively.
• Develop and maintain IDMC Cloud Data Integration mappings and workflows for complex data transformation requirements, utilizing appropriate transformation types (Router, Joiner, Aggregator, Lookup) and implementing parameterization for reusable and flexible pipeline components.
• Develop automated data validation frameworks between Informatica IDMC and Snowflake to ensure data integrity
• Have a solid understanding of delivery methodology (SDLC) and lead teams in the implementation of the solution according to the design/architecture.
• Implement Informatica PowerCenter to Informatica IDMC migration strategies for legacy data integration workflows
• Lead cross-functional engineering teams through agile methodologies, sprint planning, and delivery management
• Participate as active agile team members to help drive feature refinement, user story completion, code review, etc.
• Collaborate with Data governance team and Implement data governance frameworks ensuring compliance with banking regulations (GDPR, CCPA, PCI-DSS).
• Establish CI/CD practices for data engineering assets using tools like GitHub Actions, Jenkins, and AWS Code Pipeline.
• Optimize data platform performance through query tuning, compute resource management, and architectural enhancements.
• Develop data quality monitoring solutions that provide visibility into data completeness, accuracy, and timeliness.
• Create and maintain technical documentation for engineering processes, architecture decisions, and system configurations
• Monitor and troubleshoot production data pipelines to ensure reliability and minimize business disruption
• Establish testing practices including unit, integration, and performance testing for data engineering solutions
• Evaluate emerging technologies and recommend strategic adoption to enhance data engineering capabilities
• Define and track key performance metrics for data engineering operations, identifying areas for improvement
• Lead knowledge transfer sessions and technical training to upskill junior engineers and consultants
• Collaborate with enterprise architecture teams to align data engineering solutions with broader technology strategies
• Represent the data engineering pillar in cross-divisional planning sessions and strategic initiatives, advocating for necessary resources and priorities

Education

Bachelor’s degree in computer science, Data Engineering, Statistics, or a related field.

Certifications / Licenses (Optional)

Cloud-based analytical services, showcasing extensive knowledge of cloud platforms for data analytics and machine learning

Experience

• Minimum of then (10) years of experience in implementing large scale Data & Analytics platform in AWS, Azure, or Google Cloud, on-prem and Hybrid environment.
• A minimum of seven (7) years of hands-on experience in Datawarehouse and Data Integration (ELT/ETL)
• A minimum of five (5) years of robust experience in data engineering, data analytics, or a similar leadership role.

Other Qualifications

• Demonstrated experience designing and implementing scalable data pipelines using AWS services (Glue, Lambda, Step Functions, S3, EMR)
• Strong background and problem-solving skills with Enterprise Data warehouse, ETL/ELT development, Database Replication, metadata management and data quality.
• Experience in delivering technical solutions in an iterative, agile environment (Scrum/Kanban)
• Experience working with Data Lakes loading disparate data sources- Structured, semi-structured data (Flat files, XML, JSON, Parquet) and unstructured data.
• Strong foundational leadership in Python and / or Apache Spark for processing, analyzing, and innovating in data engineering.
• Demonstrated experience migrating on-premises ETL workloads to cloud-native solutions
• Experience migrating traditional PowerCenter workflows to Informatica IDMC Cloud Data Integration.
• Strategic leadership experience with a variety of databases, including Oracle, SQL Server, and MySQL, paired with advanced knowledge of SQL and PL/SQL for sophisticated data querying, manipulation, and database management.
• Deep understanding and hands-on experience with big data platforms such as Snowflake, Redshift, Big query, Spark, or similar transformative technologies.
• Strong data warehouse applications knowledge in preferably in financial/insurance domain is required.
• Knowledge of real-time data integration patterns using technologies like Kafka, Kinesis, or Snowpipe
• Track record of successful large-scale data migration projects from legacy systems to modern cloud platforms
• Exceptional ability to interpret complex data and communicate strategic insights to both technical and non-technical stakeholders in a leadership capacity.
• Exceptional critical thinking, problem-solving skills, and a strong vision for the analytics domain's future.
• A strong commitment to continuous learning, development, and staying ahead of industry trends in the data analytics domain.
• Stellar written and verbal communication skills coupled with a meticulous attention to detail and the ability to convey strategic visions and plans.
• Demonstrated leadership abilities in a collaborative, dynamic, and fast-paced environment, managing multiple priorities with strategic acumen.
• Demonstrated proficiency in agile-based development methodologies, championing methodologies like Scrum and Kanban for collaborative, innovative, and efficient data product development.
• Visionary ability to manage big data, with proven hands-on leadership in managing and processing voluminous, complex datasets.
• Foundational leadership in managing diverse data feeds, including batch, near real-time, and real-time, ensuring strategic, efficient data management and processing.

Region Locations

Hybrid opportunity from Rosemont, IL or San Juan, PR

Values

Base Salary & Benefits

Our base salary for this position located in Rosemont, IL generally ranges between $165,000 and $175,000 annually, depending on business needs and, on your qualifications, and experience.

Base salary does not include incentive compensation, commissions, or any other type of remuneration or benefits.

In addition to salary, Popular's well-rounded benefits package for regular full-time and part-time employees includes a comprehensive health plan, flexible time off options, and savings and investment plans.

Seasonal/temporary employees are not eligible for these benefits.”