Sr. Data Engineer

Job description

POSITION: Sr. Data Engineer
CLIENT: Fortune 150 Company; Financial Services

SUMMARY DESCRIPTION:

The Data Engineer will serve in a strategic role designing and managing the infrastructure that supports data storage, transforming, processing, and retrieval enabling efficient data analysis and decision-making within the organization.

This position is critical as part of the Database and Analytics team responsible for design, development, and implementation of complex enterprise-level data integration and consumption solutions.

It requires a highly technical, self-motivated senior engineer who will work with analysts, architects, and systems engineers to develop solutions based on functional and technical specifications that meet quality and performance requirements.

PRIMARY DUTIES AND RESPONSIBILITIES:

• Utilize experience in ETL tools, with at least 5 years dedicated to Azure Data Factory (ADF), to design, code, implement, and manage multiple parallel data pipelines.
  
  Experience with Microsoft Fabric, Pipelines, Mirroring, and Data Flows Gen 2 usage is required.

• Apply a deep understanding of data warehousing concepts, including data modeling techniques like star and snowflake schemas, SCD Type 2, Change Data Feeds, Change Data Capture.
  
  Also demonstrates hands-on experience with Data Lake Gen 2, Delta Lake, Delta Parquet files, JSON files, big data storage layers, optimize and maintain big data storage using Partitioning, V-Order, Optimize, Vacuum and other techniques.

• Design and optimize medallion data models, warehouses, architectures, schemas, indexing, and partitioning strategies.

• Collaborate with Business Insights and Analytics teams to understand data requirements and optimize storage for analytical queries.

• Modernize databases and data warehouses and prepare them for analysis, managing for optimal performance.

• Design, build, manage, and optimize enterprise data pipelines ensuring efficient data flow, data integrity, and data quality throughout the process.

• Automate efficient data acquisition, transformation, and integration from a variety of data sources including databases, APIs, message queues, data streams, etc.

• Competently performs advanced data tasks with minimal supervision, including architecting advanced data solutions, leading and coaching others, and effectively partnering with stakeholders.

• Interface with other technical and non-technical departments and outside vendors on assigned projects.

• Under the direction of the IT Management, will establish standards, policies and procedures pertaining to data governance, database/data warehouse management, metadata management, security, optimization, and utilization.

• Ensure data security and privacy by implementing access controls, encryption, and anonymization techniques as per data governance and compliance policies.

• Expertise in managing schema drift within ETL processes, ensuring robust and adaptable data integration solutions.

• Document data pipelines, processes, and architectural designs for future reference and knowledge sharing.

• Stay informed of latest trends and technologies in the data engineering field, and evaluate and adopt new tools, frameworks, and platforms (like Microsoft Fabric) to enhance data processing and storage capabilities.

• When necessary, implement and document schema modifications made to legacy production environment.

• Perform any other function required by IT Management for the successful operation of all IT and data services provided to our clients.

• Available nights and weekends as needed for system changes and rollouts.

EDUCATION AND EXPERIENCE REQUIREMENTS:

• Bachelor’s or Master’s degree in computer science, information systems, applied mathematics, or closely related field.

• Minimum of ten (10) years full time employment experience as a data engineer, data architect, or equivalent required.

   

SKILLS:

• Experience in working with large, heterogeneous datasets in building and optimizing data pipelines, pipeline architectures, and integrated datasets using traditional and modern data integration technologies (such as ETL, ELT, MPP, data replication, change data captures, message-oriented data movement, API design, stream data integration and data virtualization)

• Experience working with cloud data engineering stacks (specifically Azure and Microsoft Fabric), Data Lake, Synapse, Azure Data Factory, Databricks, Informatica, Data Explorer, etc.

• Strong, in-depth understanding of database architecture, storage, and administration utilizing Azure stack.

• Deep understanding of Data architectural approaches, Data Engineering Solutions, Software Engineering principles and best practices.

• Working knowledge and experience with modern BI and ETL tools (Power BI, Power Automate, ADF, SSIS, etc.)

• Experience utilizing data storage solutions including Azure Blob storage, ADLS Gen 2.

• Solid understanding of relational and dimensional database principles and best practices in a client/server, thin-client, and cloud computing environment.

• Advanced working knowledge of TSQL and SQL Server, transactions, error handling, security and maintenance with experience writing complex stored procedures, views, and user-defined functions as well as complex functions, dynamic SQL, partitions, CDC, CDF, etc.

• Experience with .net scripting and understanding of API integration in a service-oriented architecture.

• Knowledge of reporting tools, query language, semantic models with specific experience with Power BI.

   

• Understanding of and experience with agile methodology.

• PowerShell scripting experience desired.

• Experience with Service Bus, Azure Functions, Event Grids, Event Hubs, Kafka would be beneficial.

• Experience working in Agile methodology.

Working Conditions:

• Available to work evenings and/or weekends (as required).

• Workdays and hours are Monday through Friday 8:30 am to 5:30 pm ET.

Required Skill Profession

Computer Occupations