Role Summary
We are looking for a highly skilled Senior Data Engineer with strong hands-on experience in Python, SQL, Snowflake, Airflow, ETL/ELT, data warehousing, Fivetran, GitHub, CI/CD, API integration, and data outbound pipelines.
The candidate will be responsible for designing, building, optimizing, and maintaining scalable data pipelines and data warehouse solutions. The role requires strong engineering discipline, production mindset, troubleshooting ability, and experience working with modern cloud data platforms.
The ideal candidate should be able to work independently with architects, data modelers, business analysts, application teams, and downstream consumers to deliver reliable, secure, and high-quality data products.
*Key Responsibilities*
Design, develop, and maintain scalable ETL/ELT data pipelines using Python, SQL, Airflow, Snowflake, and modern data engineering tools.
Build and optimize data ingestion, transformation, validation, reconciliation, and outbound data pipelines.
Develop reusable Python frameworks for data extraction, transformation, quality checks, API calls, file processing, logging, alerting, and error handling.
Create and manage Airflow DAGs for orchestration, scheduling, dependency management, retries, monitoring, and production support.
Work extensively on Snowflake objects including databases, schemas, tables, views, streams, tasks, procedures, stages, file formats, Snowpipe, and role-based access controls.
Develop complex SQL queries, stored procedures, and transformation logic for data warehouse and analytics use cases.
Implement data warehouse design patterns including staging, bronze/silver/gold layers, dimensional models, fact and dimension loading, incremental loads, CDC, SCD handling, and audit frameworks.
Integrate data from multiple sources using Fivetran, APIs, flat files, cloud storage, databases, and enterprise applications.
Design and support data outbound pipelines to send curated data to external systems, downstream applications, APIs, third-party vendors, CRM platforms, operational systems, and reporting platforms.
Set up and maintain GitHub-based development workflows including branching strategy, pull requests, code reviews, version control, and release management.
Implement CI/CD pipelines for automated testing, deployment, environment promotion, code quality checks, and production releases.
Collaborate with data architects and data modelers to implement scalable and business-aligned data warehouse solutions.
Ensure data quality through validation rules, reconciliation checks, duplicate handling, null checks, schema checks, and anomaly detection.
Monitor data pipelines, troubleshoot failures, resolve performance bottlenecks, and support production incidents.
Document technical design, pipeline logic, source-to-target mappings, dependencies, operational procedures, and runbooks.
Follow enterprise standards for security, data governance, access control, data privacy, auditability, and compliance.
Required Skills and Experience
Strong hands-on experience as a Senior Data Engineer
Advanced SQL skills, including joins, window functions, CTEs, query optimization, analytical SQL, stored procedures, and performance tuning.
Strong Python programming skills for data engineering, automation, API integration, file processing, and framework development.
Hands-on experience with Airflow for workflow orchestration, DAG development, monitoring, retries, scheduling, and production operations.
Strong experience with Snowflake including data loading, transformation, performance optimization, warehouse sizing, RBAC, stages, streams, tasks, and Snowpipe.
Strong understanding of ETL/ELT design patterns, data integration, batch processing, incremental loading, CDC, and data reconciliation.
Experience working with Fivetran or similar ingestion tools for source system replication and pipeline management.
Experience with GitHub, version control, branching, pull requests, code reviews, and release management.
Hands-on experience with CI/CD pipelines for data engineering deployments.
Experience building and consuming REST APIs for data ingestion and outbound data exchange.
Good understanding of data warehousing concepts, including dimensional modeling, facts, dimensions, SCD Type 1/2, surrogate keys, and layered data architecture.
Experience in production support, pipeline monitoring, logging, alerting, and failure recovery.
Strong debugging and problem-solving skills across SQL, Python, orchestration, and data platform layers.
Good to Have
Experience with cloud platforms such as Azure
Experience with, Coalesce, or DBT or similar transformation tools.
Experience with API authentication patterns such as OAuth2, JWT, API keys, basic authentication, and token refresh.
Experience with data governance, metadata management, data lineage, and data cataloging tools.
Good to have Experience with containerization, Docker, Kubernetes, or serverless data workloads.
Good to have Exposure to real-time or near-real-time data pipelines using Kafka, Event Hub, streams, or message queues.
Key Technical Areas
Python data engineering
Advanced SQL development
Snowflake development and optimization
Airflow DAG orchestration
ETL/ELT pipeline development
Fivetran ingestion management
Data warehouse implementation
API-based data integration
Outbound data pipelines
GitHub and CI/CD deployment
Data quality and reconciliation
Production monitoring and support
Expected Deliverables
Production-ready ETL/ELT pipelines
Reusable Python data engineering components
Airflow DAGs and orchestration workflows
Snowflake tables, views, procedures, streams, tasks, and loading frameworks
Source-to-target mapping implementation
Data quality and reconciliation checks
Outbound data feeds and API integrations
CI/CD deployment pipelines
Technical design documents and runbooks
Pipeline monitoring, logging, and alerting setup
Qualifications
Bachelor’s degree in Computer Science, Information Technology, Data Engineering, or a related field.
Minimum 6+ years of experience in data engineering, ETL/ELT development, SQL, and data warehousing.
Minimum 3+ years of hands-on experience with Python and cloud data platforms.
Strong communication skills and ability to work with team
