About the Job
We are seeking an experienced Big Data Lead to design, implement, and optimize large-scale data engineering solutions on modern cloud and distributed computing platforms. The role requires strong technical expertise in Big Data frameworks, advanced SQL, data integration, and cloud-native services. The ideal candidate will have proven experience in leading data projects, building scalable data pipelines, and delivering actionable insights through analytics and visualization platforms.
Education Requirements
- Bachelor’s degree in computer science or a related technical field.
- Minimum 5 years of relevant experience in Big Data Engineering/Data Analysis.
Experience Requirements
- Strong proficiency in Python, SQL, and Apache Spark (Core Spark, Spark Streaming, DataFrame API, DataSet API, RDD, Spark SQL).
- Hands-on experience with AWS services including EMR, Glue (serverless architecture), S3, Athena, IAM, Lambda, and CloudWatch.
- Expertise in Hadoop and distributed computing frameworks.
- Advanced SQL development with Hive/Impala including performance tuning.
- Experience with ElasticSearch (OpenSearch) and Kibana dashboards.
- Familiarity with resource management frameworks such as Yarn or Mesos.
- Knowledge of external job schedulers like Autosys, AWS Data Pipeline, or Airflow.
- Experience in HBase and other Key/Value data stores.
- Physical table design in Big Data environments.
Role & Responsibilities
Data Engineering & Management
- Design, develop, and maintain scalable data pipelines across Big Data platforms.
- Perform physical data modeling and table design for high-volume data systems.
- Build and manage data catalogs and dictionaries to maintain data integrity.
Cloud & Distributed Systems
- Deliver solutions using AWS services (EMR, Glue, Lambda, S3, Athena, Redshift, Snowflake).
- Optimize big data workloads leveraging Hadoop, Spark, and cloud-native frameworks.
- Ensure efficient resource management using Yarn/Mesos.
Data Integration & ETL
- Clean, transform, and blend data from multiple sources using ETL tools.
- Implement automation for data ingestion, transformation, and monitoring processes.
- Manage job orchestration using schedulers (Airflow, Autosys, AWS Data Pipeline).
Analytics & Visualization
- Deliver self-service analytics, KPIs, and dashboards for business teams.
- Develop Tableau dashboards, including calculated fields and interactive reports.
- Support advanced analytics by integrating ElasticSearch/OpenSearch with Kibana.
Collaboration & Leadership
- Work closely with cross-functional teams to define business requirements and translate them into technical data solutions.
- Lead and mentor a team of engineers to deliver high-quality data-driven outcomes.
- Ensure effective communication between business stakeholders, data engineering, and analytics teams.
Performance & Optimization
- Conduct SQL and query optimization for high-performance analytics.
- Monitor data pipelines, troubleshoot performance bottlenecks, and implement scalable solutions.
- Perform root cause analysis and ensure reliability of data platforms.
