📍 Role Overview
We are looking for a Senior Data Engineer with strong hands-on experience in Scala, Spark, and PySpark to design, build, and optimize scalable data pipelines. The ideal candidate has worked on migration projects, operates in DevOps-driven environments, and is comfortable leveraging AI tools such as GitHub Copilot to enhance productivity.
This is a highly technical, hands-on role focused on large-scale data processing and engineering.
🔧 Key Responsibilities
-
Design, develop, and maintain scalable data pipelines using Spark
-
Develop production-grade applications using Scala and PySpark
-
Participate in data platform migration initiatives
-
Optimize Spark jobs for performance and scalability
-
Implement automation and deployment pipelines in a DevOps environment
-
Develop and maintain shell scripts for operational tasks
-
Use AI-assisted tools such as GitHub Copilot to improve development efficiency
-
Troubleshoot complex data and performance issues
-
Collaborate with cross-functional teams to deliver high-quality data solutions
✅ Mandatory Skills & Experience
-
Strong hands-on experience with Spark
-
Solid development experience in Scala
-
Proven experience in PySpark
-
Experience with Shell scripting
-
Prior involvement in migration projects (legacy to modern platforms, on-prem to cloud, etc.)
-
Exposure to DevOps practices (CI/CD, version control, automation)
-
Familiarity with AI tools such as GitHub Copilot
-
Strong analytical and problem-solving skills