Job Description

To support the build and modernization of the Data Hub within the client ecosystem— an enterprise data ecosystem — by developing, automating, and sustaining data pipelines for reporting, performance measurement, and analytics.

🎯 Key Responsibilities

• Design, build, and maintain automated, scalable data pipelines for performance and reporting datasets.
• Refactor legacy pipelines to meet enterprise data standards and improve performance. 
• Implement robust data quality routines (validation, anomaly detection, schema checks). 
• Enhance orchestration and error handling for workflow reliability.
• Support infrastructure tasks like environment refreshes, upgrades, and disaster recovery testing.
• Ensure compliance with metadata, lineage, classification, and documentation standards.
• Collaborate on datasets for executive reporting and strategic planning across the Personal & Commercial Operations (PCO), ODIA (Operational Data, Information & AI).

Job Requirements

🛠️ Core Technical Skills Required

• 3+ years in hands-on data engineering
• Tools & Languages:
• PySpark, Python, SQL 
• Hadoop ecosystem: Spark, Hive, HDFS
• Shell scripting, Linux environments
   
• Query tuning with Spark SQL, Trino, Hive
• Deployment experience across dev/test/prod
• Working with partitioned datasets, Parquet, columnar storage
• Strong grasp of data governance, metadata, and validation
   

⭐ Preferred / Bonus Skills

• Experience with event-driven or real-time processing (e.g., Kafka)
• Familiarity with vector databases, LLM-supportive architectures
• Tools: Apache Airflow, OpenShift (OCP4), AWS (for orchestration/scheduling)

We welcome candidates with the skillset above to immediately drop an email to nisha@talentspothub.com for more information and confidential conversation.

Skills