Data Engineer

Job Title: Data Engineer

Location: Lahore / Kharian

Position Type: Full-Time

Job Overview: 

This role focuses on developing and implementing data warehouse solutions across the organization while managing large sets of structured and unstructured data. The Data Engineer will analyse complex customer requirements, define data transformation rules, and oversee their implementation. The ideal candidate should have a solid understanding of data acquisition, integration, and transformation, with the ability to evaluate and recommend optimal architecture and approaches.

Key Responsibilities:

• Design Data Pipelines : Develop robust data pipelines capable of handling both structured and unstructured data effectively.

• Build Data Ingestion and ETL Processes : Create efficient data ingestion pipelines and ETL processes, including low-latency data acquisition and stream processing using tools like Kafka or Glue.

• Develop Integration Procedures : Design and implement processes to integrate data warehouse solutions into operational IT environments.

• Data Lake Management: Manage and optimize the data lake on AWS, ensuring efficient data storage, retrieval, and transformation. Implement best practices for organizing and managing raw, processed, and curated data, with scalability and future growth in mind.

• Optimize SQL and Shell Scripts: Write, optimize, and maintain complex SQL queries and shell scripts to ensure efficient data processing.

• Monitor and Optimize Performance: Continuously monitor system performance and recommend necessary configurations or infrastructure improvements.

• Document and Present Workflows : Prepare detailed documentation, collaborate with cross functional teams, present complete data workflows to teams, and maintain an up-to-date knowledge base.

• Governance & Quality : Develop and maintain data quality checks to ensure data in the lake and warehouse remains accurate, consistent, and reliable.

• Collaboration with Stakeholders : Work closely with CTO, PMOs, business and data analysts to gather requirements and ensure alignment with project goals.

• Scope and Manage Projects : Collaborate with project managers to scope projects, create detailed work breakdown structures, and conduct risk assessments.

• Research & Development (R&D) : Keep up with the latest technological trends and identify innovative solutions to address customer challenges and company priorities.

Skills and Qualifications:

•Bachelor’s or Master’s degree in Engineering, Computer Science, or equivalent experience.

•At least 4+ years of relevant experience as a Data Engineer.

•Hands-on experience with cloud platforms such as AWS, Azure, or GCP, and familiarity with respective cloud services.

•Hands-on experience with one or more ETL tools such as Glue, Spark, Kafka, Informatica, DataStage, Talend, Azure Data Factory (ADF).

•Strong understanding of dimensional modeling techniques, including Star and Snowflake schemas.

•Experience in creating semantic models and reporting mapping documents.

•Solid concepts and experience in designing and developing ETL architectures.

•Strong understanding of RDBMS concepts and proficiency in SQL development.

•Proficiency in data modeling and mapping techniques.

•Experience integrating data from multiple sources.

•Experience working in distributed environments, including clustering and sharding.

•Knowledge of Big Data tools like Pig, Hive or NiFi would be a plus.

•Experience with Hadoop distributions like Cloudera or Hortonworks would be a plus.

•Excellent communication and presentation skills, both verbal and written.

•Ability to solve problems using a creative and logical approach.

•Self-motivated, analytical, detail-oriented, and organized, with a commitment to excellence.

•Experience in the financial services sector is a plus.

ACE Money Transfer Profile: