Data Engineer - Python | DB Idyllic Services Pvt. Ltd.

About Idyllic Services Pvt. Ltd.

Job Description

As a Python data engineer specializing in ETL processes, Pyspark, and Pandas, your primary responsibility will be to collaborate with stakeholders to gather requirements, refine stories, design solutions, implement them, test thoroughly, and provide ongoing support in production environments. You will be utilizing data Behavior-Driven Development (BDD) techniques to work closely with users, analysts, developers, and other testers to ensure that we are building the right solutions. Your proficiency in Python programming, especially with libraries such as pandas, numpy, datetime, and re, will be instrumental in transforming raw data using various techniques like group by, filter, pivot, rank, merge, join, function, lambda function, and loop. Handling large datasets efficiently and creating data pipelines for reading data from APIs and storing them in databases will be key aspects of your role. Additionally, you should be adept at managing APIs in data pipelines, optimizing SQL data access through high-performance SQL scripts, and ideally have experience with Google Cloud Platform (GCP) solutions. Practical experience in building data engineering solutions, familiarity with version control systems like GIT, and a background in agile methodologies such as Scrum, Kanban, or XP will be valuable assets in this position. Your excellent English communication skills will enable effective discussions with stakeholders from the UK, US, and Germany, contributing to successful stakeholder management. Overall, your technical expertise combined with your collaborative approach and experience in data engineering will play a crucial role in driving the success of our projects.,