Position: Lead Data Engineer, Hyderabad

Department: Information Technology | Role: Full-time | Experience: 8 to 12 Years | Number of Positions: 3 | Location: Hyderabad

Skillset:

Data Architecture, Leadership, Matillion ETL, DBT, SQL, NoSQL, Python, Snwoflake, Azure/AWS, Excellent English communication skills, Strong Stakeholder management skills

Job Description:

We are looking for an accomplished and dynamic Data Engineering Lead to join our team and drive the design, development, and delivery of cutting-edge data solutions. This role requires a balance of strong technical expertise, strategic leadership, and a consulting mindset. As the Lead Data Engineer, you will oversee the design and development of robust data pipelines and systems, manage and mentor a team of 5 to 7 engineers, and play a critical role in architecting innovative solutions tailored to client needs.

You will lead by example, fostering a culture of accountability, ownership, and continuous improvement while delivering impactful, scalable data solutions in a fast-paced, consulting environment. 

Key Responsibilities

Client Collaboration:

• Act as the primary point of contact for US-based clients, ensuring alignment on project goals, timelines, and deliverables.

• Engage with stakeholders to understand requirements and ensure alignment throughout the project lifecycle.

• Present technical concepts and designs to both technical and non-technical audiences.

• Communicate effectively with stakeholders to ensure alignment on project goals, timelines, and deliverables.

• Set realistic expectations with clients and proactively address concerns or risks.

Data Solution Design and Development:

• Architect, design, and implement end-to-end data pipelines and systems that handle large-scale, complex datasets.

• Ensure optimal system architecture for performance, scalability, and reliability.

• Evaluate and integrate new technologies to enhance existing solutions.

• Implement best practices in ETL/ELT processes, data integration, and data warehousing.

Project Leadership and Delivery:

• Lead technical project execution, ensuring timelines and deliverables are met with high quality.

• Collaborate with cross-functional teams to align business goals with technical solutions. 

• Act as the primary point of contact for clients, translating business requirements into actionable technical strategies. 

Team Leadership and Development:

• Manage, mentor, and grow a team of 5 to 7 data engineers; Ensure timely follow-ups on action items and maintain seamless communication across time zones.

• Conduct code reviews, validations, and provide feedback to ensure adherence to technical standards.

• Provide technical guidance and foster an environment of continuous learning, innovation, and collaboration.

• Support collaboration and alignment between the client and delivery teams.

Optimization and Performance Tuning:

• Be hands-on in developing, testing, and documenting data pipelines and solutions as needed.

• Analyze and optimize existing data workflows for performance and cost-efficiency.

• Troubleshoot and resolve complex technical issues within data systems.

Adaptability and Innovation:

• Embrace a consulting mindset with the ability to quickly learn and adopt new tools, technologies, and frameworks.

• Identify opportunities for innovation and implement cutting-edge technologies in data engineering.

• Exhibit a "figure it out" attitude, taking ownership and accountability for challenges and solutions.

Learning and Adaptability:

• Stay updated with emerging data technologies, frameworks, and tools.

• Actively explore and integrate new technologies to improve existing workflows and solutions.

Internal Initiatives and Eminence Building:

• Drive internal initiatives to improve processes, frameworks, and methodologies.

• Contribute to the organization’s eminence by developing thought leadership, sharing best practices, and participating in knowledge-sharing activities.

Qualifications Education:

• Bachelor’s or master’s degree in computer science, Data Engineering, or a related field.

• Certifications in cloud platforms such as Snowflake Snowpro, Data Engineer is a plus.

Experience:

• 8+ years of experience in data engineering with hands-on expertise in data pipeline development, architecture, and system optimization

• Demonstrated success in managing global teams, especially across US and India time zones.

• Proven track record in leading data engineering teams and managing end-to-end project delivery.

• Strong background in data warehousing and familiarity with tools such as Matillion, dbt, Striim, etc. 

Technical Skills:

• Lead the design, development, and deployment of scalable data architectures, pipelines, and processes tailored to client needs

• Expertise in programming languages such as Python, Scala, or Java.

• Proficiency in designing and delivering data pipelines in Cloud Data Warehouses (e.g., Snowflake, Redshift), using various ETL/ELT tools such as Matillion, dbt, Striim, etc.

• Solid understanding of database systems (relational and NoSQL) and data modeling techniques.

• Hands-on experience of 2+ years in designing and developing data integration solutions using Matillion and/or dbt.

• Strong knowledge of data engineering and integration frameworks.

• Expertise in architecting data solutions.

• Successfully implemented at least two end-to-end projects with multiple transformation layers.

• Good grasp of coding standards, with the ability to define standards and testing strategies for projects.

• Proficiency in working with cloud platforms (AWS, Azure, GCP) and associated data services.

• Enthusiastic about working in Agile methodology.

• Possess a comprehensive understanding of the DevOps process including GitHub integration and CI/CD pipelines. 

Soft Skills:

• Exceptional problem-solving and analytical skills.

• Strong communication and interpersonal skills to manage client relationships and team dynamics.

• Ability to thrive in a consulting environment, quickly adapting to new challenges and domains.

• Ability to handle ambiguity and proactively take ownership of challenges.

• Demonstrated accountability, ownership, and a proactive approach to solving problems. 

Additional Information:

Why Join Us?

• Be at the forefront of data innovation and lead impactful projects.

• Work with a collaborative and forward-thinking team.

• Opportunity to mentor and develop talent in the data engineering space.

• Competitive compensation and benefits package.

Required Qualification:

Bachelor of Engineering - Bachelor of Technology (B.E./B.Tech.) - IT/CS/E&CE/MCA

With a fast-growing analytics, business intelligence, and automation company