Position: Senior Azure Data Engineer, Hyderabad

Department: Information Technology | Role: Full-time | Experience: 5 to 8 Years | Number of Positions: 2 | Location: Hyderabad

Skillset:

Azure Data Bricks, Apache Spark, Delta Lake, ML Pipeline, ADF, ADLS, Azure DevOps, Medallion Architecture, Python, Pyspark, SQL, Scala/Java, Data Governance, Performance Optimization, Project Ownership, Excellent English communication skills

Job Description:

About Us:

We provide companies with innovative technology solutions for everyday business problems. Our passion is to help clients become intelligent, information-driven organizations, where fact-based decision-making is embedded into daily operations, which leads to better processes and outcomes. Our team combines strategic consulting services with growth-enabling technologies to evaluate risk, manage data, and leverage AI and automated processes more effectively. With deep, big four consulting experience in business transformation and efficient processes, we are a game-changer in any operations strategy.

We are seeking a highly skilled and experienced Senior Data Engineer to join our growing data team. The ideal candidate will have a deep understanding of Azure Databricks, Apache Spark, and the Azure data ecosystem. You will lead the design and implementation of scalable data pipelines, enforce best practices, and ensure delivery of high-quality data products using the Medallion Architecture (Bronze, Silver, Gold).

Key Responsibilities:

• Lead the design and implementation of end-to-end data pipelines using Azure Databricks and Delta Lake.

• Build scalable, high-performance ETL/ELT processes across structured and unstructured data sources.

• Implement and optimize Bronze, Silver, and Gold architecture for data curation.

• Integrate with Azure Data Factory, Data Lake, Synapse, and other Azure services.

• Apply best practices for data modeling, partitioning, Z-ordering, and performance tuning.

• Ensure high data quality through validation frameworks (e.g., Great Expectations).

• Enforce data governance, access control, and compliance using Azure Key Vault, RBAC, and Azure Purview.

• Collaborate with data analysts, data scientists, and product teams to understand data needs.

• Mentor junior engineers and participate in architecture/code reviews.

• Contribute to CI/CD pipelines for data workflows using Azure DevOps or similar tools.

Required Qualifications:

• 5 to 8 years of experience in data engineering with at least 2+ years hands-on with Azure Databricks for pipeline development and data modelling.

Technical Skills:

o Azure Ecosystem Expertise:

1. Azure Databricks: End-to-end development including Delta Lake, notebook workflows, and ML integrations.

2. Azure Data Factory: For orchestration and data pipeline integration.

3. Azure Storage (Blob/Data Lake Gen2): Working with storage accounts for structured and unstructured data.

4. Azure Key Vault: Securing secrets and managing credentials in pipelines.

5. Azure DevOps: CI/CD for data pipelines and notebooks.

o Big Data & Distributed Processing:

1. Apache Spark (via Databricks): Advanced knowledge of Spark SQL, PySpark, DataFrames, RDDs, and performance tuning.

2. Delta Lake: Understanding ACID transactions, schema enforcement, time travel, and optimizing data formats.

o Data Modelling and Warehousing:

1. Dimensional modelling (Star/Snowflake schemas).

2. Knowledge of modern data warehousing principles.

3. Experience implementing medallion architecture (Bronze/Silver/Gold layers).

4. Experience working with Parquet, JSON, CSV, or other data formats.

o Programming Languages:

1. Python: For data transformation, notebook development, automation.

2. SQL: Strong grasp of SQL for querying and performance tuning.

3. Scala (nice to have): Useful for certain Spark applications.

o CI/CD and Infrastructure Automation:

1. Experience with Git repositories (branching, PRs).

2. Automated deployments via Azure DevOps Pipelines.

o Data Engineering & Analytical Skills:

1. ETL/ELT pipeline design and optimization.

2. Data quality and validation frameworks.

o Security & Governance:

1. RBAC and access controls within Azure and Databricks.

2. Data encryption, secure key management.

3. GDPR, HIPAA, or other compliance-aware data handling.

Soft Skills & Leadership:

o Stakeholder Communication: Translate technical jargon into business value.

o Project Ownership: End-to-end delivery including design, implementation, and monitoring.

o Mentorship: Guide junior engineers and establish best practices.

o Agile Practices: Work in sprints, participate in scrum ceremonies, story estimation.

• Education:

• Bachelor’s or master’s degree in computer science, Data Engineering, or a related field.

• Certifications such as Azure Data Engineer, Databricks Certified Data Engineer Professional is a plus. 

Additional Information:

Why Join Us?

• Lead mission-critical data projects with enterprise impact.

• Collaborate with a high-performing team in a cloud-native environment.

• Influence data architecture and engineering strategy across the organization.

• Access to continuous learning, leadership programs, and certification support.

Required Qualification:

Bachelor of Engineering - Bachelor of Technology (B.E./B.Tech.)- IT/CS/E&CE/ Data Engineering/MCA

With a fast-growing analytics, business intelligence, and automation company