Senior Data Engineer

At LifeByte, we are a dynamic and innovative collective of tech visionaries driven by a relentless pursuit of excellence. Each of us brings a unique set of skills to the table, collaborating on projects that shape the future. Founded in 2017, we are dedicated to fostering an ecosystem of seamless resource exchange, where efficiency and precision are paramount. With cutting‑edge solutions, we empower businesses to thrive and individuals to unlock their full potential. Committed to high‑tech innovation, we are actively reshaping the future, one Byte at a Time. We are looking for a highly skilled Data Engineer to help build and optimise the company's data infrastructure. This position will work closely with data scientists, analysts and business teams to improve the efficiency of the company's data processing and decision making by building efficient data pipelines and storage solutions. Job Responsibilities Work closely with business stakeholders to understand data requirements, translate business needs into scalable solutions, and ensure timely and reliable data delivery to support decision‑making. Collaborate with data analysts and data scientists to enable advanced analytics and business reporting through high‑quality, well‑modeled datasets. Together with the team, maintain, update, and strengthen the existing Snowflake Data Lake , ensuring stability, performance, cost efficiency, and governance across the data ecosystem. Design, develop, and maintain data pipelines and ELT/ETL workflows for both batch and streaming data. Implement and optimize Spark UDFs/UDAFs in Java/Scala when built‑in functions are insufficient; ensure correctness, performance, and testability while minimizing serialization overhead. Contribute to Spark performance tuning (partitioning, broadcast joins, caching, AQE) and data modeling best practices. Troubleshoot and resolve production issues to maintain data integrity, timeliness, reliability, and service‑level objectives (SLOs). Ensure adherence to data governance, lineage, security , and compliance standards; contribute to metadata management. Participate in production support rotation and contribute to ongoing system improvements. Support automation, CI/CD, and DevOps best practices for data pipelines and infrastructure (Git/GitHub Actions or GitLab CI, Docker/Kubernetes). Skills & Qualifications Strong proficiency in SQL (query optimisation, performance tuning, analytical use cases). Excellent Python programming skills for data transformation and automation. Familiar with Spark (PySpark and/or Scala) and distributed data processing; understanding of Dataset/DataFrame APIs and Spark SQL. Basic Java proficiency (collections, generics, exceptions) and the ability to write/maintain Spark UDFs/UDAFs in Java or Scala. Hands‑on experience with Snowflake (query optimisation, resource management, ecosystem integration). Knowledge of ETL/ELT tools (e.g., dbt, Airflow or similar). Experience in data modeling (Star Schema, Data Vault, or similar). Exposure real‑time streaming / CDC (Kafka, Debezium, Flink, or similar). Experience in cloud environments (AWS preferred); familiarity with containerisation (Docker) and orchestration (Kubernetes) is a plus. Understanding of data governance and metadata management tools (e.g., Collibra, Alation) and lineage frameworks (e.g., OpenLineage). Familiar with Git, CI/CD pipelines, and modern development practices. Strong analytical thinking, problem‑solving, and communication skills. Preferred Experience 3+ years of experience as a Data Engineer or in a similar role. Experience with DataOps or data platform modernisation projects. Familiarity with Snowflake advanced features (Streams, Tasks, Time Travel, Resource Monitors). Exposure real‑time analytics and event‑driven architectures. Benefits Hybrid working arrangement - 2 Days of remote work per week Opportunities for enriching career growth, including exposure to regional contexts Complimentary snacks and beverages available in the office pantry Healthcare coverage (medical, dental, optical), gym benefits Flexibility in smart casual dress code Young, vibrant and open work culture #J-18808-Ljbffr