Data Engineer
vor 20 Stunden
Job Description: As a Data Engineer, you will be responsible for designing, developing, and maintaining robust data infrastructure and pipelines.
You will work closely with cross-functional teams to understand business requirements, optimize data workflows, and ensure the reliability and performance of our data systems.
Key Responsibilities: Design, build, and optimize data processing pipelines using Apache Spark, Python, Scala.Develop and maintain data ingestion and extraction processes, including streaming data pipelines using Kafka and batch processing workflows.Implement performance tuning techniques to optimize data processing and query performance, ensuring scalability and efficiency.Collaborate with DevOps teams to deploy and manage data infrastructure on NetApp S3 (very similar to AWS S3) and Kubernetes.Containerize data applications using Docker and orchestrate deployment using Kubernetes for scalability and reliability.Develop and maintain unit tests using frameworks like pytest and junit to ensure the quality and reliability of data pipelines.Implement and adhere to best practices for data governance, security, and compliance.Utilize Behavior-Driven Development (BDD) tools like Cucumber and Lettuce to write and execute test scenarios for data workflows.Stay up to date with emerging technologies and industry trends and evaluate their potential impact on our data infrastructure and processes.Requirements: Proven experience in designing and building scalable data pipelines using Apache Spark, Python, and Scala.Strong understanding of data warehousing concepts, ETL processes, and data modelling techniques.Experience with performance tuning and optimization of Spark jobs and SQL queries.Familiarity with stream processing frameworks like Kafka and messaging systems.Hands-on experience with containerization and orchestration tools like Docker and Kubernetes.Experience with unit testing frameworks like pytest, junit, and BDD tools like Cucumber and Lettuce.Excellent problem-solving skills and attention to detail.Strong communication and collaboration skills, with the ability to work effectively in a team environment.
#J-18808-Ljbffr
-
Data Engineer
vor 2 Monaten
Melbourne, Österreich Data Privacy VollzeitContract role initially until 30th June 2022Geelong based organisation - hybrid work modelGenerous rateThis is an opportunity for a Senior Data Engineer to join an organisation undertaking a business transformation, that is not only improving the way they operate, improving customer experience, but contains data and analytics as one of its pillars.The role...
-
Gis Specialist Data Engineer
vor 3 Monaten
Melbourne, Österreich Ecf Data, Llc VollzeitCapgemini is a global strategic and technological expert collective, operating in over 50 countries with more than 350,000 professionals.They partner with renowned clients to transform businesses using cloud, data, AI, and digital engineering.The company has significantly expanded in Australia and New Zealand, with over 3,500 team members.Join their rapidly...
-
Senior Azure Data Engineer
vor 3 Monaten
Melbourne, Österreich Ntt Data Business Solutions VollzeitNTT DATA is heavily investing globally in providing state of the art analytical solutions to its clients. Our capability has been focused locally, regionally in APAC and globally by forming strategic partnerships including Microsoft Azure, SAP Analytics, Snowflake, Matillion and many more, leveraging the strength of our cloud and network assets via the NTT...
-
Data Engineer
vor 2 Wochen
Melbourne, Österreich National Film And Sound Archive Of Australia VollzeitThe Data Engineer will play a key role in enhancing the NFSA's ability to manage, analyse, and report on data across multiple domains.Working closely with the software engineering and ICT teams, the Data Engineer will drive the development of advanced data infrastructure to support NFSA's data-driven decisions during early system and process...
-
Data Engineer
vor 2 Monaten
Melbourne, Österreich Experis Manpowergroup Sp. Z O.O. VollzeitAre you a skilled Data Engineer with expertise in data modelling and a strong background in Microsoft Azure?Do you have a passion for working with large datasets and geospatial data?If so, we want to hear from you!A dynamic organisation is seeking a Data Engineer to join their team and help improve the performance of a critical database used for airspace...
-
Data Engineer
vor 3 Monaten
Melbourne, Österreich Ultradynamix VollzeitRole: Data Engineer Client: Federal Government Contract: 12 Months Extension: 12 Months(2) Clearance: baseline Location: ACT or Offsite Requirements The CIO Division provides a range of enabling services and operations delivery to the department and to Australian businesses and is seeking an experienced and motivated data engineer to assist with the delivery...
-
Data Engineer
vor 2 Monaten
Melbourne, Österreich Onyx Gaming Pty Ltd VollzeitEngineering - Software (Information & Communication Technology)Full timeOnyx Gaming are a trusted operational advisor and a leading Electronic Gaming Machine (EGM) systems company.Our commitment to excellence has made us a force in the gaming industry, supporting hospitality venues across Australia.Due to continued growth we are seeking a highly skilled and...
-
Data Engineer
vor 1 Woche
Melbourne, Österreich Go1 VollzeitThis is a Data Engineer role with one of the leading companies in AU right now 🔥 -- Go1 -- with an amazing team. They are continuing to grow rapidly. This is the chance to join right as the 🚀 takes off.More About the Role at Go1 🧐🧐**The next chapter of our growth story needs you. GO there with us.** The Data team at Go1 offers global Data and...
-
Data Engineer
vor 1 Monat
Melbourne, Österreich State Government Of Victoria, Australia VollzeitOrganisation: Office of the Director-GeneralOccupation: IT and TelecommunicationsReference: 5901About this OpportunityThe Reporting and Analytics team in the Office of Director-General (ODG) is responsible for enabling better consumption of data and reporting for stakeholders across the Victorian Infrastructure Delivery Authority (VIDA) by maintaining an...
-
Data Engineer
vor 1 Monat
Melbourne, Österreich State Government Of Victoria, Australia VollzeitOrganisation: Office of the Director-GeneralOccupation: IT and TelecommunicationsReference: 5901About this OpportunityThe Reporting and Analytics team in the Office of Director-General (ODG) is responsible for enabling better consumption of data and reporting for stakeholders across the Victorian Infrastructure Delivery Authority (VIDA) by maintaining an...
-
Data Engineer
vor 4 Wochen
Melbourne, Österreich Roy Morgan Research VollzeitDatabase Development & Administration (Information & Communication Technology)Full timeAbout Roy MorganRoy Morgan is Australia's most reputable and longest established market research company, with an unparalleled reputation for reliable, accurate, meaningful, revealing market research. At the forefront of data and research innovation since 1941, today Roy...
-
Data Bricks
vor 3 Wochen
Melbourne, Österreich Pyramid Global Technologies VollzeitAbout the job: Data Bricks - Data EngineerJob Description of Data Bricks - Data Engineer in Melbourne:Tech Stack Required:Required: 6-10 YearsExperience in Data Lake, Azure Data BricksExperience designing and deploying Databricks platforms for any of AWS, Azure, or Google CloudExperience building and deploying data engineering pipelines into production,...
-
Data Engineer
vor 1 Monat
Melbourne, Österreich State Government Of Victoria, Australia VollzeitOrganisation: Office of the Director-GeneralOccupation: IT and TelecommunicationsReference: 5901About this Opportunity The Reporting and Analytics team in the Office of Director-General (ODG) is responsible for enabling better consumption of data and reporting for stakeholders across the Victorian Infrastructure Delivery Authority (VIDA) by maintaining an...
-
Data Engineer
vor 1 Monat
Melbourne, Österreich State Government Of Victoria, Australia VollzeitOrganisation: Office of the Director-GeneralOccupation: IT and TelecommunicationsReference: 5901About this Opportunity The Reporting and Analytics team in the Office of Director-General (ODG) is responsible for enabling better consumption of data and reporting for stakeholders across the Victorian Infrastructure Delivery Authority (VIDA) by maintaining an...
-
Data Bricks
vor 3 Wochen
Melbourne, Österreich Pyramid Global Technologies VollzeitAbout the job: Data Bricks - Data EngineerJob Description of Data Bricks - Data Engineer in Melbourne: Tech Stack Required: Required: 6-10 YearsExperience in Data Lake, Azure Data BricksExperience designing and deploying Databricks platforms for any of AWS, Azure, or Google CloudExperience building and deploying data engineering pipelines into production,...
-
Senior Data Engineer
vor 4 Wochen
Melbourne, Österreich Tridant Pty Ltd VollzeitWe are seeking an experienced Data Engineer or Senior Data Engineer to design, implement, and maintain data solutions using Microsoft Fabric and related technologies. The ideal candidate will bridge the gap between business needs and technical solutions, working within an Agile framework to deliver scalable data platforms.Key Responsibilities Lead the design...
-
Senior Data Engineer
vor 4 Wochen
Melbourne, Österreich Tridant Pty Ltd VollzeitWe are seeking an experienced Data Engineer or Senior Data Engineer to design, implement, and maintain data solutions using Microsoft Fabric and related technologies. The ideal candidate will bridge the gap between business needs and technical solutions, working within an Agile framework to deliver scalable data platforms.Key ResponsibilitiesLead the design...
-
Data Engineer
vor 2 Monaten
Melbourne, Österreich Neilson Financial Services Limited VollzeitCompany DescriptionAt Neilson, we're reimagining the future of data-driven insights with our pioneering division,DataWorks.As a cutting-edge division within the Neilson Group, DataWorks harnesses the latest advancements in AI, machine learning, and predictive analytics to empower growth across diverse markets and deliver next-generation solutions in Business...
-
Senior Data Engineer
vor 1 Monat
Melbourne, Österreich Tridant Pty Ltd VollzeitEngineering - Software (Information & Communication Technology) Full time We are seeking an experienced Data Engineer or Senior Data Engineer to design, implement, and maintain data solutions using Microsoft Fabric and related technologies.The ideal candidate will bridge the gap between business needs and technical solutions, working within an Agile...
-
Data Engineer
vor 1 Monat
Melbourne, Österreich Victorian Infrastructure Delivery Authority VollzeitCompany description: VIDA is established as an Administrative Office under the Public Administration Act 2004 as a specialist organisation.The Victorian Infrastructure Delivery Authority (VIDA) is responsible for delivering the biggest transport and health infrastructure program in Victorian history.This includes more than 200 projects such as the Metro...