Data Engineer

vor 20 Stunden


Melbourne, Österreich Clearcompany Vollzeit

Job Description: As a Data Engineer, you will be responsible for designing, developing, and maintaining robust data infrastructure and pipelines.
You will work closely with cross-functional teams to understand business requirements, optimize data workflows, and ensure the reliability and performance of our data systems.

Key Responsibilities: Design, build, and optimize data processing pipelines using Apache Spark, Python, Scala.Develop and maintain data ingestion and extraction processes, including streaming data pipelines using Kafka and batch processing workflows.Implement performance tuning techniques to optimize data processing and query performance, ensuring scalability and efficiency.Collaborate with DevOps teams to deploy and manage data infrastructure on NetApp S3 (very similar to AWS S3) and Kubernetes.Containerize data applications using Docker and orchestrate deployment using Kubernetes for scalability and reliability.Develop and maintain unit tests using frameworks like pytest and junit to ensure the quality and reliability of data pipelines.Implement and adhere to best practices for data governance, security, and compliance.Utilize Behavior-Driven Development (BDD) tools like Cucumber and Lettuce to write and execute test scenarios for data workflows.Stay up to date with emerging technologies and industry trends and evaluate their potential impact on our data infrastructure and processes.Requirements: Proven experience in designing and building scalable data pipelines using Apache Spark, Python, and Scala.Strong understanding of data warehousing concepts, ETL processes, and data modelling techniques.Experience with performance tuning and optimization of Spark jobs and SQL queries.Familiarity with stream processing frameworks like Kafka and messaging systems.Hands-on experience with containerization and orchestration tools like Docker and Kubernetes.Experience with unit testing frameworks like pytest, junit, and BDD tools like Cucumber and Lettuce.Excellent problem-solving skills and attention to detail.Strong communication and collaboration skills, with the ability to work effectively in a team environment.
#J-18808-Ljbffr


  • Data Engineer

    vor 2 Monaten


    Melbourne, Österreich Data Privacy Vollzeit

    Contract role initially until 30th June 2022Geelong based organisation - hybrid work modelGenerous rateThis is an opportunity for a Senior Data Engineer to join an organisation undertaking a business transformation, that is not only improving the way they operate, improving customer experience, but contains data and analytics as one of its pillars.The role...


  • Melbourne, Österreich Ecf Data, Llc Vollzeit

    Capgemini is a global strategic and technological expert collective, operating in over 50 countries with more than 350,000 professionals.They partner with renowned clients to transform businesses using cloud, data, AI, and digital engineering.The company has significantly expanded in Australia and New Zealand, with over 3,500 team members.Join their rapidly...


  • Melbourne, Österreich Ntt Data Business Solutions Vollzeit

    NTT DATA is heavily investing globally in providing state of the art analytical solutions to its clients. Our capability has been focused locally, regionally in APAC and globally by forming strategic partnerships including Microsoft Azure, SAP Analytics, Snowflake, Matillion and many more, leveraging the strength of our cloud and network assets via the NTT...

  • Data Engineer

    vor 2 Wochen


    Melbourne, Österreich National Film And Sound Archive Of Australia Vollzeit

    The Data Engineer will play a key role in enhancing the NFSA's ability to manage, analyse, and report on data across multiple domains.Working closely with the software engineering and ICT teams, the Data Engineer will drive the development of advanced data infrastructure to support NFSA's data-driven decisions during early system and process...

  • Data Engineer

    vor 2 Monaten


    Melbourne, Österreich Experis Manpowergroup Sp. Z O.O. Vollzeit

    Are you a skilled Data Engineer with expertise in data modelling and a strong background in Microsoft Azure?Do you have a passion for working with large datasets and geospatial data?If so, we want to hear from you!A dynamic organisation is seeking a Data Engineer to join their team and help improve the performance of a critical database used for airspace...

  • Data Engineer

    vor 3 Monaten


    Melbourne, Österreich Ultradynamix Vollzeit

    Role: Data Engineer Client: Federal Government Contract: 12 Months Extension: 12 Months(2) Clearance: baseline Location: ACT or Offsite Requirements The CIO Division provides a range of enabling services and operations delivery to the department and to Australian businesses and is seeking an experienced and motivated data engineer to assist with the delivery...

  • Data Engineer

    vor 2 Monaten


    Melbourne, Österreich Onyx Gaming Pty Ltd Vollzeit

    Engineering - Software (Information & Communication Technology)Full timeOnyx Gaming are a trusted operational advisor and a leading Electronic Gaming Machine (EGM) systems company.Our commitment to excellence has made us a force in the gaming industry, supporting hospitality venues across Australia.Due to continued growth we are seeking a highly skilled and...

  • Data Engineer

    vor 1 Woche


    Melbourne, Österreich Go1 Vollzeit

    This is a Data Engineer role with one of the leading companies in AU right now 🔥 -- Go1 -- with an amazing team. They are continuing to grow rapidly. This is the chance to join right as the 🚀 takes off.More About the Role at Go1 🧐🧐**The next chapter of our growth story needs you. GO there with us.** The Data team at Go1 offers global Data and...

  • Data Engineer

    vor 1 Monat


    Melbourne, Österreich State Government Of Victoria, Australia Vollzeit

    Organisation: Office of the Director-GeneralOccupation: IT and TelecommunicationsReference: 5901About this OpportunityThe Reporting and Analytics team in the Office of Director-General (ODG) is responsible for enabling better consumption of data and reporting for stakeholders across the Victorian Infrastructure Delivery Authority (VIDA) by maintaining an...

  • Data Engineer

    vor 1 Monat


    Melbourne, Österreich State Government Of Victoria, Australia Vollzeit

    Organisation: Office of the Director-GeneralOccupation: IT and TelecommunicationsReference: 5901About this OpportunityThe Reporting and Analytics team in the Office of Director-General (ODG) is responsible for enabling better consumption of data and reporting for stakeholders across the Victorian Infrastructure Delivery Authority (VIDA) by maintaining an...

  • Data Engineer

    vor 4 Wochen


    Melbourne, Österreich Roy Morgan Research Vollzeit

    Database Development & Administration (Information & Communication Technology)Full timeAbout Roy MorganRoy Morgan is Australia's most reputable and longest established market research company, with an unparalleled reputation for reliable, accurate, meaningful, revealing market research. At the forefront of data and research innovation since 1941, today Roy...

  • Data Bricks

    vor 3 Wochen


    Melbourne, Österreich Pyramid Global Technologies Vollzeit

    About the job: Data Bricks - Data EngineerJob Description of Data Bricks - Data Engineer in Melbourne:Tech Stack Required:Required: 6-10 YearsExperience in Data Lake, Azure Data BricksExperience designing and deploying Databricks platforms for any of AWS, Azure, or Google CloudExperience building and deploying data engineering pipelines into production,...

  • Data Engineer

    vor 1 Monat


    Melbourne, Österreich State Government Of Victoria, Australia Vollzeit

    Organisation: Office of the Director-GeneralOccupation: IT and TelecommunicationsReference: 5901About this Opportunity The Reporting and Analytics team in the Office of Director-General (ODG) is responsible for enabling better consumption of data and reporting for stakeholders across the Victorian Infrastructure Delivery Authority (VIDA) by maintaining an...

  • Data Engineer

    vor 1 Monat


    Melbourne, Österreich State Government Of Victoria, Australia Vollzeit

    Organisation: Office of the Director-GeneralOccupation: IT and TelecommunicationsReference: 5901About this Opportunity The Reporting and Analytics team in the Office of Director-General (ODG) is responsible for enabling better consumption of data and reporting for stakeholders across the Victorian Infrastructure Delivery Authority (VIDA) by maintaining an...

  • Data Bricks

    vor 3 Wochen


    Melbourne, Österreich Pyramid Global Technologies Vollzeit

    About the job: Data Bricks - Data EngineerJob Description of Data Bricks - Data Engineer in Melbourne: Tech Stack Required: Required: 6-10 YearsExperience in Data Lake, Azure Data BricksExperience designing and deploying Databricks platforms for any of AWS, Azure, or Google CloudExperience building and deploying data engineering pipelines into production,...

  • Senior Data Engineer

    vor 4 Wochen


    Melbourne, Österreich Tridant Pty Ltd Vollzeit

    We are seeking an experienced Data Engineer or Senior Data Engineer to design, implement, and maintain data solutions using Microsoft Fabric and related technologies. The ideal candidate will bridge the gap between business needs and technical solutions, working within an Agile framework to deliver scalable data platforms.Key Responsibilities Lead the design...

  • Senior Data Engineer

    vor 4 Wochen


    Melbourne, Österreich Tridant Pty Ltd Vollzeit

    We are seeking an experienced Data Engineer or Senior Data Engineer to design, implement, and maintain data solutions using Microsoft Fabric and related technologies. The ideal candidate will bridge the gap between business needs and technical solutions, working within an Agile framework to deliver scalable data platforms.Key ResponsibilitiesLead the design...

  • Data Engineer

    vor 2 Monaten


    Melbourne, Österreich Neilson Financial Services Limited Vollzeit

    Company DescriptionAt Neilson, we're reimagining the future of data-driven insights with our pioneering division,DataWorks.As a cutting-edge division within the Neilson Group, DataWorks harnesses the latest advancements in AI, machine learning, and predictive analytics to empower growth across diverse markets and deliver next-generation solutions in Business...

  • Senior Data Engineer

    vor 1 Monat


    Melbourne, Österreich Tridant Pty Ltd Vollzeit

    Engineering - Software (Information & Communication Technology) Full time We are seeking an experienced Data Engineer or Senior Data Engineer to design, implement, and maintain data solutions using Microsoft Fabric and related technologies.The ideal candidate will bridge the gap between business needs and technical solutions, working within an Agile...

  • Data Engineer

    vor 1 Monat


    Melbourne, Österreich Victorian Infrastructure Delivery Authority Vollzeit

    Company description: VIDA is established as an Administrative Office under the Public Administration Act 2004 as a specialist organisation.The Victorian Infrastructure Delivery Authority (VIDA) is responsible for delivering the biggest transport and health infrastructure program in Victorian history.This includes more than 200 projects such as the Metro...