Pyspark Engineer
vor 1 Monat
Project description
We are looking for skilled PySpark Engineers to join our team, working on a high-impact data engineering project.
The project involves processing large datasets, optimizing ETL pipelines, and building scalable solutions to manage complex data workflows.
The ideal candidate will collaborate closely with data scientists, data analysts, and software engineers to drive robust, data-driven insights for business decisions.
Responsibilities Design, develop, and maintain ETL pipelines using PySpark, optimizing for performance and scalability.Work with large volumes of structured and unstructured data, transforming data to meet business needs.Integrate data from multiple sources into the data platform, ensuring data integrity and quality.Collaborate with cross-functional teams to understand data requirements and translate them into efficient data workflows.Implement best practices for data governance, monitoring, and data security.Debug and troubleshoot issues across ETL pipelines and data workflows.Continuously improve performance, scalability, and reliability of existing data pipelines.Provide documentation and training for data workflows and processes.SKILLS Must have Proficiency in PySpark: In-depth experience with PySpark for data processing and transformation tasks.SQL Knowledge: Strong command of SQL for querying and processing data.Data Warehousing Concepts: Familiarity with data warehousing, data lakes, and data integration principles.Cloud Platforms: Experience with cloud environments like AWS, GCP, or Azure for data storage and processing.Big Data Technologies: Hands-on experience with Hadoop and Spark ecosystem (Spark SQL, Spark Streaming).Data Modeling: Experience in designing and implementing efficient data models.Python Programming: Strong Python skills, particularly in data manipulation and analysis.Nice to have Experience with Airflow or Other Orchestration Tools: Knowledge of workflow orchestration tools for scheduling and monitoring data pipelines.Knowledge of Apache Kafka: Understanding of Kafka for real-time data streaming and integration.Familiarity with Data Visualization Tools: Knowledge of visualization tools like Tableau, Power BI, or similar.Machine Learning Exposure: Familiarity with machine learning concepts, particularly with integrating ML models in data workflows.Agile Methodology: Experience working in Agile/Scrum environments.Data Governance and Compliance Knowledge: Understanding of data governance frameworks and compliance standards, such as GDPR.#J-18808-Ljbffr
-
Pyspark Engineer
vor 1 Monat
Sydney, Österreich Luxoft VollzeitProject descriptionWe are looking for skilled PySpark Engineers to join our team, working on a high-impact data engineering project. The project involves processing large datasets, optimizing ETL pipelines, and building scalable solutions to manage complex data workflows. The ideal candidate will collaborate closely with data scientists, data analysts, and...
-
Pyspark Engineer
vor 1 Monat
Sydney, Österreich Tideri Jobbörse VollzeitProject description We are looking for skilled PySpark Engineers to join our team, working on a high-impact data engineering project.The project involves processing large datasets, optimizing ETL pipelines, and building scalable solutions to manage complex data workflows.The ideal candidate will collaborate closely with data scientists, data analysts, and...
-
Senior Data Engineer
vor 4 Wochen
Sydney, Österreich Allura Partners VollzeitData Contract Databricks Senior Data Engineer – 12-Month Contract - Sydney - Databricks / PySpark / SQL / ETL About the Company This Sydney-based utilities company is a leader in its sector, delivering essential services to millions across Australia.Focused on innovation, they leverage advanced technologies and data-driven strategies to enhance efficiency,...
-
Lead Python Engineer
vor 1 Monat
Sydney, Österreich Bluefin Resources VollzeitTitle: Senior Software Engineer Role Type: Contract, 12 month daily rate Responsibilities: Develop and maintain scalable microservices Implement robust multithreading solutions Collaborate with multiple teams to execute a replatforming initiative Skills and Experience: Expertise in Python development (open to different frameworks) Experience in systems...
-
Senior Data Engineer
vor 4 Wochen
Sydney, Österreich Allura Partners VollzeitUtilities company looking for a senior data engineer on a 12 months contract.29th November, 2024Senior Data Engineer – 12-Month Contract - Sydney - Databricks / PySpark / SQL / ETLAbout the CompanyThis Sydney-based utilities company is a leader in its sector, delivering essential services to millions across Australia. Focused on innovation, they leverage...
-
Senior Data Engineer
vor 4 Wochen
Sydney, Österreich Allura Partners VollzeitUtilities company looking for a senior data engineer on a 12 months contract.29th November, 2024Senior Data Engineer – 12-Month Contract - Sydney - Databricks / PySpark / SQL / ETL About the CompanyThis Sydney-based utilities company is a leader in its sector, delivering essential services to millions across Australia. Focused on innovation, they leverage...
-
Allura Partners | Senior Data Engineer
vor 3 Wochen
Sydney, Österreich Tideri Jobbörse VollzeitUtilities company looking for a senior data engineer on a 12 months contract.29th November, 2024 Senior Data Engineer – 12-Month Contract - Sydney - Databricks / PySpark / SQL / ETLAbout the CompanyThis Sydney-based utilities company is a leader in its sector, delivering essential services to millions across Australia.Focused on innovation, they leverage...
-
Hadoop Engineer
vor 3 Monaten
Sydney, Österreich Zone It Solutions VollzeitWe are looking for a Hadoop Engineer for a role based in Sydney. You will be a part of a Global Consulting firm for one of their major projects.Technical: Implementation experience in BigData Platform; preferably in Cloudera Hadoop platform.Minimum 2 years of Development experience using Hadoop ecosystem tools & utilities: MapReduce, PySpark, Kafka, Sqoop,...
-
Lead Data Engineer
vor 2 Wochen
Sydney, Österreich Talent VollzeitThey are seeking an experienced and motivated Lead Data Engineer with 8-10 years of expertise in building and managing data platforms.The ideal candidate will be instrumental in designing and implementing scalable data solutions while leveraging cutting-edge technologies and tools.This role offers an opportunity to work on complex, real-time data integration...
-
Senior Data Engineer
vor 3 Wochen
Sydney, Österreich Hays Recruitment VollzeitAre you looking for your next Senior Data Engineering contract opportunity? A leading commercial organisation is looking for a Senior Data Engineer to join their highly engaged technology team.You will be working as part of an engaging and highly focused team, collaborating to deliver across the Data Transformation Program. You will be responsible for the...
-
Data Engineer
vor 1 Monat
Sydney, Österreich Experis Manpowergroup Sp. Z O.O. VollzeitWe are seeking an experienced Data Engineer with a strong background in AWS toolsets to advise, develop, and maintain data engineering solutions within the AWS Cloud ecosystem. This role involves designing, building, and managing both batch and real-time data pipelines using various AWS services.Key Responsibilities:Design, build, and operate batch and...
-
Data Engineer
vor 1 Monat
Sydney, Österreich Experis Manpowergroup Sp. Z O.O. VollzeitWe are seeking an experienced Data Engineer with a strong background in AWS toolsets to advise, develop, and maintain data engineering solutions within the AWS Cloud ecosystem.This role involves designing, building, and managing both batch and real-time data pipelines using various AWS services.Key Responsibilities:Design, build, and operate batch and...
-
Hays Recruitment | Senior Data Engineer | Sydney
vor 3 Wochen
Sydney, Österreich Tideri Jobbörse VollzeitAre you looking for your next Senior Data Engineering contract opportunity?A leading commercial organisation is looking for a Senior Data Engineer to join their highly engaged technology team.You will be working as part of an engaging and highly focused team, collaborating to deliver across the Data Transformation Program.You will be responsible for the...
-
Senior Data Engineer
vor 4 Wochen
Sydney, Österreich Bluefin Resources VollzeitFull Time Sydney: Hybrid working Salary: up to $160k, plus super (depending on experience) We are seeking a Senior Data Engineer to join a dynamic team within a forward-thinking organisation leveraging cutting-edge data technologies.If you are passionate about data engineering, have strong mentoring skills, and excel in stakeholder engagement, this...
-
Senior Data Engineer
vor 3 Wochen
Sydney, Österreich Bluefin Resources VollzeitSpecialist employment law firm Leadership opportunity $150-170k + super Full Time Sydney: Hybrid working Salary: up to $160k, plus super (depending on experience) We are seeking a Senior Data Engineer to join a dynamic team within a forward-thinking organisation leveraging cutting-edge data technologies.If you are passionate about data engineering, have...
-
Data Solutions Engineer
vor 2 Monaten
Sydney, Österreich Experis Manpowergroup Sp. Z O.O. VollzeitOn behalf of our client, a leading financial services organization, we are seeking a Solution Engineer/Designer to join their innovative data solutions team on a contract basis.This is a high-impact role that operates at the intersection of data engineering, analytics, platforms, and project delivery, providing thought leadership and innovation in the design...
-
Senior Databricks Engineer
Vor 7 Tagen
Sydney, Österreich Cognizant VollzeitAt Cognizant, we're at the forefront of technological innovation, and we're on the hunt for top talent to join our growing databricks practice at a variety of levels to join our team. As an advisor and implementor for leading Australian organisations, you will play a pivotal role in designing and implementing cutting-edge solutions that leverage the power of...
-
Senior Engineer
vor 4 Wochen
Sydney, Österreich Westpac VollzeitPerm Full Time role Sydney, NSW Experienced Data Engineer with AWS and design experience How will I help?This is a multi-year transformation program.We are currently recruiting the development teams that will form the core of any future ramp up/down driven by the sprint back-log.In parallel to delivering a top business outcome for bank, we are looking...
-
Senior Data Engineer
vor 4 Wochen
Sydney, Österreich Suncorp VollzeitSenior Data EngineerJob No: 673501Sydney - Inner Suburbs & CBD, Brisbane - Inner Suburbs & CBD, Melbourne - Inner Suburbs & CBD, VariousPermanent Full TimeITPay Band 5Flexible working Locations of Melbourne, Brisbane or SydneyHybrid Working Environment (WFH)Implement and maintain Datawarehouse platform and data pipelinesWe're never just satisfied with how...
-
Senior Data Engineer
Vor 2 Tagen
Sydney, Österreich Airtree Ventures Pty VollzeitCompany DescriptionWhether you're working from home or on your way to the office, chances are you're probably reading this because you're seeking a new adventure, or you're up for a new challenge. GROW Inc might just be that for you!So, who is GROW Inc?Founded in 2016, we're the Australian FinTech success story you probably haven't heard about. We're solving...
