Staff Data Engineer
vor 1 Monat
Staff Data Engineer - FlinkCommonwealth Bank of Australia Sydney, Australia
Do work that matters:
We're building tomorrow's bank today.
This means we need creative and diverse engineers to help us redefine what customers expect from a bank.
Envisioning new technologies that are still waiting to be invented and reimagining products that support our customers and help build Australia's future economy.
We are a multi-disciplinary team of passionate end-to-end data & analytics delivery professionals that enables data delivery democratisation, uplift data maturity and literacy across the Group.
We provide data delivery thought leadership and create leading examples of delivering data safely and swiftly for the Group, using latest data & AI strategies, cutting edge solutions, latest technology tooling, simplified delivery patterns & new ways of working.
This role will focus on our Groups real-time stream processing services with the key focus on Apache Flink.
Key Responsibilities: Design, develop, and optimize Flink data streams to implement event buffering strategies based on fixed time intervals, sliding windows, session windows, and custom triggers.Develop and maintain robust Flink pipelines for processing and aggregating data from various real-time data sources, including Kafka, Amazon MSK, etc.Implement and manage stateful stream processing with checkpointing, fault tolerance, and exactly-once semantics.Create custom Flink Process Functions with precise time-based buffering logic and timers for controlling event triggers and processing.Collaborate with data engineers and devops to ensure seamless integration of Flink applications with downstream data storage and processing systems (e.g., S3, databases, data lakes).Monitor and fine-tune Flink job performance using built-in metrics, profiling tools, and customized metrics and alerts to ensure minimal event lag, low latency, and high throughput.Identify and resolve bottlenecks, data loss issues, and challenges related to buffering, windowing, and high-volume stream processing.Implement and configure event retention policies on streaming sources such as Apache Kafka and ensure end-to-end data consistency and accuracy.Work with cross-functional teams to architect scalable solutions for streaming data processing across multiple business units.Stay up-to-date with the latest trends in stream processing, distributed data processing, and Apache Flink technologies to bring innovative solutions to the organization. We're interested in hearing from people who: Have experience building and managing stateful stream processing pipelines, including checkpointing and state backends.Bring a strong understanding of distributed systems, data buffering, and data pipeline architectures.Provide excellent problem-solving skills with a focus on performance optimization and reliability.Have strong collaboration and communication skills for working in cross-functional teams.Have the ability to document technical designs, processes, and troubleshooting guides. Tech Skills: Expert-level knowledge of Apache Flink and its windowing mechanisms (Tumbling, Sliding, Session windows, Global windows, etc.
).Hands-on experience with data streaming platforms like Apache Kafka, Amazon MSK, and related technologies.Strong proficiency in Python for Flink application development.Experience with monitoring, logging, and alerting in distributed data processing environments (e.g., Grafana, Prometheus, CloudWatch). Preferred Qualifications: Certification with AWS Data Engineering and AWS Solution Architect Professional (e.g., AWS Kinesis Data Analytics, AWS Lambda).Experience with NoSQL or SQL databases used in conjunction with Flink.Familiarity with Infrastructure-as-Code (IaC) tools like Terraform, CloudFormation, or CDK.
Advertising End Date: 01/12/2024 #J-18808-Ljbffr
-
Staff Data Engineer Design | Sydney, Au
vor 1 Monat
Sydney, Österreich Efinancialcareers Ltd. VollzeitStaff Data Engineer DesignCommonwealth Bank of Australia Sydney, AustraliaStaff Data Engineer DesignCommonwealth Bank of Australia Sydney, AustraliaYou are an experienced solution designer with strong experience in Data modelling, Big Data platforms.We are one of the critically performing Data Engineering teams in CBATogether we can lead strategic data...
-
Staff Data Engineer Design
vor 1 Monat
Sydney, Österreich Commonwealth Bank VollzeitStaff Engineer - Data Solution Design You are an experienced solution designer with strong experience in Data modelling and Big Data platforms.We are one of the critically performing Data Engineering teams in CBA.Together we can lead strategic data warehouse applications and largely complex operational data store!Do work that matters: As an experienced...
-
Data Engineer
vor 2 Monaten
Sydney, Österreich Rich Data Co. VollzeitHybrid and flexible working arrangementsAbout RDCRich Data Co (RDC) is delivering the future of credit today! We believe credit should be accessible, fair, inclusive, and sustainable. We are passionate about AI and developing new techniques that leverage traditional and non-traditional data to get the right decision in a clear and explainable way. Global...
-
Data Engineer
vor 2 Monaten
Sydney, Österreich Rich Data Co. VollzeitHybrid and flexible working arrangementsAbout RDC Rich Data Co (RDC) is delivering the future of credit today!We believe credit should be accessible, fair, inclusive, and sustainable.We are passionate about AI and developing new techniques that leverage traditional and non-traditional data to get the right decision in a clear and explainable way.Global...
-
Data Engineer
vor 2 Monaten
Sydney, Österreich Rich Data Co. VollzeitHybrid and flexible working arrangementsAbout RDC Rich Data Co (RDC) is delivering the future of credit today!We believe credit should be accessible, fair, inclusive, and sustainable.We are passionate about AI and developing new techniques that leverage traditional and non-traditional data to get the right decision in a clear and explainable way.Global...
-
Data Engineer
vor 1 Woche
Sydney, Österreich Teachers Mutual Bank Limited VollzeitAbout the role Join us on a 6-month contract as a Data Engineer.The Data Engineer will be responsible for the design, implementation, and support of the TMB Enterprise Data Warehouse and Analytics Solutions.This role includes collaborating with Business Analysts to create technical specifications and participating in all phases of data development and...
-
Data Engineer
vor 1 Woche
Sydney, Österreich Teachers Mutual Bank VollzeitAbout the role Join us on a 6-month contract as a Data Engineer. The Data Engineer will be responsible for the design, implementation, and support of the TMB Enterprise Data Warehouse and Analytics Solutions.This role includes collaborating with Business Analysts to create technical specifications and participating in all phases of data development and...
-
Staff Engineer
vor 1 Monat
Sydney, Österreich Commonwealth Bank Of Australia VollzeitDo work that mattersTechnology is responsible for the world leading application of technology and operations across every aspect of CommBank, from innovative product platforms for our customers to essential tools within our business.We also use technology to drive efficient and timely processing, an essential component of great customer service.At the core,...
-
Data Engineer
vor 11 Stunden
Sydney, Österreich Black.Ai VollzeitSafetyCulture is a customer and product-led SAAS company, our mission is to help working teams get better every day.Our technology platform and products give front line workers a voice and leaders the visibility to make smart decisions when driving safety, quality and continuous improvement.As part of the Data & Analytics team, you'll use your skills to help...
-
Efinancialcareers Ltd. | Staff Data Engineer
vor 1 Monat
Sydney, Österreich Tideri Jobbörse VollzeitStaff Data Engineer - Flink Commonwealth Bank of Australia Sydney, AustraliaDo work that matters:We're building tomorrow's bank today.This means we need creative and diverse engineers to help us redefine what customers expect from a bank.Envisioning new technologies that are still waiting to be invented and reimagining products that support our customers and...
-
Data Engineer
vor 11 Stunden
Sydney, Österreich Sourcefit VollzeitPosition Summary: We are seeking a highly skilled and experienced Data Engineer to join our team.The Data Engineer will be responsible for designing, developing, and maintaining scalable data pipelines and systems.This role is crucial for assembling complex data sets, improving internal processes through automation, and optimizing data infrastructure for...
-
Data Engineer
vor 1 Monat
Sydney, Österreich Paytech Nexus Ltd VollzeitProject scope: Data Engineering in Credit Risk EnvironmentBackground:We are working with one of the major card schemes and a Tier 1 Australian Bank, based in Sydney. This role is a 12-month contract (likely extending to 2 years) and is for an experienced Data Engineer / Analyst with experience in data modelling and credit risk.The Role:A self-sufficient Data...
-
Data Engineer
vor 1 Monat
Sydney, Österreich Paytech Nexus Ltd VollzeitProject scope: Data Engineering in Credit Risk EnvironmentBackground: We are working with one of the major card schemes and a Tier 1 Australian Bank, based in Sydney.This role is a 12-month contract (likely extending to 2 years) and is for an experienced Data Engineer / Analyst with experience in data modelling and credit risk.The Role: A self-sufficient...
-
Data Engineer
vor 4 Wochen
Sydney, Österreich Skybeam Technology VollzeitAbout the company: Skybeam Technology is an Australia based professional services company that design and deliver tailored workforce solutions to enable businesses to experience the benefits of managing a workforce in the evolving world.Our workforce solutions specialised in recruitment, sourcing services, contract management services, outsourced payroll...
-
Data Engineer
vor 2 Wochen
Sydney, Österreich Cuscal Limited VollzeitThis is a Data Engineer role with one of the leading companies in AU right now 🔥 -- Cuscal Limited -- with an amazing team. They are continuing to grow rapidly. This is the chance to join right as the 🚀 takes off.More About the Role at Cuscal Limited 🧐🧐## **Job Description** **2x Positions Available: We are looking for a Mid-level Data Engineer...
-
Data Engineering Consultant
vor 3 Wochen
Sydney, Österreich Data Army Pty Ltd VollzeitDatabase Development & Administration (Information & Communication Technology)Are you a Mid-Level Data Engineer seeking to advance your career with a dynamic team?Join us at Data Army in Sydney!As a professional with experience in the data engineering field, we understand you're ready for the next step. Data Army is seeking a skilled Mid-Level Data...
-
Staff Security Engineer
vor 3 Wochen
Sydney, Österreich Mseq Vollzeit**Staff Security Engineer**Full time1 viewsNewShare **About Kasada** Join us in stopping bad bots, for good! Kasada protects millions of online users everyday. Founded to stop automated bot attacks, we believe the internet should be a safe place for everyone. Bad bots are highly destructive. They take over accounts, steal content, overload systems and...
-
Data Engineer
vor 2 Monaten
Sydney, Österreich Finxl VollzeitData Engineer Summary: As a Data Engineer, you will work on site with clients to design, implement, and optimise scalable data pipelines and solutions. You'll leverage modern data engineering technologies such as AWS Glue, DBT, Redshift, and Lambda to transform Lion's data ecosystem and support its data-driven decision-making initiatives.Role...
-
Data Engineering Consultant
vor 3 Wochen
Sydney, Österreich Data Army Pty Ltd VollzeitDatabase Development & Administration (Information & Communication Technology)Are you a Mid-Level Data Engineer seeking to advance your career with a dynamic team? Join us at Data Army in Sydney! As a professional with experience in the data engineering field, we understand you're ready for the next step. Data Army is seeking a skilled Mid-Level Data...
-
Data Engineer
vor 3 Monaten
Sydney, Österreich Westpac Group VollzeitData Engineer Permanent Sydney, NSW Join Westpac's Customer Analytics and Insights team Purpose/Function of Team:The Data Engineer role is part of the Customer Analytics and Insights team within our Corporate & Customer Services (CCS) division. The CCS division supports common operations and customer services for Westpac Group, using customer insights, data,...
