LakeForestRecruiter Since 2001
the smart solution for Lake Forest jobs

Senior Software Engineer - Big Data/Data Lake

Company: AbleForce, Inc.
Location: Lake Forest
Posted on: November 19, 2021

Job Description:

Please, no third parties. Permanent residents only. Although local candidates are preferred, fully remote candidates will be considered. Main Duties & Responsibilities: - Play an integral part of a team that creates and supports a highly scalable data platform services and data lake for location-based analytics. - Build the automation and tooling to support retrieval and transformation of tens of thousands of externally sourced datasets. - Collaborate with highly qualified and accomplished data and software engineers to build, enhance, and maintain our data platform that supports best-in-class products. - Develop the delivery of data platform supporting ingestion of over ten thousand datasets, stores over a petabyte of spatial, relational and raster data. - Support services serving over five billion API calls per month. - Work in an Agile/Scrum environment to deliver solutions and services for data platform to support delivery of large-scale data lake. - Participate in architecture, design and development reviews for data platform services leveraging best available big data tools and technologies such as Presto, Spark, Alluxio, etc. - Engage with cross organizational teams collaborating on data ingestion services and data engineering to develop single consistent series of services and solutions. - Work with evolving requirements based on results of teams continuing investigation, development and customer feedback. - Adhere to best practices around source code versioning, automated testing, and dependency management. - Investigate and resolve technical and non-technical issues, resolving critical incidents in timely manner and with a through root cause analysis. - Contribute to company's overall technology strategy and roadmap as an active member of its architectural leadership team. Skills & Requirements: - 7+ years of experience in software engineering. - 3+ years of experience with Big Data systems and cloud architecture. - Experience with distributed data processing systems (Spark, Dask, etc.). - Experience with distributed data storage systems (Parquet, HDFS, etc.). - Experience with low-latency data lake query architectures (Alluxio, etc.) and real-time streaming systems (Kafka, etc.) - Data lake design strategies for metadata, ontology, governance, authorization, etc. - Test automation for data quality, data flow, and API endpoints. - Data engineering techniques for big data, including data automation frameworks (such as Airflow or Prefect), metadata management (such as Amundsen) and process management strategies. - Infrastructure management and automation experience (Kubernetes, Terraform, Chef, etc.). - Cloud infrastructure management, ideally with experience in AWS, including both technical aspects, such as solutions architecture, and non-technical aspects, such as financial planning. - Modern practices around agile development, release management, continuous integration, system reliability, etc. - BS/BA in in Computer Science preferred. tags: senior data engineer, senior database engineer, senior big data engineer, senior data lake engineer, senior database developer, senior software engineer

Keywords: AbleForce, Inc., Lake Forest , Senior Software Engineer - Big Data/Data Lake, IT / Software / Systems , Lake Forest, California

Click here to apply!

Didn't find what you're looking for? Search again!

I'm looking for
in category
within


Log In or Create An Account

Get the latest California jobs by following @recnetCA on Twitter!

Lake Forest RSS job feeds