Preparing for your next Quant Interview?
Practice Here!
OpenQuant
2023-09-29

Big Data Infrastructure Engineer

logo
G-Research
Big Data Infrastructure Engineer
Texas, US
Apply Now
Job Description

This is a hybrid role, based in G-Research’s office in Dallas.

G-Research is Europe’s leading quantitative finance research firm. We hire the brightest minds in the world to tackle some of the biggest questions in finance. We pair this expertise with machine learning, big data, and some of the most advanced technology available to predict movements in financial markets.

Opened in 2022, the Dallas office is a key infrastructure hub where we work on the latest technologies in a cutting-edge environment.

The role

The Big Data Platform Engineering team is expanding and is looking for a DevOps focused Infrastructure Engineer with demonstrable experience in HDFS, YARN and the Hadoop ecosystem.

The team manages a significant distributed HDFS storage footprint and have also started to explore Apache Ozone deployment on Kubernetes. Recent Airflow adoption and support is an exciting growth area for the team, while there is also an overlap in technology support within the team, in products such as Trino, Spark, Grafana, Hive, Nifi and HBase.

You will help us deliver automation, feature enhancements and best practice security on our existing hybrid cloud platforms, as well as explore new technologies and integration with other Open Source products within our ecosystem.

There is significant focus on configuration management, container orchestration, Infrastructure as Code (IaC) and CI/CD. In return, you will gain exposure to the latest hardware and software technologies in a forward- thinking company, which values innovation, personal development and training.

Key responsibilities of the role include:

  • Developing, deploying and maintaining big data solutions within the Hadoop ecosystem, such as HDFS, YARN and Ozone
  • Deploying and maintaining Airflow
  • Delivering engineering solutions ensuring they are scalable, stable, performant, and operationally efficient
  • Providing metrics, documentation, and self-service infrastructure to help our customers work at pace and get the most out of the platforms
  • Proactively identifying and addressing system performance bottlenecks, security vulnerabilities, and areas for improvement to ensure systems are stable and robust
  • Participating in the out of hours rota, providing support and maintenance for critical systems across the BigData ecosystem

Who are we looking for?

Whatever your technical background, good coding ability is essential for this role.

The ideal candidate will have the following skills and experience:

  • Strong HDFS, YARN and Hadoop ecosystem knowledge, preferably on the Cloudera Distribution
  • Good coding skills, preferably Python
  • Experience working with CI\CD pipelines, such as Jenkins and ArgoCD
  • Experience working with Configuration Management tools, such as Ansible and Terraform
  • Strong Linux, Hardware and Networking knowledge
  • Experience deploying and running applications on Docker and Kubernetes, including the creation of Helm charts
  • Experience working with distributed systems

Highly desirable but not essential:

  • Experience working with Ozone
  • Experience working with other Big Data technologies, such as Spark, Trino, Hive, Airflow orYARN
  • Data Engineering experience, such as Spark application development and SQL
  • Team Lead experience

Why should you apply?

  • Market-leading compensation plus annual discretionary bonus

  • Informal dress code and excellent work/life balance

  • Excellent paid time off allowance

  • Sick days, military leave, and family and medical leave

  • Generous 401(k) plan

  • 12-weeks’ fully paid parental leave

  • Medical and Prescription, Dental, and Vision insurance

  • Life and Accidental Death & Dismemberment (AD&D) insurance

  • Employee Assistance and Wellness programs

  • Generous relocation allowance and support

  • Great selection of office snacks, and hot and cold drinks

  • On-site gym and car park

G-Research is committed to cultivating and preserving an inclusive work environment. We are an ideas-driven business and we place great value on diversity of experience and opinions.

We want to ensure that applicants receive a recruitment experience that enables them to perform at their best. If you have a disability or special need that requires accommodation please let us know in the relevant section

Share this job
Share On
Apply Now