Systems Engineer - Big Data
Hudson River Trading (HRT) is looking for Systems Engineers to join our growing Research & Development team. This team builds and maintains an exceptionally large and growing distributed compute cluster, many-PB-scale storage systems, operating systems, automation software, and development tools.
We are seeking a highly skilled engineer with expertise in Linux administration and experience with Big Data technologies, particularly Kafka, Trino, and Elasticsearch. As a Big Data Systems Engineer, you will play a critical role in ensuring our data infrastructure is robust, scalable, and reliable. You will work closely with our engineering and research teams to design, deploy, and maintain systems that support our research and trading activities.
Responsibilities
- Trino Administration: Set up and manage Trino clusters for large-scale data analytics. Optimize and troubleshoot query performance and ensure data integrity across the system
- Kafka Administration: Design, deploy, and maintain Apache Kafka clusters. Implement and monitor Kafka-based data pipelines to ensure efficient and reliable data flow
- Elasticsearch Administration: Configure and maintain Elasticsearch clusters on managed compute platforms such as Kubernetes
- System Monitoring: Develop and implement monitoring solutions to ensure high system uptime and reliability; utilize tools to detect and resolve issues proactively
- Linux Administration: Maintain and manage a large-scale Linux-based infrastructure; ensure system stability, security, and performance through best practices in system administration
- Collaboration: Work closely with development and research teams to understand their requirements and provide robust data infrastructure solutions, participating in cross-functional projects as needed
- Documentation: Create and maintain comprehensive documentation for system architecture, processes, and best practices
Skills
- Bachelor's degree in computer science, information technology, or a related field
- Minimum of 3-5 years of experience in systems engineering, with a focus on Linux administration
- Experience with large scale, big data query engines and technologies such as Trino, Presto, Iceberg, and Hive
- Experience with operating and managing large scale Kafka clusters, including design, deployment, and capacity management
- Experience with operating and managing large scale Elasticsearch clusters
- Experience with container orchestration platforms and technologies such as Kubernetes, Terraform, and Helm
- Proficiency in scripting languages such as Python or Bash
- Strong problem-solving skills and the ability to work in a fast-paced environment
- Excellent communication and collaboration skills
Annual base salary range of $150,000 to $250,000. Pay (base and bonus) may vary depending on job-related skills and experience. A sign-on and discretionary performance bonus may be provided as part of the total compensation package, in addition to company-paid medical and/or other benefits.