Torchora
Back to Home

Key Responsibilities and Required Skills for a Fish Wrangler (Senior Data Engineer)

💰 $140,000 - $195,000

Data EngineeringTechnologyBig DataCloud ComputingAnalytics

🎯 Role Definition

As our lead Fish Wrangler, you are the chief architect and steward of our data ocean. You won't just be casting nets; you'll be designing the entire aquatic ecosystem, from the deep-sea data lakes to the fast-moving streams that feed our analytics and machine learning models. This role requires a hands-on technical leader who is passionate about building scalable, resilient, and elegant data solutions. You will be instrumental in wrangling our most complex data assets, ensuring they are clean, organized, and ready for consumption by stakeholders across the business. If you thrive on transforming chaotic, raw data into pristine, actionable insights, this is the challenge for you.

📈 Career Progression

Typical Career Path

Entry Point From:

  • Data Engineer
  • Software Engineer (with a data-intensive background)
  • ETL Developer
  • Business Intelligence Engineer

Advancement To:

  • Principal Data Engineer
  • Data Architect
  • Manager, Data Engineering
  • Head of Data Platform

Lateral Moves:

  • Senior Machine Learning Engineer
  • Senior Data Scientist

Core Responsibilities

Primary Functions

  • Design, construct, and maintain robust and scalable fish migration routes (ETL/ELT pipelines) to move data from various sources into our central reservoir.
  • Architect and manage our primary data holding tanks and reservoirs (data warehouses like Snowflake/Redshift/BigQuery and data lakes on S3/GCS).
  • Develop comprehensive data models and schema designs to classify and tag our data "species," ensuring clarity, consistency, and efficient querying.
  • Implement rigorous data quality checks and automated cleansing processes to ensure our data is healthy, pure, and free from contaminants (data integrity and governance).
  • Fine-tune and optimize the performance of data currents and channels, ensuring efficient flow and minimizing resource consumption on our cloud platforms (AWS, GCP, Azure).
  • Build and maintain the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and 'big data' technologies.
  • Assemble large, complex data sets that meet functional / non-functional business requirements and prepare them for analysis.
  • Monitor our real-time data streams (e.g., using Kafka, Kinesis) to capture critical events as they happen, akin to tracking spawning events in real-time.
  • Create and manage standardized docks for external fishing boats (API integrations) to ensure seamless data ingestion from third-party services and partners.
  • Develop automated habitat construction scripts (Infrastructure as Code, e.g., Terraform, CloudFormation) to provision and manage our data platform resources.
  • Implement robust monitoring, logging, and alerting systems to proactively detect and resolve issues within our data ecosystem, preventing pipeline failures or data corruption.
  • Work with stakeholders including the Executive, Product, Data, and Design teams to assist with data-related technical issues and support their data infrastructure needs.
  • Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability.
  • Create data tools and frameworks for analytics and data scientist team members that assist them in building and optimizing our product.
  • Document our data ecosystem, including data lineage, schema definitions, and pipeline logic, creating a 'field guide' for all data consumers.
  • Ensure the security and privacy of our data by implementing access controls, encryption, and data masking techniques in line with industry best practices.
  • Evaluate and recommend new data management technologies and software engineering practices to constantly improve the capabilities of our data platform.
  • Lead the design and implementation of data storage and processing solutions for both structured and unstructured data.
  • Collaborate with data scientists to productionize machine learning models, building the pipelines necessary to feed them data and serve their predictions.
  • Troubleshoot complex data issues and pipeline failures, performing root cause analysis and implementing long-term solutions.
  • Drive the adoption of data engineering best practices and coding standards within the team and broader engineering organization.
  • Manage the full lifecycle of our data assets, from ingestion and processing to archival and deletion, ensuring efficient resource management.

Secondary Functions

  • Support ad-hoc data requests and exploratory data analysis.
  • Contribute to the organization's data strategy and roadmap.
  • Collaborate with business units to translate data needs into engineering requirements.
  • Participate in sprint planning, retrospectives, and other agile ceremonies within the data engineering team.
  • Provide technical mentorship and guidance to junior 'Anglers' (Data Engineers) and analysts on the team.

Required Skills & Competencies

Hard Skills (Technical)

  • Advanced Sonar & Deep-Sea Mapping (SQL): Expert-level proficiency in SQL for complex data querying, aggregation, and manipulation across large-scale relational and columnar databases.
  • Custom Net Weaving (Python/Scala/Java): Mastery of at least one high-level programming language for data engineering, such as Python (with Pandas, PySpark), Scala, or Java.
  • Ocean-Sized Data Fleet Management (Big Data Tech): Proven experience wrangling massive datasets with distributed processing frameworks like Apache Spark, Flink, or the Hadoop ecosystem.
  • Aquatic Environment Management (Cloud Platforms): Deep, hands-on expertise with a major cloud provider (AWS, GCP, or Azure) and their core data services (e.g., AWS Glue, S3, Redshift, Kinesis; GCP BigQuery, Dataflow, Pub/Sub).
  • Tidal Current Scheduling (Orchestration Tools): Proficiency in building, scheduling, and monitoring complex workflows using tools like Apache Airflow, Prefect, or Dagster.
  • Portable Fish Tank Design (Containerization): Experience with containerization technologies like Docker and orchestration systems like Kubernetes for deploying and scaling data applications.
  • Automated Habitat Construction (IaC): Solid understanding of Infrastructure as Code principles and tools such as Terraform or CloudFormation to manage data infrastructure.
  • Data Reservoir Architecture (Warehousing/Lakehouse): In-depth knowledge of modern data warehousing and lakehouse concepts, with hands-on experience in platforms like Snowflake, Databricks, Redshift, or BigQuery.
  • Real-Time Stream Monitoring (Streaming Tech): Experience with real-time data streaming technologies such as Kafka, Kinesis, or Pub/Sub.
  • Version Controlled Blueprints (Git): Strong proficiency with Git for version control, collaboration, and CI/CD practices.
  • Systematic Species Tagging (Data Modeling): Strong skills in data modeling, including dimensional modeling (star/snowflake schemas) and understanding of data normalization.

Soft Skills

  • Analytical & Problem-Solving Mindset: The ability to dive deep, diagnose blockages in data streams, and identify the root cause of a 'sick' data school.
  • Clear Communication: Can fluently translate complex technical concepts to both technical and non-technical audiences, ensuring everyone understands the currents and tides of our data.
  • Strong Collaboration: Works effectively in a team environment, partnering with data scientists, analysts, and software engineers to build a world-class data platform.
  • Ownership & Accountability: Takes pride in building and maintaining high-quality, resilient data systems, and feels a sense of ownership over the data's entire lifecycle.
  • Mentorship & Leadership: Eager to share knowledge, guide junior team members, and elevate the technical skills of the entire team.
  • Pragmatism: Balances the drive for technical excellence with the practical needs and timelines of the business.

Education & Experience

Educational Background

Minimum Education:

  • Bachelor's Degree in Computer Science, Engineering, Information Systems, or another quantitative field.

Preferred Education:

  • Master's Degree in Computer Science, Data Science, or a related discipline.

Relevant Fields of Study:

  • Computer Science
  • Software Engineering
  • Statistics
  • Mathematics

Experience Requirements

Typical Experience Range:

  • 5+ years of hands-on professional experience in a data engineering, ETL development, or software engineering role with a focus on data.

Preferred:

  • 7+ years of experience, including technical leadership or mentorship responsibilities, and a proven track record of architecting, building, and deploying large-scale data solutions in a cloud environment from the ground up.
Explore More Careers