Torchora
Back to Home

Key Responsibilities and Required Skills for a Joint Data Engineer

💰 $120,000 - $175,000

Data EngineeringBig DataCloud ComputingSoftware Engineering

🎯 Role Definition

The Joint Data Engineer is a hands-on builder and architect responsible for creating and managing the company's data infrastructure and pipelines. You'll work at the intersection of multiple teams—including data science, analytics, software engineering, and business stakeholders—to ensure data is accurate, available, and accessible. Your mission is to transform raw data into a curated, high-quality asset that drives business value and innovation across the organization. This "joint" function requires exceptional collaboration and communication skills, as you'll be the technical bridge that unites disparate teams around a common data truth.

📈 Career Progression

Typical Career Path

Entry Point From:

  • Junior Data Engineer
  • Business Intelligence (BI) Developer
  • Software Engineer (with a focus on backend or data)
  • Database Administrator (DBA)

Advancement To:

  • Senior or Lead Data Engineer
  • Data Architect
  • Staff Data Engineer
  • Manager, Data Engineering

Lateral Moves:

  • MLOps Engineer
  • Data Scientist
  • Analytics Engineer

Core Responsibilities

Primary Functions

  • Design, construct, and operationalize highly scalable, reliable, and fault-tolerant data pipelines for both batch and real-time data streams to support business-critical applications.
  • Develop, test, and maintain complex ETL and ELT processes to ingest and integrate data from a wide variety of sources, including internal databases, third-party APIs, and event streaming platforms.
  • Architect and implement robust data models within our cloud data warehouse and data lake environments, ensuring optimal performance and query efficiency for analytical workloads.
  • Build and implement comprehensive data quality frameworks and automated data validation procedures to ensure the accuracy, completeness, and consistency of our core datasets.
  • Collaborate closely with data scientists and analysts to deeply understand their data requirements, building curated data products that directly support their machine learning models and dashboards.
  • Manage and optimize our cloud-based data infrastructure (primarily on AWS, GCP, or Azure), focusing on cost-efficiency, scalability, security, and performance.
  • Write clean, maintainable, and well-documented Python and advanced SQL code for data transformation, workflow orchestration, and infrastructure management.
  • Implement and manage data workflow orchestration tools like Apache Airflow, Prefect, or Dagster to schedule, monitor, and troubleshoot intricate data dependencies.
  • Actively work with modern big data technologies such as Apache Spark, Kafka, or Flink to process large-scale datasets that exceed the capacity of traditional database systems.
  • Develop and champion Infrastructure as Code (IaC) using tools like Terraform or CloudFormation to automate the provisioning and management of all data platform resources.
  • Implement comprehensive monitoring, logging, and alerting systems for all data pipelines to ensure high availability and enable proactive issue resolution before they impact users.
  • Perform deep-dive root cause analysis into complex data-related issues, identify systemic problems, and engineer long-term solutions to prevent recurrence.
  • Partner directly with application and software engineering teams to influence upstream service design, ensuring data is produced in a way that is conducive to downstream analytics.
  • Champion and enforce data governance and security best practices, implementing data masking, PII handling, encryption, and fine-grained access controls to protect sensitive information.
  • Continuously evaluate, prototype, and recommend new data technologies and tools to enhance our data platform's capabilities, performance, and cost-effectiveness.
  • Create and maintain thorough documentation for data pipelines, data models, and architectural decisions to facilitate knowledge sharing and streamline team onboarding.
  • Systematically optimize database and data warehouse performance by fine-tuning queries, managing indexing strategies, defining partitioning keys, and configuring resource allocation.
  • Participate in a shared on-call rotation to respond to and resolve critical data infrastructure and pipeline failures, ensuring business continuity.
  • Mentor junior engineers and analysts on data engineering best practices, code review standards, and architectural principles, helping to elevate the team's overall skillset.
  • Translate ambiguous and complex business requirements into tangible technical specifications and architectural designs for new data solutions and platform features.
  • Lead the technical execution of migrating legacy data systems to modern, cloud-native data platforms, ensuring a smooth transition with minimal disruption to business operations.
  • Design, build, and manage cutting-edge data lakehouse architectures, combining the scalability of data lakes with the performance of data warehouses for maximum analytical flexibility.

Secondary Functions

  • Support ad-hoc data requests and exploratory data analysis to answer immediate and pressing business questions.
  • Contribute to the organization's overarching data strategy and technical roadmap by providing expert insights and forward-looking recommendations.
  • Collaborate with business units and product managers to translate ambiguous data needs into concrete engineering and analytical requirements.
  • Participate actively in sprint planning, daily stand-ups, and retrospective ceremonies within the agile data engineering team.
  • Develop internal tooling and automation scripts to improve the productivity, efficiency, and self-service capabilities of the entire data team.

Required Skills & Competencies

Hard Skills (Technical)

  • Advanced SQL: Deep proficiency in writing complex, highly-performant SQL queries, including window functions, CTEs, and query optimization across different database engines.
  • Python Programming: Strong programming skills in Python, especially with its data-centric ecosystem (e.g., Pandas, PySpark, SQLAlchemy, FastAPI).
  • Cloud Platform Expertise: Hands-on experience with at least one major cloud platform (AWS, GCP, or Azure) and its core data services (e.g., S3, Redshift, Glue; BigQuery, Dataflow; Synapse, Data Factory).
  • Data Pipeline Orchestration: Practical, production experience building and orchestrating complex workflows using tools like Apache Airflow, Prefect, or Dagster.
  • Big Data Technologies: Solid understanding and practical application of big data processing frameworks, particularly Apache Spark.
  • Data Modeling & Warehousing: In-depth knowledge of data warehousing concepts, including dimensional modeling (star/snowflake schemas) and modern data lakehouse principles (e.g., Delta Lake, Iceberg).
  • Database Systems: Experience working with both relational (e.g., PostgreSQL, MySQL) and NoSQL (e.g., MongoDB, DynamoDB) database systems.
  • Containerization & CI/CD: Familiarity with containerization using Docker, CI/CD best practices, and version control with Git.
  • Streaming Data: Exposure to or experience with real-time data streaming technologies such as Kafka, Kinesis, or Pub/Sub.
  • Infrastructure as Code (IaC): Working knowledge of tools like Terraform or CloudFormation for automating infrastructure deployment and management.

Soft Skills

  • Exceptional Collaboration: A natural ability to build strong relationships and work effectively across different teams, functions, and communication styles to achieve a common goal.
  • Clear Communication: The ability to clearly articulate complex technical concepts and trade-offs to both technical and non-technical stakeholders.
  • Pragmatic Problem-Solving: A knack for breaking down large, ambiguous problems into manageable, incremental steps and finding efficient, practical solutions.
  • High Degree of Ownership: A proactive mindset that takes full responsibility for the quality, reliability, and end-to-end success of the data products you build.
  • Innate Curiosity: A genuine passion for staying current with the evolving data technology landscape and a drive to continuously learn and improve.
  • Meticulous Attention to Detail: A sharp eye for detail, especially concerning data accuracy, pipeline integrity, and code quality.

Education & Experience

Educational Background

Minimum Education:

  • Bachelor's Degree in a relevant technical or quantitative field.

Preferred Education:

  • Master's Degree in a relevant field.

Relevant Fields of Study:

  • Computer Science
  • Software Engineering
  • Information Systems
  • Statistics or another quantitative discipline

Experience Requirements

Typical Experience Range:

  • 3-7 years of professional experience in a data engineering, backend software engineering, or related role.

Preferred:

  • A proven track record of designing, building, and deploying production-grade data pipelines in a cloud-native environment. Experience working within a fast-paced, agile product development team is a significant plus.
Explore More Careers