Design, develop, and optimize scalable batch and real-time data pipelines on GCP. Implement Medallion Architecture, build Python and PySpark transformations, optimize BigQuery and SQL workloads, deploy Dataflow pipelines, and orchestrate workflows with Cloud Composer and Airflow. Manage GCS storage, CI/CD, version control, IAM, data governance, security, reliability, scalability, and cost efficiency across enterprise data platforms.
This is a remote position.
KANINI is seeking a highly skilled Senior Data Engineer with deep expertise in Google Cloud Platform (GCP) and modern data architecture. The ideal candidate will have hands-on experience designing scalable data pipelines, implementing Medallion Architecture, and building robust enterprise-grade data solutions.
This role requires strong technical proficiency in BigQuery, PySpark, Dataflow, and Airflow, along with a solid understanding of cloud data governance, performance optimization, and CI/CD practices.
Key Responsibilities
- Design, develop, and maintain scalable batch and real-time data pipelines on GCP
- Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) for data processing
- Build high-performance data transformations using Python and PySpark
- Develop and optimize complex SQL queries for analytical workloads
- Work extensively with BigQuery for large-scale data processing and performance tuning
- Develop and deploy pipelines using Cloud Dataflow
- Orchestrate workflows using Cloud Composer (Apache Airflow)
- Manage data storage and lifecycle using Google Cloud Storage (GCS)
- Implement version control and CI/CD pipelines using Git-based tools
- Ensure data security, governance, and access control using GCP IAM
- Optimize data solutions for performance, scalability, reliability, and cost-efficiency
Required Skills & Experience
- Strong hands-on experience with Google Cloud Platform (GCP)
- Expertise in BigQuery (partitioning, clustering, query optimization)
- Proven experience implementing Medallion Data Architecture
- Strong programming skills in Python and PySpark
- Hands-on exposure on Java
- Advanced proficiency in SQL (complex joins, window functions, performance tuning)
- Hands-on experience with Cloud Dataflow
- Experience with Cloud Composer (Airflow) for orchestration
- Experience working with Google Cloud Storage (GCS)
- Knowledge of version control systems (Git) and CI/CD practices
- Strong understanding of GCP IAM and cloud security best practices
Preferred Qualifications
- Experience working with large-scale enterprise data platforms
- Knowledge of data warehousing and data lake concepts
- Familiarity with real-time streaming frameworks
- Experience in data governance and data quality frameworks
- Exposure to
Similar Jobs
Information Technology • Productivity • Software • Infrastructure as a Service (IaaS)
Leads and grows a distributed software engineering team responsible for developing and maintaining NinjaOne SaaS products. Provides technical direction, participates in architecture, code reviews, integrations, and deployments, and partners with product, platform, and project management teams. Oversees engineering operations, team mentoring, software quality, continuous improvement, and successful product delivery across UI, mobile, server, and agent systems.
Top Skills:
APIsAWSC++Caching SystemsCi/CdGoJavaKey-Value DatabasesKotlinPostgres
Cloud • Security • Software • Cybersecurity • Automation
Own the vision and roadmap for GitLab Dedicated’s Switchboard control plane. Partner with engineering and site reliability teams to improve infrastructure operations, upgrades, maintenance, customer configuration, networking, authentication, health visibility, migrations, and incident management. Engage directly with customers, simplify complex technical workflows, prioritize across user experience and operational efficiency, and guide product decisions in an evolving problem space.
Top Skills:
Cloud Management ConsolesDevsecopsGitlabGitlab DedicatedPrivate NetworkingSite Reliability Engineering
Big Data • Fintech • Mobile • Payments • Financial Services
Lead Affirm’s zero-to-one product initiatives in partnership with the CEO and executive team. Own product vision, strategy, roadmap, discovery, prototyping, experimentation, launch, and iteration for new products and platforms. Analyze customer behavior, define success metrics, write product content, and coordinate engineering, design, data, risk, operations, revenue, legal, and compliance teams. This hands-on role has no direct people-management responsibilities and requires strong judgment in ambiguous environments.
What you need to know about the Seattle Tech Scene
Home to tech titans like Microsoft and Amazon, Seattle punches far above its weight in innovation. But its surrounding mountains, sprinkled with world-famous hiking trails and climbing routes, make the city a destination for outdoorsy types as well. Established as a logging town before shifting to shipbuilding and logistics, the Emerald City is now known for its contributions to aerospace, software, biotech and cloud computing. And its status as a thriving tech ecosystem is attracting out-of-town companies looking to establish new tech and engineering hubs.
Key Facts About Seattle Tech
- Number of Tech Workers: 287,000; 13% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Amazon, Microsoft, Meta, Google
- Key Industries: Artificial intelligence, cloud computing, software, biotechnology, game development
- Funding Landscape: $3.1 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Madrona, Fuse, Tola, Maveron
- Research Centers and Universities: University of Washington, Seattle University, Seattle Pacific University, Allen Institute for Brain Science, Bill & Melinda Gates Foundation, Seattle Children’s Research Institute
