Build and maintain production-grade batch and streaming data pipelines in Databricks using PySpark and SQL. Develop ingestion frameworks, data models, APIs, data quality checks, monitoring, and Medallion Architecture across bronze, silver, and gold layers. Integrate diverse data sources, optimize Spark workloads, document data lineage and architecture, and collaborate with engineering and federal stakeholders. An active Secret clearance or higher is required.
This is a remote position.
Location: Remote – United States
Security Clearance: Active Secret or higher REQUIRED
Our client, a growing technology services organization supporting federal government programs, is seeking an experienced Databricks Data Engineer to support a federal technology initiative.
This is a remote opportunity available to U.S. citizens residing in the United States. Candidates must currently hold an ACTIVE Secret security clearance or higher. Candidates without an active Secret-level clearance cannot be considered.
The ideal candidate will bring strong hands-on data engineering experience within Databricks, including development of production-grade batch and streaming pipelines, PySpark and SQL transformations, data modeling, ingestion frameworks, and modern data architecture practices.
- Design, build, and maintain batch and streaming data pipelines using PySpark, SQL, Databricks Workflows, and Delta Live Tables.
- Implement Medallion Architecture across bronze, silver, and gold data layers to support data quality, transformation, and consumption.
- Develop scalable ingestion frameworks for structured, semi-structured, and unstructured data.
- Integrate data from files, databases, APIs, and streaming sources such as Kafka, Kinesis, and Databricks Auto Loader.
- Design dimensional and domain-specific data models supporting analytics and downstream applications.
- Build and consume APIs for integration with downstream systems.
- Optimize Spark workloads for performance and cost through partitioning, caching, cluster sizing, and related techniques.
- Develop and implement data quality checks, validation processes, and pipeline monitoring.
- Maintain documentation covering data flows, lineage, architecture, and integration points.
- Collaborate with engineering, platform, architecture, and federal program stakeholders throughout the development lifecycle.
Requirements
- Active Secret security clearance or higher is required.
- 5+ years of professional data engineering experience.
- 2+ years of hands-on Databricks experience.
- Strong production-level experience with PySpark and SQL.
- Experience developing both batch and streaming data pipelines.
- Strong understanding of data modeling and modern data architecture principles.
- Proficiency with Python.
- Experience working within Git-based development and CI/CD environments.
- Familiarity with Databricks Unity Catalog, including catalogs, schemas, and permissions from a data engineering perspective.
- Experience integrating data from multiple source types, including databases, APIs, files, and streaming platforms.
- Experience with Databricks Delta Live Tables.
- Experience with Databricks Workflows.
- Experience with Kafka, Kinesis, or similar streaming technologies.
- Experience working within federal, regulated, or security-sensitive environments.
- Bachelor's degree in Computer Science, Engineering, Information Technology, or a related discipline.
Similar Jobs
Financial Services
Build and operate scalable Databricks-on-AWS data pipelines using PySpark, Delta Lake, and lakehouse patterns. Optimize performance, implement data quality, monitoring, alerting, and automated remediation, and deliver curated datasets for BI and analytics partners. Collaborate with stakeholders on architecture and design while applying secure software engineering, CI/CD, agile, and operational stability practices. The role also uses AI-assisted development tools and supports workforce data analytics.
Top Skills:
AlteryxAmazon AthenaAmazon EmrAmazon S3Apache AirflowApache IcebergSparkAutosysAWSAws CloudwatchAws GlueAws LambdaBitbucketClaudeDatabricksDatabricks WorkflowsDelta LakeDelta Live TablesGitGithub CopilotJavaJenkinsOracleParquetPysparkPythonScalaSigmaSpinnakerSQLTableau
Software
Designs, builds, and operates scalable batch and streaming data pipelines on Databricks for federal missions. Responsibilities include developing Spark and Delta Lake solutions, managing clusters and workflows, implementing Unity Catalog governance and security, optimizing ETL/ELT processes, integrating CI/CD, supporting machine learning and advanced analytics, monitoring data quality, and collaborating with technical teams and stakeholders.
Top Skills:
Amazon EmrSparkAWSAzureCi/CdDatabricksDelta LakeGgplot2GitGCPHadoopHiveKafkaMlflowNoSQLPlotlyPysparkPythonSeabornSpark SqlSpark Structured StreamingSQLUnity Catalog
Aerospace • Logistics • Security • Software • Cybersecurity
Designs and implements Databricks Unity Catalog access controls, security architecture, identity governance, data classification, and compliance automation. Leads RBAC and ABAC models, dynamic views, row- and column-level security, masking, service principal integration, and infrastructure-as-code deployments using Terraform or Databricks Asset Bundles. Establishes security guardrails, templates, tagging standards, and scalable workflows while partnering with data, platform, and workspace engineering teams.
Top Skills:
AbacColumn MaskingDatabricksDatabricks Asset BundlesDatabricks LakehouseDatabricks Unity CatalogDynamic ViewsEntra IdInfrastructure As CodePythonRbacRow-Level SecurityScimService PrincipalsSQLTerraform
What you need to know about the Seattle Tech Scene
Home to tech titans like Microsoft and Amazon, Seattle punches far above its weight in innovation. But its surrounding mountains, sprinkled with world-famous hiking trails and climbing routes, make the city a destination for outdoorsy types as well. Established as a logging town before shifting to shipbuilding and logistics, the Emerald City is now known for its contributions to aerospace, software, biotech and cloud computing. And its status as a thriving tech ecosystem is attracting out-of-town companies looking to establish new tech and engineering hubs.
Key Facts About Seattle Tech
- Number of Tech Workers: 287,000; 13% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Amazon, Microsoft, Meta, Google
- Key Industries: Artificial intelligence, cloud computing, software, biotechnology, game development
- Funding Landscape: $3.1 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Madrona, Fuse, Tola, Maveron
- Research Centers and Universities: University of Washington, Seattle University, Seattle Pacific University, Allen Institute for Brain Science, Bill & Melinda Gates Foundation, Seattle Children’s Research Institute
