Magic (magic.dev) Logo

Magic (magic.dev)

Software Engineer - Pretraining Data

Job Posted 2 Days Ago Reposted 2 Days Ago
In-Office
3 Locations
100K-550K
Mid level
In-Office
3 Locations
100K-550K
Mid level
The Software Engineer will create data pipelines and web crawlers for multimodal datasets and ensure data quality through distributed computing techniques.
The summary above was generated by AI

Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone. Our approach combines frontier-scale pre-training, domain-specific RL, ultra-long context, and inference-time compute to achieve this goal.

About the role: 

As a Software Engineer working on our pretraining data, you write efficient and robust pipelines for giant, multimodal datasets. You will develop and optimize web scraping techniques to harvest and maintain data at internet-scale.

What you might work on:
  • Design & implement multimodal (video, audio, text etc) web crawlers for scraping and indexing petabytes of data

  • Create large scale data processing pipelines using tools like Ray, Apache Spark, Apache Flink, Google BigQuery etc. 

  • Implement and scale deduplication techniques across modalities and apply heuristic and model-based techniques for parsing and filtering crawled data

  • Identify new data sources for inclusion in pre/post-training datasets

What we’re looking for:
  • Strong proficiency in distributed computing and parallel processing techniques

  • Obsession with details, reliability, and good testing to ensure data quality and integrity

  • Experience with designing and maintaining high-performance, scalable data architectures

  • Ability to design, develop and operate an LLM data pipeline from web scraping to data loading

Magic strives to be the place where high-potential individuals can do their best work. We value quick learning and grit just as much as skill and experience.

Our culture:
  • Integrity. Words and actions should be aligned

  • Hands-on. At Magic, everyone is building 

  • Teamwork. We move as one team, not N individuals

  • Focus. Safely deploy AGI. Everything else is noise

  • Quality. Magic should feel like magic

Compensation, benefits and perks (US):
  • Annual salary range: $100K - $550K

  • Equity is a significant part of total compensation, in addition to salary

  • 401(k) plan with 6% salary matching

  • Generous health, dental and vision insurance for you and your dependents

  • Unlimited paid time off

  • Visa sponsorship and relocation stipend to bring you to SF, if possible

  • A small, fast-paced, highly focused team

Top Skills

Apache Flink
Spark
Google Bigquery
Ray

Similar Jobs

21 Minutes Ago
Remote
Hybrid
2 Locations
150K-270K Annually
Senior level
150K-270K Annually
Senior level
Artificial Intelligence • Cloud • Mobile • Security • Software
Lead and mentor a full-stack engineering team to deliver AI-powered voice features while fostering a culture of rapid iteration and collaboration with product and design teams.
Top Skills: AIAndroidBackendiOSMobileVoip
4 Hours Ago
2 Locations
168K-240K Annually
Senior level
168K-240K Annually
Senior level
Aerospace • Artificial Intelligence • Hardware • Robotics • Security • Software • Defense
Develop and test software applications for military systems, focusing on automation, cloud integration, and quality assurance across multiple platforms.
Top Skills: Asp.NetAWSAzureC#Ci/CdDevOpsNoSQLPythonRestful ServicesSQL ServerWpf
4 Hours Ago
Easy Apply
Seattle, WA, USA
Easy Apply
230K-260K
Senior level
230K-260K
Senior level
Artificial Intelligence • Computer Vision • Machine Learning • Payments • Real Estate • PropTech
The Senior Manager will lead customer experience application development, align software efforts with customer needs, and foster a collaborative engineering culture.
Top Skills: AIComputer VisionData VisualizationDistributed SystemsSoftware Engineering

What you need to know about the Seattle Tech Scene

Home to tech titans like Microsoft and Amazon, Seattle punches far above its weight in innovation. But its surrounding mountains, sprinkled with world-famous hiking trails and climbing routes, make the city a destination for outdoorsy types as well. Established as a logging town before shifting to shipbuilding and logistics, the Emerald City is now known for its contributions to aerospace, software, biotech and cloud computing. And its status as a thriving tech ecosystem is attracting out-of-town companies looking to establish new tech and engineering hubs.

Key Facts About Seattle Tech

  • Number of Tech Workers: 287,000; 13% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Amazon, Microsoft, Meta, Google
  • Key Industries: Artificial intelligence, cloud computing, software, biotechnology, game development
  • Funding Landscape: $3.1 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Madrona, Fuse, Tola, Maveron
  • Research Centers and Universities: University of Washington, Seattle University, Seattle Pacific University, Allen Institute for Brain Science, Bill & Melinda Gates Foundation, Seattle Children’s Research Institute
By clicking Apply you agree to share your profile information with the hiring company.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account