In a previous post, How We Run Terraform At Scale, we shared how Benchling moved from a painful, laptop-driven, apply-then-merge workflow to an automated infrastructure release system built on Terraform Cloud (TFC). That migration dramatically reduced conflicts and drift, saved an estimated 8,000 developer hours annually, and gave us the foundation to scale from 350 to 625 workspaces managing many thousands of cloud resources across 38 AWS accounts. We also walked through the ecosystem of custom tooling we built around TFC to make it work at our scale: a CLI for bulk review and approval,…
Reaching the limit of manual platform developmentBenchling’s platform handles diverse scientific data, such as DNA sequences, antibodies, notebook entries, inventory containers, workflow runs, and dozens more. Each object type carries unique domain logic: how it is validated, what relationships it holds, and what actions users can perform on it.As Benchling matured, capabilities were added that customers expected to work across all these objects including REST APIs for integration, a data warehouse for analytics, search indexing, and configuration migration tools for moving setups between…
Tactical Reflections and Self-Assessment for Your Journey from Team Management to Organizational LeadershipAre you an individual contributor (IC) who’s just transitioned into an Engineering Manager and trying to find your path to the next level? Or are you a seasoned Engineering Manager (EM) figuring out how to break through to a Senior Engineering Manager role? Read on!Most companies have a career matrix that calls out the expectations for moving from Engineering Manager (M1) to Senior Engineering Manager (M2) also known as Manager of Managers (MoM). In this blog, I’ll share reflections and…
The Multi-Model Playbook: Patterns in Agentic EngineeringBuilding production AI systems that work reliably across multiple model providers requires more than just swapping API keys. Over the past year, working on AI agents like the Data Entry Agent and Compose Agent at Benchling, I’ve learned that successful multi-provider strategies come down to understanding what’s universal versus what’s provider-specific, and designing around those constraints. The clearest revelation here was that the architectural principles underlying reliable software — modularity, separation of concerns, clear…
Managing over 165k cloud resources across hundreds of workspaces could seem daunting. But for us, it’s just another day at Benchling. Here’s how we do it.We currently have:165k cloud resources under management625 Terraform workspaces38 AWS accounts170 engineers (40 of whom are infra specialists)We perform:225 infrastructure releases daily (terraform apply operations)723 plans daily (terraform plan operations)We’ve been successfully operating Benchling’s infrastructure release system for the past two years (spoiler, it’s Terraform Cloud), over which time we’ve doubled our infrastructure…
BackgroundAt Benchling we run cloud infrastructure across several regions and environments. To coordinate and manage this complexity, our team operates a self-hosted implementation of Terraform Cloud, managing around 160,000 terraform resources across five data centers. About 50 engineers from across the engineering org release some form of infrastructure change within a given month — some are infrastructure specialists, and others are application engineers who are completely new to Terraform Cloud.Understandably, we get a lot of questions about how to use Terraform Cloud or how to debug a…
Benchling is a unified platform for scientific data. It allows scientists to collaborate on complex science, automate work, and power AI. Customers store large volumes of data on our platform, leveraging it across many applications both within Benchling and in their own infrastructure. It’s critical that customer data is accessible in a performant and scalable way.In this article, we’ll explore a recent shift in how we store and retrieve customer data. By migrating to a more compact structure, we’ve tackled key performance challenges associated with increased data volumes. This transition has…
Authors: Wonja Fairbrother and Eli LevineScience is iterative. To design the next experiment, scientists need to analyze the results of previous ones. Interactive Analysis in Benchling allows scientists to perform real-time data transformation, visualization, and analysis without having to transfer it into other systems. In this post we will describe the architecture behind interactive analysis capabilities in Benchling and give a peek into the decision journey we took along the way¹.Interactive Analysis allows scientists to:1. Select data from many sources:Benchling entity and results…
On a few occasions, we’ve needed to debug POSIX signals (SIGINT, SIGTERM, etc.). Inevitably, there’s a shell involved too. One day, we were debugging some weird interaction between signals, shells, and containers and found ourselves bamboozled by some behaviors. People who consider themselves knowledgeable about Linux have found some of the details of our investigation surprising, so read on if this sort of thing doesn’t make you want to defenestrate your laptop and become an alpaca-farming hermit.The scene of the crimeAt Benchling, we have a pretty standard testing/continuous integration…
It’s ideal to get feedback on your code faster — to make a code change and see the result instantly. But, as projects get larger, reload times get longer. Each incremental dependency or bootstrap code block that adds 200ms feels worth it, but 50 of them later and it takes 10 seconds to see the result of a code change.On the Build team at Benchling, that’s where we found ourselves one day. We used 146 packages which pull in 128 transitive dependencies for a total of 274 packages. We also spent a lot of time waiting for SQLAlchemy models to initialize. The result is our test harness took 10…
Read at the source
Your visit, your choice.
Optional Google Analytics helps us understand visits. Microsoft Clarity records masked interactions to improve the site. Optional tools stay off unless you choose them. Privacy details.