In a wildly diverse industry that touches just about every facet of life and business, tech debt is an enduring challenge for all developers at all companies. But although everyone experiences the headaches associated with rapid deployment and scale, it’s impossible to point to any single mitigation method as a catch-all solution.
To demonstrate the huge spectrum of solutions out there — and help any engineers who might be stuck in their own tech debt doldrums — we spoke with technical leaders at six different Seattle-area technology companies about how they approach these coding hangovers.
People thinking about undergoing various cosmetic treatments and surgical procedures turn to RealSelf’s all-in-one information resource and marketplace. The company’s online forums offer feedback and reviews for treatments and more than 20,000 vetted doctors and healthcare professionals who supply them. With that information, users can book their treatments through RealSelf’s portal with more confidence.
VP of Engineering Anthony Mendonca acknowledges that tech debt is a fact of life in any engineering organization, and its prudent allowance can actually increase agility over time.
What is technical debt and how does your team define it?
We recognize that every line of code our team writes has a half-life attached the minute it’s committed to our codebase. We define technical debt as anything introduced to our code base or processes that significantly reduces the agility of the team. This includes more than just lines of code, but also our deployment process or how the team triages and prioritizes bugs. It is a pretty broad definition, which intentionally leaves room for conversation about impact. Ultimately, technical debt for us is a useful analogy to have a discussion about short and long-term risk to the business.
When technical debt does occur, what process does your team use to measure and manage it?
We understand that not all technical debt is bad and that it’s a natural bi-product of software development. We instead focus on understanding the impact to team agility — and how prudent debt can actually temporarily increase team agility if executed properly. A/B testing and early release prototyping help us get to market sooner, gain consumer insights and ensure we're solving the correct problem. Iterating along the way ensures we can fine tune the right solution, with less technical debt in the long run. However, if we ignore the accrual of reckless or inadvertent debt, we will encumber ourselves with maintenance and decrease flexibility of our codebase over time. Ultimately, we do our best to create a culture of continuous improvement and empower our teams to do what’s right while acknowledging that not all debt is created equal.
When working on a problem we aren’t yet confident is valuable to our consumers, we rapidly iterate through a series of A/B tests.”
What proactive measures does your team take in minimizing technical debt?
Our iterative development process focuses less on minimizing technical debt, and more on understanding the impact on balancing short and long-term results for our customers. In order to do this sustainably, we must balance time to market, value creation and the end-to-end quality of the consumer experience among other things. We’ve adopted a framework to help set expectations across product, design and engineering so we know what trade-offs we’re making on any given project.
When working on a problem we aren’t yet confident is valuable to our consumers, we rapidly iterate through a series of A/B tests. The longer we invest in building, the more we risk creating something for a problem that doesn’t even exist. In these cases the team is empowered to accept prudent technical debt to reach consumers as quickly as possible.
Finally, once we have high confidence in our technical solution, and have validated it with customer insights, we schedule the work to pay down any technical debt that will adversely affect agility over the long-term.
Going through cancer treatment is hard enough without the interminable bureaucracies surrounding our healthcare systems. Navigating Cancer’s patient relationship management solution is designed to manage communication and data exchange between oncology doctors and patients, and the company recently raised $26 million in funding to build out its team and add new features.
Site Reliability Engineering Team Lead Arthur Kepler encourages his developers to own specific projects, which encourages them to monitor and proactively take care of technical debt.
What is technical debt and how does your team define it? Please provide a specific example.
Technical debt is a consequence of all software development, just as dirty dishes happen when making a meal. Practices to ensure periodic “clean-ups” and code reviews are essential to manage technical debt before it obstructs change and undermines morale. Technical debt can also exist in more than one variety, like defects internal to a codebase or deficiencies around maintaining a product within its ecosystem of services.
The internal variety may be due to rapid development of an MVP that needs refinements later on, or code produced without test coverage. There are times that rapid delivery outweighs maintainability, but often maintainability should be a primary concern with code that powers a business service — especially when considering factors like employee turnover.
Ecosystem defects are often related to the planning and architecture at the outset of a project. For example, at more than one company I have inherited a data ETL service home-brewed by former staff in a non-concurrent, dynamically typed language which did the job for small sets of data. When a new customer asked us to load 40 times more data, the ETL process would have taken 27 hours per day. This is an example of technical debt which had to be paid by changing underlying tools and languages, which is not a small ask to the business, despite the dividends it ultimately paid.
When technical debt does occur, what process does your team use to measure and manage it?
Technical debt is a challenging phenomenon to measure, as real-world software is complex by nature. As with financial debt, technical debt compounds over time. Additions of new business logic and handling special cases can exponentially add complexity to a codebase. The most immediate way to identify technical debt is to survey the developers working in a codebase. The approach to quantifying technical debt therefore should begin with developers continuously identifying technical debt into stories, labelled as “technical debt”. By tracking these stories at the rate they are “accrued” and “paid off”, they can give an indication of progress.
Another method to analyze “hotspots” of technical debt is examining stories and tasks that far exceed their estimate at the outset of a project. This should be an important part of retrospectives closing each sprint. Finally, a more detailed approach is to examine the version control history for changes that span numerous concerns and layers of a product — an indication of tight coupling and rigid complexity. Technical debt analysis tools such as Code Maat, Codescene, and SonarQube use similar methodologies.
Ownership of a product or service codebase is very important, as the owners will be empowered to guard against debt and decay of a code base, especially when many developers are contributing. Adequate planning, architecture, and code reviews are essential to this process, as well as ensuring test quality. If you’re at an organization with an SRE team, involve them in your code reviews and testing processes — they will thank you for it, as will customers for the improved reliability.
Trust your developers, listen to their concerns and encourage blameless retrospectives...”
What proactive measures does your team take in minimizing technical debt?
Service-oriented architecture is my top recommendation for enhancing maintainability of a software project. When done properly, SOA’s loose coupling of components provides the ability to empower change affecting the smallest necessary scope. Testing is the other lynchpin of empowering fearless change. Untested code is dangerous to change; there is no way to ensure regressions haven’t been introduced.
Finally, the best source of insight on technical debt are those who work closest to the codebase. Trust your developers, listen to their concerns and encourage blameless retrospectives for identifying proposals for maintenance projects. Discuss what debt was added and what has been paid off over the previous sprint — and keep track of it rather than letting it become a hidden trap later.
Postmates connects millions of users with its network of couriers, who deliver goods and meals from stores and restaurants around the country. Postmates is also working on an autonomous delivery robot that can safely navigate city sidewalks. The company is headquartered in San Francisco — with a sizeable PNW office in Bellevue — and has raised more than $900 million since it was founded in 2011.
Engineering Manager Chen Yang said his team is always careful to document tech debt, and devotes at least 10 percent of engineering time clearing it.
What is technical debt and how does your team define it?
In my team, technical debt is defined as something that we should do but didn't do. It can be planned or unplanned. For example, we used to have a few tests that need to be run manually for each production deployment. Automating these would require non-trivial effort — requiring a new test platform and so on — while running manually took five to 10 minutes. The same applies to services or code that are unused or has become obsolete over time.
When technical debt does occur, what process does your team use to measure and manage it?
The first step is to document it. When we add tech debt to our backlog, we also try to estimate impact. It can be the extra effort that will accumulate for the next six months. In the example I mentioned earlier, it will be multiplication of 10 minutes with the number of deployments in six months. The next step is to prioritize this together with everything else during sprint planning. We usually make sure there’s at least some tech debt work in each sprint.
Postmates has a very strong infrastructure team that promotes good practices and provides common tools across the company...”
What proactive measures does your team take in minimizing technical debt?
We enable static analysis tools like SonarQube to run automatically for new commit and inspect existing code base from time to time, on top of all the test coverage tools. During our quarterly planning, we also set aside at least 10 percent of our time to work on tech debt items. Our backlog is one source of this work. We do routine house cleaning, like keeping our software version up to date, aggressively depreciating obsolete software or hardware and auditing all errors in the logs. Postmates has a very strong infrastructure team that promotes good practices and provides common tools across the company, but every team does its due diligence to keep the tech debt minimal.
DreamBox Learning’s advanced edtech software starts from the assumption that math education isn’t a one-size-fits-all field — each individual student learns in his or her own way. The company’s software — currently in use by more than 3 million students across the North American continent — measures tens of thousands of data points for each student, and adapts in real-time to suit the needs of each child.
VP of Engineering Lorenzo Pasqualis said tech debt is inevitable, but his team’s goal is to add to it with “eyes wide open” in terms of the clearing work it will require later on.
What is technical debt and how does your team define it?
For my team, technical debt is necessary work that has been put on the back burner in order to achieve some important goal expeditiously. Some examples might include known non-critical bugs; shortcut implementations that need to be revisited; messy code that needs refactoring; missing error conditions handling; missing automated/unit tests; simplistic logic; and unoptimized algorithms.
When technical debt does occur, what process does your team use to measure and manage it?
Technical debt is part of life — the goal is not to have zero technical debt. Instead, the goal is to have a handle on what the debt is, and a plan to pay it down in one way or another. In my teams, we work using a scrum process, and we use Jira to manage backlogs. Technical debt work items are filed as Jira stories, and they get prioritized along with sprint work. Our technical leads are responsible for scheduling technical debt reduction work items in the sprints, and negotiating with the business product owners sprint-by-sprint.
...technical debt ... is a compromise we make with eyes wide open...”
What proactive measures does your team take in minimizing technical debt?
We try to not take too many shortcuts, and we have a dedicated QA function that helps us decide what the quality bar of our code and product needs to be. Deciding to put some items on the back burner to add to the technical debt pile is a compromise we make with eyes wide open, knowing exactly the risk we are taking and the issues that we are going to cause in the future if we do not address the debt.
Installing any kind of underground infrastructure — be it utilities, fiber optics, telecommunications, gas lines, or water pipes — usually involves massive disruption for the communities they serve. To limit the impact of installation and maintenance, Kent-based Digital Control has created one of the world’s most advanced devices — dubbed “Falcon” — for locating and directing underground horizontal directional drill heads.
Director of Engineering Chris Cavage outlined the multiple ways in which tech debt accrues, and his team’s collaborative approach to clearing it.
What is technical debt and how does your team define it?
Technical debt can be anything from a bad design decision early in the process to a drastic change in product requirements late in the game, causing a need to refactor code without the allotted time by management to go in and fix it. You know it when you see it, and everyone on the team starts to refer to it as “that code.” An example of this was when we rolled out our last Falcon Locator, and the protocols to the microcontrollers weren’t plumbed in correctly because of a desire to save time by reusing older protocols — which ultimately cost us time in patchwork programming until we finally just rewrote the entire section.
When technical debt does occur, what process does your team use to measure and manage it?
If the technical debt is small enough that a developer can just knock it out during his or her day-to-day sprint tasks without schedule impact, we’ll usually just advocate for that — assuming regression tests are in place and as separate commits, of course. Otherwise, larger chunks of technical debt usually find their way to our Jira boards and get triaged during our typical backlog refinement meetings. Most team-members will have a sense of tech debt’s magnitude to properly assess the story or task and address it in the next sprint or so.
We do what we can upfront to avoid technical debt and the engineering time it takes to address it.”
What proactive measures does your team take in minimizing technical debt?
We do what we can upfront to avoid technical debt and the engineering time it takes to address it. Our most successful methods have included integrated code reviews with our commit process and proper software whiteboarding design sessions before starting major feature work. We’ve been pretty successful leveraging Perforce Helix Swarm and piping code reviews into our source control process on adopting projects.
Ballard-based Pluto builds technology that allows geographically dispersed users to hang out in virtual reality. With an alpha now available on Steam, users can see digital renderings of one another’s faces and hands — the most crucial tools for in-person communication — and hear their voices. The startup has raised north of $13 million since it was founded in 2015, and is currently refining its product for a rollout to the masses.
VP of Engineering Andy Piro said the reduced speed of work caused by unforeseen tech debt is the easiest way to measure its long-term impact.
What is technical debt and how does your team define it?
We usually identify it when a design, code sample or PR comes with the caveat, “this isn’t the ideal way to do this,” or “this is what we have now but we should go back in and fix it later.” Falling into the trap of thinking, “this is the quicker way to do this,” when really it’s just the quicker way to get your code committed at this moment — while knowing it’s slowing things down in the long run and requiring work in the future that will be harder and take longer.
We have a web application that multiple native applications use and communicate with using a generic IPC interface. A couple of years ago, a single platform-specific method was added to this interface “because it’s the easiest way to get this one feature in there right now for this one platform”. Fast forward to today, and the method is still in there and is confusing, hard to use and fragile to any platform other than the one it was specifically designed for, but we still have it in there because it’s not trivial to refactor out at this point.
When technical debt does occur, what process does your team use to measure and manage it?
We typically create backlog tasks to track things that should be refactored and they are uncosted using planning poker. We will take on refactors when working on code in the same area as the refactor and since they are uncosted but still obviously take time and effort, the overall velocity of the team will go down. In the case of unknown tech debt — that is, tech debt we didn’t explicitly know about or track — we will typically underestimate the amount of work for a given task so again you’ll see a dip in the team’s velocity. This velocity reduction is the clearest way to measure the impact of tech debt, but we stay vigilant in accepting lower velocity sprint-to-sprint if it means we are improving things by paying off some of that debt.
The more engineers involved in software design, the less likely you are to deliberately take shortcuts or accidentally make bad design decisions.”
What proactive measures does your team take in minimizing technical debt?
We talk through technical design of new features and cost them out as a team, and the majority of the time we also pair program and require approval of PR’s. The more engineers involved in software design, the less likely you are to deliberately take shortcuts or accidentally make bad design decisions. We also make use of automated test suites, CI and weekly deployments, and keep PR’s small to make sure we are making small, solid incremental changes. When refactoring existing code our philosophy is to leave things a little better than you found them, so we’re always striving for continuous improvement.
