What if, instead of getting larger and more brittle over time, making changes more risky, your software could get more modular and simpler, increasing the safety of the developers working on the code? The key to soft software is focusing on creating that safety first. And the key to that is going from the traditional approach of testing last to testing first. In this talk, we will discuss how the traditional approach to building software increases complexity, slowing developers down over time, and how testing first reduces complexity and enables developers to work faster and with higher quality as time passes.
Video Transcript
Matt Van Vleet
Thank you. This will be on at the end as well, but if you need the deck, just text alignment to 5 4, 4 4, and you shall have it. My contact information is at the end as well. All right, introduction. So first, we’ll let you guys introduce yourselves. How many of you used to be a developer?
Okay, how many still are developers? You get to define that, right? So you at least get to build something once in a while, or you have a lot of people who look to you to help them know what to do and so forth. So ultimately, what we’re trying to do is learn today about what are the approaches we can take to building software so that it becomes more agile over time, easier to change, and what do we do if we already have software that’s hard to change? So how do we keep the code soft? How do we simplify complex applications? How do we improve the process as we build things and so forth? My background is a software engineer. I’ve been doing XP a long time. I started a company that wrote unit testing software and then that became free. So that wasn’t a very good business to be in.
So, then I joined a consulting company that I bought out one of the founders. I did that for 18 years, then I sold that company. Now I’m working with leading Agile, and we’re not only doing organizational transformations, but we also have a studio where we build software for people or we’ll teach them the engineering practices. My focus is about how we help enterprises get better at building software. They look at the companies they want to emulate and they have a very hard time getting there. And so what can do to help organizations get there?
Why Does Software Get More Complex Over Time?
All right, so what causes software to get more complex over time? Let’s start because ultimately we need to know why it’s getting more complex so we can then decide what to do about it. So what do you think? What causes your software to get more complex?
Audience
Features?
Matt Van Vleet
Features. So, we just have more features.
Audience
Poor planning. Code base too big.
Matt Van Vleet
Okay, poor planning. And then you said adding too big of a code base, not breaking it down, not having microservices,
Audience
Acquiring a company and having to integrate all their stuff.
Matt Van Vleet
Acquiring a company. So now you’ve got their rules. Your rules. The most complex system I ever worked on, they had acquired a business, so they did rental of uniforms. This other company sold uniforms. They combined the two, but they never merged the business units. So there were business rules on this side that conflicted with business rules on that side, but we were building one system to support both. So it’s very complex. What else?
Audience
Many multiple developers which have left developed different philosophies.
Matt Van Vleet
So, developers have left different philosophies, different approaches changed over time.
Okay, I did answer some of these myself. I think you guys got a lot of those. So if you start with fear of change, if you’re afraid to change the code, it’s just going to get more complex over time. People will copy it. People will say, well, I don’t want to break this thing that works today, so I’ll go write it over here. So you just end up with lots of complexities as soon as you’re afraid to change code. Even if you wrote it yesterday, I consider it legacy code, right? So that’s dependencies. So if you went to Mike’s talk, who’s in the back here earlier he was talking about dependencies. The more things your code’s dependent on, it’s just going to get more and more complex over time. Unplanned debt. So someone here said, look, you could make a conscious decision to say we’re going to build this in this way and then we know we’re going to have to spend some money later to fix it.
That might be the mortgage on your house. You also might just look around and say, oh, we just got that project done and we have $20,000 in credit card debt. That’s what I’m talking about, like unplanned debt. All of a sudden we just took shortcuts, we ended up with code. That’s hard to change and we didn’t even think about it along the way. If it’s a conscious decision, then it’s a business decision, and that’s fine. Schedule pressures, people talked about, nothing’s ever removed. It’s always added, right? We acquired a company, we’ve got more features, we just are putting more and more stuff into our software. What about the stuff no one’s using? We never take it out so that it just gets more complex over time and the bloated features. So we’re not building the simplest thing that’ll work. We’re building everything we ever thought of.
Because what if we’re wrong? Which again, if we get the process side and we are coming up with small features, letting the customers give us feedback of whether that’s enough or not or in much better shape. If we’re guessing what the customers want and we really don’t give feedback, then we just end up building big, bloated things that are getting more and more complex.
How to Keep Your Software Soft and Easy to Change?
Alright, so how do we keep code soft? I always talk about hardware supposed to be hard to change and software is supposed to be easy to change, and hardware is much easier to change than software the way it’s traditionally written in enterprises. So how do we make our software soft so it’s easy to change? How do we keep development teams moving fast? So what do you do? We think, how do we keep it soft? How do we keep it from getting too complex? That’s why you’re here, right?
Audience
Address tech debt
Matt Van Vleet
Spend time addressing tech debt, right? Invest in it, yeah,
Audience
Test it until you feel safe.
Matt Van Vleet
Unit testing to make you feel safe about changing code.
Audience
(unintelligible)…principles like that. DevOps.
Matt Van Vleet
So, DevOps, unit testing, invest in tech debt.
So, we got a lot of stuff I had. So robust DevOps pipelines, automated tests, automated builds encapsulate your dependencies. So if you’ve got a system, let’s say you did acquire a system, now you’ve got two systems have to talk to each other. No. Is there a clear API? Is it clearly delineated? Can you test one without the other? Right? Are you encapsulating those dependencies? The dependencies are the biggest thing that slow you down. As soon as you have a lot of dependencies, you’re going to end up with code that’s probably not able to be all managed by one team, which means now your teams are dependent on each other and everything gets harder when your teams are dependent on each other and there’s not a clear way of orchestrating between the teams.
Plan your debt and find a way to pay it off. So you guys got most of these, which is good. The biggest thing I would say is make sure that you’re treating all of those from a value focused perspective. So if you’re saying we have to play off technical debt, how do you get the business to invest in that, right? You have to be able to show how does it affect our cycle time? How does it affect our uptime? How does it affect our value density of what we’re able to build? There has to be a business reason to invest in paying off that debt, and you have interest that you’re incurring every time you try to build something as you have more and more debt. If I’m wanting to encapsulate these dependencies, how can I show that that’s going to be valuable? Now I can align my systems to my business.
We can have a more composable enterprise where you can use these systems in more interesting ways in the future. So make sure all of those things are value focused. All right, so now we’ll get to kind of the meat of the thing. So you all have big legacy systems that weren’t written with tests and builds and DevOps pipelines. You may be getting some of those things in place, but you have a lot of code that wasn’t written that way. So what do you do about it? So first, let’s say your systems architecture looks like this, right? You’ve got a bunch of systems, they have all kinds of integrations. You don’t quite know how they all talk to each other. The first thing we’re going to say is start looking at your capabilities as an organization or your products. How do you want to be organized and where are the seams between the way your business works, right?
We’re working with a private airline, and so they have planes and they have routes. Well, they don’t really have routes because every route is different, but they have airports they go to and they have owners and they have different things. So what are the things we care about? And then where are the lines and how are we going to organize the business around that? So the more we understand of how they’re currently organized or how they’re going to organize Conway’s law, the software’s going to look like the business. The challenges, like you talked about, the business has changed over 30 years, and if the systems are hard to change, the systems haven’t adjusted every time the business adjusts.
So in reality, what we find is we want to have these products, we want to be able to build it, test it, enhance it, and deploy it with minimal orchestration. So the goal would be is that we have a team, they have all the skills on it, they need as little orchestration as possible to go and build something of value that the business wants. So that’s our goal of where we would like to be. The reality is our product architecture or our capabilities don’t align to our systems architecture. We may have one big monolith that is doing a whole lot of things for different products. We may have three systems because of an acquisition that are really all related to one business capability or one product that the business wants. So we have this mess where our product architecture and our business architecture don’t align, and therefore from a systems perspective, as we’re building software, we have a lot of orchestration, right?
Priorities aren’t easily aligned; interfaces don’t exist and so forth. So what do we do about it? Well, so let’s talk about where we want to be. Has anyone heard of the memo Jeff Bezos wrote about how two systems, if you had a service that it had to have an API? And if you’re never allowed to talk to the database directly, you’re never allowed to talk to the team. It has to all be orchestrated through an API. We believe you have to have contract tests and mocks and say, okay, I need an organization where I’ve clearly defined systems. They talk to each other through an API just like if you are going to use a cloud service that’s available on Amazon or Microsoft or what have you, don’t go talk to people at Amazon or Microsoft to say, how do I consume you? If you want to look at a flight status and your company flies airplanes, I should be able to call that flight status service without talking to anyone.
I should be able to build it, deploy it, test it, integrate with it, and I should just be able to do that without talking to anyone, which means I have to have clear APIs. I have to have contract tests and mocks because if that flight status service wants to change how it performs without breaking the API, they have to be able to test their change without all of you being involved. So back again, this is how we want it to be. It’s not necessarily how it is today. So what we are going to talk a lot about is encapsulation, right? So we have these products, we think they should be encapsulated with APIs. We have code. It should be encapsulated. So I like to use an analogy. So, at the top we have chocolate chips at the bottom, we have M&Ms, and we want to build that hard crunchy shell because when heat gets applied, what happens to the system at the top is it starts to melt together.
You stop seeing where the seams are, you lose what the architectural intent was. You lose a lot of things right at the bottom right? They melt in your mouth, not in your hands, right? So this API, these tests, these mocks, they protect that. Now the question is how do I get from this to that? So first of all, you have to recognize that. So we want to get from this to that. So what we want to do is we need to start decoupling the code. So we have to refactor the clear a P. So we need to know the seam. We say you should start with your business capability model and know where does the business define the seam? And then we look in the code to see what dependencies are causing that to fail, what’s causing us to break that, right? We have some tools that we’ve written on our own using open source like code analysis tools.
There are tools like code scene where you can see what are the components of my system, which ones change the most, which ones break the most, and therefore, and then where is the tight coupling? So you can use some tools to visualize your code, but ultimately what you’re looking for is where can I have a clear API? And then you create the tests and mocks. So you build a characterization test. So you say, what is it doing today around that? So you’re starting to encapsulate it. How can I mock out the other parts that aren’t there? And you start to build these seams, and of course everything you’re doing there is going to have automated builds and tests that come along with it. So has anyone experienced this? Tried to do similar things? What do you run into?
Audience
At least when we try to add APIs, we’ve run into a lot of edge cases that were added into code. And so it takes a lot to understand all those edge cases and sometimes we don’t understand how or why they put there, but we’re supposed to still that same
Matt Van Vleet
Behavior. Sometimes those edge cases are there because they needed to be there. Sometimes they’re there because people just thought completeness. Well, what happens there? So yeah, there’s things that you’ll have to look at. Again, it’s part of that why do we never remove code? Because we we’re not sure what it does or how to change it, but you should be able to use some of the tools. You can use some of the AI tools as well to say, how would this edge case get called and what should I do to protect it? So there’s things that we’re doing there, but ultimately if you can’t encapsulate it, then your teams are going to be slow.
Sometimes the heat’s been applied so long that we’ll find a component we want to extract and wrap it and test. And it is actually at a point where the business has changed enough that we actually have to rewrite that component. So we may build tests around that component. We can’t rewrite the entire system. It’s been there too long, it’s too complex, it’s just not realistic. So a lot of times we will decide we need to rewrite a component. Well then you’re in a position for that area of code to write it with the modern engineering practices you want. So do it with test driven development. Write tests that describe the behavior you want. Write code until those tests pass. Automate the build, have a DevOps pipeline that stops you as soon as you break something that used to work and then you’ll be in good shape. So that’s ultimately there’s times where the right thing to do is to just rewrite that component. Ultimately, we’re going to go through that process again and again. And what we want to be is as soon as we have our products and our systems aligned and there’s clear APIs between them, then we’re going to be able to move much faster as an organization. So that’s ultimately where we’re headed.
Okay, so want to talk a little bit, I guess, any questions about the practicality of that? Technically how we would do it. I didn’t go deep into code or anything in this talk, but anything you want me to dig into? I’m going to talk about an example where we did it for a large e-commerce site and what we encountered, what happened and the result we got.
Reimagining Application Architecture (A Case Study)
Alright, so the situation with this client was they had a business that was mostly sold through dealerships and the new world is mostly going to be sold through e-commerce. So they went and very quickly decided that they needed to build an e-commerce capability from end to end. What happened is is they ended up ramping up. They weren’t getting throughput fast enough, they added teams. So they were up to like 30 teams across four or five vendors that are all building software that ultimately needs to take and turn a interest on the website into an order into cash. It needs to go through that overall process. And they got to the point where the more people they tried to add, the slower they got, not the faster, the more they built software, the quality went down, not up. So they started needing to do some things to fix it.
And so one of the things they did is they started adding architecture to make sure what they’d find is they had something and then the attributes of their product change. So they ended up having to rework a bunch of code. So then they made their architecture such that it was really robust and everything could change. So they ended up over-architecting it to try to future-proof it. So they never had to rewrite things, so they never had a quality issue. But the problem is if you try to predict the future, you’re going to fail and you’re going to build a whole lot of software that just creates dependencies and slows everyone down. They introduced complex management systems. So in order to get approval for this team to make a change, they had to get all the other teams to agree. And you can imagine how that slowed things down and increased lead times.
If I had a new idea, the time between we had a new idea or request from the, and we could start building code was just getting longer and longer because of all the things people had to try to think of ahead of time. And there were heavy integration cycles from when the developer believed the code was written to it got in production was months and months and months and it had to go through environments. By the time someone found a problem and it went back, no one remembered what they had done and that company that was working on it might be onto a different problem. We’re no longer as a team in this big big program. So this is how we found the team, what the current reality was. Anyone, you don’t have to reflect some realities you guys have run into in the past.
So, the first thing we had to do was get a vision of where do we want to go? And so a lot of it was what we just talked about. We want to have these areas of the system. Sorry. Okay, no problem. Are you guys able to hear, all right. So we wanted to have these areas of the system, they can move at the speed those areas of the business are changing. So if the way you configure the product is changing rapidly and the ways you can pay for it aren’t changing, we don’t have to have all of these departments dependent on each other or if they’re going in different directions. So how do we have D components separated ultimately, how do we modernize this? So someone talked earlier about having smaller components. How do we have tests and DevOps and builds around it? And then how do we get the engineering up to the level of the application modernization?
So, the first thing we really did was we looked at what are the capabilities of these teams and how do we organize teams around those capabilities? So we may have had two vendors working on parts of the system that were all part of the same capability. So how do we align that to organizationally so that the business requests tended to map to the same team that they were being served by? So a lot of that was kind of insourcing the things into the company that needed to be insourced, but then in the process of insourcing that they were afraid to bring it in from the vendor because they didn’t know how the system worked. So that’s when we built tests, we built mocks, we let them get to where they had a system of trust around that system and then when they started enhancing it, they didn’t fear changing it as much as they had in the past.
Ultimately, once we got to the point where the teams had less external dependencies, less external orchestration required, now we could get productivity back into the teams. So the teams could look at, well, instead of spending time trying to escalate everything outside, where are we spending time? Their retrospectives became more effective, their internal improvements became more effective because they weren’t constrained by all of these external dependencies. And ultimately we were able to start changing how they brought in talent and what they valued. Because instead of valuing someone who knew a particular technology or knew a particular industry, they could bring in people that knew how to write software, how to build tests, how to manage for change, and they didn’t have to go hire so many different specialists. They could bring in people who more understood how to solve problems and then they would use whatever tools were the right tools to solve those.
In the end, very quickly we were able to get a lot of results. So this is probably a worst case scenario where we got all these vendors and they’re all trying to work on the thing at the same time. But the first thing is we went from almost no automated test coverage to 93% automated test coverage and we had 70% decrease in defects. It doesn’t mean that there weren’t the same problems in the code, it was they were discovered before they left the team. So the team, if you build something and then you’re immediately told, Hey, that doesn’t quite work, it’s very fast to fix it. If you’re told three months later, Hey, that thing you were working on over there doesn’t seem to be working, it could take you days to just remember what you were doing and get it all back up in your environment.
We ultimately, the cycle time because of so much less orchestration required was nine times faster. The productivity we were getting a lot more work out of. So we ended up reducing the team size because at some point, adding people wasn’t the constraint. If we got rid of the dependencies with less people, we got more software out the door waste. There were good stats on what we decided not to do because when they were afraid I can’t, we get two releases a year, they had to have everything in that release that the customer could possibly want. When we got to monthly releases, then they said, well, let’s get this in and then we’ll find out. We’ll do a session with them to find out what they see is missing and now. So there’s a lot more of work that was not done. And so that’s a big reduction in waste. And then from an employee morale perspective, they used to be the group that everyone fled to other areas of the organization. And once they felt that they had control of their own destiny, once they were seeing that they were able to control how the software is written, they started to get people wanting to come to work for them. And the people who were there felt that they actually could make a difference and see something that they wrote in production being used by a user.
I think I went the wrong way there. We could cover that all again. Ultimately they got the software out faster, which was their number one issue, but then by insourcing it and having a smaller team, they were able to save millions of dollars and now there’s kind of an enduring practice. There’s still a fair amount of code that if they haven’t had to enhance it or it hasn’t broken, it still doesn’t have tests, it still doesn’t have all the things we would want it to have. But at the seams between the teams and as they’re changing things, they’ve got good coverage and they’re getting good results. So I feel like I went through that in record speed. So I’m probably 15 minutes left, is that right? Okay. All right. So yeah.
Q&A – Internal Developer Platform
Audience
I got a question. So you’re talking about individual product teams and defining those contracts and defining APIs. Do you have any best practices or tips around the communication or sharing what those APIs look like out there, like swagger and things like that, but consistent and up to date communication around what those APIs are huge, otherwise the teams end up having to bug each other.
Matt Van Vleet
Yeah, there’s a lot of conversation going on now for bigger organizations about having an internal developer platform and ultimately, so the first thing we should be asking our teams is the expectation should be thought of as their API is a product and its just as important as the product the customer is consuming because in the end, everyone who consumes our APIs, we’re paying money for them to consume them. So it has to be efficient. So there’s developer platform tools like
It’s not Blackboard, I’ll have to think of the name of the tool, but there’s tools that will host the APIs and document them. But the best documentation for an API is sample code and a test. So you should be thinking about it as I want someone to be able to, I’ll talk about this flight status service. I have a flight status service, I want you to be able to call it without ever talking to me. I want you to be able to know how to consume it, build it, test it. So you should be able to find my repository. You should be able to submit an enhancement saying, well, can I treat it like open source? We may have to merge it, the ones trusted with that code. But if you say, this is how I want it to be better, you should be able to use that API just like if you went to Amazon and wanted to call one of their services and theirs sometimes are hard to use, but for the most part they’re much better than our internal services.
We need to get internally to say, alright, I’ve got this team that API needs to make sense as part of the business, not just be some technical thing we came up with. But if we have flight statuses and we have crew statuses, then you should just be able to go and say, well, my application needs to show this. I shouldn’t have to have a meeting. I shouldn’t have to get permission from anyone. If I do need permission, there should be a place on the developer portal where I go and request the permission and see why I need to use it. But in the end, if it’s aligned to the business, that permission’s going to be pretty easy. But did that go where you wanted? It
Audience
It would help to just to talk about the mentality of treating your API as externally facing and those other delivery teams are your clients, right?
Matt Van Vleet
Right. And the reason you need tests around it is if you need to be able to change your implementation under the covers without talking to anyone who’s consuming it. Because even I used to do a lot of projects with enterprise application integration and we would go and everyone, it was put a new bus in, everyone’s working with us and there’s a beautiful picture of the messages come through here and they can all go to these different places. But to test it, if you need all of the people from those systems the first time you build it, they’re all available after that. You can’t go change this component without, if you can’t change it without all those people, you’re cooked, right? You might have a pretty architectural picture, but organizationally everything’s dependent. And then no one wants to change this component because what if it breaks?
We just spent a year getting the SAP team able to talk to this API now that I can’t change it, but they want something similar. So what do I do? I’ll make a copy of it and I’ll do it that way. And now my organization’s getting more and more complex. My software’s not getting better over time. It’s getting more complex, harder to change, harder to use. So in the end, if you’re afraid to change it, you are cooked. And the way to get rid of being afraid to change it is tests small components, but then we don’t want it to be hard for people to use. So one of the things, if you’re getting a internal developer platform, you’re building APIs just like it’s a product and you need to treat it like a product. If you’re building products, you’re in a lot of trouble if you’re not talking to your customers.
So, if I’m going to go build something that says, well, you can call this, but you have to call it with corba or you can only call it in this way, then no one’s going to use it. So now I’m got this idea of I’m the field of dreams. If I build it, they will come, right? And we run into a lot of developer productivity tools, APIs, things that are built internally that no one consumes. So you have to build it with getting feedback loops from your consumers who’s going to use our API? How are they going to use it? But once you set that up, and so you’re going to have parts of your big monolith, people want to call and it’s a project. So you say, all right, well I’m going to make this available to you, but I’m going to make it available in this way.
Let me put a set a mock up. Here’s how I assume you’re going to call it, and here’s what I assume I’m going to give you back. You can start writing your code against this, and then I’m going to write aside test. I’m going to build my code, and then when I turn it on, there shouldn’t be surprises. Of course there will. And then when we were wrong, we write a test to fix it and then we adjust it. But now, but we’re trying to have that collaborative conversation with who’s calling us and making it available. So it is a big conversation we have going on internally right now because if you build an API or a developer platform that can strains me too much, then I don’t want to use it because I may have to use it in a different way than you do.
But if you don’t add enough value, I don’t want to use it because I could just go write the thing myself. So well, what’s that Goldilocks thing? And the only thing we know right now is you just do it iteratively and incrementally, lots of feedback, discussion, but in the end, it’s got to be easy to change. And your legacy code isn’t by nature today. So you have to figure out, well, how do I make it easier to change? And kind of the scouting rule is right, let’s leave it a little cleaner than we found it. So if there’s an area we’re changing, encapsulate it, extract it, put an API around it and then change it. And over time, the areas of your legacy system that are never used will just sit there. The ones that never break won’t be improved, but they never break. So that’s okay. The ones that the business is changing rapidly or are breaking rapidly that are going to get prioritized to the highest level craftsmanship. Yeah.
Q&A – Encapsulated Teams & Product Owners
Audience
So, you might’ve said this, when you built these teams that worked, did you do it in XP style where you had product experts embedded and sitting alongside the developers? Or how’d you do it?
Matt Van Vleet
So, when we could, so we definitely the team’s own producing something that worked. So when you say if you had asked, did you have QA people on the team, did you have requirements people? Yes, product ownership’s interesting. And so what we did was we would have someone on the team that either knew the product, if it was really technical, if it was something business focused. We tried to get that feedback loop as quick as possible. And once the business people saw that they were getting value out of the demos and they weren’t just status meetings and that if they suggested something, they saw it the next week, they had all the time in the world to hang out with us and work on it. When they say they don’t have time to help your team, they’re saying, you’re not using my time in a valuable way, and therefore I’m going to go spend it somewhere else because in three months when the software’s delivered, it’s still going to be wrong.
So if I spend all the time now, I don’t want to. So that’s I’ll actually, when business people say they’re too busy, I said, are you really too busy or do you just not feel like what we’re doing is valuable? You get in that conversation. So then how do we make it so that their input is valuable? Our belief is that everyone on the team should learn the business. So you’re going to have business people, and ultimately there’s a decider who gets to decide. But in the end, if you are working for a company that has owners of private jets and you timeshare them, then you really ought to understand that business. And we had a project and we were doing it for a CFO, this is a different company. And the CFO was looking at the weekly demo and saying, well, this table, can you make it sort and filter and Collins be able to be reorganized?
And one of the developers asked the CFO, okay, well how does that reduce the cost of maintaining the product catalog from $300,000 to $30,000 a year? And the CFO’s like, it doesn’t, right? But the CFO is just thinking completeness, okay, I’ve seen apps. This is a CFO as money focused as possible, forgot why they were building the product. So we want everyone on the team, you have to make sure you put that developer in a position of trust where they’re allowed to ask why, right? Because in the end, if they know why, then they can use their talents to build it. The simplest thing that can achieve that purpose. And yeah.
All right. Well, we’ll end at five minutes early and if you got anything, come up and ask, I think. There’s the deck if you need it. Thank you.
