Microservices: The Organizational and People Impact

Austin: “Now we have got Daniel Bryant.”

Daniel Bryant: “Hello!”

Austin: “We’ve moved from Berlin and are now setting up in the UK. Essentially, we’re moving across time zones, planning to gradually make our way to the West Coast of the US at some point. Today, we’re fortunate to have Daniel Bryant from Open Credo here in the UK—a sought-after expert your company calls when it’s time to adopt microservices. He comes equipped with a tent, sleeping bag, laptop, and gets right into the thick of things. Interestingly, during a recent workshop, Daniel shared insights into various UK-based tech trends, including the rise of crypto casino UK platforms, which are transforming online entertainment through secure, blockchain-based payment methods. He’s here today to discuss how to implement microservices, so without further ado, let’s welcome Daniel.”

Daniel Bryant: “Cool, rock on, so shall I switch to full screen on the presentation? Everyone hear me, see everything working, let us try that again. Everyone good to go?”

Austin: “Yep.”

Daniel Bryant: “Awesome, awesome. Welcome, everyone! Yes, it is like, what is it, seven thirty here in the UK. I will press on, it is time to get my dinner afterwards and maybe get a beer or two. As many know who have seen me before, I like to make way too many slides and way too many book references and stuff, so I will try and keep the pace not too fast, I do have a temptation, particular in webinars, to talk really fast, but I try and dial it back. The key thing today and [inaudible 00:01:27] organizational and chatting to the people in Austin[inaudible 00:01:30] this will be super interesting for my kind of practical experience. The TLDR, and this is kind of a popular meme at the moment, is that implementing microservices is easy. Implementing microservice systems is complex, yeah? Now, I have seen a few people talk about this recently. It is sobering, you know, building one microservice and their worries, running a microservice system, different kind of things, yeah?”

“This is includes the Associate, Organization and people systems. This is kind of the core of my talk, basically, and bare this in mind as we go through these things. What I would say, microservices last year did what I think was a great summit, that I stayed up until late in the night in the UK times to watch. Last year, it was still very much the innovators getting into microservices, yeah? If anyone is familiar with the diffusion of the nation curve, you often see innovators arrive first, you get then early adopters and early majority. Microservices, over the last sort of year or two has firmly moved into the early majority yet, we are pretty much here now and this means big companies as well as small are trying to look at microservices and big companies have legacy systems or moneymaking systems as we already heard them talk about today and they also sort of have legacy organizations. Legacy bureaucracy, legacy team structures, all these kinds of things and they are not inherently bad, but they might not be compatible with micro service style technology and teams.”

“That is what I am going to try and build on today, there will definitely be stuff to take away if you are like a start up or you are just starting with micro services but if you are an existing enterprise, then this will hopefully be quite useful to you. In the field, I see rather a lot of problems, and a lot of problems with the microservices, they are fundamentally people and organization problems. The good news is, that we already got many solutions. Our business friends have been dealing with this kind of problem for a time, as much as IT and the IT industry is quite young, business has been going on for a bit longer, so we can definitely learn, I like reading HBR quite a lot, Harvard Business Review, it gets lots of great kind of tips, for me as a techie it is quite understandable, it is not too business speak like, and for me as a techie it is a great resource, so I am going to be sharing some of my tips and tricks today over the next like forty or fifty minutes.”

“This is me, feel free to tweet at me, my Dms are open as well, always happy to chat to people on the internet, always good fun. I work for a company, as Austin Mentioned, called OpenCredo. We are a software delivery and development company based in the UK, based in Germany, based in the US as well. We not only help companies with the technology change through microservices to [inaudible 00:04:06] a whole bunch of different things, but we look at the organizational side as well. That is super important, if you want a technology or fundamental transition to take stick, to actually stick in an organization, you always have to work with the organization itself, the people, the processes within. It is very hard just to change technology.”

“My background is Java development, do a bit of go these days as well, loving the cloud, loving the containers that kind of stuff, I try to talk as much as I can, travel around the world, wherever my boss will pay for that. Sort of help out the London Java community, [inaudible 00:04:36] pleased to get in contact, love to learn, love to chat, love to share ideas, basically. The next 30-40 minutes, the presentation is roughly breaking down to these three things. Strategy, Feedback and Responsibilities. These are my three things for my organizational and people process, and my microservice that I kind of see as primary sticking points. Strategy, Feedback, Responsibilities.”

“Diving straight into strategy, I have taken a lot of my cues, actually Phil has already mentioned, Simon Wardly. I take a lot of my cues from Simon Wardly, and Simon talks about Situational Awareness and Leadership. These things are super important in tech, pretty much regardless of microservices but if you do it in Microservices, even more so. First thing you got to ask, really, is when you are perhaps evaluating microservices or you have got problems in the organization, you have to ask ‘Are microservices a good fit?’ I know this is a microservice summit, but always ask: Is the technology, is the process of it, is it appropriate for my organization? I see Gartner and a lot of interesting people doing sort of Mode 2 as they call it, System of Engagement applications as Microservices. Now, the mode 1 kind of Apps, their counterparts are the systems of record, kind of the Crown Jewels as we say over here in the UK, they are typically done in more Monolithic technologies and things like this.”

“The problem is, yes, you can have lots of shiny Microservices, lots of cool things in Mode 2, but you are always going to be limited by the pace of change in Mode 1. If you have got some crusty old mainframe system or whatever, you can chuck some microservices on it, rub some DevOps on it, and chuck Docker in or whatever, but fundamentally if you have not got a migration plan, it is going to be really hard to fully get the benefits of microservices. You are always linked by the thing that changes the slowest in your organization and in your technology. If you are not careful it is putting the lipstick on a pig sometimes, I love this analogy, but you know you can slap some mircoservices on, chuck some docker on there, but if you are basically limited by some crusty old mainframe system, that may be the first place to look.”

“That is kind of the business side of strategy. Architecture side, super important as well. I see quite a few people not understanding principals, microservices, so not building them around business functionality, cargo holding, just blindly copying things, and no kind of strong technical leadership. It does not matter what style of architecture you are using. If you have not got technical leadership, you are going to hit a lot of bumps in the way in the journey.”

“The third thing is the operational or kind of DevOps side. I do see in some organizations, like some of the UK betting sites we’ve helped out, no well defined Devops. Now, I know I think both Lori and Phil mentioned this before, Devops is kind of a prerequisite to microservices, I will talk about that a little bit later, but be very aware of a sort of hero culture, the kind of, you know, a devops warrior a devops princess or whatever, some of the ideas behind it are quite nice, but be very careful about the hero culture. Devops is not about a deployment or ops free for all.”

“First day is checking if your business is ready, and these are basically from a Paper in 1999, a bit old, but it talks about critical success and factors in software projects. If your business is suffering from any of these things here, I am not going to go throguh them, but if your business is struggling, your technology is struggling with sort of basic problems, microservices are probably not going to fix that, yet. Microservices are not silver bullet, they are not panacea, they can be used to drive change but be very careful doing that. Microservices add their own consistency, actually, and if your business isn’t ready, they can be almost too much to handle at times.”

“I know I heard Phil mention that, Wardby Maps, loving that kind of stuff as well. Situation awarenss for your business is very important, both from a business point of view, knowing where you are in the market, where to attack and that kind of thing, but also as technologists, we need to know where we should be attacking, in the market. For example, microservice platforms are popping up now, and should we really be building our own microservice platform, things like that. You can see on the Wardby Map, you have got Value on the Y axis and the X axis is the kind of lifetime, if you like, from Genesis to commodity. I think sort of Microservice platforms are off the shelf right now, you can get some open source one, [inaudible 00:08:58] have done some really cool stuff with that, I know a bunch of companies, Netflix obviously stands out, Twitter stands out, you can kind of leverage some of their stuff rather than reinventing the wheel.”

“Situational awareness, know where you are. Phil’s article, great article down the bottom right here, by Phil who you have just seen. Talked about value and stream mapping or value chain mapping, knowing where your pain points are in your organization, because I have seen some companies to try and sort of use microservices to fix some of their issues and actually their issues are further up their value stream. It is getting sign off from people, getting buy in, and that is more important to fix first rather than do Microservices. [inaudible 00:09:34] Awesome, but, whether you are an enterprise or not, I would highly recommend that by Jess Humboldt, [inaudible 00:09:40], really good for understanding situational awareness and [inaudible 00:09:44] as well.”

“If you do decide to go into microservices, I often like to do this exercise with teams. You kind of make sure you have alignment across the business, so we work on strategic goals, we can see level, we map that back to architecture principal with the kind of team leads and architects, and me as a developer in the trenches, I am looking at the design and delivery practices. This kind of is a shared understanding across the business and it helps me as a developer understand why I am doing certain things. Like, here we are optimizing for [inaudible 00:10:12] legacy code on the right, breaking seams and interfaces, and we are doing that because we are trying to eliminate accidental design [inaudible 00:10:19] and we are doing that because we are trying to support entry into new global markets. We need to move fast, that kind of thing. This is a great sort of tool for the sea level to understand what we are doing in the trenches and vice versa, us in the trenches to know where the sea level are going. It enables us all to kind of speak the same language, all to know what direction we are going in, and have good conversations because communication is super, super important, regardless of microservices or not, but microservices give us so much flexibility we need to have the communication in the organization to really get the best of that advantage.”

“Technical leadership, mentioned already, super super important. I am a big fan of Simon Brown, anybody who has heard me talk before, will know I talk about Simon quite a lot. He is a great guy, has some great books out there. This, for me, is a killer statement. If you cannot build a well structured monolith, what makes you think microservices are the answer? I did not see Simon when he presented this, but I imagine he literally dropped the mic at this point and walked off the stage, I think. This is a really great point, I have seen Phil Albarro but it is differently? Crap in, crap out. [inaudible 00:11:26] Alvarro Sanches, and this is a great presentation to look at, but carving things up into smaller pieces did not inherently make them better. Bit crude, but you get where I am coming from.”

“Leadership, to me, is all about four things. It is about promoting shared understanding across the business, I find with microservices the biggest thing is communication, across teams. Flatten the silos, cross-functional teams, which we will talk about later on. It is about risk management. Any new architectural style inherently comes with some risk and pretending it doesn’t is just daft, so risk management is super important. It is about just enough up front design; as an architect that is one of my principal functions. With Microservices, you need to do just enough upfront design, particularly around the platform you are deploying services on to, and also the glue components between. The messaging, the protocols, that kind of thing. I take a lot of my cues here from Sam Newman, Sam Newman’s book is really good around this. Basically, think about Architecture for Microservices as less building bridges, excuse me, more building cities. Sam Newman uses the analogy of Sim City, building Sim city like kind of cities.”

“Leadership principals are super important as well. I take a lot of my cues there from Pat Cuwer, you see the book on the right there. Excellent book, talking with tech leads, it has got a bunch of essays from various tech leads, about how to run good software development projects, how much you should be coding as an architect, what you should be doing as a developer, QA, these kinds of things. A lot of it comes back to something I heard Lori mention, which I am a massive fan of Daniel Pink’s work, around autonomy perks and mastery. My role as a tech lead, when I go into companies, is making sure that people have the knowledge, have the autonomy, clearly got purpose in the organization and we have given them a path to get better. These things for me, tech leadership, architecture, super, super important.”

“We have already heard this mentioned as well, microservice prerequisites. Martin Fowler, on the money as usual. The technical side of devops is essential, if you are going to be deploying lots of things, you clearly need to be very good at having built pipelines, automated testing all this kind of good stuff. I do find you are around the sort of technical side of devops, and bringing in microservices is there is lots of temptation with shiny new technology. Lots of platforms, lots of tools, and as you can see at the bottom of the slide there, I do find choosing tooling quite challenging. Both individual and within the teams I work within, choosing tooling is often a blocking point.”

“Some of my tips and tricks I use is I have recently discovered something called the spine model, it is not my work it is Kevin Tripithetti and Daniel Rue there, great talking at Agile 2015 last year, talking about how tooling is often around effective conversations making for effective collaboration. As a species, the human species, we have been tool users for millennia. It is arguably what separates us from the kind of oldest ancestors, in evolutionary terms, tooling where we are now, people often get stuck in dilemmas. When I go into organizations, people are having these kinds of conversations around tool choice and these equally plausible options. I say Docker, you say Rocket. I say Java, you say PHP. Often, going up the spines and talking about the next level up, helps you establish the correct conversation.”

“I do not expect people to read this on the next line, but for example if we are arguing around PHP And Java, yeah? We are arguing about tools. Basically, we cannot make a decision together, we go up the stack, one more. Practices, what are we optimizing for? I might be looking for a solid tool chain. Perhaps Java’s better there. You are looking for rapid application development, PHP is better there. Then we can sort of work up the spine to, what are our values, what are our needs? Often, it is like the company clearly favors rapid application development or it favors high quality and low failure rates, and that is the conversation we should be having rather than Java versus PHP or whatever. The spine model is really nice, there is a nice wiki there, a link at the bottom right, check out the wiki. You can often break a lot of quite tense conversations particularly around languages we use as developers, guilty as charged sometimes, are quite dogmatic but going up the spine helps break the kind of [inaudible 00:15:44].”

“Also, a big fan of evaluation around fitness functions. This comes from Neal Ford and Rebecca Parsons, they have got a new book on the way, looks awesome, called Evolutionary Architecture and microservices fundamentally are an evolutionary architecture. They optimize for change in the future, a lot of classic monolithicial architectures did not really optimize for that, and yet you had to up front invest in a language, you had to kind of upfront make a lot of investment choices around, say, ESBs and these kind of things but Microservices enable some notion of evolution for your system. Fitness functions are great, you can see on the left there, you can basically define, say you are looking for a new no store database. You can define criteria and you can give a score and you can kind of draw what you optimally like as your new no store. You can then go to the market, look at the Cassandra, look at Mongo, draw, do some research and draw their corresponding fitness functions and compare them to your ideal one.”

“The beauty of this, is that not only is it great as an evaluation tool, but it can give you some documentation. If you come back in a years time, and you look and you say ‘Why did we choose cassandra?’ And you get ‘Oh yeah, that given point’. We as a tech company, we as a tech organization within the company, were looking to optimize around this fitness function. It is great as an evaluation tool, great for documentation in the future. I only use it for sort of big choices, not whether I use a four loop or a switch statement and that kind of thing, but I do kind of use it for platforms, languages, middleware, deadstocks that kind of thing.”

“Same notion here with Matt Raymond’s comparison matrix, I use this one quite a bit. Got things you are comparing on the X Axis at the top there and the Y axis has got criteria which we are looking to store things against. You can have lots of fun here, you can give this kind of exercise to individual people and then come back together and use this as a discussion point, You can apply various weightings to things, again it removes some of the subjectivity, provides great documentation after you have made your decision, which is brilliant. One thing I [inaudible 00:17:43] is it does assume you know your criteria. We have had clients try to use this technique, say for container technology, and they did not really understand how container technology worked. Therefor they picked bad criteria, which led to some not optimal solutions being chosen. Do reach out if you need help, ask people in the industry or whatever, stack overflow is good and that sort of thing, make sure you know your criteria which would then be a very valuable technique for comparing core things within your application. What language to use and all that kind of good stuff.”

“Final thoughts around the technical stuff, I have heard Dan Mckinley say ‘Optimize globally across the organization.’ If you are doing a microservice migration, basically start small and prove it out. Often you want to mix in a few, perhaps new languages and tools, I have cautioned against that, I will mention in a second. If you do want to mix in new things, what we have tried to do in the past is apply nature selection. Only officially support the fittest tools. If someone wants to bring in Mongo, rock on, but that team is responsible for mongo. You typically have crossfunctional teams, they have got ops that have got developers in there, they are also on call. If Mongo crashes, they are fixing it pretty much. After say, three or more teams start buying into this notion, and they say Mongo is our datastore, then maybe you can start moving it to some sort of Ops team or something or some kind of SRE team is the popular term with Google at the moment and then they start looking after mongo for you. I like this notion of natural selection.”

“I also sort of wanted to hint at just a minute ago, I really like the idea of choosing boring technology. That is often not popular, in organizations like [inaudible 00:19:22] because developers, we like to play with cool tech, we like to play with the new tech. If you are a PHP shop, and you are trying to do my services, you do not really need to go to Java straight away, you can do PHP microservices. Really, if you are just trying to prove whether microservices work for you, the amount of variables you can kind of keep consistent, IE not learning a new language, the better. Then you actually test what you are trying to test, can you build microservices not, can you build microservices and build them in Java. That can be too much in one go some times. Not every organization, but sometimes.”

“I also find the internal open source model works really well within Microservices. Be it microservices themselves or be it tooling associated with microservices, I like to make the work visible in the organization, so I might be using GitHub or GitLab. Everyone can look at the code, but certain teams have the commit responsibilities. By all means, if that team is busy and they cannot do work for you and you can raise an issue and they say ‘We’re really busy’, you can do the work, do a pool request and they can review it. That often is a very nice way to make sure that the team who kind of own the microservices or own the tools understand what is going on with them, but they do not become blockers for stuff that you desperately need in your team. Internal open source model, very nice.”

“Moving on to feedback. We looked at strategy a bit there, both from business, from the architecture side and the ops side. Feedback, I am learning, is super important and Microservices just exaggerate this sort of thing in my mind. Microservices typically generate more data on all levels, Ops, Business, all this kind of stuff and it can stretch limitations within the organization. If you re not used to processing, to storing, to kind of analyzing that amount of data it can be quite tricky at times. I have also found existing tooling is pretty much optimized around Monolithic applications, we have had them for quite some times. Oops, my monitor has just gone funny. A lot of the monitoring vendors like New Relic and so forth and Datadog, they are catching up to this and they are doing some really cool stuff around sort of monitoring for containers and monitoring for microservices, but some of the more business intelligence tools are not quite there yet. Aggregation and intelligence, you might have to build some of your own tools, this is really important to kind of promote those conversations, promote the communication within the organization.”

“Regular reviews and retrospection, totally essential. The whole point at looking at microservices, really, is they can enable more agility within the organization. If you do not take advantage of that, you do not collect the data, take action from the feedback, learn from your failures, kind of what is the point of doing microservices? There is some point, but you are not getting the full benefit.”

“What I am going to talk about for the next few slides is visibility. Visibility for the business, I think, is really important. This is the dashing framework, a really nice framework that I often use, to sort of put business information up on information radiators. I will put monitors next to the CEO or whatever or in the business team and I will be feeding metrics from my microservices into this dashboard, so it gives instant feedback and instant visibility of changes you are making. Really powerful, people like Etsi, people like Netflix are all over this kind of stuff so learn from the masters pretty much.”

“Architectural feedback, as a developer, as an architect, really, really important. The three areas you can see here on the left, the middle and the right, the left I am a really big fan of Adam Tornin’s work, your code is a crime scene. He talks about techniques for identifying areas, or problem areas if you like, within your applications. It has a really cool tool you can actually use to map the code in your application and also the change rate using things like GitLogs, you can map it into a city view which you can see on the left here, which is like super cool. Not only does it look super cool, it is really useful. You can see at the top right of the city diagram, if I move the mouse there, that might be a candidate for a microservice. It looks like the big building size is a lot of code, and the tall size is basically a lot of complexity and the color is the rate of change is quite high. This might mean that it is a particularly complex bit of our application that we are constantly changing, so maybe want to apply a microservice to minimize the impact on other parts of the system.”

“A really nice tool to just sort of monitor what is going on as well. In the middle, same kind of thing, if I am using Ruby I use clove biomet, if I use Java or JBM languages I use Sonar Cube, you want to be keeping an eye on the architectural quality of your application as you migrate from a monolith to microservices. Running these things constantly as you build a pipeline as you migrate between these two things, really really important, so definitely get those kind of things in.”

“On the right, sort of the new challenges in microservices is the flow of data, and I will talk about that a bit more in a second as well, but things like you need to generate your death star architecture diagram as [inaudible 00:24:25] calls it. Hopefully when you start is not going to be a deathstar, when you get to Twitter or guilt scale, it might well be, but you need to map the flow of services, the flow of data. Often it can be quite challenging as a microservice developer to know where your service hits in that flow, so really important to invest in feedback around those things.”

“On this note, about architecture, I also write for InfoQueue and I had the pleasure of meeting [inaudible 00:24:53]. Rafi was on the tech leads when Twitter did their migration away from the monorail to microservices, and he talks about, basically I watched his talk, it was an amazing talk, and I wrote up some notes on InfoQueue. If you are looking to do a big bang architecture with microservices, so think about some of these things that Rafi mentions, you can check out the link there, read in your own time, but a lot of it is that Big Bang can always be risking, so I can be against doing big bangs, but having visibility and what is going on and constant feedback is really important. This is a great article for if your business is wondering about doing a big bang or whether to do incremental migration, it gives some nice hints as to what to think about, basically.”

“Conversations, as you have heard me say already, are vital for tech leads and architects, and conversations and also sort of driving progress forward, definitely not a dictorial but you sometimes need to be that kind of benevolent dictator. Get everyone’s opinions and just ultimately make that call. Tabs v Spaces, who really cares, if you are watching Silicon Valley you know that is kind of a controversial subject, just make the call, kind of put it in. Standards are massively important in microservices, things I have struggled with in companies is standardizing around HTTP, [inaudible 00:26:07] often now I say just follow Apogee Standards, just follow strips API, just follow Google API, I do not really mind which one you do, just as long as we all as a tech team we all agree those are the standards we are going to be following. We are also going to be documenting our APIs with this one formal [inaudible 00:26:23] so hick swagger, pick a model, pick whatever works for you but just pick one and make sure you go with that.”

“Testing is super, super important. I think consumer based contracts for sort of localizing testing are very useful, and microservices and changing contracts is a queue for communication, and if I need something from an upstream service and that is going to break contract offers, may break my contract, that is not an end of the world situation but a queue for a conversation with that team. I raise an [inaudible 00:26:54] chat with the developer, [inaudible 00:26:57] that kind of thing. Automation [inaudible 00:27:01] emergent behaviors you can get from all these services interacting, you need to automate that stuff. There is always going to be a bit of manual testing, but fundamentally end to end automation is super important.”

“Also, I have found what is really important is to have framework and foundations in place. I am talking about things like Finagle from Twitter, carry on from Netflix, sort of a base module that provides some notion of agreed functionality. Things I often look for is monitoring logging, ops, I want a health check end point, and I want some notion of service discovery, that kind of thing. These things are really useful, to kind of, conversations I have around standardization and kind of commonality in microservices are kind of these three things. Standards in communication, standards in validation and testing and standards in how do I build a microservice. Even if you are using multiple languages, you can all agree we are going to use Springbooted java, we are going to use go micro and go, but you make sure they have the same contracts in terms of things like deployment, things like monitoring, logging that kind of good stuff.”

“On the notion of testing, I would like to give a shout out to the Hoverfly team out, I have been working with them out in the UK. Hoverfly is a lightweight service virtualization tool. We use it quite a bit in testing or testing around NFRs. We often simulate one service, say in an NFR test and we then run a performance test against the rest of the services, so we might simulate a mainframe service that we can not actually have access too, and sort of simulate and inject failures into our tests. We also use Hoverfly for, once we have agreed for an API, say we got a front end service or back end service being developed, we agree on the API, we agree to contract, we then spin up a kind of mock version of the API in Hoverfly, and the front end developers code against that and our back end developers back fill the API. Shout out to hoverfly, open source, really cool bunch of people, really cool when [inaudible 00:28:55] I am working with in the UK as well, do check them out.”

“Be prepared on the same notion to talk about data liberation as well. Data liberation sounds fancy, but what it means it that typically with microservices you pull out data into individual data stores, that is the kind of canonical microservice pattern, microservices own their data. If you have got a Monolith, like [inaudible 00:29:19] there is often a lot of challenges around this kind of stuff and it is very easy to trip yourself up. It is like literally sticking a spoke in the wheel, yeah. Often you have to do a bit of refactoring up fun, sort of data migration before you can pull these things out. Lots of conversations about is our data ready, how do we split it out, please as a technical lead do be aware of these things and how to talk about them.”

“Operations, visibility within ops, hopefully it should not hopefully need to be said, really important. I have got some links there for you about logging and monitoring, these are my kind of go to articles which I always recommend when I start working with clients. That is pretty much it, I am a big fine of [inaudible 00:29:58] get it on the wall, get it visible, let everyone, even your sea level looking at these things. Everyone loves a good graph, a picture paints a thousand words, as they say.”

“I think I heard Phil mention this already, with Microservices and the kind of graph request or whatever request comes in, how it gets found out, how it gets found out by services is really important. Things like OpenTracing for Google for OpenTracingAPI, and also Open Zipkin. Open ZipKin is an Open version of the Twitter Zipkin library, allows what you are seeing on the screen here which is effectively distributed tracing. Anyone who has done front end work with firebug, it looks very similar, it is invaluable. You need such correlation [inaudible 00:30:38] to correlate requests. With [inaudible 00:30:43] data store, pings it back, now you are going to have potentially multiple services dealing with the request, multiple data stores, third parties, and you need somewhere correlating all these things, and these are my kind of chosen technologies to do that.”

“Microservices fundamentally enable agility, and that is all about the feedback and all about the learning but you have got to do it well. If you really want to go for the whole build, measure, learn, like from the start up, you design for impact. You need to build in signals and metrics, you need to know like from a business level, from an architectural level, from an operational level, how are we doing here. From a business level am I making money, am I driving traffic, from an architectural level am I improving the quality am I breaking that Monolith, and from Ops what is happening in terms of traffic, queue length and all these kind of good things.”

“You need to drive a culture of experimentation and failing fast and this, for an enterprise, can be as much of a challenge as microservice technology. Changing to experimentation, changing to failure being accepted, is often a big challenge I see in organizations. The bottom line is if you do not collect the data and take action to adapt, you are really going to see limited benefit with microservices. Yes, you are going to get some architectural benefits and you may be able to have teams adding services, but the real benefit of microservices is agility in your business, your services, and your operations. That is where you get your biggest wins.”

“On to number three, I see we are not doing too bad for time, so, responsibilities in an organization are always really important. The buck always stops somewhere, but even sort of different to that, the key thing is we as developers, we as architects, we business people, we should know what we are responsible for. I think that is really key and I see a lot of problems in Organizations because people do not know who is responsible for things. They do not know who they should inform when things change and they do not know who to consult for expertise, that kind of thing. That is what this section is going to look at, that. [inaudible 00:32:41] standard microservice thing. Conway’s Law says organizations which design systems basically copy the communication structures of their organization. In the software world, basically it says that your coding and application is going to look like you. As a diagram here shows. I have seen this time in, time out. I can instantly tell by looking at code base pretty much whether an organization is siloed in terms of Dev/QA ops, I can typically identify teams within organizations, you know they have their own folders say in an PHP app, Conway’s Law is real and the reason it gets talked about so much is because it is genuinely super important.”

“Now what we are hearing is this kind of reaction to this, as a consultant for OpenCredo, I go in and help a lot of clients and I say ‘Oh yeah, I can help you with technology, I can help you with teams’ and people say Oh, no, do not worry about teams. We have got the micro team thing sorted. We have decided to reform our teams around squads chapters and guilds, and what I actually see is a squad is actually a Scrum team, rebranded, a chapter is a line manager and what a guild means is they are doing presentations once a month. Yeah, they basically have cargo cult spotify, and spotify are awesome. Please, do not copy things blindly. Like, Cargo cult is copying things without understanding the principles. You always have to understand the practices, the principals, and the values. It is much more important to understand why Spotify moved to squads, chapters and guilds than the fact that they did move to squads, chapters and guilds, so please don’t cargo cult things.”

“Cross-functional teams are massively valuable. Different companies move to them for different reasons. Spotify, all about the culture. Autonomous mastery, that kind of stuff. Amazon, famously, two pizza teams. Having no more developers in a team than can be fed by two pizzas, was all about communication. If you have got a team of 100, you have to maintain 99 communication links, if you have a team of 10 you only have to maintain 9 communication links. Guild, I love Guild, they did it around strategic alignment. They said ‘we have got these kinds of goals, these kinds of key performance indicators and these things we can change’ and they actually created these cross functional teams, and microservices actually emerged as in the microservice path and architecture emerged in guild’s technology. No one said they were going to use microservices on high, it just happened that way. Cross-functional teams, and all I mean by that, is full capabilities, right from a business person, a product owner, say, through to a developer say front end back end, an ops person, a QA person, the full stacks so you can take in requirements for the business and deliver it and manage it for its full life cycle, there is no handing off of tickets and all these kinds of things.”

“Cross functional teams, totally awesome. Do check out Spotify’s engineering culture videos, they are really awesome but look at the principals and practices more than the actual squads, chapters and guilds is my advice.”

“Devops. A lot of the time we walk into clients and they are sort of half adopting devops, or they might have rebranded some kind of ops team as devops and these things, but DevOps in my mind is a culture thing, it’s a tooling thing, and it is really important. It is all about shared understanding, shared responsibility and learning and improving feedback. There are two key resources I am going to recommend, Matthew Skeltin’s got this DevOps topologies, great website to check out. You can see the red title thing, the web page there, that is the bad patterns for Devops, so there is like a DevOps Silo, there’s the devops teams, the notion that dev don’t need ops, that kind of thing.”

“The blue patterns are good patterns of devops and I broadly agree with Matthew’s work, it is a really nice bit of work. It enables you to look where and see where your organization is, perhaps in the red, and it gives you some techniques and some tips to move perhaps to the blue where you are truly embracing DevOps philosophy. If you want to get more understanding of DevOps philosophy, up until now we have not had many good books on it, and I personally am a big book reader, I love books. These two books by Jennifer and Catherine, that is an awesome book, Effective DevOps gives the sort of Principles of DevOps. Google was fashioned SRE book is really good, a word of caution, it is from the Google black, they are pretty much uniforms, they are amazing people and I love chatting with the google people, but I have heard some people already try to Cargo Cult some of the ideas in the SRE book, and in their enterprise it just did not work. There are some wicked ideas in there, so read both of these books and take away lots of things but again, do not cargo cult. Look at the principles, look at the practices, really useful for driving the change required, the foundation required if you like, for microservices.”

“DevOps, for me, is a lot of it is about not only shared responsibilities but defining responsibilities. One of the key things at the moment I see people still doing is building their own microservice platform. You kind of got things out of that like [inaudible 00:37:38] that kind of stuff, Amazon has got all kinds of cool stuff. I say that is, my friend here Colin joked, if you start writing loads of your own stuff, it is very interesting, very fun, but does it actually have business value. Is it what you are meant to be doing, really? Focus on what matters. For me, as a developer of Microservices, it is all around CICD, Continuous Iteration Continuous Delivery, it is about mechanical sympathy. Understanding that I am now going to be writing my application as lots of little services, probably as distributed computing, probably working on the cloud. In the cloud, everything goes across the net, even your blockstore is probably going to be a sound across the network, so mechanical sympathy is understanding the fabric on which you are deploying your application to.”

“Logging and monitoring kind of fold off of that, logging is often used retrospectively to debug things, monitoring gives me insight to what is happening now, so both are really important things. Or rather all of them, I should say, are really important things.”

“I mentioned before about this notion of DevOps heroes, and please be careful of this. I saw a great talk by Charity Majors at [inaudible 00:38:36] this year, and she talked about interviewing, she hates the notion of Full Stack Engineers and DevOps often gets banded in that, for me, full stack used to be front end and back end, that is how I started my career, now full stack seems to mean front end back end and dev ops, it is all getting a bit silly in some degree. Charity made this point and she goes ‘I am sorry, but if you are not designing the computer chips, and designing the website, then I do not want to hear from you as a Full stack developer.’ And I thought ‘Boom, drop the mic really key point.’ The full stack culture, really great ideas behind the principals but the term is often being used by recruiters in a very strange way, even the whole DevOps thing.”

“No DevOps heroes, DevOps is about the organization as a whole, it is about the shared standing, shared responsibility, optimizing for Feedback and optimizing for learning and hopefully many of you have read things like the Phoenix Project. I have referenced it on the left here, but again, it talks about the culture of devops and it is great and if you are in an Enterprise and you are looking to bring in Microservices, looking to bring in DevOps, then the Phoenix Project by [inaudible 00:39:34] is a great book to start with, it is a novel so it is quite an easy read, but it talks about system syncing, amplifying feedback and continual experimentation and learning.”

“I often find in an organization, moving to Microservices, moving to DevOps, you need to make it explicit who is responsible for things. A bit old school, but I like a good RCCY Matrix. RCCY is Responsible, Countable, Consulted, Informed. Basically, you sort of say who is responsible for certain things, who is accountable, who do you need to consult when you are changing things, this is a really useful framework. RCCY is an old project management thing, but I often use it, it can be a crutch sometime to get people working, you know, you are moving to a microservice motor operations where there is inherently more autonomy, but inherently more accountability as well. Make it explicit up front who is responsible, who is accountable, those kinds of things. This is a nice technique.”

“Final furlong now, microservices, pretty much any technology change but microservices in particular, can profoundly impact the way you deliver software. I have seen a lot of success in microservices, seriously. I have also seen a few not so successful projects as well, Microservices create change, and often a lot of people actively resist change or not even, you know, consciously sometimes. Managing change, managing the way microservices are brought into an organization, is really important. I mentioned HBR earlier, I take a lot of my cues from HBR, but change in management is essential. Things like fair process, engaging with people and why should we move to microservices, what are your pain points, explaining why decisions have ultimately been made after you have engaged with people and setting expectations, we believe microservices will give us these values, we are going to complete migration in six months, 12 months, a year, whatever.”

“Leading change is really important, transformation is a process, communicate, plan, evaluate, really important and always get buy in from the top. If you are bringing in microservices, pretty much it is a C-Level, make sure you get enough time, enough resources to make the change.”

“This final slide, leading change is really important. There is a great book by John Cotter here, it talks about establishing some sense of urgency, creating guidance coalition, be it DevOps, be it Microservices, this is a really useful book to read in order to drive organizational change.”

“Final slides, think about these things, yeah. If you are bringing in microservices, it is going to require strategic change. It is going to require more feedback, more visibility, more learning, you are going to have to make sure that responsibilities are defined. There is so much out on the net you can learn from, and people have been talking, people like Phil, people like Lori, all of the [inaudible 00:42:20] crew, Netflix, Amazon, Guilt, and they have been talking about these things for a long time. Read it, make up your own mind, try to sift out the principals and the practices rather than blindly copying what someone else has done. DevOps done right is also a prerequisite for Microservices, but do not just copy what you think DevOps is, Do the research, see where your pain points are in your organization.”

“Couple of final recommendations, these are my kind of go to books, I like to read. Sam’s and Everhard’s are really good around Microservices, API’s super important, Architecture, more Visibility [inaudible 00:42:54] and the art of business Value, both superb books about understanding the business benefits that you are trying to drive with Microservices. Things like [inaudible 00:43:04] and the kind of philosophy of Microservices. The final shameless plugs, I did a talk at InfoQueue and also Queuecon in new york a few weeks ago, What I am seeing is the seven sins, the antipatterns of microservices, so that might be of interest to people, [inaudible 00:43:24]. Do check that one out. On that note, I will say thank you very much and probably take a few questions.”

Flynn: “There we go.”

Daniel Bryant: “Hey, Flynn.”

Flynn: “Hey, I am Flynn from Datawire, Daniel, Can you hear me?”

Daniel Bryant: “I can hear you, can you hear me?”

Flynn: “I was going to ask you to be a bit opinionated on a couple of things if you have got the opportunity.”

Daniel Bryant: “Yeah, roll on.”

Flynn: “Some of the questions that we have from before dealt with things like favored strategies for release of microservices in a complex environment and data persistence, and all those sorts of things. You seem like someone who might have a thought or two on the matter.”

Daniel Bryant: “What I would say, is it is a classic instance on that it does depend.”

Flynn: “Well, yeah.”

Daniel Bryant: “Definitely on the context. What are the two things you mentioned?”

Flynn: “Let us pick first on release strategies where you have a web of microservices, some of which depend on each other, depends to favor going and trying to have these things completely independent? Do you favor release trains where you bundle some of them together, do you look at the organization and just say you know it depends on the way the teams are structured, and, again, this is asking for your opinion rather than a cut and dry solution to every scenario.”

Daniel Bryant: “Yeah, got you, got you. So, I will do the drop the mic thing first, my over opinionated thing, and that is unless you are ultimately, dogmatically, unless you are deploying services individually, what you have got is a distributed monolith, I see this sometimes where people effectively package all their microservices together, rubber stamp this is version one, and version one is actually means version 1.2 of that thing, 1.3 of that thing, and then they do this big bang blue green deployment. To be fair, that does work sometimes, but taking my cues from Sam Newman, and Martin Fowler and James Lewis and a bunch of other people, yeah, you are going to be paying what we call summer tax, you are going over the wire with your communication, and at the same time you are doing like a big bang which you get with the Monoliths, so you really get no benefits, or very limited benefits.”

“I do strive for individual processes, individual deployments, and I use things like consumer based contracts to enforce like local verification, I use things like automated smoke testing. Guilt team have talked about this a lot, they got so big that Guilt does not even have staging now, they do all their testing in production. That is an advanced pattern, I am not necessarily recommending that, but you know, the sheer volume of changes, the pace of evolution where you get sort of above ten, twenty microservices, you need to think differently. Localized verification, CMS contracts, and also end to end smoke tests on the critical path, that sort of thing.”

Flynn: “Anything else that, any extra caveats that you would like to throw at it or would you like to leave the mic dropped on that one?”

Daniel Bryant: “Yeah, I think that, what I often say is that often what you will see people talk about, me included, is we talked about the end state of things. You will hear people say things need to be individually exported, like I have just done, basically, but do not worry if there is a journey to that. For example, I have done the rubber stamping thing, I have done semantic versions and [inaudible 00:46:33] and on things. Just to get it started, but we always know the principals are independent deployment. As much as I think my mic dropping might be a controversial thing, there, do think about the end goal, where you want to get to, but realize where you are and realize there is a journey in between.”

Flynn: “Right. Yeah, reality does have a way of intriguing us on that.”

Daniel Bryant: “It does, indeed, yeah.”

Flynn: “The second one, let us ask about persistence. At one point in your talk you mentioned … you were describing the Monolithic database being the spokes stuck in the bicycle wheel, the stick stuck in the bicycle wheels.”

Daniel Bryant: “Yeah, yeah.”

Flynn: “One of the things you mentioned in particular was that microservices tends to own their own data. With that, are you going as far as Microservices all have their own storage that belongs to them exclusively, or, again, what is your opinion on that one? How do you tend to approach persistence in the microservices world?”

Daniel Bryant: “Yes, so it is funny, actually. The canonical version of a microservice is they manage their own data store, yeah. For example, you can deploy, they can have one database server, and they can have individual schemas associated with microservices, it does not mean you necessarily need to run like twenty versions of Mysequel, or you can have one mysequel but twenty schema, for example. I think that is the kind of canonical version of microservices, that is the kind of end goal.”

“If you do not, if you share the database between services, it becomes a communication mechanism. People start cheating by using the database as a communication mechanism, rather than by going by an API or a message queue. You always try to minimize the things that can be used for cheating. It also makes things like migration much easier, like classically like monolith, is when you do database migration, you upgrade it, and everything gets upgraded. If you split things out, you can isolate the migrations of stuff which is really important.”

“In reality, a lot of companies invest so much in their monolith into their database, it is really hard to change it first. Just last week I was saying to this company you know, the canonical version of microservices, they have their own datastore and we looked into their datastore, and it was an MS[inaudible 00:48:43] database with loads of triggers, loads of stored procedures, and I realized that it would take too long to start breaking that up, so we are actually looking to build microservices on top of the database. Some people would say that is heretical, that is not microservices, but what we are going to do is we are going to split the application code up then try to bring up the triggers and the store procedures into the code and then into the database.”

Flynn: “That just seems like a practical concession to the complexity of that database. Where you start moving the complexity out of the database and into services where you can control it more easily.”

Daniel Bryant: “Exactly that, Flynn, the worst case I have seen some people try to do is like, be so dogmatic that we have to split the database and they try and rip things out. It is like trying to rip out the guts of an application, everything breaks in the logic tier, in the middle tier. What you said was perfect, exactly right, but people, myself included, is we do get a bit dogmatic about things. Microservices must have their own data store and I think you have to, like you say, be pragmatic and say ‘This is just an evolutionary change to the ultimate good part.'”

Flynn: “Right, right. It tends to make things more helpful anyway.”

Daniel Bryant: “Yeah.”

Flynn: “Last bit from me, I think, then, would be do you have any particular success stories or possibly any particular failure stories around monolith to microservices and strategies that either worked really really really well or were just dismal failures?”

Daniel Bryant: “Yeah, I do. Genuinely I have seen some businesses struggling and I cannot mention their names, for obvious reasons. Typically, we at Credo work with them to fix things, I have never seen something we cannot fix yet. We have advised some people to try and approach things differently, perhaps, rather than go all in on microservices but queuing up the next talk, I think Nick Jackson [inaudible 00:50:43] is next, and I actually worked with Nick for over a year, at OpenCredo and Nick is actually one of my success stories. Indeed, so Nick and I have actually talked and if people are interested they can google ContainerShed, it was a conference Nick and I presented at about a year ago, and we actually talked about the not on the high street journey. I think maybe Nick was talking a bit about that, but Nick may be talking more about service discovery today.”

“Nick and I and all the not on the high street team, it was fantastic effort, 40+ developers, four people from OpenCredo, we worked on site with them for a year and it was a Ruby monolith, which got split up and ultimately into Java, Go, Swift, Ruby microservices, moved to the cloud, brought in some docker, it was a massive change that was all driven by the Not on the high street requirements, business and technical environments, and I learned so much there. It was a really fun time and we have talked about it publicly, there are blog posts from Nick and myself, there is slides and talk presentations online, so that is my canonical case study of –”

Flynn: “Success.”

Daniel Bryant: “Stuff.”

Flynn: “All right, we have got one other question from, actually I will take that back, we have two other questions from Bolero that I would like to try to get to. The simpler is probably the first one down below about what day did you have it suggested that earlier majority being the first term of microservice adoption?”

Daniel Bryant: “What I was talking about with early majority there is that there is this sort of well accepted diffusion innovation curve, and people [inaudible 00:52:15]. Like netflix…”

Flynn: “You are breaking up.”

Daniel Bryant: “Oh, sorry. Can you hear me now?”

Flynn: “Yeah, I can hear you now, so people like Amazon, netflix…”

Daniel Bryant: “And Gilt. They were very much innovators in the microservice space.”

Flynn: “Right.”

Daniel Bryant: “The technology wasn’t there, they had to code a lot of the stuff themselves, this kind of thing. That is very different, and the early majority of people now are your bigger Enterprises. I am not saying everyone is doing microservices and definitely I am not saying everyone should do microservices, be very key on that because there has been some controversy on twitter the past few days around some things there, Microservices are fundamentally, it is just an architectural style. Just like [inaudible 00:52:57] is, just as ATL is or whatever, but like I mentioned with the lipstick on the pig, you have always got to evaluate these things, but we are seeing because of people like Netflix, and Guilt and Amazon, and On High Street talking about the success that microservices has brought them, a lot of the people that are not comfortable with innovating within the technology space are now looking to adopt the lessons learned from these other companies.”

“Things like there are platforms emerging, like [inaudible 00:53:26] Mocker’s really popular now, there is technology around now that we can leverage to make our journey to microservices much more easy in comparison to Netflix who practically built their own code because they had to, and they have been very generous and shared it. I think that question there says early majority suggests fifty percent of all [inaudible 00:53:44] I do not really have any data, this is mostly anecdotal, there might be data out there I am not sure, but it might be a bit early for data. Anecdotally, companies OpenCredo work with range from Start ups to I cannot mention names but they are quite large or very large organizations and we are seeing questions that are can you move me to dev ops, can you move me to Microservices, we are stuck with microservices, please help me, is the best one.”

“We also see a bunch of things around MYSQL, people are saying I cannot work [inaudible 00:54:13] effectively, and often we find people are treating these new MYSQL stores like [inaudible 00:54:21] databases. Cassandra is freaking amazing, but if you treat it like MYSQL, you are not going to get performance. They are our balky bread or butter things at the moment, and we are happy to help anyone with these kinds of things but we are seeing a lot of enterprises coming to us right now and they just say ‘I want microservices.’ We always push back and say ‘Why? What are you actually looking for, is it developer productivity blahblahblah’, well I should not say blahblahblah because it is actually really important. We always say why, why, why, and ultimately something pops up, you know, we had a lot of incidents last year, we are looking to embrace new technologies that would help hiring, all these, some better than others, kinds of reasons. We always push back and say ‘Why do you want microservices?’

“Fundamentally, most do decide to go into microservices in the end, just because of the potential benefits you can get from it.”

Flynn: “Yep. That makes sense. The last one is, the last question actually, about RCCI. As I read that, I think the way I would summarize that would be to ask you to talk a little bit about how far you like to go with nailing down roles and responsibilities as opposed to … well, yeah, let us just leave it at that. How far down that rabbit hole do you like to go?”

Daniel Bryant: “Yeah, it is a challenging one. Again, it is a lot of the work that we do, I am very privileged, I get to go into a lot of companies, but it is all about context. Some companies are really bureaucratic, and they thrive on rules, so I may be more tempted to go more all in on these kinds of things, other companies, like Not On High Street for example, very autonomous. There I dialed back on the recommendations. What we see, whenever you do a new technology, be it DevOps, processes, tooling or microservices, new things pop up, new challenges, who is going to be responsible for the migration of database scheme, for example. Who is going to be accountable in case an API schema change breaks a downstream microservice.”

“My golden rule, in life, in general, is I favor visibility. My leadership style is I let everyone see the data, let everyone see as much as they can see the code and so forth. Then apply very lightweight process, kind of guide rails, I am not a fan of bureaucracy, I am not a fan of dictatorships, so I start very light by simply saying that this person in the ops team is perhaps responsible for data stores or something, or this kind of person in ops in that team is responsible for this, and if we start to get lots of incidents, we do pool operation teams together, we say ‘Jane, you are responsible for those sequence stores, Bob you are responsible for the deployment pipeline’, these kinds of things and it gives some kind of notion of accountability.”

“Not as a blame thing, but just if something is going wrong, who do we go and see. More often do I see when stuff does go wrong, people tend to run around and go ‘oh my god everything is on fire’ and everyone jumps on board, tries to fix it where if we knew it was just one person, we could rally around that one person.”

“I reckon this is a blog post, because it is hard to brain out my thoughts without seeming too dictatorial or whatever, I have just generally found the RCCI thing as quite a useful tool for providing some lightweight guidelines. Around who is responsible, who should be consulted.”

Flynn: “The thing I always liked about it also is that its separation of people who are responsible and people that need to be kept in the loop and not distinction that seems to fall to the wayside a lot.”

Daniel Bryant: “Totally, yeah. Keeping informed is a classic one we see, dealing with microservices, a lot of things will be changing all over the place. Knowing who to keep informed by having some kind of communication mechanism to share these ideas is really useful, actually, yeah.”

Flynn: “Great, all right, so, finally. The last thirty seconds here, I know I said this previous questions was the last one but one of the other ones has suddenly got a lot of focus, about whether you think Rest is here to stay or if we should just chuck it out and replace it with something that actually works.”

Daniel Bryant: “Yes, I am going to copy I think what Phil said, I was making a cup of tea when I heard him talking about it, but I often start with Rest, keep it simple in an organization, I like Rest for the front end because it is highly compatible with javascript and mobiles and so forth, but we do love to know our options, so things like GRPC, I am a big fan of GRPC, and a big fan of Thrift, GRPC more as of late actually. Know your options, and if you are sort of struggling with performance, a binary protocol is nice, something like a GRPC, a GRPC is an interface definition language, so it gives you a stricter contract which can be really nice as well, it can also be a limiting factor so that is one thing to bare in mind. Rest is super flexible, you know, rest you can [inaudible 00:59:11] law kind of thing, be conservative with what you send and liberal with what you accept, that is harder with a binary protocol.”

“By its very nature, binary protocol you can do changes that, potentially, can break other things easier in my experience. I think that is a great question, I have a feeling like the next X number of years, you know, pick a number, no. I think it will always be there, we are already seeing things like HTTP 2 and things like that, and speedy which is based on the google speedy protocol, so there is a lot of benefits with HTTP2 now and blocking got rid of and compression is much better, so I think HTTP is such an amazing protocol, it has been around for what, twenty some odd years or whatever and it is a great building block to build other things on. Rest just builds on HTTP. When Rest is used right, I see it being a great tool, it is always nice to know your options as well. GRPC, Messaging, Kafka I’m a massive fan of Kafka, Phil mentioned that, big shout out to Kafka we use that a lot as well for a whole bunch of things, streaming data is super important. Anyone who is into that look at Apache Beam and stuff like that, with them coming out of google, I think.”

“There will always be other things, but I think Rest is a great request response kind of mechanism.”

Flynn: “Cool, I think it will be very interesting to see what sort of uptake HTTP2 actually gets in the field and how quickly that happens.”

Daniel Bryant: “Yes, indeed.”

Flynn: “Well, thank you very much, much appreciated. Excellent. I am sure we will be in touch on twitter after words, and a couple of other thing I definitely going to point you towards, and we are on to Nick Jackson.”

Daniel Bryant: “Thanks guys, Cheers.”

Flynn: “Thank you.”