Mosaic Microservices at Zalando Tech

Austin Gunter
“All right we’ve got Dan Persa and Lauri Apple from Zalando Tech. Going to talk about their journey in micro services now. Without further ado, I’ll let you guys kick it off.”

Laurie: “Yeah.”

Dan: “Yeah.”

Laurie: “I’m going to use this button. That’s Dan, and that’s me. Dan is … you want to tell them what you do here?”

Dan: “Yes. I’m a software engineer at Zalando. I’ve been working for Zalando in the last three and a half years and in the last year I have been working at the project called project Mosaic, and that’s what I’ll be talking about today.”

Laurie: “This is me. I’m Laurie Apple. I’m a opensource evangelist at Zalando and also doing some work now with a team near Dan, so I’m say the agile project manager for short, so that makes sense to everyone. I’ll be talking to you a bit about our ride towards [inaudible 00:01:21] services and then a bit about what we’ve seen on the open source side, since we’ve gone in the micro services direction.”

“I’ll tell you first a little bit about Zalando. I’m American and I know many Americans might not know what we do or have ever heard of us. Our blanket statement is that we connect people with fashion. What does that actually mean? It means a lot because Zalando is a pretty huge company and we do a lot of different things with fashion and offering fashion to the world. Specifically … we were founded in 2008. We’re up to more than 10,000 employees and more than 1200 of them are technologists, and they work across 7 different tech hubs in Europe. This map shows you our presence that we were huge. We are in 15 European countries, with more than 18 million customers and 19 million downloads of our app as of today. We’re publicly traded, so we’re headquartered in Berlin, Germany and we also have tech hubs in Helsinki, Finland, Dublin, Ireland and then a couple others in Germany.”

“We’re the fastest European company to reach a billion in revenue. That’s something like the [inaudible 00:02:49]. In our early days, so how was it in our early days as a technologist and what were we working with? We worked with a pretty limited staff as many start ups do. This monolithic, based on Magento PHP, MySQL. We hit the wall in about 2010. In that year we started to really grow as a business quickly. In about three or four months we completely moved everything to Java and we got our own system. Java, Spring Postgres, that was a data base. This worked for awhile. We added new features to the stack for 5 years. It went okay. Then we hit the wall again, because we grew even faster and bigger.”

“We were relying on this Monolith for way too much, so we saw that just having this monolithic applications, started to hinder our productivity. With all the dependencies, the coordination, laws and [inaudible 00:04:07]. We saw all of these signs as we tried to work around this. On innovations we started to have a lot of side effects. A lot of maintenance. A huge code base meant that bugs became dense. Finding them was harder and fixing them was even harder. The processes became rigid and we were taking care of the monolith, which meant that we couldn’t innovate as quickly and agily. Even in terms of [inaudible 00:04:44] was a hindrance. Slow to onboard people because we had to show them how to work in this environment with this huge code base. The tech stack got kind of old, and Java is great but we couldn’t really intro [inaudible 00:04:59], efficiently. [inaudible 00:05:01] made it a lot less interesting to work here, so it was harder to even hire people because they wanted to work in different languages and learn. That’s what we do as technologists.”

“In spring of 2015, we officially launched this mindset or management strategy or whatever you would like to call it. We call it radical agility. This is a nice drawing we have to illustrate what this means for our teams. It takes Daniel Pink’s three pillars from drive which is a pretty popular book on innovation and what creative people like to have as their work environment. We base radical agility and our tech management strategy on these three pillars of purpose, mastery, and autonomy. Then you have thrust to hold everything together.”

“On the Autonomy side, we want to have autonomous teams deliver amazing products efficiently at scale. Autonomous teams means not that they can go to a cave or something and completely ignore the rest of the organization, but that they can make their own decisions about the language that they use. Sort of other supporting technologies that they want to apply, and start delivering. We found that this way to have autonomous teams, was really one of the ways that we can try to counteract Conway’s law, which you’re probably all familiar with. We saw that with the monolith, this structure started to basically hold us in place. With autonomous teams we can subvert this trend but constantly challenging the way that we organize and keeping it fresh.”

“With Radical Agility we want to have a purpose driven organization composed of these teams working, ideally in as much as synchronized as possible and this requires a lot of communication. That’s how we can then deliver products have clear definitions and deliver on time. One of the main prerequisites for micro services is Martin Fowler who’s identified as rapid provisioning. Rapid application deployment and basic monitoring. These are all things that we have been working on since we introduced radical agility. I should say that with radical agility we also introduced 5 rules of play, which are technical decisions that enable us to support those pillars. One of them is having an API first mindset. Another is going towards micro services in every possible opportunity.”

“When you go into this kind of micro services environment you have to expect some level of failure, because a lot of teams aren’t acquainted with working in this environment before. They may not also be familiar with the responsibilities that it takes to actually own their micro services. When we say end to end responsibility, we really want the teams to own their micro services so anybody on the team ideally can answer questions about the work that’s being done. It’s not, you’re the front end person over here, you’re the back end person over here, and you’re the project manager. We want to have that familiarity with a bold purpose, as a balancing even across the team as possible. Product also needs to be aware of this, and we need to have engineering and product being the same as well.”

“As an open source evangelist, one of the things that I see since radical agility is being introduced, there’s a lot more freedom from the teams to deliver open source projects. Suddenly the diversity or projects they can work on is greatly expanded. They’re not tied to specific languages anymore. Have at least 6 in use and in development currently. This enables for a lot more diversity of what the teams work on, how they work on it, and then also being able to outreach to the specific language communities and get contributors that way. It also means with end to end responsibility, that micro services in force reinforces, the teams are starting to develop more of a product mindset you might say. They want to deliver something that’s useful to the community, that the community could easily be installed. There’s no dependencies internally. They can easily start contributing.”

“One of the ways that we guide this development along is by developing … or one of the ways we have guided it is by developing these open source first principles. We really want to encourage teams to think of their work as something they could actually deliver publicly as much as is possible. As an eCommerce company, we have some tactical considerations. We’re in Germany so privacy is a key topic that we need to talk about and think about. We have pretty strict privacy laws. We have to make sure that anything we release does not allow private customers data out into the world. We also have to be audit compliant because we’re now publicly traded so in addition to the customer data and regular audit compliance requirements we have as a business, we also have to take extra care to make sure that we don’t fail in these times.”

“We can’t open source everything, but we do try to open source as much as we can that is useful to the public. These principles reinforce that by encouraging developers to think about open sourcing their work from the start. This can help them develop really strong, solid independent projects. We encourage them to take ownership, so this goes back to the end to end responsibility. You make sure that you meet compliance and security guidelines. You make sure that your code is of high enough quality. You make sure that you have delivered something you’re really proud of and that somebody else can use, without too much friction.”

“I talked about the safety and the quality aspects, security aspects, also providing documentation. When you’re working in a micro services environment as well, if you’re teams are trying to understand how different, what you have, what you’re working on, having good documentation enables them to do that. It’s a little bit different from the monolith where everybody is kind of on the same team. You want to have clearer cues for other teams to be able to ask what you’re doing. Can they actually work on the same micro service, or spin off a different version of it. Those are really important.”

“Asking for help also. Like a said, micro services architectures require extra communication so teams can find out what they’re doing, find out what’s happening and stuff. Hopefully we can share code that way so they avoid duplication’s and just work better. Asking for help can be a really good way to find out what’s happening around you. We ask people to promote their work as well. We have an open source guild where we come together every two weeks, a lot of us from the team and talk about the kinds of projects that we’re working on.”

“Open source activity has accelerated since we started working more in the micro services mindset. This is the top, above the fold image of the open source dashboard that we have. This tracks different statistics from all the repo’s across our Zalando organizations. You can see the search function shows some of the language diversity that we’ve been able to support since moving towards micro services. That’s not a complete list. We actually have as you can see the statistic, about 32 languages so far. Clearly for enriching the diversity of your tech environment [inaudible 00:14:02].”

“This is what that dashboard shows. It shows you the diversity of language and even the diversity between them. The projects that the teams are working on speak to different supplementary learning and different specifications so we have the gamut of pretty much what’s happening in the industry represented in our portfolio. Now I’m going to hand it over to Dan, because he and his team are working on some of our key front end targets in the micro services.”

Dan: “Yes, thank Laurie for doing such a nice first part of the presentation.”

Laurie: “Thanks.
”
Dan: “In the next meetings you have to get used to my Romanian accent. Let’s begin. As Laurie says, Zalando had lots of departments. Here is a picture of them. Obviously this picture is really simplified and on top you can see that we have several sales channels like shop, mobile lounge and then we have lots of other services. Back end services connected to each other. As part of the fashion store department today, I will concentrate on our new architecture in the fashion store. We’ll see how these new architecture relates to micro services.”

“More than a year ago we started to migrate our monolith to microservices. Every company with a code base older than a few years has a monolith, and our monolith is so monolithic, that we gave it a name. We call it Jimmy. As I said, we started to migrate Jimmy to micro services a year ago and right now we migrate our first big feature which is check out. There are still lots of other features to be extracted out of Jimmy. In Jimmy we have thousands of Java classes and documented features and business logic on all of the layers of the system, including the data base, and we have no clear ownership of the code and the features. We have product people, [inaudible 00:16:31] and now nobody knows and really care about those features. Engineers can’t remove those features, for different reasons like CEO.”

“A few years ago, we used to be a big shopping, and now with radical agility we have lots of small things. What we want is to be able to add even more. Of course it’s obvious for us that going forward with Jimmy is not possible, as we are not able to scale the number of things we do have and allow them the things that radical agility promises. To be independent and autonomous.”

“On the other hand, Jimmy is generating lots of money for the company right now. We got to be successful because of Jimmy. Our business has a constant growth year after year and having the revenue of 3 billion Euro last year is not something to ignore. The question is, why would we change something which works? What are the goals behind these changes, reaching from new architecture is rather a huge investment. Where in migrating the engineers will be busy with the migration, instead of adding new features to the users, and you don’t do that only for the sake of being trendy. In the end what we want is to have more benefits after the migration than we initially started with. What are our goals? What do we want to achieve here?”

“First of all we want to be able to scale our architecture. We want many things working at different features in parallel without interfering with each other. We want team autonomy. In Jimmy this is of course not possible. We are bound to weekly release cycles. We always have code translates with other things and there is no real code and feature ownership. We also want programming language diversity. We want to use the right tool for the job. We’ve started with PHP some years ago, then we continued to be a Java based company and now we are putting more and more languages into production. Also, adding and removing features really fast drives the product forward. Experimenting with features is a great way to innovate and to drive the product forward.”

“We also want to be able to scale our tech organization. What does this mean? We want to attract new talented people to grow our team. Right now we have plenty of interesting products to build and we want more engineers to be able to build all of those services. Of course it doesn’t make sense to hire more engineers if you have more people leaving the company than were coming. Team happiness is also something we care about. Putting somebody to spend 4 hours to deploy our monolith doesn’t make anybody happy, or having our front end developers waiting for Jimmy to compile because Jimmy is a Java based monolith so we need … it needs a few minutes to compile everything.”

“We also want diverse people who are able to see the same thing from different perspective. The more ideas you have, the more likely it is to find the best one. Diversity leads to innovation. We also want to actively encourage innovation. We want to change our architecture. Now a days, if you have any question about product, or architecture there is only one answer. Microservices. It’s just impossible to ignore this trend. The question is do micro services solve the problems we have? Let’s look at what micro services promise. They promise to work autonomously to be able meet different tech stacks and independent release cycles.

“This certainly sounds good but the biggest part of Jimmy, is the front end part. Do these apply to front end also? Let’s see what options we have here. Let’s assume that we want to avoid using micro services on front end. The back end side, well this is easy. We address this by creating back end API’s owned by teams. What do we do with the front end? One possible solution without micro services would be to have a front end monolith. As teams own APIs now, we can call those APIs from a more light weight front end monolith. Maybe written in note instead of Java. In the end we’ll have the web app with contributions for different teams. For multiple front end teams. Let’s look at the things we are trying to achieve.”

“Are front end teams working autonomously? No they are not. They are still dependent on each other. Can they use and mix different tech stacks? Hardly. Can they deploy independently whenever they want? No, they are still bound to a release cycle of front end monoliths. It seems that avoiding front end micro services and solving the challenges you have is not an option for us. We somehow have to make it work. Right now, meet the Mosaic project. Micro services on the front end. Mosaic is a set of services and libraries together with the specifications that defines how it’s components interact with each other to support the micro service style architecture for large scale websites like we have. While the composing of back end into micro services is a widely adopted approach nowadays, to achieve flexibility in development and in operations, most front end solutions are still running as a monolith application.”

“Mosaic addresses this issue by using fragments that are served by separate services and are composed together a drawn time according to some template definitions and we’ll dive deep into this a little bit later. As a result, fragments can be iterated on very rapidly and be more flexible in technology choices and can benefit from the extreme development base of today’s front end technologies. As we have this fragment of course, we solve the problem of back end with having a typical micro services there in which we put API’s and we’ll talk about this a little bit later also. On top of this fragment, we put a layout service which is able rid some templates and base on those templates, they’re able to put together this fragment.”

“The assembled content is streamed to the client through a router and we have names for all of these components as they are part of the mosaic project, so our router is called skipper. We have inkeeper as a route management API, Tailor is our layout service and we have Quill as the API for managing the templates. Here is how we use mosaic at Zalanda. We have a fit of skippers and while skipper, our router is the anti point for every request from the browser, excluding the static resources. Skipper acts as the reverse proxy and will delegate the request based on some internal routes to one of our new micro services or in Jimmy, on the duration in which we do the migration. As I said, we have a fit of skipper instances and it’s important for all of them to have the same routes at the same time. In the job of Inkeeper, another micro service could do this. Next we have Taylor and Quilt. Taylor is our layout service who’s calling other micro services which provide HTML. I already said that we call those fragments, and put everything together and returns response in the form of a stream. Taylor calls Quilt the template management API to retrieve the templates.”

“Let’s go a little bit more in detail with each of the components we talked about. First of all Skipper. What Skipper does is forward requests to different end points based on request properties like the host, the path, wait a second. Based on host, path, method, cookies, etc. and it also adds security feature. It decrease and increase sensitive cookies at the edge. Also provides XSRS protection. It streams the content from the end points and is able to do round time updates of it’s routing table, and does the [inaudible 00:26:06] have a URL for it as Skipper is an open source component. In order to define routing Skipper, we use the Eskip routing definition language. Eskip is a DSL for describing route expressions, route definitions and complete routing tables.”

“Let’s take a look at one of these. We have route for catalog, which has a path so this would match everything which has category at the end, and would forward the request to the catalog example.org beck. With the product page is something similar. We have … we are able to match something there and we have another one, user account and in the end we have a more interesting route. Note fun which has as the matching part a star which means catch all. Everything which is not handled by the previous route would be caught by this route and what we are doing here is applying a filter, remote path which will rewrite the path to the note fun HTML and we use the static page in order to render something.”

“Here are the predicates for Skipper. The predicates are the first part of the route. We have here an example so we have a path to an API and a resource and we have the heather which is an [inaudible 00:27:41] heather. We separate all of this predicates with a double end. In order for a request to be matched, all of the predicates have to be matched. We have lots of built in predicates for pretty much everything around HTTP requests, so we have regular expressions for a path. We are able to match on the matted or on the hatters or on anything. It’s pretty easy to create new predicates for Skipper. The second part of a route are the filters. Filters are used to augment the incoming request and the outgoing responses or to do other useful stuff. For example setting the response or serving some static resources. With filters, we kind of are able to manipulate the request. We are able to change the status to redirect and lots of other stuff. Of course the Skipper filters are also easily standable. It’s really easy to create new ones.”

“Skipper is flexible and open and it can be used in a lot of many ways. As an API gateway to spill the mobile and desktop traffic. We can speed the traffic based on the version heather. We can do [inaudible 00:29:14] authentication reverse process, so we can put Skipper in front of certain service that doesn’t support Owans and redirect on authenticating request to another end point for example. We can do weighted traffic control, use the same route with different end points who support [inaudible 00:29:34] cookies, and it can act as an HTTP server or an HTTP compression proxy.”

“Inkeeper is a route management API for Skipper. This is like the next component which we have. When a new instance of Skipper is started, it will connect to Inkeeper and ask for all of the routes and initialize it’s internal route stable. Then every few seconds, Skipper will pull Inkeeper for modified routes and updated internal table structures. What’s the reason behind having this API? When we initially started Mosaic, as we had just a few routes, we started to use ETCD as a storage for routes. ETCD in case you don’t know is a distributed key value storage. The next step was to allow the teams to push Eskip definitions, the ones we just saw in the previous slides to a [inaudible 00:30:32] repository. The routes were taken from that repository, by a [inaudible 00:30:39] job and pushed to ETCD from where Skipper was able to pick them up.”

“As our architecture evolves and we start migrating more and more parts of Jimmy to Mosaic, we feel the need to allow the things to manage their own routes. One reason is of course the team autonomy. Right now the teams depend on our team to do the reviews for those requests and merge them. Now the reason is that a gift hub repository doesn’t really scale. We’ll soon have teams pushing thousands of routes for their CMS’s and of course, we want to … we want the teams to put so many routes but we don’t want those routes to collide with each other. We want to identify if one route tries to override another route, and of course it’s kind of hard to do this without having a center point where you can push the routes. The API of Inkeeper is pretty simple, and what we have is path.”

“Each team’s path belongs to teams. Each team can create a lot of these path, and for each path is able to create routes. What Skipper does, we have two end points which are specifically designed for it, the current route, where as Skipper is able to retrieve the whole route table. The whole routing table and we have the updated routes on which Skipper are able to get a data of modified routes. That IDUFC and not making routes is actually a date and what date that it signifies is that while it gives me all of the routes which are modified since, well that date. Another component … Inkeeper is also open source.”

“Another component in our Mosaic architecture is Tailor. Tailor is a layout service that uses streams to compose a web page from fragment services. Loads the content of all fragments from the template in parallel and offers nice error handling and fall back features. Let’s see how this works. We have a request for Zalandod/cart from Skipper, and Skipper knows that this URL should be handled by Tailor and forward the request to it. Tailor needs a template for the /cart URL and in case it doesn’t get cached, it asks Quilt for the rest of the API which is responsible for managing the templates. Tailor gets a template, looks at it and sees that it has to call the header fragment, the cart fragment and the tracking fragment. It does all of those calls in parallel, puts everything together and returns a response to Skipper as a stream.”

“It doesn’t wait for everything to finish but streams as soon as it has something. We’ll see a little bit later how a template looks like. The fragments are the micro services developed by the different teams, so looking back to this slide we have the header fragment, cart fragment and tracking fragment which are composed together by Tailor. What’s happening inside of a fragment, we can render HTML for Tailor, we can return ajax requests for Skipper can pause fragments directly, and the fragment can call some other API’s. Of course none of these actions are mandatory.”

“Here is how our Tailor templates look like. They are … what we did is I got lots of inspiration from Facebook’s big pipe but we changed it a little bit to optimize it for CEO. What you see immediately is a primary thing. This is called the primary fragment. What Tailor is actually doing is it does request to all of the fragmenting parallel, but it starts streaming stuff only after it get’s 200 response from the primary fragment. We also have another peg they are acing, so those are the kinds of pegs which doesn’t have to put in order. For example, the tracking fragment can answer a little bit slower than the others and this acing fragment will get to the base through Java script, instead of being rendered and streamed directly on the server side.”

“How does it look together? We have the header fragment, the cart fragment, the tracking fragment and the base assets fragment, which renders some techs from Java script and CSS. Not every fragment has to be visible. We can have some invisible fragments like the tracking fragment or the base assets fragment. One more important thing here. We also want to have a consistent user experience. What this means, is that our customers should think that our fashion store was developed by one big team, and by lots of small teams like we have. Shaker contains tiles and components specific to Zalando, and because of that it’s kind of closed source. If somebody else would like to use Mosaic, I see they creating their own set of components so they would provide their customers consistent user experience. This is a really important thing which a shop like us has to provide.”

“The Mosaic architecture solves half of the problem. The browser facing part. How do we solve the back end problem? As well this is a software architecture problem. The answer is simple. With more micro services. In this case restful APIs. Our APIs need to last for a long time. API’s can only be involved in certain ways, and bigger changes are really hard and expensive. Our API’s are the most pure expression of what our systems do, so we prefer our API’s to be designed outside of the code, and be reviewed by peers before starting to implement them. One of the reasons of API first is to decapulate the [inaudible 00:37:48] from technology.”

“Each team has it’s own tech stack. They can choose their own programming languages. The common ground for communication between teams are API’s. Of course we use our APIs for communicating outside of the company also. The Mosaic project together with restful API’s solves the problem of having many things working in paralleled at different features without interfering with each other. We also solved the language diversity problem. Each API can be implemented in a different language. Fragments can be theoretically implemented also in any language. I say theoretically because well as you want to have several side rendering usually and react is like, a really good choice for that. Note JSC is the easiest way to go to when you want to implement a fragment. Something which renders stuff.”

“Are the micro services the silver bullet? Of course not. They fit our use case. By using them, we will be able to scale the number of teams, make our developers happier, create language diversity, allow the teams to innovate more, and be more autonomous. This doesn’t mean that micro services will fit every case and every company. I mentioned that switching to micro services from a monolith is a big investment. It really is. Also for a single small team, starting a new project, micro services might bring too much overhead, or not. As a company you should always evaluate the pros and cons of every alternative solution. If the answer is yes, you want to go with micro services, Mosaic might help you bring them to front end also. In the end I’d like to thank everybody who contributed to the Mosaic project, and especially to my team, the pathfinder team who made Mosaic possible and put lots of passion into making it a reality. Questions?”

Laurie: “We might have some coming in.”

Dan: “Okay, let’s look. When micro … oh. Okay. There is another one. On autonomous teams and ability to use the language, how do you handle future support updates if the core team leaves work and you don’t have other teams with same tools, expertise? Actually we have plenty of teams and this wasn’t the case until now. We also do lots knowledge sharing.”

Laurie: “I think the question below it. The diversity of tooling, how to developers collaborate across teams or do they strictly work in silos? We actually have part of the purpose pillar that I mentioned, the autonomy master and purpose. Part of the mastery actually is, we call it the tour of mastery which we encourage people to work on different teams so they can pick up new skills and get a more well rounded view of what a company like ours does. We encourage teams, people to move and switch teams when it’s appropriate. We also like Dan said, do a lot of knowledge sharing. We have teams teaching other teams the languages. Pathfinders was working with one of our platform infrastructure teams to get up to speed on go, so that the teams could then work together on Skipper.”

Dan: “We also have lots of study groups. For example we have a scholar study group, a closure study group, and constant meetups organized in our offices, so this also contributes a lot to knowledge sharing and getting people interested in different other technologies.”

Laurie: “One of the advantages of working in a company of this size is that you can have lots of … we have a tech academy. It’s kind of like an internal school for learning new skills. We have classes on get and github, scholar, closure, and then the guilds too so people teach each other.”

Flynn: “Some of the other questions there were getting into the whole polyglot micro services world. How do you handle dependencies, how do you handle developers leaving when they’re the only ones who know that particular tool? Is there anything else you can add about that?”

Laurie: “This is something that we’re working on actively right now, so we just launched this past week an intersource pilot. Intersource is a collaboration model that Paypal kind of put the name on, but it’s pretty much creating an open sourced environment but internally for your own organization. One of the reasons why we thought this was a great idea is some of us have worked in companies where this was just how you did things. Another reason is that we saw some of the repositories have what I like to call a knowledge bottleneck so you see one repo where one person that has 500 commits and the next person it’s going to be 2025. The way to balance out knowledge in a single team, is to encourage more of this collaboration and then also opening these repositories developed by a team or maybe in reality, one person on a team to other teams to then foster collaboration across teams.”

“That’s how you start to get the knowledge bottlenecks, you start getting the knowledge shared more evenly across the same team, and then other teams over time. It’s not going to happen overnight with some of the teams. We still have radical agility but we still have teams working more in the older way. It’s not something that you just do overnight. It takes a little bit of time to bring the teams up to speed and really help them be autonomous. We still see some of these bottlenecks in the organization, but hopefully intersource will help them become more autonomous by becoming more open.”

Flynn: “Is there anything you wanted to add on dependencies in particular.?”

Laurie: “Different teams?”

Flynn: “More the problem about micro services that may depend on other services? How have you been finding yourselves coping with the fact that you have to make sure that you don’t break one service by reeving another one? That sort of thing.”

Laurie: “Yeah, I mean …”

Dan: “First of all we have this API first thing, which makes sure that before any line of code, we talk to each other and we get our API reviewed, but the most important stuff is that before, while designing the API you should start speaking with your stakeholder and ask them, is this really what you need? They can say oh yes it is, or they can say oh no and it makes it easy to redesign it or to make it slightly different if you didn’t invest any time into writing code. We take lots of care into making the right API’s and of course if you somehow didn’t do it correctly, it will send lots of mails and be sure that everybody updates stuff and this is even harder if you’re API is used outside of the company. Zalando wants to be [inaudible 00:45:59] and offer services to other companies also. With those API’s we have to take even more care to do the right ones. Of course there are some ways to evolve the API’s but yeah, it’s kind of critical to try to do them as well as possible.”

Laurie: “The first time.”

Flynn: “Yeah, definitely. Okay. I think that that is all that we have time for.”

Laurie: “Okay.”

Flynn: “Thank you both very much. Wonderful job. We appreciate it.”

Laurie: “Yes. Thanks. Bye.”

Dan: “Bye.”

Citation: https://highestpayoutonlinecasino.net/