As a conclusion to my notes and reflections on the CA Microservices Conference, I wanted to share the MicroS Way that was presented there.
1. Establish the right boundaries in your code and organization
2. Have a system that balances easy and speed with safety
3. Have the right processes and standards
4. Steer the system and measure it
5. Accept that right, easy, and safe are a product of time and context
Showing posts with label microS. Show all posts
Showing posts with label microS. Show all posts
Monday, June 27, 2016
MicroS - lessons learned part 3
This is a continuation on my notes from (and reflections on) the CA conference on microservices.
How to Change
Use shock and awe to make change. Be a little outrageous. Do not pivot by spinning on your heel --sprint with a two-by-four, slam it in the ground, and spin round to a new direction. An enterprise is about maintaining stability, so it won't change just because you casually mention it. You need to prove this change is happening and motivate the teams. Have a crazy challenge motivator.
And make the rules of change embody your bold statement. For instance, Gilt moved to microservices by saying you cannot use the main database or commit to the Ruby on Rails depo. These two rules helped to build a completely autonomous system. The trick is to find something simple that forces a new direction.
Besides changing the technology focus, change the way you frame the problem. Gilt doesn't say "do this" but instead gets the best out of a team by asking, "how do you solve this?" You want to tell your team where you are headed, but not how to get there. This is partly because it encourages inguinity, but also because the people on the ground floor always have great knowledge of how things work. Even if management rises from within things can change and those that know the day-to-day need to be consulted.
Another way to get the best out of a team is to have the right team skills. Agile teaches that you put the key players on the team. You choose the engineers and developers that have the skills for the software and technology that you need. The expertise is built within the team so they don't have to reach out and wait for someone outside the team to have the time to help. But there's an expertise that can be overlooked in the team: the business. Having close allignment with the business is automatic when someone with business background sits as a member of the team. Furthermore that person will gain an understanding of the pressures on IT so that the whole team can make wise decisions on what is both business valuable, but not shortsighted of long term IT cost.
Part of long term IT cost is the fact that once something exists it seems to be able to justify its continued existence based only on the fact that it exists. "Someone might need it" justifies going to large cost to maintain an unnecessary system, while half of that cost will not be spent to build a necessary system if it does not exist yet. There are very human reasons for thinking this way that all of us are prone to. The way to combat this natural tendency is to have a Sunset Team. This team is charged with responsibily shutting down systems that are no longer needed. The entire organization should celebrate shutting down an old service as much as they celebrate a new one.
Building Teams
It can sound easy to just put together a team, but there is a lot to consider. Besides what I already spoke to about having the right skills on the team there is also attitude and size to consider. At the conference a team size of four to five people and a department size of around 20 was recommended. If you do not have at least ten engineers working on microS, then your group likely won't be able to handle the increased overhead that comes with microS. As for choosing the people watch how they naturally gravitate to each other. For instance, watch for those that want to work on the new and shiny stuff versus those that want to build a solid infrastructure that is hardened. Put them in separate teams and give them the projects that fit them best.
But do not build new teams and ask them to build microS right from the start. The team should always better understand the work than the technology. In other words you can be new to building a shopping cart, but you better be experienced in building websites and never vice-versa. Your first microS should be ones that you build to solve a business problem that your team already understands. Always know the business better than the technology.
One reason for knowing the technology better than the business is that debugging is always more difficult with something you do not know. This is especially true of microS, but true of anything. If you know the business part well you at least have a firm understanding of what going right means even if you don't know why something is going wrong when a bug emerges.
Focus on team culture. Change that goes against the culture will not be lasting and likely will not even take place. It is an attitude challenge. Culture can cause a team that always codes in Java to write Python as if it were Java. Changing technology does not change culture. Over time a balance will grow in a group between culture, technology, and structure. When you change one the balance is lost, but the group will try to find the balance they once knew by modifying the other two to make up for the change in the one. Unless of course the change to all three is thought out and the team is properly prepared for it.
You properly prepare by knowing that these three steps must happen in this order:
1. The technology and culture is stabilized. Things in flux are actually less able to change because everyone recognizes that the last thing you want to do with a shaky house is change the foundation.
2. Optimize them. Once you have stabilized you can build on that and have a better system, which helps motivate the team to believe in their ability to create effective change.
3. Transform them. Now you are ready for real change.
Embrace Change and Risk
I wrote in a previous post about how to talk about failure. Netflix has a great culture that embraces this at all levels. The fact that they can openly talk about Quickster proves they understand that innovation comes with risk and the best way to learn from it is to talk about it openly. They trade lessons learned in an open way, which transfers knowledge between teams in a radically better way than in a workplace where competition means not admitting what went wrong or could have been better.
How to Change
Use shock and awe to make change. Be a little outrageous. Do not pivot by spinning on your heel --sprint with a two-by-four, slam it in the ground, and spin round to a new direction. An enterprise is about maintaining stability, so it won't change just because you casually mention it. You need to prove this change is happening and motivate the teams. Have a crazy challenge motivator.
And make the rules of change embody your bold statement. For instance, Gilt moved to microservices by saying you cannot use the main database or commit to the Ruby on Rails depo. These two rules helped to build a completely autonomous system. The trick is to find something simple that forces a new direction.
Besides changing the technology focus, change the way you frame the problem. Gilt doesn't say "do this" but instead gets the best out of a team by asking, "how do you solve this?" You want to tell your team where you are headed, but not how to get there. This is partly because it encourages inguinity, but also because the people on the ground floor always have great knowledge of how things work. Even if management rises from within things can change and those that know the day-to-day need to be consulted.
Another way to get the best out of a team is to have the right team skills. Agile teaches that you put the key players on the team. You choose the engineers and developers that have the skills for the software and technology that you need. The expertise is built within the team so they don't have to reach out and wait for someone outside the team to have the time to help. But there's an expertise that can be overlooked in the team: the business. Having close allignment with the business is automatic when someone with business background sits as a member of the team. Furthermore that person will gain an understanding of the pressures on IT so that the whole team can make wise decisions on what is both business valuable, but not shortsighted of long term IT cost.
Part of long term IT cost is the fact that once something exists it seems to be able to justify its continued existence based only on the fact that it exists. "Someone might need it" justifies going to large cost to maintain an unnecessary system, while half of that cost will not be spent to build a necessary system if it does not exist yet. There are very human reasons for thinking this way that all of us are prone to. The way to combat this natural tendency is to have a Sunset Team. This team is charged with responsibily shutting down systems that are no longer needed. The entire organization should celebrate shutting down an old service as much as they celebrate a new one.
Building Teams
It can sound easy to just put together a team, but there is a lot to consider. Besides what I already spoke to about having the right skills on the team there is also attitude and size to consider. At the conference a team size of four to five people and a department size of around 20 was recommended. If you do not have at least ten engineers working on microS, then your group likely won't be able to handle the increased overhead that comes with microS. As for choosing the people watch how they naturally gravitate to each other. For instance, watch for those that want to work on the new and shiny stuff versus those that want to build a solid infrastructure that is hardened. Put them in separate teams and give them the projects that fit them best.
But do not build new teams and ask them to build microS right from the start. The team should always better understand the work than the technology. In other words you can be new to building a shopping cart, but you better be experienced in building websites and never vice-versa. Your first microS should be ones that you build to solve a business problem that your team already understands. Always know the business better than the technology.
One reason for knowing the technology better than the business is that debugging is always more difficult with something you do not know. This is especially true of microS, but true of anything. If you know the business part well you at least have a firm understanding of what going right means even if you don't know why something is going wrong when a bug emerges.
Focus on team culture. Change that goes against the culture will not be lasting and likely will not even take place. It is an attitude challenge. Culture can cause a team that always codes in Java to write Python as if it were Java. Changing technology does not change culture. Over time a balance will grow in a group between culture, technology, and structure. When you change one the balance is lost, but the group will try to find the balance they once knew by modifying the other two to make up for the change in the one. Unless of course the change to all three is thought out and the team is properly prepared for it.
You properly prepare by knowing that these three steps must happen in this order:
1. The technology and culture is stabilized. Things in flux are actually less able to change because everyone recognizes that the last thing you want to do with a shaky house is change the foundation.
2. Optimize them. Once you have stabilized you can build on that and have a better system, which helps motivate the team to believe in their ability to create effective change.
3. Transform them. Now you are ready for real change.
Embrace Change and Risk
I wrote in a previous post about how to talk about failure. Netflix has a great culture that embraces this at all levels. The fact that they can openly talk about Quickster proves they understand that innovation comes with risk and the best way to learn from it is to talk about it openly. They trade lessons learned in an open way, which transfers knowledge between teams in a radically better way than in a workplace where competition means not admitting what went wrong or could have been better.
Sunday, June 26, 2016
MicroS - lessons learned part 2
At the CA Microservices Conference, Vijay Alagarasan spoke about some anti-patterns. I have reworded them to be patterns to follow and added a couple of my own thoughts.
Automation is Litmus
The litmus test on whether you can handle microS is if you can automate. Automated testing and deployment is key to microS, but beyond that if what you have is not regular enough or so eccentric that automation is not possible then you either still in R&D land or you simply are not ready yet. You do not need to automate everything, but if there is something about the software that inhibits automation then that will be a problem for stability, scale, or your own sanity when problems occur.
Centrally Manage Config
Design a configuration manger. There should be one server (not literally one, you're allowed redundancy) that can send configuration changes to your services. If you don't start from day one then you will be unable to control scale. The time to centrally manage the configuration of a service is when you deploy the first instance. Once you are running 15 instances of it, you are likely to miss an instance when you need to change a setting on it. Just think to the last time you were debugging an issue that was intermittent and you checked everything only to finally discover that one of the servers was configured differently. Sometimes it is obvious, but given that you always assume someone was diligent and set them all up identically you can often overlook that sort of bug. Avoid the headache and always manage configuration centrally.
The one argument you can make against central config is security. As I write this it does occur to me that a malicious actor who merely wanted to reek havoc on your network could do it through the config server, but that is more the case of a disgruntled employee with system knowledge. Chances are your hack is going to be by someone that wants to find their way to something valuable and changing something like a rate limit on an API is not likely to help much. Maybe talk to your security officer about this if you are concerned.
API Gateway
An API Gateway has many benefits. It can reveal a lot about network traffic and help to control it with things like caching, routing, throttling (limiting particular calls), and authorization. It can also help you transition from one version of a service to a another by slowly moving traffic to the new one. For example: consumer A is routed to version 2, but all other consumers are routed to version 1. Then consumer B moves to version 2 and so on. A good version strategy allows you to to introduce new versions of a service with a minimum of risk.
However all of the stability a gateway can offer can be reversed through misuse. Every bit of business logic that works its way into the gateway is a potential production issue. The point of the gateway is to abstract you from the business logic so that you can change your business logic carefully (as mentioned with transitioning to a new version of a service). If the business logic is in the gateway then you cannot change that logic without risking all of your traffic being negatively impacted. There will be some business logic in your gateway, but assume anything you put in there may need to be changed and cause a full system outage. Each IF statement should be put in place with that realization.
I'm unfamiliar with these products, but CA Layer 7 and AWS Mobile Gatway are examples of API Gateways.
Question Every Layer
Abstraction is a useful tool, but it needs to be used purposefully. For instance, having a policy to abstract the connection to every database is not good. Each use of abstraction should be for a strong reason about that particular case. Unnecessary layers should be avoided. A coworker told me "no one ever wants to be the middle man" and you should think of layers like middle men. Most of the objections to middle men are the same reason you don't want extra layers in your software. They introduce complexity and the potential for translation issues, they constrain your message into their terms (admitidly this can be a pro), and they consume resources. People would always rather go to the source, but they will put up for a middle man when there's a good reason. Don't just have an extra layer in place like in Office Space.
Do not separate your layers by bussiness logic, data access, or orchestration. One layer should likely embrace all three of these things. The boundaries of a service should be based on a business solution and not a technical one. You are building services to solve business needs. Being able to switch your database out to another one should not be the guiding star by which you design your services. If you have a good version strategy then that should be all you need for technical changes.
Automation is Litmus
The litmus test on whether you can handle microS is if you can automate. Automated testing and deployment is key to microS, but beyond that if what you have is not regular enough or so eccentric that automation is not possible then you either still in R&D land or you simply are not ready yet. You do not need to automate everything, but if there is something about the software that inhibits automation then that will be a problem for stability, scale, or your own sanity when problems occur.
Centrally Manage Config
Design a configuration manger. There should be one server (not literally one, you're allowed redundancy) that can send configuration changes to your services. If you don't start from day one then you will be unable to control scale. The time to centrally manage the configuration of a service is when you deploy the first instance. Once you are running 15 instances of it, you are likely to miss an instance when you need to change a setting on it. Just think to the last time you were debugging an issue that was intermittent and you checked everything only to finally discover that one of the servers was configured differently. Sometimes it is obvious, but given that you always assume someone was diligent and set them all up identically you can often overlook that sort of bug. Avoid the headache and always manage configuration centrally.
The one argument you can make against central config is security. As I write this it does occur to me that a malicious actor who merely wanted to reek havoc on your network could do it through the config server, but that is more the case of a disgruntled employee with system knowledge. Chances are your hack is going to be by someone that wants to find their way to something valuable and changing something like a rate limit on an API is not likely to help much. Maybe talk to your security officer about this if you are concerned.
API Gateway
An API Gateway has many benefits. It can reveal a lot about network traffic and help to control it with things like caching, routing, throttling (limiting particular calls), and authorization. It can also help you transition from one version of a service to a another by slowly moving traffic to the new one. For example: consumer A is routed to version 2, but all other consumers are routed to version 1. Then consumer B moves to version 2 and so on. A good version strategy allows you to to introduce new versions of a service with a minimum of risk.
However all of the stability a gateway can offer can be reversed through misuse. Every bit of business logic that works its way into the gateway is a potential production issue. The point of the gateway is to abstract you from the business logic so that you can change your business logic carefully (as mentioned with transitioning to a new version of a service). If the business logic is in the gateway then you cannot change that logic without risking all of your traffic being negatively impacted. There will be some business logic in your gateway, but assume anything you put in there may need to be changed and cause a full system outage. Each IF statement should be put in place with that realization.
I'm unfamiliar with these products, but CA Layer 7 and AWS Mobile Gatway are examples of API Gateways.
Question Every Layer
Abstraction is a useful tool, but it needs to be used purposefully. For instance, having a policy to abstract the connection to every database is not good. Each use of abstraction should be for a strong reason about that particular case. Unnecessary layers should be avoided. A coworker told me "no one ever wants to be the middle man" and you should think of layers like middle men. Most of the objections to middle men are the same reason you don't want extra layers in your software. They introduce complexity and the potential for translation issues, they constrain your message into their terms (admitidly this can be a pro), and they consume resources. People would always rather go to the source, but they will put up for a middle man when there's a good reason. Don't just have an extra layer in place like in Office Space.
Do not separate your layers by bussiness logic, data access, or orchestration. One layer should likely embrace all three of these things. The boundaries of a service should be based on a business solution and not a technical one. You are building services to solve business needs. Being able to switch your database out to another one should not be the guiding star by which you design your services. If you have a good version strategy then that should be all you need for technical changes.
Subscribe to:
Posts (Atom)