So far in this blog series (See previous post: PowerShell – The Patch Solution – Part 2), we’ve explored some of the thoughts and questions that come to mind when thinking about patching your infrastructure. Now we need to design deployment strategy. Keep reading in order to get your mind and ideas flowing for how you can achieve this in your organization.
As with your personal development plan that you may have to do with work, you need to have a set of achievable goals. Without having a list of goals in mind, how are we going to measure our success!?
Here are some goals that you can easily achieve with The Patch Solution
- Report on outstanding patches
- Report on patches that will be applied in the next maintenance window
- Configuring maintenance windows per machine
- Configure change freezes/change stops
- Keeping costs down – The Patch Solution is free!
- Uses WSUS or Windows Updates
- Run it locally or from a central file server
- Option to control it via group policy
- Uses native Windows Updates API
- Runs as a scheduled task
- Automatically runs
- Can automatically patch 100% of your Windows servers
- Reduce number of employees doing late night patching and reduce overtime costs
Which goals do you have?
The difficulty in designing an update strategy is getting buy in from the server owners. I’ve run across a fair share of server and service owners that would probably have their teeth pulled or give up their first born in order to have their non-highly available server up 24×7. Like we learned in the first post of this series, it’s very simple to update a machine and reboot it, but we’re not at home. We’re at work, and they want to be up 24×7 and make money. In the second post of this series, I mentioned that you won’t get to 100% of patching, but what you do end up doing is patching more servers, keeping more servers up to date with updates and patches. This gives us more time to work with the business in flushing out schedules, creating fault tolerant designs and patching the “manual” servers.
To make a goal measurable, we need to measure it against something. This shows us how well we’re doing. We may be on our way to reaching a goal, but without a baseline to measure it against, we cannot tell whether we are underachieving, meeting or exceeding our expectations.
To figure out a baseline using The Patch Solution, it’s quite easy. Implement it, but set all machines to “Manual”. Then we can run a full report and see how many patches our servers (and workstations) are missing.
How many different environments are there in your company? Do you have any DEV or QA type environments? Can we release patches to those environments right away and see how they impact those environments? If there are no impacts, then we can release them to the production environment.
The Patch Solution can handle this in several ways.
- Different maintenance window per machine
- If an IP is within a specific network, we can always patch it
- If a computer name matches a set of characters, we can always patch it
They say timing is everything. When thinking about timing, there are several types of phrases that are thrown around.
- Maintenance Windows
- Scheduled Maintenance
- Change freeze
- Freeze Period
- Moratorium on Changes
- Change stop
Some may argue there are technical differences between a few of these, but at the end of the day, these all define dates and times. The question is, can we continue patching during these dates and times? Even though from an operations perspective of updating and patching is routine, documented and a repeatable task, during these windows, one should not routinely update machines. Released patches are regression tested, but there have been times where an update or patch has created instability or broken a system. This is where your knowledge of the company and usage of the servers comes in.
Services / Products
This topic circles back to timing. Some of the services in your environment such as AD, DHCP, and Exchange are fault tolerant by design. What happens if you were to patch and reboot all your AD servers at the same time? All your DHCP or DNS servers? You see the problem here… I hope!
What about different tiers of a service. Do some of your services have a web tier with dependencies on the application and database tiers? What we need to do in this case is find out if each tier is load balanced or highly available and again, provide different maintenance windows to servers that host a service.
The goal here is to keep the service up and running while other servers or nodes in the server are patching and rebooting!
This has been an area I would love to find time to work on and update the code to leverage cluster aware updates. Alas, the Patch Solution doesn’t handle it. Aside from time, people are leery about automatically clusters. They’re clustered for a reason, high availability and precious workloads. Generally, they’re left in the manual patch group and are handled by humans. I can recall probably a handful of clusters across several organizations where we have patched clusters nodes with The Patch Solution. We essentially set different times on the maintenance windows for the cluster nodes. When a node was cleanly shutdown for reboot, it will fail everything over to another node. This could easily be done in your DEV/QA environments to keep them up to date, and then opt to manually update your production cluster nodes.