Mastering Database DevOps: Session 3 – Monitoring the DevOps Pipeline

In the previous video, we talked about the development process, and the development process is really where it all starts. That's where change happens. That's where the innovation happens. That's where our development teams are writing code and maintaining code that is required for the business application to become a reality. So it's a very important process. However, moving from a directional approach, where we have a slow really cycle into a much more DevOps Agile approach with a much shorter cycle takes some effort. And it requires ultimately an investment in time and effort to make sure that developers can pretty much do what they're doing before, but build in elements of what they do so that they can become automated. So automating the testing of their code, automating the reviews of the quality of the code, and so on, and so forth. So performance of the production application is super critical, because if the application runs slow, it's bad enough if it's an internal application. But if it's a public facing application, like a website, then if the application runs slow, because you haven't taken care of performance. That means you got a lot of latency. People get frustrated, and start thinking about, OK, let me look for an alternative. Because this isn't really cutting it. It's too slow, and so, yes, monitoring performance is really important. And monitoring the database performance is actually where most of the application performance problems come from. The main challenges of monitoring a database environment and DevOps pipeline is a lot of the traditional approach to monitoring databases tends to be focused on the production database, which is fine. I'm not saying don't manage your production database, because that's super important. But in DevOps pipeline, you could actually use monitoring tools to mitigate the risks of these changes. Now they're happening faster, and make sure that we're not missing any likelihood of performance impacts, scalability impacts, impacts due to changes in the database structure that might then cause a problem in production. It gives us an opportunity to do scalability testing and monitor the impacts of those before we go into production. In other words, it helps to mitigate any risks of the unforeseen before we commit them to your live production database. If you're not monitoring your database environment in your DevOps pipeline, then you're assuming that everything is fine, and you're ready to deploy all these changes that you've just made into your production environment. But unbeknownst to you, you might have introduced some issues when you made those changes way back in the development phase. For example, you might have modified am SQL statement. And therefore, when that statement runs in a production application, that transaction is now running slower than it was before. But because you're running in an environment where you've got a lot more data, a lot more users, the effect of that is amplified significantly. And next thing you know, you've got significant latency in an application because of something that somebody changed. And you never captured it, so that's the downside. It's all about risk mitigation. We have a solution cross software. It's called Foglight for Databases, and it's a 24/7 database monitoring tool. It works against many of the mainstream database vendors, Oracle, Microsoft SQL Server, DB2, My SQL, as well as some of the NoSQL databases that are becoming more popular, especially Mongo D.B. and Cassandra. It enables you to monitor, like I said, not just your production database, but all of your databases in your non-production environment as well and identifies any potential problems. And the way it does that, it enables the DBA, the database administer, to set some workload profiles. What is the normal behavior that a DBA would expect to see in that database? And then any resource consumption that goes outside of that profile is brought to the DBA's attention through an alert, and that alert is raised inside of a dashboard. So the DBA sees a very high level dashboard that shows in all of the databases they're monitoring, regardless of which type of database it is, or where it is. It can be in the data center. It could be in the cloud. It doesn't really matter and then see a visual alert. And that visual alert will tell them in real time that something has happened on a particular database. And then the DBA can then drill into more detail screens and effectively triage the root cause that problem in fine detail really quickly. Once they understand what it is, then they can understand what they need to do to resolve the problem. The key thing is they fixed that problem before ever got to production. The ROI of this is if you look at the downside of not monitoring your databases, and you've got an unforeseen performance problem happens, either a scalability issue or just a transaction that's now running slower, you might end up at best with slow latency on your application. But at worst, it could cause a data corruption. It could cause a major outage. And when we consider these days, a lot of applications, and financial services, and what have you costing thousands of pounds every minute that system is down. That's going to add up to a lot of money really quickly. So if you can do something to mitigate that risk, then you minimize any chance of that happening. So with Foglight for Database, it's a very easy application to get up and running. You connect to all of your databases that you have. You see them visibly in a single dashboard, single pane of glass. You can instantly drill down as soon as you start seeing something happen. And then the DevOps pipeline, there's a really cool feature, which actually captures the effect of a change. So you've got a baseline deviation capability, which basically if something unforeseen happens, you can compare it historically again, something