Name: VSWUG: A Gallery Walkthrough into Evidence-Led Monitoring
Uploaded: 2026-08-27T16:16:29.298Z
Duration: 1 h 13 min 44 s
Description: VSWUG: A Gallery Walkthrough into Evidence-Led Monitoring

Transcript for "VSWUG: A Gallery Walkthrough into Evidence-Led Monitoring": Hello, everyone. So, I'm just gonna do a quick introduction of myself here. So I am Marie, and I'm working, with SolarWinds. So I'm helping out in the TWAC community. And thanks so much, to everyone for joining us today. So, I'm very, very grateful for one of our SolarWinds MVP who is, you know, kind of kindly giving up his time today to speak with us. So you might know him as I like eggs in in Twack and have probably seen the dashboard series he he posted on the Twack community for those who are Twack members. So, just a little information about, about Daniel, our social speaker for for today. He is a regular member since '2, 02/2012, and Daniel specializes in SQL, SWQL, scalable alerting, custom dashboards, and evidence led monitoring models that gives engineers clear defensible operational context. And, yes, he really does like eggs too. And we're really looking forward, to your talk, Daniel, and I will hand over, to you now. Hello? Can you hear me? Oh, oh, hi there. Don't know if anyone can hear me. So, yeah, I'm I'm I'm Dan. Just like to say kinda hello to everybody. First time kinda doing a virtual swag. So I guess I'm probably a bit nervous, but I'm gonna try and get through this as best as I can. So, yeah, hi, everyone. I am Dan. I've been working with SolarWinds since, 2012. Most of that time has been has been spent designing and leading SolarWinds monitoring for customers for small environments to large global environments. I'm now a site reliability engineer at Creative ITC, Soluent MVP, and a long time community, contributor. If you know I like eggs on, then that is that is me. The question running through today's presentation is kinda simple. How do we make, what solar wind absorbs more useful to engineers, and what what can they actually do next? So this is essentially our, our, menu. We could call it that. So this dashboard itself, I just wanna quickly go into, a small part of the dashboard, just to show a it's kinda like a proof that this is, in fact running, as as natively inside of SolarWinds as possible. This is just generally a SolarWinds table widget, and we can see the first few lines of the code. It's just SQL, that I'm essentially running into into a table resource. It's all select only, And, really, I just wanted to show that there's nothing kinda hiding behind this. There's nothing there's no other application or other configurations made on the SolarWinds box itself. It's it's essentially fully native inside of the platform. I won't really go too much further, but as we can see, if I edit resource again, we can preview the results, and we can see the dashboard has been outputted. I can also update the source, just a normal SolarWinds table, and click save, and that brings us to, the the the dashboard again. So this is really our menu for today. We'll kinda be walking through the SolarWinds, SolarWinds gallery. To be honest with you, like, this is just a bit of fun, on how far, you can take the platform. Engineering sometimes can be too much monitoring data. So I just really wanted to show today, like, what, what SolarWinds can do with its own data and how we can present that data essentially. So the first one we'll be looking through is the SolarWinds, gallery, then we'll go on to an interface intelligence dashboard. We'll then go ahead around the SolarWinds polar dashboard. We'll then look at how I apply context to dashboards. We'll then go into time aware monitoring and what that means, and then we'll finish up with the evidence led monitoring, piece of work that I've been working on for for a few for one too many years that I would I would like to say. But, yeah, everything that we're gonna be looking at today is normally a problem that I faced or a customer has requested, and it's just really the approach that I've kinda taken. So first, we'll go to the gallery. And really here, I wanted to start by thanking the community. I also would want to thank, the, MVPs who have helped shape some of these dashboards. Some of the feedback that I've gathered from the MVPs has been incredible, and the community have been so supportive around, building the dashboards even though deep down I have been pattern matching the community just to see what patterns do work for the community. And this is really just, just to show that all of the MVPs are connected to one single platform, which is the SolarWinds platform. So we're we're not all disconnected. We're all connected to to one single platform. The next area I wanna go into is our session guide. This is Bob. He started off really just to allow me to make transitions between different pages. He then he then went into, becoming a wizard. He's he's come into becoming a Yoda. He's picking different different, sort of costumes that he may be putting on during the gallery. But, really, this is just a bit of fun. It's just literally just a bit of fun that we can kinda just, like, have a bob. You know? And he doesn't like a bob. You know? So he'll be our session guide. This is fully inside the SQL using the method that I just shown. It's, SVG, and CSS and HTML, that we're using to generate him. There's no JavaScript. There's no nothing that actually goes and check or that affects the page, or there's there's there's, yeah, he's just chilling. So, really, the the first dashboard really in the, the gallery well, the third the third dashboard in the gallery is essentially a way for me to present, the solar wind services inside of, one single view rather than an a certain application monitoring where we have table list and we have CPU and all these other things inside of the component. I wanted to just really show, how a control pane, log pipeline, the queue transport, polling engine, and work process kinda all work together. And if one of those has a problem, then it'll be highlighted kind of on the dashboard. So it's kind of like using the visualization to really show where the the issues are inside of the SolarWinds platform without going into many tables, like going into checking when this the poll was up. And we also have Bob, which is, just down here. Here we can say that the platform is operating in integrated state. 94% of the evaluates of the validated signals are healthy. Really, this is just a dashboard that didn't kinda quite make it into the other ones, but I just love this dashboard. And the reason really for the gallery is just to kinda show you how, different, weights can affect an overall kinda posture, for application monitoring. I do know that the site is curried down. Don't worry about that. This next this next one, is really also one of my favorite one of my favorite dashboards, really. This is, node status. This is currently, nodes being pulled that are currently down, and Bob just comes in and out as just a bit of fun, and just changes the color. And we can see that, it is running on a solo it is running on live data, so we can see the widget just here as well. Bob is just, a wizard. I tried to create Harry Potter, but, yeah, it's it's actually quite difficult to create Harry Potter in here. This isn't a topology. It's just really a pattern that I like just to kinda watch and and kind of like view. I promise I would spend hours just looking at this particular dashboard, but we'll we'll we'll keep that for later. So, really, the next dashboard that I kinda wanted to show you guys, was really around, what I would normally send to a customer. So this isn't the exact same thing that I would send, or maybe I will. But, really, what we're what we're kinda doing here is building a dashboard to give us our total number of nodes that are being monitored, the current nodes that are currently down, the current interfaces that are down, the the hardware sensors that are being affected by this current scope, and any volumes below 20%. We also have availability, which has given us our posture for the last seven days, and then we're essentially looking at the last thirty days, and then we're doing some delta calculations in there to give us a small forecast around what we would expect to see next week based on what we've previously observed. We also have the control tower, keeping inside of the world tour theme. This is just our triggered alerts, over thirty days, and these are our hardware sensors, which come onto the TorHealthcare. These are just monitored nodes and those with problems. We have information around volumes. We we also have some support in context. The monitoring to observability route is holding a flight readiness of 89% over the last seven days. Controlled tower turbulence is one trigger alert with previous periods of delta of plus one grounded objects, zero nodes, and and so forth. The next dashboard, I will be completely honest with you. Like, I will I will I will be honest with you. I don't know why I created it. I don't know what even come over me to create the next dashboard. But if I if I started with the next dashboard, the way that I would have finished this presentation would have been crazy. So I'm just gonna get this one out of the way immediately, just because it's I don't know why. But, so here we have two Bobs. As I mentioned before, Bob will dress up in different costumes. He will give us, just a bit of visualization. Really, what we're looking at here is the the total count of nodes that are up and the the total counts of the nodes that are down. Yoda is node up, and Darth Vader is node down. So, really, I'm just allowing them to have a small jewel, where we can just see the overall state, and I think, it's just really just a bit of fun. I just really wanted to just to show what the play what the platform has been capable of, but I don't think the, I don't think this will be used for a customer environment anytime soon. And sometimes I just find myself just watching them. After I build them, I'm just kinda watching them, and and I'm looking at the SVGs, and I'm looking at the, SMILs, and I'm just looking at the different components that connect them together. I'm looking at timing, and I'm kinda looking at if the thing actually makes operational sense. I know this one completely doesn't, but, really, I just wanted a way to show, a jewel between nodes up and nodes that are currently down. That is that is it. The that is kind of a bit of the fun, around the gallery. Each dashboard will contain, like, a small footer that would take us back home into the hub. But, yeah, it's it's just really just a bit of fun. Gallery is just a small piece of fun. It doesn't essentially, mean anything. Like, these themes, generally, when I send them to customers, they normally only change the theme. So they're more of a a lighter based, platform, and and they don't, interfere with with what the information I'm trying to deliver to that customer, using the dashboards. And, again, this is pure SQL. It's running SVGs inside, HTML, CSS. So, essentially, the these dashboards can withstand, system upgrades, And these these dashboards, I've been running them for approximately, three or four years between that time. And I've never seen a problem with the the the the sis with the platform itself. So one thing to keep in mind that the dashboards, they do, use SQL, so they are querying. So the dashboards do have a wait time when the, when the page is rendered and the execute the SQL is actually executed. So we'll just pop back to the hub. And, hopefully, this kinda feels like a dashboard experience. I wasn't really too sure what a dashboard experience essentially is gonna be. I've never kinda been on a a dashboard experience before, so, hopefully, the menu is helping to tie things together, and stuff like that. The next area I kinda wanna look at is more around, interface intelligence. So these dashboards are slightly different from the other dashboards that I've kinda shown. They've all been more around, like, the SVGs and and stuff like that and HTML. But this one here is a slightly different approach. We are currently looking, at the interface and a current interface utilization, essentially. The question, I actually want to answer is whether the pressure and what is the in which direction is that pressure coming in at. So every interface becomes essentially its own lane. On the left, we have the inbound, and on the right, we have the outbound, with the current split in the middle. So what we're looking at here is live SolarWinds sorry. Live SolarWinds Orion data, and we can kinda see that if I hover over one of these, we can see tool tip, that's been generated from the SolarWinds platform for its particular entity. On the right hand side, we have state summary. We can say that the overall, flow of this dashboard is currently degraded, and we know that because of the top exceptions. We can also see the current evidence flow. So we can see the inbound share is 58%. We can see the outbound share is 41%, and this is all live SolarWinds data. Again, we're looking at SQL at this point. So these are SQL queries that I'm essentially writing and and pulling this data from the platform. We do have, a key down here. So healthy is either blue or green. Degraded, it is purple. So depending on the policy that's been assigned to the interfaces, the policy will also be reflected inside of the dashboard as well. And to be honest with you, the reason I kinda built this, one of our customers said, oh, do you know what? Could you let me know what the DC links are doing right now? I was like, fine. I can tell you what those DC links are doing. I gave him the report, and I thought to myself, do you know what? Actually, having a table, worth of of of of percentage utilization Doesn't really give me much around, where the pressure is building up inside of his scope that he's given me. So these are all of our Internet circuits, and currently, the dots will yeah. Also, I probably took it too far again. Sorry. But the dots themselves, are based on the actual utilization that is actually transferring through those interfaces. So if it if an interface is, at, say, 70%, the the actual dot itself will increase. And if it slows down, then the dot will actually, keep its own own timing. It's just really another way to show how data can be kinda presented. When I'm so close to the data. It it's, like, unreasonable for me not to use the data as well in the visualization. So I really wanted to build both of both, the value and also the visualization to complement one another, just to give essentially a two or three second glance what the live view looks like for those interface things. So, just gonna take a if you if you guys are happy for me to go back, I'm just gonna pop back into the hub. I'm just a bit conscious that I'm just gonna take forever talking about these things because if you catch me in the office, you you hear, oh my god. He's coming back. He's gonna talk about that dashboard again. And I and and I do talk about him probably way too much. The next one really I wanna go into is the SolarWinds polar dashboard. This is probably one of the newer dashboards that I've been kind of experimenting with. This one, does take a small amount of time to load just simply because what we're asking of the system, is is quite a hefty task. We're essentially asking the system for what is its polling distribution, what are its, what is its freshness policies? So if nodes are, outside of a freshness policy, meaning that are we getting polling issues between nodes, interfaces, volumes, and server and application monitoring? So once it kinda loads, it is is a fairly hefty query. I'll agree with that. So here we go. Here is our polling estate distribution. This is kind of like a new dashboard that I've kind of been experimenting with, and again, you can probably see that I just love looking at the data. The the idea really is just to have something that is pleasing for me to kind of look at, just because sometimes, yeah, IT is not boring, but sometimes it's nice to add a little bit of something, into the environment. So fine. Sorry. On the right on the left, sorry, we have the engine. We have end or our polars. We have a total number of elements that each, engine is currently polling. We have our polling completion of each engine. We have our posture, and then we also have the latest polling cycles, of those engines. On the right hand side, we have the total number of nodes that are being monitored in this environment, and we also have interfaces that are monitored in the environment, and we have volumes in this environment, and we have SAM inside of this environment. This is now complete scope of of of the of the monitoring platform. We can say that we can comfortably say that everything is current. Our our complete, polling completion is 99.5%. The query did take quite long because it did have 32,000 objects that it has a query. I know the next question after here is fine. We know how the complete platform looks, but what if I wanted to know how this one is doing down here that is currently red? So now we can click into, the dashboard itself. This is not JavaScript. This is pure SQL and the same method I've been using throughout. Here we can say engine 11 currently has 616 devices. 99.668 of them, currently had a freshness, but we can see interfaces. 4,000 of those, are currently, being monitored by this platform, but 7474% of them are fresh, but the remaining is degraded, meaning that it's to really show, polling issues inside of each category. And we can go I mean, go through each of these and just check how each poller is doing and how fresh its own data is. I just to be honest with you, I just love these kind of dashboards. They they provide me quick and easy access to know how my pollers are doing in the environment, and they generally just look good on the monitors. Someone's walking past, like, oh, what's that? Like and it's it's just it's like, oh oh, yeah. We're gonna talk about that for a few minutes. But, yeah, all I'm trying to do is rather than giving someone a table or asking them to go to this table or go to this part of the SolarWinds platform, I'm trying to make everything just more concise and give it to them in in just a one single place. So really behind this, how it works is all I'm doing is essentially changing the, the the shadowing, essentially, and based on the grouping. So I'm saying go to a 10% occupancy, and while this one stays at a 100%, the values themselves are values that are basically behind these counts. But once I click on them, the other counts go, and these ones come into into into the page. So this is yeah. It's just a different way to show monitoring data, and how the how fresh that data is based on policy. Currently, the policy is if any of these nodes interfaces volumes or some objects, are currently older than twenty minutes or haven't updated their own data in twenty minutes, then then present that. The lines themselves also do change in their motion. So if we see a freshness, or a degrade a degraded freshness, then the color will will change, and also the little dot will also slow down, which kinda gives the illusion that the the monitoring platform is slowing down in what it can pull. Again, visualization is only really complementing the data, and it really is just pulling all of this data into into one single view. I haven't really been looking at the querents, but, yeah, hopefully, I can kinda get to everything at the end. But let's go into, the next area. So I'm just gonna head back into the swag. So this is kind of the stage where I'd be like, you know what? Is there any questions? Because before we're about to go into, more of, like, more of a reasoning, more of context dashboard, more time aware monitoring, more evidence led monitoring. I don't know if there's anything that I would that anyone wants to to say or anything or, I don't know. I don't actually know how long I've been speaking to you before. If not, then then that's that's fair. The next one I really wanna go to is, around context dashboarding. Here is where I start to start to get into the reasoning behind the the the the dashboards themselves, like showing interface utilization, showing the polling, showing Yoda, going, against, dark Vader, is is all great, until the customer actually wants something sensible. So these dashboards are really just to give enough context, to to to managers and people that don't normally stay with inside of the platform. So for a moment here, I just wanna take a zoom in approach. This dashboard here, is essentially a project that I recently completed for one of our customers. We would get in several thousand, APs flapping almost, every month. And the customer, approach with a set of rule sets, or a policy that we could apply to, the DNA that they would that would be sufficient enough for them to to give them a plausible action. So here, all we're basically doing is we are building a dashboard that gives the reasoning behind the policy. So we can say the policy for the Cisco DNA, we are checking if it's the duplication, so making sure that the AP itself isn't, a duplicated AP within, like, a fifteen minute window. We're also looking for APs that are flapping. We're also looking at is there a, three is it three, APs that are this is exactly the same that has triggered in the last five minutes? And that should give, one particular ticket. Where it starts to get interesting is, we also wanna know if the wireless LAN controller itself is down. Because if the wireless LAN controller is down, the APs are gonna go down at the same time. So we don't want another storm to say, we've got this many APs, and we know that wireless LAN control is down, and we don't want the platform, to essentially waste, engineering effort and engineering time, so we suppress those. We also suppress the, the the APs if a core device is down, and then we have a fallback to say, fine. If none of these rules have been, have are successful, give it give it a ten minutes, isolation and then send an isolated alert. So each of these rules really have to be satisfied. The reason of the dashboard, though, why it's important for this type of dashboard is to show the reasoning kind of behind the rules. Because if something is suppressed, and I'm I'm trying to show the customer that this thing is in fact suppressed, if there is site, if there is a site device suppression, I also wanna show that. So the dashboards have started moving in a way that I can now provide the customer with the actual reasoning that SolarWinds is going through to to give a certain amount of, attention to the platform. And it I've tried to make it as super simple for the customer as as possible. I've given them the current APs that are currently, down, the site parents that are down, the repeated disconnections, the isolation suppression, and just a a nice little in, animation of just the AP, and identity and dependency policies. Just a quick way for them to kinda see how their environment is doing, and it is completely live. So every time the customer refreshes the dashboard, they would see what policy, has been satisfied for those those alerts. Bear with me for a second before moving to, the next area. Really, the next one was for an alert maturity assessment that I done for a recent customer. So, normally, what we would do or what we would do is we would say, how many alerts have triggered in the last seven days? How many of those alerts have been reoccurring, say, over the last thirty days? That is useful, but still we need to grab, a copy of the table inside of the database or inside of SWQL or whatever method we use. But I wanted to basically have the dashboard, reason on its own. I wanted the customer to be able to come into the platform and look at their own maturity assessment of the alerts. So I've kinda built this in a way where it has, I'll kinda just go through it very briefly. So we can see this is so, yeah, again, this is real data. I've tried to normalize things as much as possible. I'd wanna give the most authentic experience. So some things, are probably not gonna look, as as they should, but, it's fine. It's fine. So here we can say, this customer has 8,611 alerts in the last thirty days. The daily rates, have been 287. The sixty day rolling, count is 289, and the ninety days is 300. What I'm also analyzing here is quite important because I'm also, analyzing that if the last thirty days compared to sixty days, has the alerts improved? I also wanna say based on the last thirty days and if I compare it to ninety days, are the alerts improving? And we can see here, that the trigger counts are in fact really in in improving, by 0.96%, which isn't a great number. And the latest thirty days compared to a ninety days is 4.64% lower. Really, the executive summary is just to give a high level view. The latest thirty days, generate alerts, of the per day rate. The latest rate, of a six day rolling, window, is below the ninety day rolling window, and six ovular in definitions for approximately or definitions account for approximately 80% of the current volume. 21 show increase in signals, and 19 are reducing. So I'm analyzing ninety days worth of day up, but I'm only really focusing so much on the thirty days. I'm only really focusing on the 80% worth of the alerts that are worth reviewing. Everything that falls outside of that area isn't something that I really look at too much because they may be, isolated issues, and they don't, form an a reoccurring pattern. So we can also give, the latest contributor. So the last thirty days, we received 3,961 triggers, and that's roughly a 132 alerts per day. And that currently sits at eight 46% of the current volume. And to be honest with you, it is actually stable, funny enough. We can see that it is currently stable. We haven't seen a massive decrease in the data. We can see as a problem review or next step. So as part of the assessment, is to give kind of like a next step of what we would suggest the customer to do. So or we would actually do I would do. So we have 300 3,961 triggers, occurred across thirty days of thirty days. A 100% activity coverage, 35 of the 43 of of the repeated objects, are classed as repeated alerts. So I'm also doing the same analysis really for everything here. So we can say 832 triggers across twenty seven days, which is a ninety day, percentage of current, day coverage, and 47 of the 47 of the 192 affected objects, is repeated coverage. So really we're looking at alerts and we're also looking at how frequently they trigger and how repetitive they trigger and where they are isolated. And that's really what I wanted the maturity assessment to really give. I wanted to give an analysis that the system can do. I didn't wanna give another table or a spreadsheet that says, fine. These are the values for seven days. Now let's go and ask, copilot to analyze these for us and give us an analysis that potentially the platform can't support. And that's the biggest thing. Right? I really want the platform to be able to support the decisions that it makes, almost making it defensible in its own right because, really, the platform is built on trust, belief. And I know that everyone in in monitoring knows that. So, really, I'm trying to surface as much of the reasoning behind what I'm showing to build that trust, and the dashboards really do help me, to build that trust because the system we are analyzing the system, and the system is telling us what it's observed. We're then looking at those observations as an evidence to say, okay. What did you observe and what evidence can you support? So, really, here we have a review queue because of number one, two, and three. It says start at this point as it's triggered this many times. Investigate this particular area because it's, higher rate than the other days, and very verify a also, which is really crucial here, the system must also say if something has been increasing and it's improving, it must also say that something is reducing. Our analysis must also consider the fact that something can increase, but something can also decrease, and decreasing can also be a problem. So verify the reduction of of VM post days in VMware. Its latest rate is 26% below its sixty day rolling average, and that may be a problem. We may see a case where the alerting itself is maybe been disabled or the object inside it may be, in a maintenance mode or or something along those lines. So the next one really is around the context, dashboarding. This one here is probably one of my favorite sentences that SolarWinds probably has ever produced, and because it simply goes beyond saying something has high utilization. So here we can see, a interface circuit utilization. Normally, this would be something that I would send to a customer on a monthly basis, and it would give them an executive summary, and it would give them the same kind of evidence layout that we've given before in the alert maturity assessment. So I'm starting to build a theme that the customer can kinda see. So here we can say, based on the last thirty days, the action basis that SolarWinds can now say is this. The same clock hour between one and 2PM exceeds 80%. On nine of the 31 observed days, which is 29% of the observed days, meet in the reoccurring pattern rule of at least four days of twenty observed days. The monthly peak was 93%. Next, review the service workload and traffic contributing around that same clock hour. You know, like, now I feel once I created something like this, I feel like monitoring kind of changed for me, just being though honest here. Now the system is saying something else. It's no longer saying that if something is 93% or what what it is. It's saying 93%. It says 86% for this interface. It says 95% for this interface. But if you notice here, 95% is not at the top. You know? And the analysis or the evidence is basically saying to us the reoccurring time pattern between one and two exceeded 80% on nine of the last thirty one observed days. We can also say okay so let's look at 95% High exposure, five of a seven seven hundred and forty four observed hours exceeds 90%. That's a 0.67 exposure where the interface was greater than five, that was greater than 80% for five hours. Now that we're starting to really go into data around time, and I really believe so when this has a massive amount of data, it has a it's so it even has tables in the database that have hourly data. It has, daily data. Once we start to look at this data as an hourly or or or, a daily or hourly pattern, we can start to identify different source of information that the SolarWinds platform can support saying, you know. I really just wanted to show you the context afterwards. They they are more answering a different question from the virtual ones. So, yeah, I just wanna kinda show what those, kind of dashboards look like and what the kind of customer experience would kinda be around the analysis, and what we're trying to kinda show. I just need to grab some water. Just give me a few a few minutes. Fine. Thank you. So the next one really is I wanna introduce something else into the platform. So I'm not going to say this is anomaly detection. I'm not gonna say anything like that. This is simpler than that. This is essentially a time aware node profile. That's that's all this is. All we're really doing here is we are essentially comparing, the last CPU for the last hour, the last memory for the last hour, the last response time for the last hour, and the last packet loss for the last hour. What we're then going ahead and doing is we are then comparing the last hour to its own thirty day, same hour of the day context. So we are essentially saying this node has given 23% of CPU. Compare that to its baseline or its normal band or its normal workings. This this dashboard here essentially just sits on node ID, so it can sit on a note details page. Currently, time aware monitoring is really just giving us one answer. It's saying, is the current CPU lower, is it usual, or is it higher for the current hour? I'm then looking at the history of this. And now now that I have the ability to say what the current hour is I can say that it's usual. 23% is usual for current hour and 48% is also usual for the current hour. Now we can look at the same the same hour behavior over the previous thirty days. So we can say CPU is normally around the range of a band between 049%. We can see what its variations are in that data. We can also look at memory to say the band is between 3068%. This is normal cadence of this particular, metric. Response time, we can see the band is between twenty five and forty two milliseconds. And here for response time, I use something called, interquartile range, which essentially moves 125% of one side and 25% of another and just move just keeps the spread for 50%. The actual bands themselves are just an average and a standard deviation, and I basically multiply that by a policy, which is three. That allows me to create that spread rather than allowing the baseline to become too narrow when the metric is too narrow, it's too consistent. This is an anomaly, and I say that. But in saying so, this is not anomaly without policy. Once we introduce policy into here, we can say, what is our normal range looking like for this hour, and how far or how many deviations or what is the zed score outside of this particular range? We can then essentially alert on those metrics. Those alerts can feed information around. This happened in the last thirty days. This is what the normal pattern looks like. This is the current value, and this is what we've measured. There's there's there's no AI here or anything like that. It's just averages, standard deviations, and baselines, and same clock hour of the day and comparing the same clock hour to that particular time frame. Because really yeah. I just wanted a way to show if something is not normal on this platform. Sometimes a customer will say, I don't know what I want, but tell me what's not normal. Fine. We'll create advisory led alert in or a temporal aware monitoring alert. And I thought this allows me to produce those requirements, and it also gives a dashboard that gives an engineer the context and gives the history at the same time, or the temporal context as we could call it. Fine, that pretty much brings me to forty four minutes. I've only got like one minute so you don't kick me out. I just wanna pop into, evidence that monitoring quickly. I will try and do this one as quick as possible, but as you know or as I've tried to to go through every one of these alerts, oh, sorry, every one of these dashboards, it really can be their own session, and they can really be, they can really be their own analysis, like the time aware monitoring, the polling distribution dashboarding. Each of them is its own session, really, to really go into high into a low level. So, hopefully, I'm trying to keep it as as high level as possible, and and no one has fallen asleep yet, so thank you for bearing with me. So really the next one really is evidence led monitoring, So this is something, that I've kind of been working on. The original requirement was simple. The engineers are getting too many alerts. So what does the engineer do when he gets an alert? So here I've compared an event led ticket to an evidence led ticket side by side both using the same source, both using the same alerting, history or one is using one of the alerting history and the other one is using current state. So the event led ticket on the left hand side is saying, so when it's recorded a VM port state in in virtual machine host zero forty two, and it can give us a time stamp. What the what this model can also say is please investigate. Fine. So now we know that we're getting more and more noise, and all of our tickets are saying, please investigate. So engineer will then go away and say, is the threshold still breached number one ticket closing activity? The issue is is is no longer, active, so we're gonna close the ticket, and we're not we're gonna wait for the next occurrence of that ticket. But if the ticket is still active, then the engineer will ask themselves, is there a time focus of this particular thing happening, say, CPU? Is the CPU spiking at this particular time? Is there a pattern? Has the pattern changed? Where should they start? From what they what they currently know, where should they start? They know that their threshold is active. They can they can work out if there is a time pattern. They can say, oh, it normally happens around 2AM. The engineer at that point has got to go away and review charts, review the analysis, and stitch all of those things together, pull it in a ticket, ticket closes, another ticket comes through. Now the engineer has to do the same thing there to say, fine. Is the is the threshold still breached? And so forth so forth. And you can see Bob has gone through each of those steps, and accumulated the available evidence, based on the charts of of that. So an evidence led ticket is a bit differently. Here, what we are basically doing is we are grouping, the alerts by policy. So we can say, one single alert may have a policy of four alerts in seven days. It must have a freshness of, say, two days, which essentially means that the alert, has triggered more than more than, four times in seven days, but the most recent alert is within the last two days. Meaning that we also use freshness as a way to say, don't close this event, this evidence led ticket. Only close it when there is no new evidence that's been hit in the platform or that's been observed by the platform. So this allows us to do something else. So because we are looking at policy and we're looking at reoccurrence, we can start looking at pattern matching alerting itself. And, again, I love this line that SolarWinds can produce. So a thirty day wider pattern of 31 alerts, 21 of those alerts have triggered in the same clock hour. The evidence recall records at between two and 02:59, a 67% of a concentration that's happened inside of that hour. Now the system can defend or can can defend itself in saying, this is when we most recently get the alerts. This is where we should focus our engineering attention. This allows level one engineers to say, fine. We're going to look at this particular hour, and we're gonna check these particular jobs around this hour. It allows them not to be is the threshold still breached? Is there a time pattern? Is there more information that we've got accumulate before we can, make an analysis? We also do a seven day pattern assessment. The seven day pattern assessment says, how many alerts did we receive in the last seven days? We can say there's been 12 alerts. Of the 12 alerts, seven of those fell into a supported time hour between two and three, and that's 58%. What we can also do at this lay at this layout is we can compare the last seven days with the previous seven days. So we can say things like, across the matching of non overlapping seven days, the two to three hour contains five of the, map triggers, and the previous seven days had seven of the 12. And we can see that there is a change, in this value from 62% to 58%, and that's a point 4% percentage point. We can also analyze the thirty day pattern. So we can also do a pattern delta, against the thirty day pattern. So what is the pattern like the thirty days, and what is it like in its previous thirty days? So action basis is simple here. Engineer gets the ticket, and they are aware of the time, the time metrics that we put in. Also, we don't only look at one hour. We don't just say, oh, yeah. Is there a dominant clock hour? We also look at, is there a day of the week? Is there a dominant day? Is there a dominant weekend? Is there a dominant night? And we look at dates, like, in a different way and start to mold it into something that SolarWinds can can say based on the weighted of this particular alert or this event and how frequently it's happening. This is the tension that we would believe that the, engineer should go. So I'm just gonna read the the supported state. So con supported state is currently satisfied as it's four alerts in seven days. Use the two to four use the two to three as a first review context because the seven day seven day assessment and current thirty day wider pattern independently support that focus. So the platform is now starting to kind of reason about itself, what am I telling you and what am I trying to interpret for the engineer to look at. The last part really is the reasoning behind this. So when I say it took me a little bit of time to build these things, like these are this is a live alert here, these are live tables that we're currently looking at, and we're currently looking at a complete evidence led framework from a view where it allows me to kind of show people what it actually means. If I say, oh, yeah. We only trigger alerts four times in seven days, that's like, what does that look like? So really this here is showing, that we received 12 events or 12 observations. Those observations became evidence. We then asked two questions. What's the current state? And we can see that as 12 alerts in in in oh, there's four alerts in seven days and 12 of those currently active. We then ask another question of pattern, current pattern. What is the thirty day current pattern? These are all essentially going into action basis. What I also do is problem memory. Problem memory is the first time that I've been able to take an assessment that's currently today and compare it to an assessment that was in the past so we can start to look at what pattern it produced previously. And I thought that's so important because of SolarWinds only retains ninety days worth of data. The assessment can change. So I needed to take a snapshot of the assessment that essentially gave me a a time a point in time assessment that I can come back to and and compare in the in in the future. And that allows us to give more context to the alert. We can now say a ninety day, pattern has been this or the, one hundred 80 dot a hundred and eight day pattern has been this. The system can now start to reason about its own memory. I'm not saying this is like an AI or anything like that. I'm really, if I want to go down that route, I could say this is a deterministic reasoning model built inside of the platform, meaning one thing goes in and one thing comes out. This can work on all of the alerts. It doesn't matter if it's a note down. It doesn't matter if it's interface utilization. It's a matter of its component. Policy drives the alert. If the policy is essentially, one alert in in two days, then fine, but we still account for the quiet time. Action base is important because once a system can say we've got this much amount of evidence, we can say 21 of the 31 pattern observed between two and 3AM. We can what we don't allow or what I'm trying to not let the platform say is what the decision was. What did does it what did the decision that the engineer took after the, event or the evidence ticket was generated? Decision and action and outcome are all where the authority ends for SolarWinds. This is kinda where, I kinda wanna kinda take us take take the platform and give it a boundary. The boundary is really, really important to me because the platform should never really claim more than the available evidence support, and that's crucial. Because if if if it says that the system requires a reboot in the ticket and the system didn't require a reboot, the platform can't defend that. All of a sudden, trust comes into the game, and trust is always looking at us. So, we just keep looking forward for now. But yeah. So, really, we can say an alert asks for attention while the evidence earns that attention. So let's go I'm just gonna grab some more just for two secs. Sorry. I'm not really too sure to say about that. So after all of the SQL, the dashboards, one slightly over employed Bob, the the point is actually quite simple. SolarWinds already observed huge amount of data. The opportunity to turn those observations into useful evidence without asking the platform to come to to claim more than it can prove. This is why I believe SolarWinds should be an observation and evidence engine. It can give us a strong action basis, but the human still owns the decision, action, and outcome. I would also like to thank SolarWinds for inviting me, hopefully these are problems that answer real world questions. I would like to thank the the Black community for constantly giving me good feedback, positive feedback. I'd also like to thank, my girlfriend who has heard me talk about SolarWinds for an unreasonable amount of time. I would also like to thank, Grant, my manager, Tom, Ben Carroll, Lester, Alex, as all of these engineers have really just given me, just just the space, really, lunch breaks, and, and, of course, the frac community. I wouldn't have I wouldn't have done any of this if it wasn't for, the feedback I got from from the community. So I really appreciate sharing it with everyone. So thank you for listening. Thanks very much, Dan. It was it was a great talk and great to get an insight into your, you know, your knowledge and experience. And we will now have a brief q and a for those who are in, in the room. So there's a few question and questions on the q and a tab. Dan, you can tap into one of them if you'd like to answer them. I think there's one from Mark Roberts and. Martin course. some. of them. You know how I see it? Okay. So Mark Roberts says, can you confirm if you are using any graphing libraries? That was one of them. So in here, all of the all of the all of the artwork that I'm creating is essentially created in a platform called Inkscape. Essentially building the the artwork and as an SVG plain plain, template and then exporting, the the template into essentially Versus code. And then with the, components of the SVG, I'm essentially tying in the sequel. Yeah. I don't know I don't know if that answers the question. We got another one here from Martin Russell. Yeah. Thank you. What tips would you give if you want to learn how to develop these types of dashboards? They all see they're all SQL. They're all everything is here is in SQL. That would be the best place to kinda start. I would I'm happily kinda show some examples on on, probably first on the SolarWinds Slack community. But, yeah, I think getting into them is really it's all SQL. They're all SQL queries that I've been showing. And because that's a native language to SolarWinds, it really allows me to pull data that's the actual data from SolarWinds rather than going to another platform, and asking, any other kind of any other can you do this? Yeah. I'm just trying to keep everything inside of the platform as much as possible. Best place to start is SQL, to be honest with you. I'm I'm happy to kinda help, where I kinda can. And and yeah. K. Just another question there. So this this this particular platform is running, in twenty twenty six point two point one, and it it's it's been running from probably about three, three years. So it's got a few a few SolarWinds Ryan platforms. Oh, yeah. The other reason to start it again. It is on a bit of a loop. But, yeah, it's it's it's the self hosted SolarWinds platform. There's no SolarWinds, agent used in this build. It's fully offline, and everything that I'm currently showing, can be deployed in an offline environment. I think that's actually a really good question, to be honest with you, because most of these things are are more than likely achievable using, the the platform connection. But for customers that don't have that platform connection and only have self hosted environment, some these customers still need the ability to have time aware monitoring and evidence led monitoring, context dashboarding, lovely kind of dashboards that a self hosted platform can produce. And to be honest with you, like, this the self hosted platform is a true powerhouse. I've used many platforms, but the self hosted platform, really is one of the best platforms that is is generally on the market. And I'm not just saying I could've got a set of this t shirt on, but it's generally because I do believe in this platform, and I do believe it is providing good context, without, the need for an AI to to to interject. But that's not to say that's a bad thing, and and and that's fair. Okay. Another one here for you then from. Jeremy Mayfield. Can I make it for the virtual reality, and can I and can I walk through and add Jarvis? We can definitely work together, I guess, and we can do, something. Having got four d classes, though, so I have to go scare off some of those and definitely give that a shot. But, yeah, I think some of the dashboards would look beautiful in in four d. One of my work, massive monitors, I don't normally just go there and put one of these dashboards on there and just to review it, but, sometimes I do. But with four d glasses on, I would definitely do that. Okay. And we got I think Mark I think here Mark Roberts. Oh, sorry. There you go. They're dark team. There can be. There there there definitely can be. There can be, dark theme. There can be customer themes. There can be essentially anything, imaginable. There can be a Star Wars theme. There can be, a SolarWinds World Tour theme. Yeah. Any anything, and and we can add toggle to it. I should have probably included that now I think about it. Some of the dashboards will look very clean against a a, a darker background. Some of it I really just wanted to show what I actually show customers as well, and dark and I think sometimes the customer like, well, I don't always get dark theme in this. And, Yeah. I just wanted to kinda show you what exactly I'm kinda showing. But, yeah, we can do dark freedom. Okay. Another one here from Philip. Yeah. Would you be able to build it using to be honest with you, I don't think it's possible. I'll be absolutely completely honest with you. I don't think this is is possible in in SWQL. Yeah. I don't I don't think it is possible. It it probably could be. It probably could be. I think I've got, about ten years of experience really with SQL, and and I do play around with SWQL. And me and me and another engineer at work, we have a always a debate about the automation happens in SWQL and and the analytics happens in SQL. But I'm not too sure. I'm actually not too sure because I've never really tried. I'll I'll, I'll give it a shot. Thank you. yeah. Stewart has left, I think, but for everyone who wants to be able to know the answer to this question, I'll share this. What's the load on the t u? Yeah. So because we're using essentially so we're using we're going here. We're using a custom table. Really, the load is coming, from the the the the system actually pulling the data into, into the report. So, really, it just depends on the complexity of the query. The query itself does offload the SVG and animation to the browser, meaning that the dashboard can stay stationary while the browser, can produce the HTML and and the SVG and animation while the SQL is updating. From my calculation, it updates every hun two hundred and forty seconds. Yeah. So it sometimes just depends on actual SolarWinds itself and and and where the user is and how far they are and all those other, like, SolarWinds related kind of things. And to be honest with you, the actual performance is is nothing really noticeable, which has been one of the most surprising factors for me, especially, the polling dashboard. Like, this is a a massive query, we're asking a lot of information from it, so it's it's really if you are writing sequel into the platform, it it just naturally just takes time to pull that data back from SQL back into, back into the the presentation layer. But really these dashboards are really a a reasoning layer. They are presentation layer, but they're a way for me to show my own reasoning. So my reasoning can get quite advanced, though sometimes the actual query itself can can take, a substantial amount of time. Here's a dark mode one, purposely kinda created in a in a darker in a darker mode, and just given additional kind of context. And you can see, like, we're querying the 2,000 different objects, and we're asking for freshness of each of those individual categories. So really it does depend on the query that we're running it running against the platform. And again, it's a SolarWinds native table that we're running this through. Like, it's not custom, JavaScript. It's not a custom source files. We're keeping everything completely in house, and I think it's really important to do so as it as something I have kinda seen is I don't wanna keep the tool sprawl. I don't wanna have another application pulling this data. I wanna pull our customers back into SolarWinds, and that's really important because SolarWinds has so much to offer. And if we can keep all of the customers inside of one platform, there isn't really no other reason to go to to I'm just gonna say it. Power BI or Grafana, all these other platforms, it really just keep the customer where the data is being observed, where the evidence is being accumulated, where policy is. Everything lives here. Keeping the dashboards in the same place to provide reasoning beyond that is value to be added and keeps the customers, like, keeping the customers, subscriptions up. You know? Like, that always that all helps. And tool sprawl is real. You know? Having these other applications, I'm trying to remove them from the equation. I'm really trying to put the emphasis on how powerful self hosted platform is. I don't know where I was going with that. I'm sorry I can I can just waffle on, but yeah? Thanks, Darren. Alan Moore has one query here. Of course, of course. Where do you get? To be honest with you, some of the inspiration come from, the Google SRE. If you've had an opportunity to read the Google SRE, that's like a observability kind of like, like documentation. And in there, there is, like, operational art. And when I saw the operational art, it was so amazing. You know? Being able to present data in a way that's, like, immersive, like, really caught me, and I love a lava lamp. I love, like, things that are going up and down in in and out, and, really a lot of inspiration probably comes from Tron. These kinda remind me of the Tron lines. I don't see Tron on his little motorbike going through these, but every now and then, I could do. Iron man is also another one. Star Star Wars, just really things that I just find that I've just been yeah. I think some of my personality comes out a lot in this work, and really I'm just also showing a side of my personality that's the creative side, and how I'm applying the creative side into into my into my work, really. But, yeah, we could probably say, Iron Man and Tron, Star Wars, and and Harry Potter. I I did try to create Harry Potter. I did try. But yeah. And I think we have the I think it was probably is another one from Brian. Okay. The first question is, how do we do something like this awesome any post regards? So I did review so I did do a SolarWinds series, on. It really just goes through different sorts of dashboarding and really gives an understanding of what evidence monitoring platforms are. And and the best place to kinda start, is probably SQL, and I I will document this. I thought I do believe the community do deserve to have this sort of, knowledge. Once it is stable enough, I'm I'm completely happy to to run it through the, the the Solent's MVPs, see how they feel about it in their own environments. As I feel there is there's a lot of value to be added here. Even though we have touched upon these these dashboarding, they're really the underlying thing is saving engineering time by creating an evidence led monitoring, creating dashboards that answer what the system is doing within a few seconds. That's really important to me because engineering attention is something that is really, really extremely valuable to me. And in creating alert noise is something that, like, just is not possible anymore. When we have AI analyzing things and all these other things, doing things and sending our information outside of the sideways platform, so it needs something that is in in in house. Sorry. I don't know where I was going with that, but my my brain's pretty fried. It's like fifty five minutes. But I will definitely document this, and I'll definitely share it with these the the community as I think it is very valuable. It is complex. Don't get me wrong. But, it's definitely something that I believe the community should have, because their customers will also see the value. Yeah. Okay. Thank you. Alright. Thank you so much, Danielle, for, you know, a lovely presentation. And to everyone, to all of the, participants who's, who's there, you have a little bit over time, but thanks for tuning in up until. Sorry. now. So, thanks a million guys. So first of all, first of all, well, I wanna thank Daniel, our speaker, all of the community team who actually made this event possible. For those who attended, you will get your TWAC badges and your points as well loaded into your profile. For those who haven't signed up the TWAC, you, you can you can do so. So just go to go to TWAC, for that one. And, I suppose we have a coming, for community. We have events coming in. I suppose our monthly missions, we have one coming up in September. I know there's one still on at the moment, the September IT pro day mission on, but we have one coming up, in September 14, for those who's looking forward to that. So hope you love all the prices that we have in there. Another v swag after, Daniel's awesome presentation, we'll be having another one for the fourth quarter. So it's gonna be on the November 5. And upcoming swags as well happening, every quarter for next year. And don't forget, guys, solar Wednesday coming up on the October 14. So, that will be a great, talk and topic. So, thank you so much, guys, and we look forward to having you again soon. Thank you.