00:00:00:07 - 00:00:22:02 Speaker 1 Hello, everybody. Welcome to this link to the live session with me, Greg Leffler and Kelly Fitzpatrick from Renmark. We're going to talk today about observability, what DevOps users should know, software developers, authorities and all those sorts of people. What do you need to know about observability? As I said, I'm Greg Loeffler. My titles on the screen there. I work at Splunk, which you've probably heard of. 00:00:22:02 - 00:00:44:27 Speaker 1 We saw logging software and security software, and we also sell observability software. And I wasn't sorry at a another company you've probably heard of called Link ten for several years. I did not work on this feature that we're using right now, but I worked on traffic congestion, content ingestion, all sorts of things that involved talking to the Internet was one of the things I did. 00:00:45:14 - 00:00:50:10 Speaker 1 So I'm really excited to be here with Kelly from Redmond. I'll let her introduced herself and then we'll jump into it. 00:00:51:11 - 00:01:11:17 Speaker 2 Yeah. Thanks. Thanks, Greg. And I love the fact that you may have heard of which I mean, I definitely have happy to be here today. My name is Kelly Fitzpatrick. I am a senior analyst at Redmond. Redmond, if you're not familiar with it, is a developer focused industry analyst, firm, meaning that we follow a lot of tech trends. 00:01:11:26 - 00:01:40:11 Speaker 2 We're very interested in what people are building, what they're using to build it, to run it, to maintain it. And we are specifically interested in the point of view of the developer kind of specifically and practitioners more broadly. So what is what is tech mean for developers and kind of tech practitioners? In my former life, I taught computer science majors at Georgia Tech, and even before that, I like many, many, many, many lifetimes ago, I was a test and release manager. 00:01:40:11 - 00:01:51:22 Speaker 2 So, so, so many of the technologies that we are kind of looking at today and even kind of talking today, we're talking about the sort of ability or things that like I wished I have had back in the day. 00:01:52:06 - 00:02:05:21 Speaker 1 Yeah, totally. I think it's easy for people to sort of forget about how hard a job tests is. So much respect from someone else who was in the trenches like that. That was that's a tough thing that have to do. 00:02:06:15 - 00:02:27:18 Speaker 2 Yeah, absolutely. And the interesting thing is I clearly I've moved on to a different, different profession, but like so many of my colleagues are, either they have job health now with either DevOps or like Asari in the titles, they've kind of moved for these like test releases like this jumping off point to go do some different things, which I think are the interfaces that are extremely interesting right now. 00:02:28:24 - 00:02:44:16 Speaker 1 Yeah, well, it's, it's actually, you know, one of the first things that I wanted to talk about was just sort of like, what is observability? You know, we put it in the title. That's what I say all the time. But like you see it in the industry, you see it with people you talk to. So what are people talking about when they talk about observability, too? 00:02:44:17 - 00:03:07:01 Speaker 2 You have this and this is a really interesting question because one of the things that that for me, observability is one of those things that I think about it differently when I am doing my job as an analyst and when I try to imagine my job as back, you know, back when I was a practitioner. So it specifically having that kind of practitioner point of view when thinking about observability. 00:03:08:15 - 00:03:27:10 Speaker 2 So I think that it's interesting to kind of think of it, think of it that way is there's almost this like bifurcated way of thinking about it. But even from an analyst perspective, it is extremely difficult to get a single useful definition because the industry just uses it in so many different ways. And I think you've we've talked about this before, but I've been an analyst for about four years. 00:03:27:19 - 00:03:45:27 Speaker 2 And when I started, it was the it's kind of like I learned about observability from my colleagues. This is the type of thing where we were talking to people, we were going to vendors and telling them observability is something that you need to care about. About two years ago, there was maybe a shift where suddenly we have vendors coming to us telling us about observability. 00:03:46:17 - 00:04:06:27 Speaker 2 And that's that's a big shift when you're an analyst between like when you're the person who's who's kind of talking about this and it's new and people are coming to you to tell you about it. And now it's like we have vendors coming to us with like competing understandings and definitions of reserved abilities. So it seems like there's just a lot of ubiquity out there, a lot of inconsistency there, actually. 00:04:07:06 - 00:04:41:12 Speaker 2 I feel like it can be very confusing for someone who's never, never seen this concept before. So your first question is really tough and I feel like I didn't even start answering it yet in terms of what I think observability is. And there are there are just so many ways that you can kind of approach it in kind of especially from a partition or as you were kind of making your way through these often vendor driven definitions of observability, they seem familiar to familiar already with concepts like instrumenting your code or telemetry. 00:04:41:12 - 00:05:04:27 Speaker 2 And it can seem kind of very daunting because you're like, I don't know what any of these things are. And also someone is trying to sell me a box of observability, which I don't know if I even need, but there's so many different places like you can start with, like the systems, the kind of control theory which you can go to Wikipedia page and see that it's about, you know, understanding systems from your outputs and about things that you can kind of understand and manage. 00:05:05:11 - 00:05:28:21 Speaker 2 There's a very popular, which I know we've talked about, like the three pillars model of observability, where it's a kind of combination of like metrics, logs and traces. And then there are folks who are like, we start defining observability by saying it is not just metrics, logs and trees that you always have to go beyond that kind of that three pillars model. 00:05:29:12 - 00:05:52:12 Speaker 2 But for me, I think it's very much about thinking about the benefits of observable what are the actual tangible things that you can get from some of the technologies that are falling under this observability category and kind of thinking about what can devs or Azeris get from this and what kind of tools do they need to do their job? 00:05:52:28 - 00:06:16:04 Speaker 2 Because when you're looking at the software industry, which largely we're making more changes and we're making them to systems that are increasingly more distributed and just more more kind of more complex. So the job of observability should be to help us understand those systems and troubleshoot them work more easily. I'm going to pause there because I know you have thoughts on observability as well. 00:06:16:04 - 00:06:33:22 Speaker 1 I mean, I think that last sentence really sums it up. I one of the one of the things that I do here at Splunk as well is sometimes the sales enablement stuff. So talking to our salespeople about, you know, how to sell the product and we have to define observability, too. And it's a joke that I'll ask what is observability? 00:06:33:29 - 00:06:56:03 Speaker 1 And somebody will inevitably say, metrics traces the logs. And it's like a you're like, that's not solid observability. It's those are things that you need to have observability. But observability isn't the data source is right. Observability is the engine and the insights that you get right. So you also mentioned complexity, which is a huge drive for all of that stuff, right? 00:06:56:12 - 00:07:18:09 Speaker 1 Like back in the day, you know, the first application I supported like professionally was advertising and it was an API for partners to update prices of products. Right. It's pretty straightforward, honestly. Like it ran on one machine. Well, I mean, it was cloned, but like the whole app could fit on one machine. I could keep the state of that whole system in my head. 00:07:18:14 - 00:07:47:03 Speaker 1 Very simple, right? There's ingress, there is material, there's database. Now, if we kill a cat, the simplest service, you know, it's 20, 30 microservices to display a web page, right? So like understanding what's going on, where problems are, it's so, so hard. So for me, I guess to get back to like what is observability, it's being able to take data from your whole system and then to get useful things out of it, right? 00:07:47:03 - 00:08:00:12 Speaker 1 Like and to do it without having to say in advance like, this is what I'm going to want to see, right? Like on a monitoring approach, you want to say, okay, this part could fail, right? So observability is going beyond that, I guess, is what I would say. 00:08:01:07 - 00:08:23:21 Speaker 2 Yeah. And I think you bring up another good point, like talking about the difference between observability and monitoring and. Well, what we have absolutely seen is like almost like a shift in categories. It going from this like almost like monitoring mindset of, you know, you're basically saying, here's what I'm going to look for and let me know if it happens. 00:08:24:01 - 00:08:47:03 Speaker 2 Where is another aspect of observability? It's like, I need to understand this system and I need to be able to see what's happening when it's for something. I don't know what to look for because most of most of the things that that are going to break, that are going to take out a system are going to be things that you were not planning for, because if you knew to plan for them, you would probably, probably work around that in a much better, much better way. 00:08:48:08 - 00:08:49:16 Speaker 2 I guess I would say. 00:08:50:06 - 00:08:55:16 Speaker 1 As a necessary it's it's one of the most terrifying things is to get a page about a service you've never seen before it. 00:08:55:16 - 00:08:57:19 Speaker 2 Right at 3 a.m. it's going to be three and. 00:08:58:09 - 00:09:16:15 Speaker 1 Of course it's going to be Etherium or it's going to be in the middle of Christmas that was one of those years that I was at LinkedIn. There was an outage that was at like 11 a.m. on Christmas morning. So that 1.1 was fun. But the hard part is you get exposure to, you know, somebody calls you and says Service X is broken. 00:09:16:15 - 00:09:33:01 Speaker 1 And that's that's the best you get, right? So it's like, okay, what do you mean broken? Right. Like, is it completely unresponsive? Is a e broken for half the users is a canary like what has happened. And so like I think the next thing I want us to talk about is sort of like, why do you care about this? 00:09:33:01 - 00:09:52:27 Speaker 1 But one of the things that I think we really need to why we care about this is because you can't figure out what's going on that just like you could before. Right. You need to figure out where is the problem, who is experiencing the problem, what version of the code is running, you know, what type of customer are that? 00:09:52:27 - 00:10:15:10 Speaker 1 There's so many factors now that it's just so overwhelming to try to troubleshoot something, especially with no experience for it and no exposure to it. And like, yeah, we also I've gotten many an email forwarded from some executive that's like a screenshot of an error message or even worse, just a screenshot of a page and question mark. And that's it. 00:10:15:10 - 00:10:38:17 Speaker 1 And you're like, Oh yeah, I don't know, man. Or What's he said that? What's going on? So my, you know, my perspective on caring about observability is like the software in the environments that we operate are so complicated that like needed, right? So is that what you're saying as well as that what you say, what you think, like what's what's your rationale for why folks should care about this? 00:10:39:10 - 00:10:59:09 Speaker 2 Yeah, I think complexity is probably used that term a lot. But in this case, it is it is absolutely one of the kind of key things to point to when you ask why should why should anyone care about observability? Why are the things that we were using five or ten years ago not sufficient for what we are we are doing today? 00:10:59:09 - 00:11:30:29 Speaker 2 And, you know, it is that systems are becoming more complex and the way that we use software is becoming more complex as well, you know. So for instance, one thing, the things that we learned from, you know, 2020 and 2021 is that the demand on systems can change in an instant. And suddenly the number of users who need to access the same system that was usually getting like very kind of like light usage suddenly surges in, in cases where they were not necessary, predictable or not predictable in the long term. 00:11:30:29 - 00:12:00:23 Speaker 2 So one of the cases that one of the cases people always use in terms of kind of fluctuating load and user use is you know, the day after Thanksgiving when everyone is shopping and suddenly like e-commerce sites know this and it's ready, ready to we're ready to deal with it. But, you know, suddenly with the COVID 19 pandemic, we're in a situation where people are using things in different ways that they weren't before and the demands were different in the problems that were creeping up were also also extremely different. 00:12:00:23 - 00:12:22:10 Speaker 2 And then we suddenly have people becoming or, you know, trying to become software shops that weren't before. So even like your mom and pop store across the street now has like had like now does have online ordering because it's something that they introduced in kind of 2020 and someone somewhere has their job is to make sure that that thing is happening and it's kind of still working. 00:12:22:10 - 00:12:32:00 Speaker 2 So it's like the way we use things are changing and the things that we're using are becoming kind of more complex. And I will pause there so I don't talk through the rest of our entire session. 00:12:33:08 - 00:12:53:21 Speaker 1 I mean, I think you're right. Like I after this call, I need to go pick up groceries, which I ordered online. All right. And I will tell you, I had no clue that my grocery store had an app. I had no interest in having a grocery store app on my phone until the pandemic. And now I order groceries every week. 00:12:53:21 - 00:13:15:10 Speaker 1 I haven't set foot inside the grocery store in in a long time. Right. Right. Like, you know, even those vendors, like they the spike in demand must have been insane, right? Like and also, like restaurants. Every restaurant near me that's still open has online ordering. Even ones that before were, like, very high end. And, you know, they didn't they don't make reservations. 00:13:15:12 - 00:13:31:29 Speaker 1 You know, it's like, you know, you show up if you want to and good luck. Well, now you can just go online to toast and tick the stuff you want. Go pick it up. Right. It's it's night and day, but restaurants and grocery stores don't have it. I mean, I guess stereotypically typically don't have good operations teams, right? 00:13:31:29 - 00:13:42:21 Speaker 1 They don't have people who think about this stuff. So like, are you seeing people that come to you as an analyst, like from those industries? Like, are you seeing people reach out from places they haven't before? 00:13:42:21 - 00:14:04:25 Speaker 2 You know, we were in terms of the conversations we're having, I would say that in terms of who we're speaking to and where they're working is absolutely changing whether or not we're speaking to them just as a like a client perspective or, you know, just speaking to them because, oh, my goodness, the people who are running and building software are running the world in some ways. 00:14:05:21 - 00:14:43:28 Speaker 2 But there absolutely is this. I would say that there's an increased demand for people who have skills to run and build software and to keep it running. You know, to the point of why does observability matter and why is the tooling around it important? It's because there's there's nobody wants their systems like to fall over, whether they see their system as being super important and critical for the world to keep functioning, or if it's just to keep your business running and we're seeing a greater need for it and troubleshooting is almost always somebody's problem. 00:14:43:28 - 00:14:48:13 Speaker 2 And if it ends up being yours, it's just best to have the best tools possible. 00:14:49:26 - 00:15:11:03 Speaker 1 Yeah. Like one of the things that I wish we had had, you know, some of the observability products that we have, it's like when I was the Nets are, you know, LinkedIn was pretty far ahead of the curve in a lot of things. But, you know, when we were moving to a more microservice model in which it was always it was never a monolith, right? 00:15:11:03 - 00:15:31:09 Speaker 1 But there were definitely more microservices came as time went on and we didn't have distributed tracing initially. Right. So when something went wrong, we couldn't find it easily, right? It's like I had a custom script that would say to the entire infrastructure and grep through logs it which like, oh, that made me feel really good because I could find the problem. 00:15:31:09 - 00:15:51:18 Speaker 1 But like, I mean that's just not, you know, that didn't scale at the time and it certainly wouldn't scale now to however many hundreds of thousands of machines they have. Right. So like I think distributed tracing is something I really wish we had had when I was supporting systems, but also with rum and synthetics, you know, some of the more niche observability things. 00:15:51:18 - 00:16:10:25 Speaker 1 But I mean to me they fit in. There is like what's happening with our users because so many modern apps or Java scripts and you know, the web apps and browsers like what we're using to do this platform is a JavaScript app, right? So like if there were exceptions, if they were problems like the people that operate this platform wouldn't know unless they have a systematic threat. 00:16:10:28 - 00:16:23:14 Speaker 1 So that's another thing that I think I wish I had had. Like, what do what do people ask you about? Or what are key things that you think are important for DevOps folks to have like what observability capabilities or tools would you say are most important? 00:16:25:01 - 00:16:51:04 Speaker 2 I think that has I hesitate to be like here is the general prescription for what you need to start with. Because I think one thing talking about how observability has changed is one thing we've seen is this category collapse of monitoring. APM is now in profitability, but also logs, metrics increasing as much as that is not that does not constitute observability. 00:16:51:04 - 00:17:15:00 Speaker 2 But you know, vendors who started out with those have moved into the observability space. So, you know, often wherever, wherever, wherever you are, you probably are using something that has some part of of the complexity that is observability, so to say, okay, just cut whatever you have and go use X, Y and Z regardless of your use case. 00:17:15:11 - 00:17:35:29 Speaker 2 I think that can be that, that was very tricky. I don't I don't think I would ever recommend that. What I would I think I would say start with is observability is not something you want to go out and just acquire and just have and have it just have it to have it. So absolutely start with the problems that you're trying to solve. 00:17:35:29 - 00:17:54:19 Speaker 2 Like what are what needs do you have that you think something like observability or some specific tooling can solve and kind of go from there? But Greg, I feel like you probably you probably do have some very specific, even more specific advice on where people might want to start in terms of what tooling they would have. 00:17:55:14 - 00:18:18:23 Speaker 1 Yeah, I would say, I think to me the most important thing is that this is going to be essential for your business as it continues to grow. And if you don't necessarily think you need like observability tooling today. So what I would say to get started is really focus on having open instrumentation for your applications. So using open telemetry or something like that. 00:18:18:23 - 00:18:36:24 Speaker 1 But realistically, if it's telemetry, it's pretty much the option so that if you decide to adopt an observability tool, you don't have to redo any work like you're ready out of the box to start using that. So that's like from a, like, what can I do today? Like that's what I would say is something you can do today. 00:18:36:24 - 00:18:41:02 Speaker 1 I really think that synthetic monitoring is underrated as, as a tool. 00:18:41:14 - 00:18:49:19 Speaker 2 You've said this. You said this on every every time we talk this is that you you say this and I'm beginning to believe it. It's like so, so much. 00:18:50:12 - 00:19:08:21 Speaker 1 I mean, it's just like so if you're not familiar with what synthetics are for the people who are watching, it's basically simulating a user interaction with your site or with your mobile app or something like that. And so you can have synthetic tests that go through your whole checkout flow or you're member registration or whatever is critical to your business. 00:19:09:06 - 00:19:29:21 Speaker 1 And you can find out about it tailing before your customers find out that there are problems, right? You can type it into your CI pipeline so that if you push out a new release and it breaks something, you can automatically roll it back. And you know, we always say the best ETR is zero, right? If you resolve the problem before that happens to a customer that great, you did it. 00:19:30:09 - 00:19:51:27 Speaker 1 But I would say I would say those are the things that pretty much anybody can use. Right? You can use synthetics pretty much no matter what you do. And open telemetry is a great way to sort of get started and like get your feet wet into the world and then you can choose where you go from there. I do want to just you know, we could talk all day about this, but I think we want to get some time for questions. 00:19:51:27 - 00:20:01:24 Speaker 1 And one of the things I would say before we jump into those is just sort of like final takeaways, like, do you have something you want the audience to know? Like what's what's top of your mind about this? 00:20:03:07 - 00:20:31:28 Speaker 2 I think for the way we think about how observability can be a bit ubiquitous and a bit daunting, I think, like my takeaway is like, don't let that scare you. That especially as a practitioner, there are tools out there that can be extremely powerful. And one thing that I'm expecting to see in the near future is the kind of like increased focus on observability and tooling that can be useful for developers and to about practitioners. 00:20:32:08 - 00:20:40:03 Speaker 2 So it's kind of like just this is this is still an evolving, you know, kind of field in some ways. Don't be afraid of it. I think it can only help. 00:20:41:15 - 00:20:45:02 Speaker 1 A You stole mine. I was going to say pretty much the exact same thing. 00:20:46:17 - 00:20:48:06 Speaker 2 Thank you for letting me go first. 00:20:48:14 - 00:21:08:28 Speaker 1 Yeah, I. I totally think, like, it's just don't be scared. Your architecture is probably way scarier than any observability tool. It's kind of like. And just to know that the one of the things you can really do with an observability platform is adapt it and pick and choose the components that are right for you and to write for your application. 00:21:09:24 - 00:21:27:17 Speaker 1 You know, we sell an integrated product, so a lot of our competitors sell integrated products. But at the end of the day, you really need to think about like, what are the things that you need to be done well and to make your buying decisions based on that, right? But if you have a good foundation with open telemetry, for example, then it doesn't matter who you choose, right? 00:21:27:17 - 00:21:49:28 Speaker 1 You can use whoever is best for you. So we did have a question in the chat that I am going to say is directed at me since it asks for Splunk position on something. I am I am not allowed by company policy to tell us to tell you our position on anything. What what I will say is, you know, cripple has a lot of ex-smokers to work there. 00:21:50:22 - 00:22:19:26 Speaker 1 They have they have their approach to observability and we have ours. I think the the differentiator that I will say that I think is really important is our platform operates in real time, right? So with the smoke observability cloud products like you are getting alerts, you're seeing changes, you're seeing them happen within seconds. Right. And that's something that not really anybody else is able to do at the speed and scale that we can. 00:22:19:26 - 00:22:35:19 Speaker 1 Right. So you know, I'm not going to say anything bad about cripple like that. You know, that's just that's just not true. But that is a big difference that we have, is that, you know, we can perform generally we can perform faster. And then we did have somebody ask a question by sending me a message, which is cool. 00:22:35:28 - 00:22:45:23 Speaker 1 We also had chat, but of how do I convince my manager I need observability and I want to sell that one to you, I think. Do you do it? Do you have an opinion on that? 00:22:46:24 - 00:23:10:07 Speaker 2 Well, so one of the things that Redmond has been speaking about for a long time is something called developer led adoption, which is the developers actually have a lot more influence and technical practitioners like kind of, you know, more broadly a lot more influence in the decisions that they're like the tech teams in the organization make and those tech buying decisions than they did however many years ago. 00:23:10:07 - 00:23:45:21 Speaker 2 So it's kind of like, first of all, don't think it's impossible, you know? And I think a lot of the stuff that the great, you know, been talking about is been like the increasing complexity that we are seeing in applications today in what kind of like keeping them running and then figure out what is going on. And even with like, you know, in bicycles and the expectations you know, around releases and things like testing and production like observability is one of the kind of like key things that is going to help us deal with all of this as it becomes even more, you know, kind of complex, because it's not we're not going to go 00:23:45:21 - 00:23:55:15 Speaker 2 back to the days of like one simple application where you never have to worry about like user context or anything like that. Like it's just not going to happen. 00:23:55:15 - 00:24:13:14 Speaker 1 Yeah, I think I did have a little bit of a heartbeat. Skip When you said test and production, it just still gives you know, it still gives me the willies, but, you know, it's just how you have to do it. Now, I think it's not realistic to duplicate the scale and the data volume and the transaction volume of production, right. 00:24:13:14 - 00:24:29:29 Speaker 1 So, you know, we're getting better at figuring out ways to do that and flags and canneries and all that sort of thing. So it's a little less scary. It was at the time, but um, well, I really, I really love this conversation and I hope all of you in the audience did as well. We appreciate those of you who are engaged. 00:24:30:08 - 00:24:42:16 Speaker 1 If you have follow ups, mainly to messages are open. So you can feel free to reach out. I'm not going to volunteer Kelly for that, but I really enjoyed it and I had I had a great time. Kelly Anything else you want to say before we sign off? 00:24:43:06 - 00:24:53:12 Speaker 2 Feel free to reach out to me on LinkedIn. I hope I'm with you. I'm like happy to have you chat and thanks for having me today. This I learned a lot in this in this like 25 minutes. 00:24:54:06 - 00:25:02:07 Speaker 1 I think it was awesome. And I really appreciate you coming on. And again, thanks to everybody who joined us and enjoy the rest of your day. Thanks.