Do’s and Don’ts of Observability Transcript 00:03 Greg Leffler: Hey everyone. Welcome to our LinkedIn live today talking about the do's and don'ts of observability. I'm Greg Leffler from Splunk. My title is observability practitioner director, which doesn't mean much. What it does mean is that I have been an SRE before working on large scale systems. And now, I am tasked with teaching everybody about observability, and why you need it and why it's awesome. And all that fun stuff. And I'm super, super happy to be here again with Kelly Fitzpatrick from RedMonk, who I will let you introduce yourself. 00:33 Kelly Fitzpatrick: Hi, everyone, Kelly Fitzpatrick with red monk. We are a developer, developer focused industry analyst firm. That probably sounds like a bunch of like, what does that mean to you? Basically, my job is to follow tech trends, especially around stuff that people are using to build software, and with a specific focus on developers and other technical practitioners. So thanks again, for having me here. Greg. Like our last conversation was so fun. I was like, very excited when I had the chance to do this again. 01:01 Greg Leffler: Yeah, me too. And I hope that if the audience agrees, I think it's gonna be we have a good session lined up today. So I'm looking forward to it. 01:09 Kelly Fitzpatrick: Yeah. And so we're talking to do's and don'ts. And I feel that starting with the dues, maybe some some kind of positive, you know, prescriptions might be a good way to start. So Greg, what, what are some of the good things that people should do when they are thinking about starting an observability journey? 01:29 Greg Leffler: Well, it's interesting, I think, I think it's almost a prerequisite to, if not an SRE practice to at least have an SRE mindset before you do and observability journey, right? You know, a big part of that is sort of accepting that things will fail. You know, there's nothing as 100% uptime, nothing is going to be as reliable as you would like it to be. And you should also be focused on and interested in getting rid of manual work, you know, that toil phrase that we all hate so much, right? observability sort of enables you to do those things better, but I think you really have to be convinced that they're things that are worth doing. Before, it's sort of doing the whole observability thing, right? You know, in my experience, it's easy to let yourself get dragged down in the weeds when you're starting an observability journey, to say like, oh, there's all this data, there's so many things going on, I don't know what to do with it. And if you don't know your own architecture and your own infrastructure, you know, observability tools may be more complicated than you can deal with, right? Like, you need to sort of understand how your app works. And like have this mindset of, I'm gonna know that parts of it might break. And that's okay, right? That's not going to kill my users. It's not going to make my experience bad. It's all going to work. So I'd say that's a big one. The The other thing I would say is, like, the best place to start is with instrumentation. And it is one of the things that is like, it's kind of toil, it's kind of slog a little bit, some products are easier than others, right? Instruments specifically using open telemetry, right? We, we, obviously, at Splunk are big pushers of open telemetry, and it's not the right word. But we're big fans of open telemetry, let's use that word. And supporters. Yeah, supporters. We invest a lot of resources, both in money and personnel and open telemetry. So like, it's really important to us. But like, I would say that even if I didn't work at Splunk, I would say that regardless, because it is the industry standard at this point, like every observability vendor is either on board or they're being drugged on board kicking and screaming, but like everybody's coming on to open telemetry and like, a lot of instrumentation work is manual by nature, right? Like, we do have some auto instrumentation support for some common things. But like, no auto instrumentation can figure out a critical business workflow, like you have to tell it what is a critical business flow. So doing that once is like key to the SRE philosophy is that you're not doing this first with vendor A, and then we switch to vendor B. So now we have to do it again. And always switch to vendor C, you know, a third time, right. So that's also probably another place that I would say. And then I think the last thing is just really using resources to sort of learn about the market and the product, and just sort of figure out what parts of an observability-like platform or system that are really going to give you the most value, right? A lot of it depends on how your architecture is how your application is instrumented or written. You know, if you have a cloud-native 300 microservice application like, you have a very acute need for observability. you specifically need an APM tool. So you can figure out where things are breaking and stuff like that. If you just started a cloud migration, and you have a classic three-tier architecture that you sort of lifted and shifted, then you know, you don't really need the observability yet, but you can adopt the infrastructure monitoring right and get the advantages of integrating all that data into one place and having a good foundation for when you expand later. So I suppose are some things that I would say, or things where I would start with or some do's? You know, I guess I'll flip it back to you real quick, like, do you have anything that you would recommend people do to get started? 05:14 Kelly Fitzpatrick: You know, I'm going to I agree with everything you said, you know, especially the kind of, it's the starting point is not gonna be the same for everybody. Because not everybody is like starting with the same existing applications or tools or teams or knowledge. So to feel like that there's some type of cookie cutter path that you need to take. That's, that is not the case. Anything, Greg, you were really good about kind of laying out some of those, those variables to kind of think about in terms of like, your type of applications that you have. And here's here are places where, you know, observability may or may not be necessarily suitable. But you know, I also think one of the almost like, implied dues in everything that you just said is, if you are interested in observability, you really, really need to look at open telemetry. And you know, for those of you who have never heard of open open telemetry before, but you may have heard of observability, because observability is, it's something that we hear more and more in the industry, open telemetry, less. So if you're not already in that kind of observability universe, or an observability, you know, kind of practitioner, you know, an open Telemetry is an open source project, it's actually housed within the Cloud Native Computing Foundation, the CNCF. And, Greg, everything you were saying earlier about it being vendor neutral. You know, I think that is one of the things that you can't emphasize enough that this is not technology that a single vendor, you know, has control over or can just make decisions about, because it truly has become something that is backed by an entire community that includes a whole number of organizations and vendors that are like throwing support behind it. 06:52 Greg Leffler: Yeah, you know, we open Telemetry is the second busiest CNCF project, you know, behind with the tiny little, you know, skunkworks project called Kubernetes. So, it's, it's a really busy thing. But, you know, one thing that I was really excited to ask you, specifically, since you talk to so many people, like, what are you hearing about observability? Like, you know, what are people asking you? You know, there seems to be some hype behind it, like, is the hype real? Like, do people come to you saying, hey, Kelly, I want observability? Or are you saying, like, Hey, you should look at observability? Like, where's that sort of split that you're seeing? 07:29 Kelly Fitzpatrick: You know, I would say, you know, however, many years ago, three, four years ago, it would have been us going people being like, there's, here's this thing that you need to know about, or you need to care about. And now we just have so many people coming to us asking, what what do we need to care about observability? What if not, why do we need to care about it? But what should we know about it? Where do we start? What are the types of things that we should care about? And I think for me, one of the most interesting kind of advancements in the way people have been thinking and talking about observability came from, I was at CubeCon a couple of weeks ago. So CubeCon, cloud native con, which is like the kind of flagship conference that the Cloud Native Computing Foundation kind of puts on. And it was there, the 2019 is last time was there. And it was very much this is about cloud Kubernetes was, of course everywhere because it is in fact is in fact called con. You know, this time around the number of vendors who had observability offerings or we're talking about observability, or the you know, the interest around open telemetry, which as you pointed out is like one of the most popular projects within the CNCF. The change that was something I was not expecting. So I was I wasn't surprised that all of these things were more top of mind for people that people were added please like CubeCon talking about observability. But like, I shut up hoping to get an open to eliminate opentelemetry t shirt so that I could wear it to the slot in our live stream tonight. And by the time I got to the opentelemetry booth, there were none there were no t-shirts left. So that was that was definitely a bit of something I was absolutely not expecting. It looks like we lost Greg. I'm gonna continue chatting just for a little bit and hopefully we'll kind of get Greg back. So in addition to all of the things that like I saw at but and Greg is back, I'm just gonna I'm just gonna keep going without you and just like tell everybody about more of my opentelemetry thoughts adda a coupon but since your back you know kind of question for you because I want to make sure that we get to this because we did promise people do's and don'ts of observability 09:43 Greg Leffler: We did 09:44 Kelly Fitzpatrick: We did and I think I want to make you start talking about the don'ts like I'm gonna give you the talk about the negative thing. So what for you? What are the things that you think people should avoid when they start looking at observability 09:58 Greg Leffler: Thank you for giving me the negative part. I appreciate that. 10:02 Kelly Fitzpatrick: You got to start with the positive part too, I feel like… 10:06 Greg Leffler: I did. I did. I also, I think we forgot to say that we are going to take questions at the end. So you know, LinkedIn audience, like, if you have questions, please ask us. I think we forgot to say that. But I think the biggest don't is going to be kind of related to one of the biggest do's which is maybe cheating a little bit. But it's, I would say it's like don't rely on, on magic for for lack of a better term, right? There's, there's a lot of vendors who will say like, Oh, if you just drop our magic agent on your box, then we instrument everything, and you don't have to do any work, just install the thing, and it's done. Like, I would, my caution for that is that a lot of those agents abstract away a lot of what you actually need to do to get data. And if you decide to change your platform later, if you decide to do something open source or in house, right, like, you may find it very difficult to sort of shift, you know, like, once your architecture is more complicated once you have more services. And, you know, once you sort of bought into that world of oh, it's magic, like that magic can be taken away really easily. And it's why a lot of smart companies don't deploy their whole infrastructure into one cloud provider, right? Like, if they raise the prices, or they decide they don't want to be in the cloud business tomorrow or whatever, right? Like, you know, you have the option to move to the platform. But if you've locked yourself into a proprietary agent, like it's really tough. So that's, that's probably the only one that is the opposite of a do. Right is like, you know, don't not use opentelemetry phrase that. But I think another another sort of, kind of related one is to try to stop out tool sprawl pretty early, right? Like, you know, I imagine you probably hear this at least sometimes, right? But a lot of the observability tools are built in a really slick self service model where you can go to vendor.com, and slap down your corporate annex, and boom, you have observability for you know, some part of your environment. The problem with that is, if I do that from the front end team, and Bob does that from the back end team, and Mary… 12:15 Kelly Fitzpatrick: My team has chosen something else that we want to go and in and use 12:19 Greg Leffler: Yeah, right. And so you have, you know, five different products that, of course, are not integrated at all. And when there's a problem, it's almost never just solely in one place, right? You have to look at all these different tools to figure out what's going on. So it's really, really complex. And it also leads you down the trap of oh, well, we instrumented service x with Agent y. And it's not compatible with the format that we need for tool Z, right? Like, it gets really confusing. And I mean, I would just like, Have you have you heard that? Is that a real problem, and I'm making that up in my head. 12:51 Kelly Fitzpatrick: No tool sprawl is an absolutely real problem, not just not just around observability. But observability tooling is one of the places where that steps in, you know, even more so because as we've seen, kind of a category smooshing, if you will, into observability. So, so what used to be disparate, you know, kind of tools for, you know, logs and metrics and tracing, it's now you have tools that are offering or to do all of these different things. But you still have the tools that you were using. So going beyond the the model were developed, separate disparate teams have decided to use different solutions. It's kind of like trying to figure out and integrate with the stuff that you already have and deciding, deciding what to use and what to shift on that we see a lot of that and it is it is certainly it. I don't know that you can completely avoid it. But I think you can kind of be you can prepare yourself for it, you can be like is this is something that we can mitigate in some way? 13:47 Greg Leffler: Yeah, I think another way that you can sort of do that is to have a, you know, Production Readiness reviews. If you're adopting the SRE mindset, you know, something you'll have and one of those things that should be a part of that is, is this instrumented with open telemetry, right? Like, can we use observability for the service layer, even if we don't need it right now, right? Like, the biggest benefit of open telemetry is it's so flexible that even if you aren't sending stuff to an observability platform, per se, right now, like you can still use it to collect metrics and send those to a monitoring platform, right? You can do tons of work, just if you have the foundation in place, but like, like a house, you can't go back and remodel the foundation later without tearing the whole house down. Right. So like, I would say another don't is, you know, don't not have to have a standard interpretation. But like having a unified expectation and like, enforcing that you should have this in everything you deploy, even if you're not using it right now. Like it's a lot less work to add it at the beginning of this added at the end, right? And I think, yeah, geez. We have burned through a lot of our time. So I will I will shift to sort of final thoughts, I think and we'll again, we'll have q&a. So if you would like to ask stuff, feel free to Ask Us, I would say my final thoughts are really more really resources are things that I would want people to check out. So we have sort of a QR code and screen that we'll put up here in a second for you. There's two things that I think you should check out. And one is, the QR code on the left goes to a game that we produced at Splunk called pipe storm. The URL is pipestorm.splunk.com. And that's a chance for you to build an open telemetry pipeline, you can like literally see, what are the components of a telemetry pipeline, say that 10 times fast? Like, what are processors? What are receivers? What are exporters? How do they fit together, and like you're not actually building a pipeline, you're not writing any code, it's a game, but he will learn sort of how the components fit together and how they work. So it's a really soft intro to open telemetry and kind of a fun way. And then on the right, we have a book that we wrote called Amir and the Magical Lens, which is a children's book beautifully illustrated about open telemetry and sort of about observability. And it's really good like I was, I was amazed at how well it explains this. And I actually sent it to my mom and was like, Hey, this is what I work on all day. And she's like, Oh, I get it, like I understand what it is. So if you're looking for a really nice intro into that, scan that QR code, or go to splk.it/amirandthemagicallens, all one word, all CamelCase Good luck, it's on the screen. So you know, take a screenshot and check it out. ads, you know, I think those are two resources, I would check out. My real final thought is, like, do it like the time to start, you know, the best time to plant a tree was yesterday, the second best time is today, right? Like, that's what I would say about opentelemetry instrumentation as well. So Kelly, I'll give you some final thoughts, what are your what do you want to say about do's and don'ts or, or anything [Kelly’s Final Thoughts and Q&A Starts] 16:52 Kelly Fitzpatrick: I think I'm going to I'm going to end with like a final final don't. And it's going to, it's going to kind of play off with what you're saying. It's like just don't don't put off figuring out whether wet like your observability your position on observability. Like, it's something that you should look into now. It's something to follow, it's something that I think is going to be come more and more just integral to the way that we can build and run and manage software. So it's like a just like, Don't Don't delay. I know that sounds a little little infomercial, but just like, don't look into looking at this whole thing. And Greg, I know that we were going to jump into some some q&a, but I feel like I'm gonna like, take advantage of the privilege of being a speaker, I'm like, I'm going to jump in and do the first question for you. And I hear that you're going to RE:Invent in a couple of weeks like you and something like 50,000 of your closest friends in Las Vegas. What what are you looking forward to for RE:Invent? 17:47 Greg Leffler:: Well, it's gonna be my first reinvent. So sort of just getting exposure to an event at that scale. I haven't been to an event that scale ever, but I have a lot going on myself at reinvent. I mean, Splunk, of course has a huge presence, you know, come by our booth 3516, next to the developer lounge, right. But like, additionally, like I am hosting a game day session, which I'm really excited about. It's Monday morning. And it's basically a way for people to come in and learn and play with like AWS stuff, right? So a lot of this stuff, or AWS services and new tools and things that they want to teach people about. But then we also have part of that session is doing observability, right, it's instrumenting a machine, it's instrumenting a service, it's turning on code profiling and finding a performance issue. And like, the code profiler can show you like this line of code in this file is the problem, right? And like, it's really, really magical. When we did our internal tests, we showed people this and people that worked here didn't know that we could do this. So like, I'm really excited to see, you know how people react to that and how people respond. In a similar vein, we're also doing the jam lounge at AWS. So we have a challenge there, that involves getting observability into serverless lambdas. So, you know, the gameday session is a little more traditional. It's a Java application, you know, maybe not quite super cloud native, but still really interesting. The Jam Lounge is as cloud native as it gets, right? You know, it's lambdas, it's integrating with Splunk, opentelemetry, Splunk, observability. With opentelemetry. I'm doing a speaking session, I'm gonna be at the booth. Like, it's gonna be it's gonna be a lot of fun. I think it's gonna be busy but fun. And then, you know, since you asked me a question, I'll ask you a question like, What do you think is? What's coming next year for observability? Like, what are trends? What are the things people need to look for that are like, if you're already a believer, right? Like, what is something you're gonna want to think about for next year? 19:31 Kelly Fitzpatrick: Well, I think in this fits into larger trends that I think we're going to see in the in the tech industry, and among like, at least application development at large is that, you know, the way kind of like, economic conditions are looking. We had been talking about the great resignation. And then for a while we were talking about the great onboarding as people were like, kind of, you know, everyone's just going to like new jobs. What what I anticipate we're gonna see now is something more of a great reshuffle, we're seeing, you know, a number of people, even in tech kind of being laid off, but then there's still a need for tech workers and other places. So the kind of moving of places of workers from, you know, kind of place to place. And, you know, if you're one of the people that have been laid off here that you're, you know, out there looking for a new job, you know, my heart goes out to you, if you are in one of those places that are bringing on those those kinds of new workers, especially if those who've been laid off, oh, my goodness, if thinking about the way that you're onboarding is the probably one of the most important things that you can think of, and you think around what type of tool you're doing, what type of observably practices you have, and how you're going to like the introduce these these kinds of new colleagues into how you do things and explain it. I think that's going to be especially important for observability specifically, and for just the tech industry at large. So just having that that in your mind that the tech tech industry is gonna go through some shifts, it always does. But that's one of the things that I anticipate not anticipate with joy, but expect to see in 2023. 21:09 Greg Leffler: Yeah, I would agree. I think, you know, have play pipe storm, right, get get some skills, I would say in general, like learning new skills, and staying on top of things is always sort of how it is in our industry, right? Like you have to be, really, even if you don't have a personal interest in stuff, you kind of have to to have your skill set up to date. And like observability is still definitely like the new hotness, right? Like if you can talk instantly about it, you haven't implemented like, it's going to help you in your career. So that's why I did it. I, you know, I started talking about observability. Because I think it's really important. And I might like my personal passion is I want I want the site to never be down. But more importantly, like I want the customers to enjoy and get benefit from whatever application we're running. Right. And so that makes it work faster. Say of all that sort of stuff. So I didn't see. Do we have any questions? Did you see anything? Come in?would agree, I think, you know, have play pipe storm, right, get get some skills, I would say in general, like learning new skills, and staying on top of things is always sort of how it is in our industry, right? Like you have to be, really, even if you don't have a personal interest in stuff, you kind of have to to have your skill set up to date. And like observability is still definitely like the new hotness, right? Like if you can talk instantly about it, you haven't implemented like, it's going to help you in your career. So that's why I did it. I, you know, I started talking about observability. Because I think it's really important. And I might like my personal passion is I want I want the site to never be down. But more importantly, like I want the customers to enjoy and get benefit from whatever application we're running. Right. And so that makes it work faster. Say of all that sort of stuff. So I didn't see. Do we have any questions? Did you see anything? Come in? 22:04 Kelly Fitzpatrick: I did not. But Hi to everyone who's saying hi in the chat. Great to see a lot of hi’s. 22:08 Greg Leffler: Hello, thank you for joining us. I guess we'll wait a couple more seconds. See if anybody has a last second question they want to squeeze in. But if not, thank you for attending. I hope you really enjoyed it and got some value out of the conversation. Please check out pipe storm and the near and magical lens book. Please install open telemetry everywhere you can. Even if you don't use Splunk’s product we would like you to but if you don't, you know open telemetry really is gonna benefit here. So that's what I would say to wrap things up. Kelly, you have any last things you want to say? 22:41 Kelly Fitzpatrick: I mean, I think you said you said all the good things. A lot of pressure pressure me to come up with something you know something extra that you know other things and then Greg especially Do you have a have a great time at reinvent it sounds like you have you're gonna need a vacation after that. 22:56 Greg Leffler: I'm actually going to Sydney the week after to go to SRE Con Asia Pacific. So not exactly a vacation. Should be fun. I'm looking forward to it too. 23:08 Kelly Fitzpatrick: Great. 23:09 Greg Leffler: And yeah. And so recon APAC, if any of you are in Sydney, come join me. They're talking about open telemetry. So very, very good. Follow on for this, but it looks like there's no questions. So we'll let everybody go. Thanks again for coming. Kelly, thank you so much for these. I love these. I would do these whenever we want. Like I think it's a really great time. So thank you again for your expertise. 23:27 Kelly Fitzpatrick: Thanks for having me 23:28 Greg Leffler: Bye, everyone Do’s and Don’ts of Observability Transcript 00:03 Greg Leffler: Hey everyone. Welcome to our LinkedIn live today talking about the do's and don'ts of observability. I'm Greg Leffler from Splunk. My title is observability practitioner director, which doesn't mean much. What it does mean is that I have been an SRE before working on large scale systems. And now, I am tasked with teaching everybody about observability, and why you need it and why it's awesome. And all that fun stuff. And I'm super, super happy to be here again with Kelly Fitzpatrick from RedMonk, who I will let you introduce yourself. 00:33 Kelly Fitzpatrick: Hi, everyone, Kelly Fitzpatrick with red monk. We are a developer, developer focused industry analyst firm. That probably sounds like a bunch of like, what does that mean to you? Basically, my job is to follow tech trends, especially around stuff that people are using to build software, and with a specific focus on developers and other technical practitioners. So thanks again, for having me here. Greg. Like our last conversation was so fun. I was like, very excited when I had the chance to do this again. 01:01 Greg Leffler: Yeah, me too. And I hope that if the audience agrees, I think it's gonna be we have a good session lined up today. So I'm looking forward to it. 01:09 Kelly Fitzpatrick: Yeah. And so we're talking to do's and don'ts. And I feel that starting with the dues, maybe some some kind of positive, you know, prescriptions might be a good way to start. So Greg, what, what are some of the good things that people should do when they are thinking about starting an observability journey? 01:29 Greg Leffler: Well, it's interesting, I think, I think it's almost a prerequisite to, if not an SRE practice to at least have an SRE mindset before you do and observability journey, right? You know, a big part of that is sort of accepting that things will fail. You know, there's nothing as 100% uptime, nothing is going to be as reliable as you would like it to be. And you should also be focused on and interested in getting rid of manual work, you know, that toil phrase that we all hate so much, right? observability sort of enables you to do those things better, but I think you really have to be convinced that they're things that are worth doing. Before, it's sort of doing the whole observability thing, right? You know, in my experience, it's easy to let yourself get dragged down in the weeds when you're starting an observability journey, to say like, oh, there's all this data, there's so many things going on, I don't know what to do with it. And if you don't know your own architecture and your own infrastructure, you know, observability tools may be more complicated than you can deal with, right? Like, you need to sort of understand how your app works. And like have this mindset of, I'm gonna know that parts of it might break. And that's okay, right? That's not going to kill my users. It's not going to make my experience bad. It's all going to work. So I'd say that's a big one. The The other thing I would say is, like, the best place to start is with instrumentation. And it is one of the things that is like, it's kind of toil, it's kind of slog a little bit, some products are easier than others, right? Instruments specifically using open telemetry, right? We, we, obviously, at Splunk are big pushers of open telemetry, and it's not the right word. But we're big fans of open telemetry, let's use that word. And supporters. Yeah, supporters. We invest a lot of resources, both in money and personnel and open telemetry. So like, it's really important to us. But like, I would say that even if I didn't work at Splunk, I would say that regardless, because it is the industry standard at this point, like every observability vendor is either on board or they're being drugged on board kicking and screaming, but like everybody's coming on to open telemetry and like, a lot of instrumentation work is manual by nature, right? Like, we do have some auto instrumentation support for some common things. But like, no auto instrumentation can figure out a critical business workflow, like you have to tell it what is a critical business flow. So doing that once is like key to the SRE philosophy is that you're not doing this first with vendor A, and then we switch to vendor B. So now we have to do it again. And always switch to vendor C, you know, a third time, right. So that's also probably another place that I would say. And then I think the last thing is just really using resources to sort of learn about the market and the product, and just sort of figure out what parts of an observability-like platform or system that are really going to give you the most value, right? A lot of it depends on how your architecture is how your application is instrumented or written. You know, if you have a cloud-native 300 microservice application like, you have a very acute need for observability. you specifically need an APM tool. So you can figure out where things are breaking and stuff like that. If you just started a cloud migration, and you have a classic three-tier architecture that you sort of lifted and shifted, then you know, you don't really need the observability yet, but you can adopt the infrastructure monitoring right and get the advantages of integrating all that data into one place and having a good foundation for when you expand later. So I suppose are some things that I would say, or things where I would start with or some do's? You know, I guess I'll flip it back to you real quick, like, do you have anything that you would recommend people do to get started? 05:14 Kelly Fitzpatrick: You know, I'm going to I agree with everything you said, you know, especially the kind of, it's the starting point is not gonna be the same for everybody. Because not everybody is like starting with the same existing applications or tools or teams or knowledge. So to feel like that there's some type of cookie cutter path that you need to take. That's, that is not the case. Anything, Greg, you were really good about kind of laying out some of those, those variables to kind of think about in terms of like, your type of applications that you have. And here's here are places where, you know, observability may or may not be necessarily suitable. But you know, I also think one of the almost like, implied dues in everything that you just said is, if you are interested in observability, you really, really need to look at open telemetry. And you know, for those of you who have never heard of open open telemetry before, but you may have heard of observability, because observability is, it's something that we hear more and more in the industry, open telemetry, less. So if you're not already in that kind of observability universe, or an observability, you know, kind of practitioner, you know, an open Telemetry is an open source project, it's actually housed within the Cloud Native Computing Foundation, the CNCF. And, Greg, everything you were saying earlier about it being vendor neutral. You know, I think that is one of the things that you can't emphasize enough that this is not technology that a single vendor, you know, has control over or can just make decisions about, because it truly has become something that is backed by an entire community that includes a whole number of organizations and vendors that are like throwing support behind it. 06:52 Greg Leffler: Yeah, you know, we open Telemetry is the second busiest CNCF project, you know, behind with the tiny little, you know, skunkworks project called Kubernetes. So, it's, it's a really busy thing. But, you know, one thing that I was really excited to ask you, specifically, since you talk to so many people, like, what are you hearing about observability? Like, you know, what are people asking you? You know, there seems to be some hype behind it, like, is the hype real? Like, do people come to you saying, hey, Kelly, I want observability? Or are you saying, like, Hey, you should look at observability? Like, where's that sort of split that you're seeing? 07:29 Kelly Fitzpatrick: You know, I would say, you know, however, many years ago, three, four years ago, it would have been us going people being like, there's, here's this thing that you need to know about, or you need to care about. And now we just have so many people coming to us asking, what what do we need to care about observability? What if not, why do we need to care about it? But what should we know about it? Where do we start? What are the types of things that we should care about? And I think for me, one of the most interesting kind of advancements in the way people have been thinking and talking about observability came from, I was at CubeCon a couple of weeks ago. So CubeCon, cloud native con, which is like the kind of flagship conference that the Cloud Native Computing Foundation kind of puts on. And it was there, the 2019 is last time was there. And it was very much this is about cloud Kubernetes was, of course everywhere because it is in fact is in fact called con. You know, this time around the number of vendors who had observability offerings or we're talking about observability, or the you know, the interest around open telemetry, which as you pointed out is like one of the most popular projects within the CNCF. The change that was something I was not expecting. So I was I wasn't surprised that all of these things were more top of mind for people that people were added please like CubeCon talking about observability. But like, I shut up hoping to get an open to eliminate opentelemetry t shirt so that I could wear it to the slot in our live stream tonight. And by the time I got to the opentelemetry booth, there were none there were no t-shirts left. So that was that was definitely a bit of something I was absolutely not expecting. It looks like we lost Greg. I'm gonna continue chatting just for a little bit and hopefully we'll kind of get Greg back. So in addition to all of the things that like I saw at but and Greg is back, I'm just gonna I'm just gonna keep going without you and just like tell everybody about more of my opentelemetry thoughts adda a coupon but since your back you know kind of question for you because I want to make sure that we get to this because we did promise people do's and don'ts of observability 09:43 Greg Leffler: We did 09:44 Kelly Fitzpatrick: We did and I think I want to make you start talking about the don'ts like I'm gonna give you the talk about the negative thing. So what for you? What are the things that you think people should avoid when they start looking at observability 09:58 Greg Leffler: Thank you for giving me the negative part. I appreciate that. 10:02 Kelly Fitzpatrick: You got to start with the positive part too, I feel like… 10:06 Greg Leffler: I did. I did. I also, I think we forgot to say that we are going to take questions at the end. So you know, LinkedIn audience, like, if you have questions, please ask us. I think we forgot to say that. But I think the biggest don't is going to be kind of related to one of the biggest do's which is maybe cheating a little bit. But it's, I would say it's like don't rely on, on magic for for lack of a better term, right? There's, there's a lot of vendors who will say like, Oh, if you just drop our magic agent on your box, then we instrument everything, and you don't have to do any work, just install the thing, and it's done. Like, I would, my caution for that is that a lot of those agents abstract away a lot of what you actually need to do to get data. And if you decide to change your platform later, if you decide to do something open source or in house, right, like, you may find it very difficult to sort of shift, you know, like, once your architecture is more complicated once you have more services. And, you know, once you sort of bought into that world of oh, it's magic, like that magic can be taken away really easily. And it's why a lot of smart companies don't deploy their whole infrastructure into one cloud provider, right? Like, if they raise the prices, or they decide they don't want to be in the cloud business tomorrow or whatever, right? Like, you know, you have the option to move to the platform. But if you've locked yourself into a proprietary agent, like it's really tough. So that's, that's probably the only one that is the opposite of a do. Right is like, you know, don't not use opentelemetry phrase that. But I think another another sort of, kind of related one is to try to stop out tool sprawl pretty early, right? Like, you know, I imagine you probably hear this at least sometimes, right? But a lot of the observability tools are built in a really slick self service model where you can go to vendor.com, and slap down your corporate annex, and boom, you have observability for you know, some part of your environment. The problem with that is, if I do that from the front end team, and Bob does that from the back end team, and Mary… 12:15 Kelly Fitzpatrick: My team has chosen something else that we want to go and in and use 12:19 Greg Leffler: Yeah, right. And so you have, you know, five different products that, of course, are not integrated at all. And when there's a problem, it's almost never just solely in one place, right? You have to look at all these different tools to figure out what's going on. So it's really, really complex. And it also leads you down the trap of oh, well, we instrumented service x with Agent y. And it's not compatible with the format that we need for tool Z, right? Like, it gets really confusing. And I mean, I would just like, Have you have you heard that? Is that a real problem, and I'm making that up in my head. 12:51 Kelly Fitzpatrick: No tool sprawl is an absolutely real problem, not just not just around observability. But observability tooling is one of the places where that steps in, you know, even more so because as we've seen, kind of a category smooshing, if you will, into observability. So, so what used to be disparate, you know, kind of tools for, you know, logs and metrics and tracing, it's now you have tools that are offering or to do all of these different things. But you still have the tools that you were using. So going beyond the the model were developed, separate disparate teams have decided to use different solutions. It's kind of like trying to figure out and integrate with the stuff that you already have and deciding, deciding what to use and what to shift on that we see a lot of that and it is it is certainly it. I don't know that you can completely avoid it. But I think you can kind of be you can prepare yourself for it, you can be like is this is something that we can mitigate in some way? 13:47 Greg Leffler: Yeah, I think another way that you can sort of do that is to have a, you know, Production Readiness reviews. If you're adopting the SRE mindset, you know, something you'll have and one of those things that should be a part of that is, is this instrumented with open telemetry, right? Like, can we use observability for the service layer, even if we don't need it right now, right? Like, the biggest benefit of open telemetry is it's so flexible that even if you aren't sending stuff to an observability platform, per se, right now, like you can still use it to collect metrics and send those to a monitoring platform, right? You can do tons of work, just if you have the foundation in place, but like, like a house, you can't go back and remodel the foundation later without tearing the whole house down. Right. So like, I would say another don't is, you know, don't not have to have a standard interpretation. But like having a unified expectation and like, enforcing that you should have this in everything you deploy, even if you're not using it right now. Like it's a lot less work to add it at the beginning of this added at the end, right? And I think, yeah, geez. We have burned through a lot of our time. So I will I will shift to sort of final thoughts, I think and we'll again, we'll have q&a. So if you would like to ask stuff, feel free to Ask Us, I would say my final thoughts are really more really resources are things that I would want people to check out. So we have sort of a QR code and screen that we'll put up here in a second for you. There's two things that I think you should check out. And one is, the QR code on the left goes to a game that we produced at Splunk called pipe storm. The URL is pipestorm.splunk.com. And that's a chance for you to build an open telemetry pipeline, you can like literally see, what are the components of a telemetry pipeline, say that 10 times fast? Like, what are processors? What are receivers? What are exporters? How do they fit together, and like you're not actually building a pipeline, you're not writing any code, it's a game, but he will learn sort of how the components fit together and how they work. So it's a really soft intro to open telemetry and kind of a fun way. And then on the right, we have a book that we wrote called Amir and the Magical Lens, which is a children's book beautifully illustrated about open telemetry and sort of about observability. And it's really good like I was, I was amazed at how well it explains this. And I actually sent it to my mom and was like, Hey, this is what I work on all day. And she's like, Oh, I get it, like I understand what it is. So if you're looking for a really nice intro into that, scan that QR code, or go to splk.it/amirandthemagicallens, all one word, all CamelCase Good luck, it's on the screen. So you know, take a screenshot and check it out. ads, you know, I think those are two resources, I would check out. My real final thought is, like, do it like the time to start, you know, the best time to plant a tree was yesterday, the second best time is today, right? Like, that's what I would say about opentelemetry instrumentation as well. So Kelly, I'll give you some final thoughts, what are your what do you want to say about do's and don'ts or, or anything [Kelly’s Final Thoughts and Q&A Starts] 16:52 Kelly Fitzpatrick: I think I'm going to I'm going to end with like a final final don't. And it's going to, it's going to kind of play off with what you're saying. It's like just don't don't put off figuring out whether wet like your observability your position on observability. Like, it's something that you should look into now. It's something to follow, it's something that I think is going to be come more and more just integral to the way that we can build and run and manage software. So it's like a just like, Don't Don't delay. I know that sounds a little little infomercial, but just like, don't look into looking at this whole thing. And Greg, I know that we were going to jump into some some q&a, but I feel like I'm gonna like, take advantage of the privilege of being a speaker, I'm like, I'm going to jump in and do the first question for you. And I hear that you're going to RE:Invent in a couple of weeks like you and something like 50,000 of your closest friends in Las Vegas. What what are you looking forward to for RE:Invent? 17:47 Greg Leffler:: Well, it's gonna be my first reinvent. So sort of just getting exposure to an event at that scale. I haven't been to an event that scale ever, but I have a lot going on myself at reinvent. I mean, Splunk, of course has a huge presence, you know, come by our booth 3516, next to the developer lounge, right. But like, additionally, like I am hosting a game day session, which I'm really excited about. It's Monday morning. And it's basically a way for people to come in and learn and play with like AWS stuff, right? So a lot of this stuff, or AWS services and new tools and things that they want to teach people about. But then we also have part of that session is doing observability, right, it's instrumenting a machine, it's instrumenting a service, it's turning on code profiling and finding a performance issue. And like, the code profiler can show you like this line of code in this file is the problem, right? And like, it's really, really magical. When we did our internal tests, we showed people this and people that worked here didn't know that we could do this. So like, I'm really excited to see, you know how people react to that and how people respond. In a similar vein, we're also doing the jam lounge at AWS. So we have a challenge there, that involves getting observability into serverless lambdas. So, you know, the gameday session is a little more traditional. It's a Java application, you know, maybe not quite super cloud native, but still really interesting. The Jam Lounge is as cloud native as it gets, right? You know, it's lambdas, it's integrating with Splunk, opentelemetry, Splunk, observability. With opentelemetry. I'm doing a speaking session, I'm gonna be at the booth. Like, it's gonna be it's gonna be a lot of fun. I think it's gonna be busy but fun. And then, you know, since you asked me a question, I'll ask you a question like, What do you think is? What's coming next year for observability? Like, what are trends? What are the things people need to look for that are like, if you're already a believer, right? Like, what is something you're gonna want to think about for next year? 19:31 Kelly Fitzpatrick: Well, I think in this fits into larger trends that I think we're going to see in the in the tech industry, and among like, at least application development at large is that, you know, the way kind of like, economic conditions are looking. We had been talking about the great resignation. And then for a while we were talking about the great onboarding as people were like, kind of, you know, everyone's just going to like new jobs. What what I anticipate we're gonna see now is something more of a great reshuffle, we're seeing, you know, a number of people, even in tech kind of being laid off, but then there's still a need for tech workers and other places. So the kind of moving of places of workers from, you know, kind of place to place. And, you know, if you're one of the people that have been laid off here that you're, you know, out there looking for a new job, you know, my heart goes out to you, if you are in one of those places that are bringing on those those kinds of new workers, especially if those who've been laid off, oh, my goodness, if thinking about the way that you're onboarding is the probably one of the most important things that you can think of, and you think around what type of tool you're doing, what type of observably practices you have, and how you're going to like the introduce these these kinds of new colleagues into how you do things and explain it. I think that's going to be especially important for observability specifically, and for just the tech industry at large. So just having that that in your mind that the tech tech industry is gonna go through some shifts, it always does. But that's one of the things that I anticipate not anticipate with joy, but expect to see in 2023. 21:09 Greg Leffler: Yeah, I would agree. I think, you know, have play pipe storm, right, get get some skills, I would say in general, like learning new skills, and staying on top of things is always sort of how it is in our industry, right? Like you have to be, really, even if you don't have a personal interest in stuff, you kind of have to to have your skill set up to date. And like observability is still definitely like the new hotness, right? Like if you can talk instantly about it, you haven't implemented like, it's going to help you in your career. So that's why I did it. I, you know, I started talking about observability. Because I think it's really important. And I might like my personal passion is I want I want the site to never be down. But more importantly, like I want the customers to enjoy and get benefit from whatever application we're running. Right. And so that makes it work faster. Say of all that sort of stuff. So I didn't see. Do we have any questions? Did you see anything? Come in?would agree, I think, you know, have play pipe storm, right, get get some skills, I would say in general, like learning new skills, and staying on top of things is always sort of how it is in our industry, right? Like you have to be, really, even if you don't have a personal interest in stuff, you kind of have to to have your skill set up to date. And like observability is still definitely like the new hotness, right? Like if you can talk instantly about it, you haven't implemented like, it's going to help you in your career. So that's why I did it. I, you know, I started talking about observability. Because I think it's really important. And I might like my personal passion is I want I want the site to never be down. But more importantly, like I want the customers to enjoy and get benefit from whatever application we're running. Right. And so that makes it work faster. Say of all that sort of stuff. So I didn't see. Do we have any questions? Did you see anything? Come in? 22:04 Kelly Fitzpatrick: I did not. But Hi to everyone who's saying hi in the chat. Great to see a lot of hi’s. 22:08 Greg Leffler: Hello, thank you for joining us. I guess we'll wait a couple more seconds. See if anybody has a last second question they want to squeeze in. But if not, thank you for attending. I hope you really enjoyed it and got some value out of the conversation. Please check out pipe storm and the near and magical lens book. Please install open telemetry everywhere you can. Even if you don't use Splunk’s product we would like you to but if you don't, you know open telemetry really is gonna benefit here. So that's what I would say to wrap things up. Kelly, you have any last things you want to say? 22:41 Kelly Fitzpatrick: I mean, I think you said you said all the good things. A lot of pressure pressure me to come up with something you know something extra that you know other things and then Greg especially Do you have a have a great time at reinvent it sounds like you have you're gonna need a vacation after that. 22:56 Greg Leffler: I'm actually going to Sydney the week after to go to SRE Con Asia Pacific. So not exactly a vacation. Should be fun. I'm looking forward to it too. 23:08 Kelly Fitzpatrick: Great. 23:09 Greg Leffler: And yeah. And so recon APAC, if any of you are in Sydney, come join me. They're talking about open telemetry. So very, very good. Follow on for this, but it looks like there's no questions. So we'll let everybody go. Thanks again for coming. Kelly, thank you so much for these. I love these. I would do these whenever we want. Like I think it's a really great time. So thank you again for your expertise. 23:27 Kelly Fitzpatrick: Thanks for having me 23:28 Greg Leffler: Bye, everyone