[00:00:00] Mike: When you do a fire suppression test in your data center room, there’s a, what they call an active test and a passive test. The test you want to run is passive… Unfortunately, this person decided to do an active test with, which put our data center into full shutdown mode, emergency shutdown.

Introduction

[00:00:18] Jonathan Crowe: Hello, and welcome. Please come in. Join me. I’m Jonathan Crowe, Director of Community at NinjaOne and this is IT Horror Stories. Brought to you by NinjaOne, the leader in automated endpoint management.

[00:00:32] Jonathan: Hello everyone, and welcome to another episode of IT Horror Stories. I’m your host, Jonathan Crowe, and today’s guest is Chief Digital Information Officer of Netskope, Mike Anderson. Mike, thanks so much for joining us on the show.

[00:00:46] Mike: Excited to be here, Jonathan. Thanks for having me.

[00:00:48] Jonathan: So Mike, you come to the show with a ton of experience, over 20 years of being an IT leader (for) Fortune 500 global companies. Here you are a courageous, fearless, hardened, experienced IT leader. Surely that hasn’t always been the case. Tell us a little bit about your background and how you got here.

[00:01:10] Mike: Yeah, so my path to being a CIO is a little bit non traditional. I started out as a software developer in the 90s when we were all going through the dot com boom. Had the great fortune to have my own startup, which unfortunately a great learning opportunity, or kind of got, some of that courage tested was, you know, being caught up in the downfall of all the dot coms, which was a big learning experience for me.

[00:01:31] But that led me down a path where I was in technology companies or in technology roles for most of my career. But I would say for the first half, it was really around, you know, leading business development, sales, engineering, you know, sales teams, running operations for a Microsoft joint venture. Ultimately being a GM for a regional system integrator.

[00:01:51] Where, one of my customers recruited me to come be CIO, which was interesting because I was, you know, in my past when I was running a sales team, I was the person that brought Salesforce in and was the shadow IT person. ‘Cause I couldn’t stand the on-prem CRM system we were using, but I think it brought a right level of empathy into the roles as a CIO. And I’m, you know, starting year 12, being a CIO in my third company in that capacity.

[00:02:15] And, you know, it’s been a really exciting journey, but a lot of learnings along the road. Hopefully we’ll be able to talk about a few of those during our conversation.

[00:02:22] Jonathan: Let’s talk a little bit about your first CIO role. I believe the horror story, the first we’re going to get to anyway, took place shortly after you took that mantle. Is that right?

Fire Suppression Test Gone Awry

[00:02:34] Mike: Yeah, you know, within the, I was there for about five years and, you know, one of the, it was about year two, you know, we were back in the day before, you know, this whole public cloud thing and really taken off. You know, cloud was really, you know, early in the Microsoft 365 days, you know, we were on that journey and, you know, ServiceNow was, you know, still an early growing company.

[00:02:53] And, so we had our own data center and it happened to be in our building. We had a fire inspector come in, and we had a normal fire marshall we knew, an inspector and they would inspect all the fire systems in the building. And they had a new person that was a study, and our facilities group let the person into our data center room.

[00:03:09] And when you do a fire suppression test in your data center room, there’s a, what they call an active test and a passive test. The test you want to run is passive, which just makes sure all the systems are functioning, as expected. Unfortunately, this person decided to do an active test with, which put our data center into full shutdown mode, emergency shutdown.

[00:03:27] And we had a lot of systems. Unix systems that hadn’t been shut down in years, and it took us a couple weeks to really recover. Our CEO was a former CIO himself for major companies like Dell and PepsiCo, and he just looked over at me and shook his head, and I was, it was really one of those big kind of humbling learning moments, as a CIO.

[00:03:48] You know, we recovered from it. We changed some of our process around whenever someone goes in the data center, make sure they have someone from our IT team that’s accompanying them to make sure we don’t emergency shut down the data center in the middle of the business operations. But yeah, great learning opportunity. I look back now and laugh. It’s kind of funny, but at the time it was just a bit embarrassing. I have to say.

[00:04:10] Jonathan: Well, we really appreciate you sharing this story now that enough time has passed, so it’s not too painful. That fire suppression system, I mean, am I picturing things correctly – I mean, I almost imagine sprinklers going off and ruining everything. It wasn’t that regard, was it?

[00:04:24] Mike: No. Fortunately, there were no sprinklers. A lot of times you would use like a halon gas, that would put things out, so it did go into that type of mode. It didn’t suck all the air oxygen out. That’s usually, we didn’t have that advanced of a data center. Some people have those, where it sucks all the oxygen out so it puts flames out. We weren’t that advanced, but it did trigger the emergency shutdown so that, you know, servers weren’t still spinning up and overheating and causing, you know, worse damage.

[00:04:49] Jonathan: That risk that you’re taking in house you’re having your own data center is a great example. Having these older devices that are still running that no one has rebooted in years and years, and you basically are tiptoeing around them. You don’t want to touch them. Do you think that that’s, has that changed at all? I mean, having your role as a CIO now, and having, everything, you know, migrating to the cloud and removing some of that risk, do you feel like, you have an easier job in some regards or are things, is risk just shifted?

Shifting risk

[00:05:23] Mike: It’s really risk has shifted. I mean, I would tell you that after that experience, our journey to the cloud accelerated. Two things, you know, on the risk side, just because you move it to public cloud doesn’t mean that you’ve removed responsibility.

[00:05:35] I’m providing a service to them, and to our customers. And I can’t just pass blame on someone else and say, well, I’m sorry, they went down. It’s not my fault. It’s my job to make sure we have resilient operations, from a technology standpoint. So it’s still the same risk. It’s just, it’s moved around. But you have to inspect what you expect as well. You can’t just move it and then not inspect what you want to happen. Verifying then trusting, not just trust and verify.

[00:05:57] Jonathan: We think about horror movies, and there’s definitely the cases where there are some characters on the screen who are clearly doing something dumb. They are making a mistake. They are going to pay the consequences for that. In a lot of these IT horror stories, the IT team is suffering the consequences of actions that aren’t done on their part. And yet, as you pointed out, you can’t just say, well, AWS is down. Or I mean the fire inspector, you know, look what they did to us. You actually have to still step up and perform and correct the situation.

[00:06:32] Mike: What we say in IT, you know, there’s always the phrase, if it’s not broke, don’t fix it. You know, it’s the, you know, things break when things change. Now, those things can be changes in applications. It can be hardware that fails. That’s a change. It could be the fire marshall coming in and running the wrong test. That’s a change. Those are all different scenarios that we have to think about, and I think that has become even more prevalent with a lot of the security risk we see today. You know, we’re constantly having to think about, you know, if this happens, how do I react and recover to that? A lot of our efforts have shifted to, you know, you assume breach and assume that, you know, I have to figure out how do I recover when the worst things happen? And then how do I prioritize what comes up first, second, third, based on, you know, its impact to our organizations. And so that’s a lot of how we’ve been wired now, especially with how the threat landscape has changed over the last, you know, last decade.

[00:07:21] Jonathan: Absolutely. I mean, I think maybe one of the positives is, as those risks have increased and gained more notoriety, we have seen wider adoption of planning ahead of time. But let’s talk about security risks, because that brings us to our next horror story that you were wanting to share. You were suffering your first major, cyber incident.

Every CIO remembers their first cyber incident.

[00:07:43] Mike: Yeah. At this point for me, it was again, my first CIO role. You know, we had our first incident and everyone has one. If someone is a CIO and they said they’ve never had a major security incident occur, I don’t think they’re being 100 percent honest with you because, either that or they don’t know and it’s already happening. Which is, you know, you just assume that it’s already happening and you’re trying to figure out, you know, how do you make sure and, recover, respond and recover to it.

[00:08:05] In this case, it was early. It was, you know, fortunately before ransomware had really taken off. I think during my time, my first CIO role, we had a couple of people that had their individual machines ransomed, but this was, you know, your traditional, this is the credential harvesting phishing attack.

[00:08:19] We had a user fall prey to that. Fortunately, we had telemetry that helped us figure out and mitigate the spread across the organization, and we took the right actions. But you know, you’re looking like who do I go to? You’re going to your legal counsel. You’re talking to partners. You’re trying to figure out who’s going to help me in this scenario.

[00:08:36] And so that’s where we really had to lean in is, I went to my boss, you know, he had been a successful CIO. And he had some contacts that said, how do we respond to this? And he had a lot of good counsel. And fortunately, we were able to control the blast radius and make sure and mitigate it from being a much bigger issue.

[00:08:52] But that’s always a very humbling experience. The first time you go through it.

[00:08:57] Jonathan: It sounds like you were lucky enough to have those resources available. Did you have a feeling of, Oh, no, wait, the, the buck to some degree stops with me. Everything we’ve been preparing for, this is actually happening now.

[00:09:08] Mike: Oh, absolutely. You know, and then it makes you realize the importance of tabletop exercises and scenario planning. What happens if this occurs? I mean, one of the most interesting ones is like, what happens if you’re single sign on provider goes down or gets compromised. How do you get into your systems?

[00:09:22] If your active directory environment, if you’re using active directory, how do I recover if that’s compromised? Because I can’t go to email because they could be read by the threat actor. So how do I communicate in my organization? So going through those tabletop exercises and playing those out. You know, has to be part of, has to be part of your role today.

[00:09:40] Ultimately the accountability is with you, but it’s your job, security is everyone’s responsibility in an organization. It’s not just the CIO. It’s not the CISO. It’s everyone in the organization, including the board of directors. And it’s your job to make sure you’re bringing forward the risks, and having conversations about what’s your risk appetite and how much you’re willing to invest around the various controls you need to have and why certain controls are more important than others, depending on the business you’re in and the risk if those business operations are disrupted.

[00:10:06] Jonathan: The raised awareness around cyberattacks and risk, also with compliance, you know, the changes with cyber insurance and everything like that, I’m sure, are things that have helped you get buy in with leadership, but that’s something that’s specifically around tabletops and everything.

[00:10:21] Is this something that you’ve always had buy-in from an executive team for? Is it something that you’ve had to work towards? Any advice for folks who are out there who know this is something that they’d like to drive forward in their organization, but maybe they’re facing some headwinds?

[00:10:37] Mike: Yeah, I think it’s definitely picked up pace over the past several years. A lot of times because you’re, when you look at security questionnaires coming from your customers or when you’re looking at insurance, you’re asked, are you doing tabletop exercises? Are you doing pen testing? Are you doing all the things that you should be doing?

[00:10:54] And so that kind of forces those things to happen. I’d say if you rewind the clock 10 years ago, they were less common than they are today. I think it’s, you know, highly regulated industries like financial services and insurance and others, you know, they have large security teams. So they’ve been doing it for quite a while. But I think if you look across the broad spectrum of enterprises, it’s really, you know, caught steam in the past 10 years. Every year it’s gotten more and more important. I would say if you’re not doing it today, it’s something you definitely need to look into and do it at least on a quarterly basis, if not more.

[00:11:23] And it may not be the entire board. It may be, you know, part of your executive team. It may be a function, but you know, going through those scenario plannings, what do we do if this happens? And it really goes into not just security, but you know, your whole business continuity and disaster recovery. Going back to examples, there’s been times when, you know, major SaaS applications have gone down. You know, we had a, you know, field service application, you know, this is, you know, Salesforce had an outage several years ago and our field service application went down. And we didn’t have a backup. Our field service technicians couldn’t figure out where to go because it was all in the system. They couldn’t go work jobs.

[00:11:56] So we basically lost a day of revenue from a field service standpoint. And so how do you run your operations if the technology is not available? And those are things you also have to ask yourself as well as a CIO and business leader.

[00:12:08] Jonathan: When you think back to some of these stories and improvements that had to happen, maybe the hard way. Do you think that there is a flip side to some of these, incidents or some of these horror stories where… you do come out stronger on the other side?

[00:12:25] Were there some initiatives that you’d been asking for, for a while that suddenly got greenlit or anything like that?

[00:12:32] Mike: We definitely got security and the security investment that we put forward on the next budget cycle definitely got approved. Because we went through it. So that helps, you know, a lot of times it’s, don’t let a disaster go to waste, I think is one of the phrases we always use in the world.

[00:12:45] How do you leverage it to grow, you know, both your team and budget sometimes, you know, in that case. Things are going to happen. Things are going to go wrong. It’s how you respond to it and how you recover. It’s the resilience of your team. You know, when I, a lot of times when I go to universities and I work with people that are early in career, I’m bringing people on the team, but one of the things I’m looking for is resilience. I want to ask people like, tell me a time that you failed in your life or in your career and then what did you learn from it and how did you respond?

[00:13:12] And I want to learn, I want to hear their stories. Because it’s those… I know we’ve all had them. And we need to be able to understand what happened and what do we learn for how did we grow? The one thing I’d say that I see is a common problem is, things will occur and we spend a lot of our energy focused on trying to assign blame on someone who’s at fault for this, you know, this problem.

[00:13:33] And I feel like that’s a lot of wasted energy. We should focus our energy on why did that occur? And then how do we make sure it doesn’t happen again? And what did we learn from it? But don’t spend time trying to assign blame because that’s working in reverse. Figure out why it happened. And then how do you solve for it in the future,

[00:13:49] So it doesn’t happen again. And I think that’s where people need to be solution focused. Don’t be, you know, looking at assigning blame.

[00:13:58] Jonathan: I’m sure it’s just like any horror movie, right?

[00:14:00] Some things seem obvious from the perspective of the audience or in hindsight. Of course that went wrong. You can see all the things that are leading up to this crisis or this thing that happened, in hindsight, but some things are truly, you know, like you say, things are going to happen.

[00:14:18] There are going to be surprises that are going to be human error. There’s going to be mistakes that happened. And what really matters then is, you’ve got processes that are flexible, you’ve got the resiliency. The people on your team who are able to, to feel like they’re trusted and can respond and will be supported.

Experiment, and fail fast.

[00:14:35] Mike: You want to fail fast, you know. And, you know, you want to test and learn. And, you know, that means that not everything’s going to work. And that’s okay. You need to make sure that, you know, you have that expectation, but if you don’t experiment, then the innovation is directly impacted by that.

[00:14:51] And so, you know, a lot of times, you know, as I was listening to another podcast myself and a lot recently and also in a book I read, you know, we think about like the VC mindset. A lot of times in organizations, we don’t take the big risks. We take the small risk because if it doesn’t pay off, the ramifications can be impact us personally, can impact our job.

[00:15:11] But if we don’t place those big bets, we don’t experiment. Then the big days where it’s transformational for our organizations, those things aren’t going to occur. And so I think we have to make sure we’re making calculated risks, but we also have to make sure we’re creating a culture where people can experiment because that’s how we’re going to drive real innovation inside of our organizations.

[00:15:29] Jonathan: So let’s talk a little bit more about that, in your role as a CIO. Are you thinking about taking risks in terms of where you’re leading the IT org, making some big bets that way? Or is your role, do you think of your role more of, IT is there to support the organization when they decide to make those big bets?

[00:15:48] Mike: It’s got to be both. I mean, I look at it. IT is just another function in the business. We’re all part of the business we run today. And, you know – we have digital natives in our workforce today. That’s our new work population. So technology is part of everyone’s job in the organization.

[00:16:06] And so our job is CIOs to be, to be the, you know, custodian of that process, put the right governance and the right processes in place. You know, we are taking calculated risk, but we’re doing it together. If I’m sitting down with legal, what are we doing to drive efficiency within the legal process?

[00:16:20] How do we remove, you know, choke points within our business processes. So we can create a better buying experience for our customers. There’s two things I look at. One is, how do we make it easier for customers to do business with us first and foremost?

[00:16:32] And so how do you take friction out in the buyer journey? Where are the bottlenecks and the choke points that are causing friction? The second thing I look at is how do you make life easier for your frontline employees so that they can be more productive?

[00:16:44] They don’t want to spend time, you know, how do we meet people where they’re at? How do we drive self service? How do we automate process? How do we remove unnecessary approvals? You know, and so I think that’s the, and so when I think about calculated bets, I’m looking at, you know, agentic AI and saying, okay.

[00:16:59] We need to be thinking about that today, because if I look in our own world, you know, we have all these different bots we interact with within our Slack environments. Well, as an employee, it makes sense to us as IT, because we live in it every day, but for our employees, they don’t. And so how do I start thinking about the interaction with those bots?

[00:17:15] And can I, if I think about agentic, can I create the experience where I go to one place, the Netskope Agentic AI bot, and then it decides, Oh, it looks like you want a PTO request. Well, I need to go talk to the Workday bot for you and make that request happen. And so I think, you know, we’re looking at that as like, that’s a calculated bet that can have a direct impact on the experience of our employees, their productivity.

[00:17:36] And then as a result, they’re going to be able to create a better experience for the customers that they’re facing off to every day. And so those are the kind of bets and things that we’re thinking about from a technology standpoint.

[00:17:46] Jonathan: I wonder if the bigger horror story is by taking some of these big bets, maybe you’re exposing yourself to some of those things, but you could argue that you’re always exposed to those things and I wonder if the bigger horror story is, IT not being involved in any of those decisions to begin with. You know, you don’t have a seat at the table and so, any advice for IT leaders who are listening, who are maybe not at that CIO level yet, but, what are the skills, what are the things they should be working on if they have aspirations to get there?

A few notes for aspiring CIOs.

[00:18:19] Mike: You know, so there’s a few words of advice I have. First and foremost is you have to be an expert in the business you’re in. So, you know, the first thing you need to be doing if you want to be, you know, if you have aspirations of being a CIO is get out and learn your business. You know, if you’re in manufacturing, walk the plants, meet the people, understand – one of things I tell my team is follow the revenue. How does your company make revenue? And then what is your impact on that process? And so make sure you really understand how money is made inside your organization. That’s a skill set to have. You know, I’ve always encouraged people that are on my teams that have aspirations to move outside of IT into a role where you have a profit and loss where there’s a P and an L, not just an L associated to it.

[00:18:59] Because you start to understand as a person that’s running a P and L that has a revenue line, when they go make investments in technology, they’re either signing up for a higher revenue number to cover the cost of whatever that investment is, or they are making a decision not to hire someone or to cut an expense in their area to fund whatever it is you want to go do.

[00:19:17] And so you have to make sure when you have that, you start to have more empathy for, well, yeah, I want to go hire five more people for my team, but at what cost to the organization. So you have to be, think of the bigger picture and realize that, you know if I’m getting, if I’m getting a dollar, someone else isn’t getting a dollar.

[00:19:32] It’s coming from somewhere, and it’s either coming from revenue growth, or it’s coming from cost in someone else’s area, and so you always have to think about the tradeoffs and that kind of how you balance those investments that’s one of the areas you really have to think about as well.

[00:19:45] Jonathan: Whenever we ask IT people to name the villains in their own horror stories, often they’ll think of users, and it sounds like what you’re advocating here is really bridging the gap. Making sure that you have empathy and understanding of how the business operates and how IT can be better partners.

[00:20:03] Mike: Yeah, I mean, I think one of the things is you think about a lot of companies or teams have shifted from a project operating model to a product operating model. I think the evolution, which you need to think about is and this all rolls around kind of an agile mindset and agile principles. If you can start to create product fusion teams, which aren’t just members of your IT organization, but they become cross functional in nature. So when you think about traditional, like an agile team, you have a product owner and scrum master that may be part of that. But, you know, the person that holds the designation of developer doesn’t have to be a developer from IT. It could be, you know, a person that’s a subject matter expert within a function in the organization. It could be someone in your HR team. It could be someone on the finance team. It could be someone that owns part of that capability.

[00:20:47] So I think if you really want to drive innovation, I’m a big believer that dedicated teams get stuff done. And so if you can create a small team that owns an outcome and then can move quickly and you eliminate those handoffs, that allows you to move really fast. And so a lot of times that’s what I’m trying to set up inside Netskope.

[00:21:04] And the same thing we did when I was a Schneider Electric. Was really setting up, you know, the squads or cross functional teams that own an outcome and it didn’t matter where you reported in the organization. You were part of that team driving the outcome together. That’s a winning recipe for success.

[00:21:20] Jonathan: What it sounds like too, it’s one way to, shift things from, the reactive, you know, purely seeing IT as the, the reactive cost center, they’re there to put out fires, nothing more really. It sounds like a way to shift things away from that.

[00:21:35] Mike: If you think about a startup, you know, oftentimes we’ve been asked the question, how do we become more nimble and be like a startup? You know, I can’t tell many times I’ve had that question come up. Well, if you look at a startup, you’re struggling to survive. And by definition, you have a small team and you’re working together to a common goal. And everyone sits together and says, okay, you take this task. I’ll take this one. Can you sweep the floors? Yeah, I can sweep the floors and everyone kind of does the role.

[00:21:57] But as you grow your organization, then naturally, functions get created, which creates silos. And then we tend to operate and get comfortable working in the silos. There’s a great book by Matt LeMay called Agile for Everybody, where he describes some of those laws of organizational gravity.

[00:22:12] And one of those is we get comfortable in those silos. And so if we break those silos down and start thinking, my job is to develop people with certain technical disciplines. The finance organization’s job is to create, you know, and develop skill sets around the finance. So if we have an outcome that requires finance and technology.

[00:22:27] Let’s create a team together with those two different groups, working on an outcome and think about ourselves as developing talent with certain skill sets, not as “This is my silo. Stay out of my silo. You stay in yours.” So that’s how we act more like a startup is create those teams, a team of teams and those small teams that work together that can own an outcome and move very quickly inside the organization.

[00:22:47] Jonathan: Any other. recommendations for IT leaders who are listening to that and they’re nodding their heads and maybe they are in a position where their role as an IT leader, the expectations as a CIO are a little bit different that they’re more traditional. Let’s say. How [should they] start chipping away at that or maybe changing that mindset and getting buy in for a different approach like that?

“We’re all people first.”

[00:23:16] Mike: There’s 3 words of advice, 3 things of advice I’d give to people from a leadership standpoint. You know, the 1st one is your job as a leader is to ignite a fire in someone not under them. And so if you’re constantly, you know, do this, do this, you have to get done by this day, you know, you’re lighting a fire under someone instead. If you tell them, Hey, this is where we’re trying to go and this is what winning looks like. So when you know, when you get there, right, that’s the first thing is, you know, light a fire in people and make sure people understand what winning looks like so they know when they get there. The other one is we’re all people first. So connect with your teams and everyone is a person first. Break bread with people, get to know them. And I think that’s one of the things we’ve been challenged a lot in the hybrid world, is, especially people that are fully remote is when you get to connect with people as people. It’s harder, but you can do it over Zoom or over Teams calls, but it can happen. But get to know your team. Take care of your team as people first. And then the other thing to realize is you want to empower the people closest to the problem to solve the problem. I used to tell my teams, you know, I like to go on vacation. And if I’m expected to be the one that solves the problem and makes all the decisions, I can never go on vacation and really disconnect and enjoy. And nor am I the person that has all the information because I’m not living that problem every day. The person that’s working with it every day is.

[00:24:30] So what are they recommending? And so as a leader, we need to empower the people closest to the problem to solve it. And that empowerment will help you create a high performing team. And I think what people often forget is that if you want to move up and grow in the organization, A) you have to have a successor, but it’s really your reflection is how are you building a hyper high performing team?

[00:24:49] That’s really what your job is. It’s not to be the problem solver. It’s not to be the hero. It’s really to empower your team to do amazing things. And to me as a leader, that’s the most satisfying thing as leader for me is seeing someone that’s been on a team before that’s worked on a group that I’ve had the opportunity to lead and then see them do amazing things.

[00:25:04] I love investing in people. I love early in career programs. I’m the executive sponsor for our, early in career employee resource group here at Netskope, you know, I love, you know, developing talent in the organization. And I think that’s something you really have to think about as leaders. How are you developing your team?

[00:25:18] How are you investing in them? Because that’s what’s ultimately going to be the measure of success.

[00:25:23] Jonathan: I mean, you clearly have an investment, an authentic investment in people.

[00:25:28] I wanted to give you time to talk a little bit about some of these additional causes that you’re volunteering for.

[00:25:34] Mike: Yeah, so 2 areas I’m passionate about. One we talked about is the how do we develop more talent in the, you know, in science, technology, engineering, mathematics, so in STEM. So that’s one of the areas where I contribute time. And it’s really because I feel like we have to build our future workforce and we need a diverse workforce that we can pull from.

[00:25:51] That’s an area that I’m very passionate about, you know, and so it’s a, you know, we have to grow a higher population of people from diverse backgrounds and different genders that we can then pull from. So when interviewing [from] the candidate pool, it has equal representation, right?

[00:26:03] You know, because if the candidate pool, 90 percent of the people look the same way and 10 percent have the diversity or different gender that doesn’t give us that. That’s not what we’re looking for. And we have a talent shortage, honestly, especially in security. The other areas around mental health, you know, I had, unfortunately, lost my niece by suicide back in 2018.

[00:26:21] So it was a family trauma for us and it just, reinforced how important it is to look after mental health and look at suicide prevention. And I think a lot about, I’ve been a big advocate for that, you know, since that time, being able to use the platform that I’ve been given as a leader to talk about that, you know, and one of the things that I tell people, look, it’s okay to not be okay.

[00:26:39] You know, I’ve had those moments where I have to be mindful. I have to step away. You know, I have to kind of get recenter myself. I think we’ve all had those kind of moments where we’ve had anxiety or had issues that come up. And so it’s okay to not be okay. And I think we have to show that vulnerability to our teams because they see us as a real person then.

[00:26:54] And so, and it helps us again, connect to people as people.

[00:26:58] Jonathan: I appreciate you calling that out because it is something that I think is prevalent all across the board, no matter what field you’re working in. But certainly in tech and IT, I mean, we, this is a topic that comes up a lot. Self care is something that’s hard to do in jobs like this and especially in hybrid roles and everything too. And so glad that you.. you know hats off to you for doing that and glad you had a chance to talk about it here. Having put that in perspective, let’s close with another a final horror story that I’d love for you to share. And this is, going a little bit further into your career, a little bit more recently, I believe. You know, this is another one of the type of horror stories that I find that we keep coming back to where not everything can be fixed digitally, you know so much of the focus is on obviously software and information technology itself, but in a lot of these cases. You know, IT work is very, tangible. You know, there are some physical things that you’re working with here. I think this is a really good example of it.

[00:28:00] Mike:Yeah. So, in my previous CIO role before Netskope, we had a distribution plant in Pennsylvania, it happened to be the last day of the month. Luckily, it wasn’t the last day of our quarter, but it was the last day of the month. And, there was an outage with the provider that provided the entire fiber connectivity, internet connectivity for most of the, you know, Hershey, Pennsylvania area and Harrisburg and some of the surrounding areas, the whole backbone was down.

[00:28:22] So lots of customers were impacted. It was one of those things, those tales of like, it just can’t get any worse. And I remember being on the call, you know, one of the beliefs always had is I would never have my team do something that I wouldn’t do myself. So I remember I was on the red alert call all day, until we got the issue resolved. I remember the provider that provided the service is a sub to the company we worked with. The card that they needed to replace that failed they thought they had a spare in the office where… in the location where the outage was at. In fact, that actually wasn’t there.

[00:28:53] The closest one was in Washington, DC. So they were gonna have to, and it took them a while to figure that out. So they were actually gonna have to have someone drive up. Meanwhile, it so happened that we had put forward a plan to put backup cellular connectivity into all of our plants and all of our distribution facilities.

[00:29:08] We did it in a few spots, but the full project wasn’t funded. and so we had no cellular backup at this distribution facility. And so we were basically because everything was running in cloud or data centers that weren’t located on prem, you know, we were basically dead in the water.

[00:29:22] We couldn’t, you know, get shipping systems. And so we were in a bit of a panic. So we ended up having someone from Nashville basically get in their car and drive a cellular modem up to the distribution center. So we could get backup connectivity, which we did. It was albeit a bit slow. This is pre 5G. This was LTE days. But we were able to start to get things going back up at a very slow pace. And then fortunately, by that evening, we got, they got the card up from DC and things started running again. We did have a little bit of slippage of some revenue from that month to the next. It could have been a lot worse, especially if it was end of quarter.

[00:29:54] But a lot of lessons there. I can tell you that entire cellular backup plan got funded immediately after that. The question was, how quickly can we get this rolled out? And, so, you know, I think that gets the lesson from that is, you know, one leaning into your people, but then two, it’s when you it’s to really talk about what’s the risk.

[00:30:11] And so when you sit down with your peers, if you’re talking to a supply chain peer, say, if we don’t this is the investment. The risk is if the, if… if something like this does occur, here’s the impact that’s going to happen. Your operation and how much is, what is that going to cost the company?

[00:30:24] In revenue and time and because you’re paying people to stand around in a facility with no connectivity, they can’t do anything. And so there’s a cost absorption there as well. So making sure when you think about technology investments and you’re thinking about risk, what’s the risk if this occurs? And then what’s the investment that’s required to mitigate that risk? And that’s how we have to make all of our decisions.

[00:30:43] Jonathan: I imagine, too, maybe, part of that contingency plan is having a fast car somewhere available for someone if they need to take it. I’m just imagining someone, really keeping an eye on that speed limit, hoping they don’t get pulled over, but really pushing it as much as they can.

[00:30:59] Mike: Yeah, I think maybe Uber helicopter, you know, maybe there’s something that’s something there that can be had, you know, we’ll see, maybe we’ll get particle acceleration, you know, figured out and we can just, you know, you know, beam it over there. But, you know, you know, we’ll definitely get there.

[00:31:11] You know, I think when manufacturing, it’s always interesting because you never have a manufacturing facility in the middle of a big fiber hub. It’s always in the middle of a rural area, an industrial area. And a lot of times there’s one pipe coming in that building. And so you have to really think about how do I provide redundant connectivity?

[00:31:25] So which is, I think one of the more important things when you think about applications with everything running in the cloud, if the internet goes down, you basically just shut off your business. So how do you make sure that your critical operations have resiliency when it comes to connectivity within all your sites.

Closing

[00:31:39] Jonathan: Any advice when it comes to, you know, a situation like that, you’re relying on others, you know, you can plan as much and have, contingencies yourself, but, you know, you’re working with that external provider who, had to be in DC and get the card up and everything, any advice in terms of working with partners in that regard and trying to distribute and, and reduce risk that way?

[00:32:01] Mike: Yeah, I think I always encourage people, our partners are part of our team. They’re an extension of our group. And so make sure you know, there’s you either have vendors or partners. I always like to think about it as partners. We’re in this together. And the question I always ask is when there’s a, when there’s a problem, are they in the foxhole with me? Are they standing on the sideline in the peanut gallery?

[00:32:16] Basically everything you do that’s positive, or when you’re helping someone you’re making a deposit every time you’re, you’re asking for something, you’re making, or do something bad. you’re making withdrawal. And so think about all your relationship accounts with people, including your partners.

[00:32:29] And how are you making deposits? So when you need to go make a withdrawal, you’ve got a bank account balance to draw from. And I tell that to our salespeople. When you’re calling a customer, if you’re at, when you’re asking for a meeting, you’re asking for a withdrawal where you’ve never made a deposit. So how are you making deposits in relationships?

[00:32:44] So you have bank account balances to draw from when you need to, when you need to make a request.

[00:32:49] Jonathan: Well, Mike, on that note, I think, we’ll call this one a wrap. Thank you so much for taking the time and really appreciate all your insights. Appreciate you reliving those horror stories with us. And most of all, sharing your outlook and your advice for up and coming leaders.

[00:33:04] Mike: Thanks, Jonathan. It’s great to be here. And, you know, it’s the horror stories that help shape who the people we are today because that helps build our resilience and test our perseverance. And that’s how we learn. You know, we learn more from the failures in life than we do from all the positive things that happen.

[00:33:18] Jonathan: Extremely well put. Thank you again, Mike. And thank you everyone for listening. We’ll see you next time.

[00:33:24] Music transition.

[00:33:24] Jonathan: Thanks for listening to this week’s episode of IT Horror Stories. For even more information and resources on how you can beat IT misery and transform your IT, check out NinjaOne.com. Also, in case you missed it, check out the recording of Backups and Bandwidth, our D&D inspired tabletop exercise that we put on for World Backup Day.

[00:33:44] We had a party of IT experts face a diabolical backup disaster scenario and rise to recovery. Head over to youtube.com @NinjaOne.Official for more. That’s all for this week. We’ll be back with more soon.