In this episode, information security analyst in the professional sports world, Matthew Schuster, joins host Jonathan Crowe to discuss Matthew’s harrowing IT horror story from the early days of the Covid pandemic. Matthew shares how his small but mighty team chased down and shipped out laptops to support his organization’s quick transition to fully remote work (cutting a 12-month planned rollout down to just three days), how a dedicated leadership team helped his team knock it out of the park, and lessons learned from operating under pressure.
IT Horror Stories
The scariest stories in IT.
About This Episode
Host
Jonathan Crowe
Director of Community, NinjaOne
Guest
Matthew Schuster
Information Security Analyst, Professional Sports
About Matthew Schuster
Matthew Schuster has spent six years navigating the world of IT operations and information security within the fast-paced professional sports industry, steadily rising from an intern to various technician and analyst roles along the way. In addition to all things ITOps and security, he’s passionate about cars, motorcycles, baking, and 3D printing!
Audio Transcript
[00:00:00] Matthew: And so, you know, came in, got everything done and, we didn’t have the biggest space at the time. It wasn’t like a full lab set up, but it was to the point where every outlet, every ethernet port, every square inch of carpet and desk that we had was just taken up by laptops. Everything beeping away, just going and imaging everything we need to do. Like those classic scenes in the library where the ladder is moving back and forth and you’re just going from one side to the other, just one laptop to the other, putting in credentials, entering in configurations, and stuff like that.
Introduction
[00:00:26] Jonathan Crowe: Hello, and welcome. Please come in. Join me. I’m Jonathan Crowe, Director of Community at NinjaOne and this is IT Horror Stories. Brought to you by NinjaOne, the leader in automated endpoint management.
[00:00:41] Jonathan: Welcome to another episode of IT Horror Stories, the show where we invite IT leaders to relive their most harrowing moments for our listening pleasure and entertainment. I’m your host, Jonathan Crowe, and I have with me Matthew Schuster. He’s an information security analyst in the professional sporting world. Matthew, thanks so much for taking time. Welcome to the show.
[00:01:02] Matthew: Thank you very much for having me, Jonathan. Happy to be here.
[00:01:05] Jonathan: We’re happy to have you and before we get down into the nitty gritty here and listen to your tale of terror, let’s learn a little bit more about you. Tell us more about yourself and your role now, currently as an information security analyst.
Leading off
[00:01:19] Matthew: Yeah, so as you said, I’m an information security analyst currently with a team of two on the security side, but we have a pretty large IT department otherwise that we also help with or help and work with. So I’ve been doing this currently for coming up on six years now. Started my way up all the way as an intern, kind of worked my way up through the farm system, as we say, through technician role, another analyst role on the ITOps side, and then now officially over on the information security side.
[00:01:44] We kind of equate it to the whole farm system thing. So, it’s, you know, starting out in minor league, working your way all up through the big leagues, or to the big leagues, rather.
[00:01:50] Jonathan: Let’s get right into it and set the scene for your horror story. What stage was this at in your career? What was it looking like in terms of the team? Kind of like, what was your, what was your status?
[00:02:04] Matthew: At the beginning of 2020, a time that I’m sure a lot of people are familiar with. The team was a little bit smaller than we currently are now. I was… had just finished my one-year mark as an intern at the time, over on the IT operations side. And had worked with a lot of the different sub departments in IT, such as, you know, networking and security, stuff like that, and was primarily still on the help desk side, you know, doing the whole day to day stuff.
[00:02:24] Jonathan: And so you said you were just finishing interning. Were you, was this a full-time role or were you part-time?
[00:02:29] Matthew: Yeah, so it was a part-time role. At the time, I was also a full-time student at the university.
[00:02:33] Even back then, the three full-time people we had, had more than just IT operations side. So our manager also did full-time telecom with mobile devices, which that’s a full job in and of itself while also managing the help desk, you know, two roles there.
[00:02:50] Jonathan: So you had a team around you. Everyone’s kind of doing, wearing multiple hats, doing different things. Tell us a little bit more about your mindset here at this outset. Are you still trying to prove yourself at this point in your career?
[00:02:59] Matthew: Oh, absolutely. I mean, as everyone, everyone that’s kind of still in college and learning a lot, and then also kind of, you know, jumping in as an intern is still absolutely trying to prove themselves. But at the same time, I was just trying to be an information sponge, trying to absorb as much as I could. And that’s one of the benefits that we had of wearing a lot of those hats, is that you kind of got cross trained on a lot of stuff and you had a huge opportunity to learn a lot of stuff, which you know, just entering the IT world, there’s a lot of sub departments in there. And so it kind of helped me figure out, okay, do I want to go networking? Do I want to go software? Do I want to go, security? And obviously ended up on the information security side. So it’s a huge benefit.
[00:03:31] Jonathan: So, when we talk about horror stories in IT, people can quickly go into different things that make their job challenging or make it scary for them in various ways. That can include the org they’re in, the team, the users they’re having to deal with. Tell us a little bit about your satisfaction with the org at the time. Were there any scary elements that were already existing prior to this particular story?
[00:03:55] Matthew: We had a great, you know, actual team that I worked with, and then above us we had an amazing leadership and still do have an amazing leadership team, which truthfully is one of the best things you can have. And so, you know, as an intern. Coming in, especially if it’s your first business job, things can be daunting. You know, trying to learn the business side of things while also trying to absorb as much information as possible.
[00:04:12] Jonathan: It’s a fast paced, a lot of going on. But positive environment, we have bright eyed Matthew, young in his career looking to make a difference here. Then what happened? Let’s get right into the incident. What changed on this fateful day you’re going to be talking about?
The day of the incident
[00:04:28] Matthew: Yeah. So I want to emphasize what you said. Fast paced, absolutely fast paced, everything, you know, it’s not just, you know, for us, of course, we’re in the sports industry and, you know, it happened to be February and March at the time, and that’s the off season. But there’s still a lot of stuff going on.
[00:04:41] So I’m sure again, it’s 2020, a time that a lot of people in and outside of IT are pretty familiar with, happened to be the onset of COVID. And so we were tasked, with from the leadership team of setting up, and transferring all of our current work, like at the stadium employees, to being able to work from home full time remote.
[00:04:57] Before that, we had, you know, policies and procedures like other companies where you have mandated work from home days just to make sure systems and work and stuff like that. But this was full scale work from home where office shut down, everything like that. And at the time, we had probably, I want to say about a hundred to 200 people that had full desktops here at the stadium.
[00:05:18] That needed to be swapped over to laptops. and the time frame of that was three days. And so you can imagine a team of, you know, three full timers and two part timers at the time while also doing the day to day stuff of, hey, you know, what’s going on? I need help with my keyboard. You know, my printer stopped working.
[00:05:35] Also trying to set up between 100 and 200 laptops.
[00:05:38] Jonathan: You’re acknowledging 2020, the pandemic hitting, everyone has their story, right? Everyone is remembering what it was like for them. But for you guys, those three days, where did that come from in terms of, did this, was this a very sudden kind of onset of a project? Was there any kind of anticipation towards leading up to it?
[00:05:58] Matthew: Yeah. Oh, I mean, there was so much speculation going on, you know, everywhere. Like, no matter where you were at, no one kind of knew what the full picture was. Thankfully, again, we had a great leadership team, and they came to us with the proposition of, Hey, this is what we’re looking at doing. What do you guys need to enable that?
[00:06:13] Rather than just saying, Hey, you know, we’re shutting down in two days. This is how much time you guys have. They came to us and said, Hey, you know, we need a timeline here. What can you guys get done? And so we ended up saying, okay, we need a little bit more time. Obviously. And just trying to get that done as fast as possible, while not being, Oh, you know, we need like a month to do this. You know, you gotta be a little more realistic sometimes, and you gotta be super fast paced and just adapt.
[00:06:34] Jonathan: So tell us a little bit more about what this looks like in your environment. How disruptive potentially was this looking like for your team?
[00:06:44] Matthew: Yeah, so it was more so just like the day to day business stuff is what they were using it for. It was the off season, thankfully, so we didn’t have a lot of different, you know, obviously didn’t have to worry about the product that we were putting out on the field, but we had to do a lot of the day to day business stuff like any other company has to do, you know.
[00:06:57] Trying to market our product, trying to get people paid with finances, trying to make sure that people are employed and staying employed with HR, and then, you know, of course, trying to sell our product on the ticket sales side of things. And primarily that’s what we were trying to enable was the ticket sales folks of just being able to go home and do what they need to do.
[00:07:11] We had a process beforehand for setting up and imaging these laptops from scratch. But as we’ll explain later, it wasn’t the easiest and probably the most efficient process at the time.
[00:07:21] Jonathan: I think a lot of people probably had something on paper there and then they get to see, I mean, this is, real life testing. They got to see how, really kind of pressure test those things in real time…
[00:07:31] Matthew: Yep.
[00:07:32] Jonathan: With the pandemic, like, as you said, a lot of things being up in the air for a while, people not being really sure what was happening or what wasn’t. And then I remember a key point of course was, Tom Hanks in Australia, but also the NBA, shutting down games and it really, that being a point of like, okay, this is actually a thing that’s gonna be very impactful.
[00:07:51] Matthew: And that’s kind of the transition period that we were at is we knew things were going to be, things were going to hit pretty soon. We didn’t know exactly how hard they were going to hit, but we knew we had to be prepared for it.
The 72-hour countdown begins
[00:08:01] Jonathan: You were given initially those three days. You have to transition everyone who is used to using a desktop, you have to get them laptops. That clock… you’re able to successfully push back a little bit, get a little bit more time, but now the clock’s ticking.
[00:08:14] Matthew: So immediately just started working, you know, hit the ground running, set the treadmill to 10 and started running at 11 kind of thing, you know? It was an all hands on deck thing. Again, like I said, the other full time people that we had on the team, were also doing these other telecom and security and coordination. And they didn’t just put those to the side. They kind of overclocked themselves and did all of those things while also doing all of these laptop imaging and stuff like that. It was a complete and utter, you know, group collaboration thing that wouldn’t have happened if it weren’t for everyone total.
[00:08:41] Jonathan: And so that’s the team’s effort. What about yourself? Personally, do you remember the first things you were doing?
[00:08:46] Matthew: I remember being extremely thankful at the time because I was on spring break. And they announced, during school announced that, Hey, you know, we’re going to go on an extra week of spring break. And, you know, none of us really thought about the consequences of that. We were just like, Hey, you know, an extra week off is pretty sweet.
[00:09:00] And thankfully that lined up at the same time that I was, that we needed all these laptops done. And so, you know, came in, got everything done and, you know, we didn’t have the biggest space at the time. We just had, you know, a bunch of desks and everything. Didn’t have like a full lab set up, but it was to the point where every outlet, every ethernet port, every square inch of carpet and desk that we had was just taken up by laptops.
[00:09:19] And just everything, you know, beeping away, just going and imaging everything we need to do while just kind of, like those classic scenes in the library where the ladder is moving back and forth and you’re just going from one side to the other, just one laptop to the other, putting in credentials, you know, entering in configurations and stuff like that.
[00:09:35] Jonathan: So as an IT person working at a professional sporting stadium, like a location. Yeah, what does that setup look like?
[00:09:43] Matthew: Depending on the sport you’re in and depending on the team you’re in, and depending on when the stadium was built, there’s a lot of different factors that you’ve got to take into account. And thankfully here, we have a relatively new stadium, you know, all things considered, and so we have dedicated office space here at the stadium.
[00:09:55] And so because of that, you know, it’s nice that we’re here and able to support everything, but obviously, the big attraction of the stadium is the seating and stuff like that. And so, you know, it’s not just we have offices in one central location, we have offices throughout the entire stadium. And so, you know, trying to run around of like, hey, you know, I’m setting up this laptop. Now the person on the other side of the stadium also needs help doing something.
[00:10:14] Jonathan: You’re getting your steps and you’re getting your calf workout for sure.
[00:10:17] Are you guys staying on top of things, knowing how you’re going to break down this task. It’s putting laptops into place, but, did you have basically like an assembly line going out? Like, how were you tackling this?
“We had the procedure. It was just a matter of doing that procedure 100 to 200 times.”
[00:10:30] Matthew: That’s exactly it. Thankfully, again, we had the procedure. It was just a matter of doing that procedure 100 to 200 times. And so from a literal standpoint, it started at imaging the laptop, based off of a base image that we had, over the network. Which, it was a pretty large image, and so each laptop, the imaging process itself, completely hands off would take anywhere between one to three hours, depending on what type of laptop it was.
[00:10:53] With the square inches instead, or square feet that we had, and then also the amount of, you know, network that we had available, you could only do so much at one given time. Everyone just started imaging as much as we can. And then from there we’d break it down of, okay, you know, we’re going to have you work over on the configuration side of the profile. We’re going to have you keep imaging stuff or just keep kicking them off as you can. And eventually it got to the point where we just have, you know, the entire process down just sped up and everything ready to go.
[00:11:17] Jonathan: A big question I should have asked earlier. You had all these laptops just lying around, ready to go?
[00:11:22] Matthew: Thankfully, we were already looking at moving everyone over from desktops to laptops. Had all these laptops already in stock in our warehouse here at the stadium. We were just waiting to, or like we would roll them out over the 12 months rather than three days. And so that’s a huge, huge benefit on our side. Thankfully, it was still early enough I feel like in the speculation of COVID, where we probably could have gotten those laptops if we needed to, it just would have added complexity to the whole ordeal, which thankfully we didn’t have to.
[00:11:48] Jonathan: Were there any other hurdles that you had to get across? Anything that didn’t go as planned?
[00:11:53] Matthew: The other hard part is, now trying to teach the users of hey, you know, a laptop is pretty much just a desktop with a keyboard attached to it, right? So, like, the hardware itself, not much different. The hard part’s teaching them the different things you need in order to work at home. So, like, using a VPN, for example, and connecting to business resources here at the stadium. That’s probably the hardest part. Because, again, a laptop, you set it and forget it kind of thing. It just takes time. Every user and every person will understand and kind of adapt to the teachings differently.
[00:12:22] Some people, you can just hand them a page of documentation, say, here, here you go, go for it. Some people obviously are going to need a little more help than that. And mind you, we were doing all of this while still setting up those laptops. So it’s, you know, we get one laptop done. Hey, here you go. I’m going to give it to John Smith.
[00:12:36] And now I’m going to teach John Smith how to connect to our VPN and, you know, how to use the phone system from home and stuff like that. Use softphone and all of that.
[00:12:44] Thankfully, again, leadership team’s fantastic. We have that budget that we can be proactive rather than reactive. And so we have a very set time frame, of like, hey, you know, your laptop’s X years old, we’re going to give you a new one. One hurdle that was unintended from that was now every three years or every four years or however long it is.
[00:13:00] We have a very large list of laptops to refresh. You know, versus 2024 versus 2025, there’s like a hundred laptop delta there, which, you don’t really think about cause you just kind of push that under the rug kind of thing. But it comes up later on.
[00:13:14] Jonathan: So, you guys were running around, you’re operating under controlled chaos. Everyone’s working together to try to make this happen. What was the end result?
The end result
[00:13:24] The end result, I mean, we got it done, thankfully, as everything you have to, just get it done. And obviously everything with, you know, the situation in COVID after that kind of transitioned, but during that time period where everyone was working from home, that gave us time to kind of think about and sit back of like, hey, we got through it. If we had to do this again, which hopefully we don’t, how can we make things better? And not only just adapting, you know, like that emergency plan. But also just improving our day to day kind of stuff. And so, the changes that we made to that plan, transitioned to our actual laptop refresh and desktop refresh strategy.
[00:13:59] So now, the entire process is way more streamlined and way more efficient without having to rely on, Okay, we’re going to go back to the old way of doing it, even though it was slow, it worked. No, we’re going to improve things whenever we get the chance and, you know, we’re not just going to sit there and say, Hey, it works. We’re just not going to make it better because that’s not the case. You can always make things better and you got to find that line of when you, when you can and when you should.
[00:14:21] Jonathan: Do you think this experience brought you guys together as a team? When this was finally all done.
[00:14:26] Matthew: Oh, absolutely. You know, of course it took a lot of energy and time to get through that, so we didn’t have the most energy to have a celebration at the end of it. But there was definitely this sense of just accomplishment. And of course, you know, it’s kind of like fighting through the trenches together. At the same time, it brings everyone closer. We already had an amazing and super close team. And don’t get me wrong, I definitely didn’t do it and absolutely could not have done it alone were it not for, you know, the help. Shout out Nikki, Jocelyn, Kaylee, and John. But definitely got closer from that.
[00:14:52] Jonathan: The shared trauma of the IT horror story, keeps you together. Tell us a little bit about the aftermath in a longer term perspective. So you’ve stayed with the organization, you’ve moved up the ranks and into a different role.
Growth opportunities since
[00:15:05] Matthew: That’s a lot of the culture here is that a lot of people that have been around in the higher positions have been here for a while, and they’ve kind of started low, got their way up. And because of that, you have this like very good culture inside the company and a lot of knowledge to kind of spread around.
[00:15:19] If you’re an analyst on my side or you’re an intern starting off for the first time, there’s just so much knowledge to go around and everyone that we have, you know, started out either in some smaller role and made their way up or proved to everyone that, you know, they belong on the team and they know what’s going on.
[00:15:35] Jonathan: Are there any other pieces of advice or I guess, keys that you could identify that helped you all under this challenging circumstances. [To] make sure that you guys were able to come together and pull this off versus crumble?
[00:15:48] Matthew: One of our leadership people say is that, you know, you got to know when to fail and because it’s going to happen. We have a saying in the information security world of, you know, it’s not if you get hacked, it’s when you get hacked. So you got to be prepared for things and be proactive but also learn how to fail and learn how to fail quickly, is the most important thing and learn from those mistakes.
[00:16:07] And that’s been a key theme for us for a long time around here. Thankfully, I would say we don’t fail that often. I’m probably biased saying that. But when we do, we make sure to learn from those mistakes and to fail quickly and then move on.
[00:16:19] Jonathan: So let’s think about your role in security now, a whole new, fresh set of horrors to keep you up at night. What are some of the things that you focus on now and how are you thinking about, relaying like the things that you’ve learned through this past experience and others to help make sure that you’re going to be able to face further possible horror stories.
[00:16:41] Matthew: One of the greatest things is that because I have that shared experience and knowledge on the IT operations side of things, while being on the security side, I can look at the changes that we’re going to make and the policies that we’re going to change and how that’s going to affect other departments, whether it be inside IT or outside of IT.
[00:16:55] I know that if I, for example, turn off everyone’s USBs, the help desk is going to get swamped from that. And so there’s kind of like this, this balance that you kind of got to walk. Whereas if you have someone that doesn’t have a lot of that experience, not saying they’re never going to know that, but at the same time, you know, they might not have that breadth of knowledge and kind of know where to find that balance, or at least have those conversations, you know, you’ve got to bring all of the stakeholders to the table and say, hey, this change that we want to make is going to affect you guys.
[00:17:20] So we want your opinion on it as well. Kind of thing.
[00:17:22] One of the things that keeps me up at night is AI, both from a scary standpoint and also from a super helpful standpoint, there’s a lot of benefits from it, but there’s a lot of detractors from it. So on the security side, you know, one of the biggest things is automating tools and stuff like that for people, for attackers to use, to instead of having to sit down and target you, they can use AI to kind of learn your environment and kind of pivot as they need to.
[00:17:43] On the blue team side, you can use AI to kind of go through data and find heuristic data, without having to just read it yourself and go through logs the entire time. And so, AI, you know, it’s a buzzword obviously, but at the same time, there’s a lot to be learned there. And there’s a lot, a lot going on.
[00:17:57] You as the security people, if you want to be successful, you have to learn how to use it, but also how to adapt against it and make sure that you’re, you’re prepared for it. You know, again, on the data governance side, you’ve got things like different, large corporations have different products coming out.
[00:18:10] That’ll kind of ingest your data on your company and make different, you know, policies or different suggestions on it. But one of the things people don’t realize is like, they’ve got access to your data. What are they doing with that kind of thing? So, you know, your rights and compliance and governance, making sure that everything is locked down while still being super helpful.
[00:18:27] You know, you can’t just say no to things. You have to enable people to be able to use stuff in order to make them better too.
[00:18:32] Jonathan: Well said. We joked at the beginning, forcing you to relive this terrible experience, you can tell that you actually, I mean, you have pride in it, what you were able to accomplish, right? What are some specific things, that stand out that kind of make this a story that you like telling?
[00:18:47] Matthew: Just seeing how, you know, not only how we held or handled the situation rather, but how we made ourselves better from it. A lot of the results, and consequences of that, you know, or from that event, rather, we learned and adapted from. And so, you know, one of my favorite things is just improving that entire process and having fun with it.
[00:19:06] Like now, at the same time, we have different scripts and stuff that we can run. And, you know, for example, we have something called the Everything Bagel, on our fun side, where that runs all of the software now, versus the Plain Bagel that runs some of the software, right? And so rather than having to focus on an imaging software that could take one to three hours, you know, you’ve got super lightweight scripts that are super fast, everything you want to do, and adaptable.
[00:19:26] So just like seeing, seeing something that was not really a fun event, you know, four years ago at this point, transitioning to something fun and silly like the Big Mac or, the Everything Bagel or the Plain Bagel, just seeing reminders of that, but in a fun light rather than, oh man, like, I don’t ever want to think about that again.
[00:19:42] Like, no, it’s, it’s something that happened. You’ve got to learn from it and, you know, have fun, have a sense of humor.
[00:19:46] Jonathan: Yeah, and it seems like too, like you’re not one to run away from a challenge, you actually seek those out because you’re a problem solver at the heart of it.
[00:19:54] Matthew: That’s a very, very common theme in IT is that at the end of the day, problems exist, and, you know, you gotta learn how to solve them. And, me personally, I don’t like leaving a question or a problem unsolved. It kind of bothers me, and so when I get that, that one issue that really bothers me, I put all of my resources to it and just kind of jump at it and attack it until I can figure it out.
[00:20:14] And then when you do figure it out, it’s almost like a runner’s high kind of thing. After running 20 miles or whatever it is, your ultramarathon, you know, you’re physically dead at the same time, but your brain is just going wild kind of thing.
Closing
[00:20:26] Jonathan: All right, Matthew, I’m going to ask you one last question and, it’s putting you on the spot a little bit, but if you were going to have a survival kit that was going to help you ensure that you were going to survive an IT horror movie, what’s going to be in your kit?
[00:20:45] Matthew: Ooh, that’s a good one. What are the, what are the stakes here? Am I on like a deserted island, or am I in the middle of Manhattan where I’m being attacked by killer robots, you know?
[00:20:53] Jonathan: I think you can let your imagination run wild with this one.
[00:20:56] Matthew: Well, the first thing I gotta have is a laptop, because I gotta be portable. Even if I don’t have internet connection, I still can do a lot of things with a laptop and, you know, everything like that. And then obviously I gotta have something to charge that laptop with, so, you know, hopefully like a solar panel or an inverter, something like that.
[00:21:10] And then from there, you know, if we’re going full survivor apocalypse, I feel like it’d be cool to just have a machete. You know, I don’t know what I’d use that machete for, but you know, it’d be cool, hey, you know, there’s something going on in this network closet, shut it down. No, I’ll just take my machete and just, you know, chop off the entire stack of cables kind of thing, right? It’s kind of a cop out answer, but…
[00:21:29] Jonathan: So, laptop, something to plug the laptop in, power source, and machete. I think that’s a pretty solid trifecta.
[00:21:35] Matthew: I’m just saying, if anyone else needs anything more than that, I don’t think they’re doing it right. You know, poor carpenter blames their tools.
[00:21:44] Jonathan: Matthew, thank you so much for taking the time to share your horror story with us. Any final words that you’d want to share?
[00:21:51] Matthew: Emphasis on having that good leadership, no matter where you are, if you have a great leadership team, it’s going to enable you to do good things, and then having that cohesion between your team enables you to do good things as well.
[00:21:59] Jonathan: Well thank you once again for taking the time and joining us. And that’s all the time we have today. And thank you everyone for listening! We’ll see you next time.
Introduction
[00:00:26] Jonathan Crowe: Hello, and welcome. Please come in. Join me. I’m Jonathan Crowe, Director of Community at NinjaOne and this is IT Horror Stories. Brought to you by NinjaOne, the leader in automated endpoint management.
[00:00:41] Jonathan: Welcome to another episode of IT Horror Stories, the show where we invite IT leaders to relive their most harrowing moments for our listening pleasure and entertainment. I’m your host, Jonathan Crowe, and I have with me Matthew Schuster. He’s an information security analyst in the professional sporting world. Matthew, thanks so much for taking time. Welcome to the show.
[00:01:02] Matthew: Thank you very much for having me, Jonathan. Happy to be here.
[00:01:05] Jonathan: We’re happy to have you and before we get down into the nitty gritty here and listen to your tale of terror, let’s learn a little bit more about you. Tell us more about yourself and your role now, currently as an information security analyst.
Leading off
[00:01:19] Matthew: Yeah, so as you said, I’m an information security analyst currently with a team of two on the security side, but we have a pretty large IT department otherwise that we also help with or help and work with. So I’ve been doing this currently for coming up on six years now. Started my way up all the way as an intern, kind of worked my way up through the farm system, as we say, through technician role, another analyst role on the ITOps side, and then now officially over on the information security side.
[00:01:44] We kind of equate it to the whole farm system thing. So, it’s, you know, starting out in minor league, working your way all up through the big leagues, or to the big leagues, rather.
[00:01:50] Jonathan: Let’s get right into it and set the scene for your horror story. What stage was this at in your career? What was it looking like in terms of the team? Kind of like, what was your, what was your status?
[00:02:04] Matthew: At the beginning of 2020, a time that I’m sure a lot of people are familiar with. The team was a little bit smaller than we currently are now. I was… had just finished my one-year mark as an intern at the time, over on the IT operations side. And had worked with a lot of the different sub departments in IT, such as, you know, networking and security, stuff like that, and was primarily still on the help desk side, you know, doing the whole day to day stuff.
[00:02:24] Jonathan: And so you said you were just finishing interning. Were you, was this a full-time role or were you part-time?
[00:02:29] Matthew: Yeah, so it was a part-time role. At the time, I was also a full-time student at the university.
[00:02:33] Even back then, the three full-time people we had, had more than just IT operations side. So our manager also did full-time telecom with mobile devices, which that’s a full job in and of itself while also managing the help desk, you know, two roles there.
[00:02:50] Jonathan: So you had a team around you. Everyone’s kind of doing, wearing multiple hats, doing different things. Tell us a little bit more about your mindset here at this outset. Are you still trying to prove yourself at this point in your career?
[00:02:59] Matthew: Oh, absolutely. I mean, as everyone, everyone that’s kind of still in college and learning a lot, and then also kind of, you know, jumping in as an intern is still absolutely trying to prove themselves. But at the same time, I was just trying to be an information sponge, trying to absorb as much as I could. And that’s one of the benefits that we had of wearing a lot of those hats, is that you kind of got cross trained on a lot of stuff and you had a huge opportunity to learn a lot of stuff, which you know, just entering the IT world, there’s a lot of sub departments in there. And so it kind of helped me figure out, okay, do I want to go networking? Do I want to go software? Do I want to go, security? And obviously ended up on the information security side. So it’s a huge benefit.
[00:03:31] Jonathan: So, when we talk about horror stories in IT, people can quickly go into different things that make their job challenging or make it scary for them in various ways. That can include the org they’re in, the team, the users they’re having to deal with. Tell us a little bit about your satisfaction with the org at the time. Were there any scary elements that were already existing prior to this particular story?
[00:03:55] Matthew: We had a great, you know, actual team that I worked with, and then above us we had an amazing leadership and still do have an amazing leadership team, which truthfully is one of the best things you can have. And so, you know, as an intern. Coming in, especially if it’s your first business job, things can be daunting. You know, trying to learn the business side of things while also trying to absorb as much information as possible.
[00:04:12] Jonathan: It’s a fast paced, a lot of going on. But positive environment, we have bright eyed Matthew, young in his career looking to make a difference here. Then what happened? Let’s get right into the incident. What changed on this fateful day you’re going to be talking about?
The day of the incident
[00:04:28] Matthew: Yeah. So I want to emphasize what you said. Fast paced, absolutely fast paced, everything, you know, it’s not just, you know, for us, of course, we’re in the sports industry and, you know, it happened to be February and March at the time, and that’s the off season. But there’s still a lot of stuff going on.
[00:04:41] So I’m sure again, it’s 2020, a time that a lot of people in and outside of IT are pretty familiar with, happened to be the onset of COVID. And so we were tasked, with from the leadership team of setting up, and transferring all of our current work, like at the stadium employees, to being able to work from home full time remote.
[00:04:57] Before that, we had, you know, policies and procedures like other companies where you have mandated work from home days just to make sure systems and work and stuff like that. But this was full scale work from home where office shut down, everything like that. And at the time, we had probably, I want to say about a hundred to 200 people that had full desktops here at the stadium.
[00:05:18] That needed to be swapped over to laptops. and the time frame of that was three days. And so you can imagine a team of, you know, three full timers and two part timers at the time while also doing the day to day stuff of, hey, you know, what’s going on? I need help with my keyboard. You know, my printer stopped working.
[00:05:35] Also trying to set up between 100 and 200 laptops.
[00:05:38] Jonathan: You’re acknowledging 2020, the pandemic hitting, everyone has their story, right? Everyone is remembering what it was like for them. But for you guys, those three days, where did that come from in terms of, did this, was this a very sudden kind of onset of a project? Was there any kind of anticipation towards leading up to it?
[00:05:58] Matthew: Yeah. Oh, I mean, there was so much speculation going on, you know, everywhere. Like, no matter where you were at, no one kind of knew what the full picture was. Thankfully, again, we had a great leadership team, and they came to us with the proposition of, Hey, this is what we’re looking at doing. What do you guys need to enable that?
[00:06:13] Rather than just saying, Hey, you know, we’re shutting down in two days. This is how much time you guys have. They came to us and said, Hey, you know, we need a timeline here. What can you guys get done? And so we ended up saying, okay, we need a little bit more time. Obviously. And just trying to get that done as fast as possible, while not being, Oh, you know, we need like a month to do this. You know, you gotta be a little more realistic sometimes, and you gotta be super fast paced and just adapt.
[00:06:34] Jonathan: So tell us a little bit more about what this looks like in your environment. How disruptive potentially was this looking like for your team?
[00:06:44] Matthew: Yeah, so it was more so just like the day to day business stuff is what they were using it for. It was the off season, thankfully, so we didn’t have a lot of different, you know, obviously didn’t have to worry about the product that we were putting out on the field, but we had to do a lot of the day to day business stuff like any other company has to do, you know.
[00:06:57] Trying to market our product, trying to get people paid with finances, trying to make sure that people are employed and staying employed with HR, and then, you know, of course, trying to sell our product on the ticket sales side of things. And primarily that’s what we were trying to enable was the ticket sales folks of just being able to go home and do what they need to do.
[00:07:11] We had a process beforehand for setting up and imaging these laptops from scratch. But as we’ll explain later, it wasn’t the easiest and probably the most efficient process at the time.
[00:07:21] Jonathan: I think a lot of people probably had something on paper there and then they get to see, I mean, this is, real life testing. They got to see how, really kind of pressure test those things in real time…
[00:07:31] Matthew: Yep.
[00:07:32] Jonathan: With the pandemic, like, as you said, a lot of things being up in the air for a while, people not being really sure what was happening or what wasn’t. And then I remember a key point of course was, Tom Hanks in Australia, but also the NBA, shutting down games and it really, that being a point of like, okay, this is actually a thing that’s gonna be very impactful.
[00:07:51] Matthew: And that’s kind of the transition period that we were at is we knew things were going to be, things were going to hit pretty soon. We didn’t know exactly how hard they were going to hit, but we knew we had to be prepared for it.
The 72-hour countdown begins
[00:08:01] Jonathan: You were given initially those three days. You have to transition everyone who is used to using a desktop, you have to get them laptops. That clock… you’re able to successfully push back a little bit, get a little bit more time, but now the clock’s ticking.
[00:08:14] Matthew: So immediately just started working, you know, hit the ground running, set the treadmill to 10 and started running at 11 kind of thing, you know? It was an all hands on deck thing. Again, like I said, the other full time people that we had on the team, were also doing these other telecom and security and coordination. And they didn’t just put those to the side. They kind of overclocked themselves and did all of those things while also doing all of these laptop imaging and stuff like that. It was a complete and utter, you know, group collaboration thing that wouldn’t have happened if it weren’t for everyone total.
[00:08:41] Jonathan: And so that’s the team’s effort. What about yourself? Personally, do you remember the first things you were doing?
[00:08:46] Matthew: I remember being extremely thankful at the time because I was on spring break. And they announced, during school announced that, Hey, you know, we’re going to go on an extra week of spring break. And, you know, none of us really thought about the consequences of that. We were just like, Hey, you know, an extra week off is pretty sweet.
[00:09:00] And thankfully that lined up at the same time that I was, that we needed all these laptops done. And so, you know, came in, got everything done and, you know, we didn’t have the biggest space at the time. We just had, you know, a bunch of desks and everything. Didn’t have like a full lab set up, but it was to the point where every outlet, every ethernet port, every square inch of carpet and desk that we had was just taken up by laptops.
[00:09:19] And just everything, you know, beeping away, just going and imaging everything we need to do while just kind of, like those classic scenes in the library where the ladder is moving back and forth and you’re just going from one side to the other, just one laptop to the other, putting in credentials, you know, entering in configurations and stuff like that.
[00:09:35] Jonathan: So as an IT person working at a professional sporting stadium, like a location. Yeah, what does that setup look like?
[00:09:43] Matthew: Depending on the sport you’re in and depending on the team you’re in, and depending on when the stadium was built, there’s a lot of different factors that you’ve got to take into account. And thankfully here, we have a relatively new stadium, you know, all things considered, and so we have dedicated office space here at the stadium.
[00:09:55] And so because of that, you know, it’s nice that we’re here and able to support everything, but obviously, the big attraction of the stadium is the seating and stuff like that. And so, you know, it’s not just we have offices in one central location, we have offices throughout the entire stadium. And so, you know, trying to run around of like, hey, you know, I’m setting up this laptop. Now the person on the other side of the stadium also needs help doing something.
[00:10:14] Jonathan: You’re getting your steps and you’re getting your calf workout for sure.
[00:10:17] Are you guys staying on top of things, knowing how you’re going to break down this task. It’s putting laptops into place, but, did you have basically like an assembly line going out? Like, how were you tackling this?
“We had the procedure. It was just a matter of doing that procedure 100 to 200 times.”
[00:10:30] Matthew: That’s exactly it. Thankfully, again, we had the procedure. It was just a matter of doing that procedure 100 to 200 times. And so from a literal standpoint, it started at imaging the laptop, based off of a base image that we had, over the network. Which, it was a pretty large image, and so each laptop, the imaging process itself, completely hands off would take anywhere between one to three hours, depending on what type of laptop it was.
[00:10:53] With the square inches instead, or square feet that we had, and then also the amount of, you know, network that we had available, you could only do so much at one given time. Everyone just started imaging as much as we can. And then from there we’d break it down of, okay, you know, we’re going to have you work over on the configuration side of the profile. We’re going to have you keep imaging stuff or just keep kicking them off as you can. And eventually it got to the point where we just have, you know, the entire process down just sped up and everything ready to go.
[00:11:17] Jonathan: A big question I should have asked earlier. You had all these laptops just lying around, ready to go?
[00:11:22] Matthew: Thankfully, we were already looking at moving everyone over from desktops to laptops. Had all these laptops already in stock in our warehouse here at the stadium. We were just waiting to, or like we would roll them out over the 12 months rather than three days. And so that’s a huge, huge benefit on our side. Thankfully, it was still early enough I feel like in the speculation of COVID, where we probably could have gotten those laptops if we needed to, it just would have added complexity to the whole ordeal, which thankfully we didn’t have to.
[00:11:48] Jonathan: Were there any other hurdles that you had to get across? Anything that didn’t go as planned?
[00:11:53] Matthew: The other hard part is, now trying to teach the users of hey, you know, a laptop is pretty much just a desktop with a keyboard attached to it, right? So, like, the hardware itself, not much different. The hard part’s teaching them the different things you need in order to work at home. So, like, using a VPN, for example, and connecting to business resources here at the stadium. That’s probably the hardest part. Because, again, a laptop, you set it and forget it kind of thing. It just takes time. Every user and every person will understand and kind of adapt to the teachings differently.
[00:12:22] Some people, you can just hand them a page of documentation, say, here, here you go, go for it. Some people obviously are going to need a little more help than that. And mind you, we were doing all of this while still setting up those laptops. So it’s, you know, we get one laptop done. Hey, here you go. I’m going to give it to John Smith.
[00:12:36] And now I’m going to teach John Smith how to connect to our VPN and, you know, how to use the phone system from home and stuff like that. Use softphone and all of that.
[00:12:44] Thankfully, again, leadership team’s fantastic. We have that budget that we can be proactive rather than reactive. And so we have a very set time frame, of like, hey, you know, your laptop’s X years old, we’re going to give you a new one. One hurdle that was unintended from that was now every three years or every four years or however long it is.
[00:13:00] We have a very large list of laptops to refresh. You know, versus 2024 versus 2025, there’s like a hundred laptop delta there, which, you don’t really think about cause you just kind of push that under the rug kind of thing. But it comes up later on.
[00:13:14] Jonathan: So, you guys were running around, you’re operating under controlled chaos. Everyone’s working together to try to make this happen. What was the end result?
The end result
[00:13:24] The end result, I mean, we got it done, thankfully, as everything you have to, just get it done. And obviously everything with, you know, the situation in COVID after that kind of transitioned, but during that time period where everyone was working from home, that gave us time to kind of think about and sit back of like, hey, we got through it. If we had to do this again, which hopefully we don’t, how can we make things better? And not only just adapting, you know, like that emergency plan. But also just improving our day to day kind of stuff. And so, the changes that we made to that plan, transitioned to our actual laptop refresh and desktop refresh strategy.
[00:13:59] So now, the entire process is way more streamlined and way more efficient without having to rely on, Okay, we’re going to go back to the old way of doing it, even though it was slow, it worked. No, we’re going to improve things whenever we get the chance and, you know, we’re not just going to sit there and say, Hey, it works. We’re just not going to make it better because that’s not the case. You can always make things better and you got to find that line of when you, when you can and when you should.
[00:14:21] Jonathan: Do you think this experience brought you guys together as a team? When this was finally all done.
[00:14:26] Matthew: Oh, absolutely. You know, of course it took a lot of energy and time to get through that, so we didn’t have the most energy to have a celebration at the end of it. But there was definitely this sense of just accomplishment. And of course, you know, it’s kind of like fighting through the trenches together. At the same time, it brings everyone closer. We already had an amazing and super close team. And don’t get me wrong, I definitely didn’t do it and absolutely could not have done it alone were it not for, you know, the help. Shout out Nikki, Jocelyn, Kaylee, and John. But definitely got closer from that.
[00:14:52] Jonathan: The shared trauma of the IT horror story, keeps you together. Tell us a little bit about the aftermath in a longer term perspective. So you’ve stayed with the organization, you’ve moved up the ranks and into a different role.
Growth opportunities since
[00:15:05] Matthew: That’s a lot of the culture here is that a lot of people that have been around in the higher positions have been here for a while, and they’ve kind of started low, got their way up. And because of that, you have this like very good culture inside the company and a lot of knowledge to kind of spread around.
[00:15:19] If you’re an analyst on my side or you’re an intern starting off for the first time, there’s just so much knowledge to go around and everyone that we have, you know, started out either in some smaller role and made their way up or proved to everyone that, you know, they belong on the team and they know what’s going on.
[00:15:35] Jonathan: Are there any other pieces of advice or I guess, keys that you could identify that helped you all under this challenging circumstances. [To] make sure that you guys were able to come together and pull this off versus crumble?
[00:15:48] Matthew: One of our leadership people say is that, you know, you got to know when to fail and because it’s going to happen. We have a saying in the information security world of, you know, it’s not if you get hacked, it’s when you get hacked. So you got to be prepared for things and be proactive but also learn how to fail and learn how to fail quickly, is the most important thing and learn from those mistakes.
[00:16:07] And that’s been a key theme for us for a long time around here. Thankfully, I would say we don’t fail that often. I’m probably biased saying that. But when we do, we make sure to learn from those mistakes and to fail quickly and then move on.
[00:16:19] Jonathan: So let’s think about your role in security now, a whole new, fresh set of horrors to keep you up at night. What are some of the things that you focus on now and how are you thinking about, relaying like the things that you’ve learned through this past experience and others to help make sure that you’re going to be able to face further possible horror stories.
[00:16:41] Matthew: One of the greatest things is that because I have that shared experience and knowledge on the IT operations side of things, while being on the security side, I can look at the changes that we’re going to make and the policies that we’re going to change and how that’s going to affect other departments, whether it be inside IT or outside of IT.
[00:16:55] I know that if I, for example, turn off everyone’s USBs, the help desk is going to get swamped from that. And so there’s kind of like this, this balance that you kind of got to walk. Whereas if you have someone that doesn’t have a lot of that experience, not saying they’re never going to know that, but at the same time, you know, they might not have that breadth of knowledge and kind of know where to find that balance, or at least have those conversations, you know, you’ve got to bring all of the stakeholders to the table and say, hey, this change that we want to make is going to affect you guys.
[00:17:20] So we want your opinion on it as well. Kind of thing.
[00:17:22] One of the things that keeps me up at night is AI, both from a scary standpoint and also from a super helpful standpoint, there’s a lot of benefits from it, but there’s a lot of detractors from it. So on the security side, you know, one of the biggest things is automating tools and stuff like that for people, for attackers to use, to instead of having to sit down and target you, they can use AI to kind of learn your environment and kind of pivot as they need to.
[00:17:43] On the blue team side, you can use AI to kind of go through data and find heuristic data, without having to just read it yourself and go through logs the entire time. And so, AI, you know, it’s a buzzword obviously, but at the same time, there’s a lot to be learned there. And there’s a lot, a lot going on.
[00:17:57] You as the security people, if you want to be successful, you have to learn how to use it, but also how to adapt against it and make sure that you’re, you’re prepared for it. You know, again, on the data governance side, you’ve got things like different, large corporations have different products coming out.
[00:18:10] That’ll kind of ingest your data on your company and make different, you know, policies or different suggestions on it. But one of the things people don’t realize is like, they’ve got access to your data. What are they doing with that kind of thing? So, you know, your rights and compliance and governance, making sure that everything is locked down while still being super helpful.
[00:18:27] You know, you can’t just say no to things. You have to enable people to be able to use stuff in order to make them better too.
[00:18:32] Jonathan: Well said. We joked at the beginning, forcing you to relive this terrible experience, you can tell that you actually, I mean, you have pride in it, what you were able to accomplish, right? What are some specific things, that stand out that kind of make this a story that you like telling?
[00:18:47] Matthew: Just seeing how, you know, not only how we held or handled the situation rather, but how we made ourselves better from it. A lot of the results, and consequences of that, you know, or from that event, rather, we learned and adapted from. And so, you know, one of my favorite things is just improving that entire process and having fun with it.
[00:19:06] Like now, at the same time, we have different scripts and stuff that we can run. And, you know, for example, we have something called the Everything Bagel, on our fun side, where that runs all of the software now, versus the Plain Bagel that runs some of the software, right? And so rather than having to focus on an imaging software that could take one to three hours, you know, you’ve got super lightweight scripts that are super fast, everything you want to do, and adaptable.
[00:19:26] So just like seeing, seeing something that was not really a fun event, you know, four years ago at this point, transitioning to something fun and silly like the Big Mac or, the Everything Bagel or the Plain Bagel, just seeing reminders of that, but in a fun light rather than, oh man, like, I don’t ever want to think about that again.
[00:19:42] Like, no, it’s, it’s something that happened. You’ve got to learn from it and, you know, have fun, have a sense of humor.
[00:19:46] Jonathan: Yeah, and it seems like too, like you’re not one to run away from a challenge, you actually seek those out because you’re a problem solver at the heart of it.
[00:19:54] Matthew: That’s a very, very common theme in IT is that at the end of the day, problems exist, and, you know, you gotta learn how to solve them. And, me personally, I don’t like leaving a question or a problem unsolved. It kind of bothers me, and so when I get that, that one issue that really bothers me, I put all of my resources to it and just kind of jump at it and attack it until I can figure it out.
[00:20:14] And then when you do figure it out, it’s almost like a runner’s high kind of thing. After running 20 miles or whatever it is, your ultramarathon, you know, you’re physically dead at the same time, but your brain is just going wild kind of thing.
Closing
[00:20:26] Jonathan: All right, Matthew, I’m going to ask you one last question and, it’s putting you on the spot a little bit, but if you were going to have a survival kit that was going to help you ensure that you were going to survive an IT horror movie, what’s going to be in your kit?
[00:20:45] Matthew: Ooh, that’s a good one. What are the, what are the stakes here? Am I on like a deserted island, or am I in the middle of Manhattan where I’m being attacked by killer robots, you know?
[00:20:53] Jonathan: I think you can let your imagination run wild with this one.
[00:20:56] Matthew: Well, the first thing I gotta have is a laptop, because I gotta be portable. Even if I don’t have internet connection, I still can do a lot of things with a laptop and, you know, everything like that. And then obviously I gotta have something to charge that laptop with, so, you know, hopefully like a solar panel or an inverter, something like that.
[00:21:10] And then from there, you know, if we’re going full survivor apocalypse, I feel like it’d be cool to just have a machete. You know, I don’t know what I’d use that machete for, but you know, it’d be cool, hey, you know, there’s something going on in this network closet, shut it down. No, I’ll just take my machete and just, you know, chop off the entire stack of cables kind of thing, right? It’s kind of a cop out answer, but…
[00:21:29] Jonathan: So, laptop, something to plug the laptop in, power source, and machete. I think that’s a pretty solid trifecta.
[00:21:35] Matthew: I’m just saying, if anyone else needs anything more than that, I don’t think they’re doing it right. You know, poor carpenter blames their tools.
[00:21:44] Jonathan: Matthew, thank you so much for taking the time to share your horror story with us. Any final words that you’d want to share?
[00:21:51] Matthew: Emphasis on having that good leadership, no matter where you are, if you have a great leadership team, it’s going to enable you to do good things, and then having that cohesion between your team enables you to do good things as well.
[00:21:59] Jonathan: Well thank you once again for taking the time and joining us. And that’s all the time we have today. And thank you everyone for listening! We’ll see you next time.
