
HACKING FOR GOOD, BEING SPIED ON, AND CYBERSECURITY
October 6, 2022
S03 - E04
James Flom and RSnake dig into the history of ha.ckers.org, SecTheory, and the incredible amount of attacks sustained. James and RSnake dig into some stories about how hackers tried to hack ha.ckers.org (big surprise), how the ISPs got involved and many other crazy stories. RSnake and James further discuss FOIA requests, Carnivore boxes, and being infiltrated by spys. They also talk about what it took for RSnake to become a good person. This is an insider view into the incredibly perilous lives of professional security researchers and hackers.
VIDEO TRANSCRIPT
Robert Hansen
Today, I'm with James Flom. James is a security expert and former business partner. We give you an insider view into the incredibly perilous lives of professional security researchers and hackers. Some details are going to go over people's head, and I think that's okay. If you continue to listen, you'll get the broader concepts.
James has historically stayed behind the scenes and isn't much of a talker. But I think you'll see why we get along so well. We dig into the history of ha.ckers.org, SecTheory, and the incredible amount of attacks we sustained.
We dig into some of our stories, how hackers tried to hack into ha.ckers.org, big surprise, how our ISPs got involved, and many other crazy stories.
We discussed FOIA requests, Carnivore boxes, and being infiltrated by spies. We also talk about what it took for me to become a good person. With that, please meet James Flom.
Hello, and welcome to The RSnake Show. Today, I have with me a very good friend of mine, James Flom. How are you, sir?
James Flom
Good.
Robert Hansen
Good to see you. Thanks for coming all the way down.
James Flom
It was long, a 15-minute drive.
Robert Hansen
You and I have known each other for more than two decades, I think. Something like that.
James Flom
We met in April of 2001.
Robert Hansen
Okay. Yeah. So, more than 20 years. You and I have been friends. We've been business partners. We've done a lot of really cool experiments in science and technology and hacking, a lot of hacking.
James Flom
A lot of weird stuff.
Robert Hansen
A lot of things. Out of all of my friends that I have ever had in my life, I think you're the third longest friend I've stayed in touch with all these years, which is really saying something because that's a long time to stay in touch with somebody and be as friendly as we have been all this time.
For those who have no idea who you are, which I suspect is virtually every single person listening, this is an interesting one. Because you and I have worked together and on almost the exact same things at the same time. But you from a very different perspective than mine, which is cool.
My expertise has primarily been on the web application and browser side. And yours is?
James Flom
Network security, host security, random other security, breaking into buildings.
Robert Hansen
Yeah, you did. I forgot all about that.
James Flom
Several times.
Robert Hansen
Yeah. There's some good stories around that, too. But I remember long ago when we were very first talking back when we still had an office and were working out of that office together, we used to sit down and have these very intense, long conversations. Super intense, multi hour-long conversations.
I remember at least two or three times you'd say something like, “Ah, we should have recorded that. That would have been really interesting. I think a lot of people would have gotten a lot of value out of hearing all of that.” And I totally agree.
I think Raymond Kaminski, actually, encouraged me to actually go through with it. He finally had a podcast studio. He's like, “Come on down. You should do some stuff in my studio.” But I think it was you who was the seed in my brain originally.
You made me think, “It is annoying that this information is lost. We're never going to get back. We can't recreate that conversation. It's literally impossible.” Do you have the same feeling?
James Flom
Yeah, I think it's living in that experience at that time that trying to recreate that knowledge is very difficult. I think a lot of the times it was like we're dealing with a hack or something. So the insights that we came up with during it, there were applicable then. They're probably applicable to a lot of other things.
Robert Hansen
Absolutely. Well, you have an interesting designation in my brain as being the most dangerous person I know.
James Flom
Thank you.
Robert Hansen
Which I think is really saying something. I like to think of you as a walking nuclear bomb. You are capable of a lot of really interesting things online that most people just can't even really comprehend. How would you possibly do these things? It's almost magic.
I realize it is not anywhere near magic, unless you're talking about the literal sense of a magician where they have just a bunch of tricks up their sleeve. So that makes you an interesting character for all kinds of reasons, I think.
One of the reasons you never got a whole lot of limelight is you chose to be fairly behind the scenes.
James Flom
100%.
Robert Hansen
You actually even chose a handle ID, and ID turns out to be one of the most impossible things on the internet to search for. I think you took quite a bit of glee in that disappearing from the internet.
Let's talk about the beginning. What got you into hacking? What made you decide that this was something that you wanted to entertain or do or whatever?
James Flom
More of something that I just started doing. I'm going to date myself here. Prodigy, which was owned by Sears, of all places-
Robert Hansen
Was it really?
James Flom
Yeah.
Robert Hansen
I didn't know that.
James Flom
We got the free trials and all that. I wanted to figure out a way to get from their service onto the internet. I had to basically break through their software to figure out how to get to newsgroups. And I guess that's probably how I started.
Robert Hansen
So it was out of necessity or you needed to do something and you decided to go and accomplish it?
James Flom
I was nine.
Robert Hansen
I remember you telling me this one time about a story about your dad oversees and working on some system. Somebody came up, and they were shutting down the facility or something. And he basically was like, “Thank you so much for helping me out.” Tell the story.
James Flom
Yeah, him and my uncle were both over in Vietnam. I was in Vietnam too during the war as a baby. They thanked the guard. And the guard is like, “My job is to kill you if we were ever breached to save the data.” Because they didn't want them to be tortured into giving it up, which is fairly insane.
Robert Hansen
That's the ecosystem that you grew up in. That was your family. That was your life. You may not have been aware of it, obviously, at the time as a baby. But as you grew up, you were around very technical people who had access to computers and access to the internet as early as the internet really existed for consumers.
I did not have that exact life story. But it's not dissimilar, a lot of very technical people around me. I also was trying to get online and figure out how to do interesting things. So I think that's a common story from back in those days you can get away with.
James Flom
It was before hacking, and I hate to use the word anymore. Before computer security was about what it is today. It was mostly people exploring, finding things. Yes, I was bypassing controls to get to things. But very different than people think of it today.
Robert Hansen
How so?
James Flom
Well, nobody was monetizing. You could go all the way back to the first big hack you heard of that someone was trying to monetize, probably Mitnick. But there was a long time before that that it was just simply the first server I ever broken to besides Prodigy, maybe AOL was a South African university that was running an early version of Linux.
It was just about exploring. I didn't want to hurt anybody or do anything. There was no monetary value to it. It was just you're a kid, and you need to explore.
Robert Hansen
Yeah, I also found myself exploring a lot in the early days. I think, for me, it was largely about the visual aspect of it. I really was curious, how did these pixels end up on my screen? Why is this server able to change my computer to do something different than it was doing a minute ago?
That all seemed very foreign and interesting and strange. Then for me, the marriage of that plus the psychological aspects that I saw, why does this color elicit certain behavior? Why are links shaped the way they're shaped? Why did they choose this language to make me want to go from one page to another?
That really opened my mind up to a realm of possibilities. Those two things opened me up completely like, “Oh, I bet I could do all kinds of things. This is wide open. This is the Wild West.”
James Flom
I think one of the differences between you and I is that I'm a little bit older than you. I learned to program before the internet was really a thing. In ‘79, we had our first computer, which is way before it was a consumer-type thing. So we probably view things very differently just from how we ended up growing up.
Robert Hansen
Sure. I think that's probably true. Also, I took a lot of engineering classes. And there was always something about the bits and bytes of it that seemed fairly abstract and strange and distant to me. Whereas the visual components, stuff I could actually see and interact with, felt very near and approachable.
When I looked at your world, the networking world, the host world, I understood pieces of it. I understood well enough to speak intelligently to it to other people. But I sure didn't understand really how it worked, the underpinnings of it.
I couldn't write TCPIP. I couldn't get on a switch and configure it, nothing like that. But that's your whole world. That's your bread and butter. That's where you cut your teeth and still do to this day.
James Flom
Yeah, to some extent.
Robert Hansen
How you and I first met, we first met at Digital Insights. Sorry, Digital Island rather. Digital Insights is a totally different company. I wasn't even in security at this point. I was just getting into security a day later.
Up until that point, I had been a programmer.
I recall you and I were very early on involved in one of the largest tasks in the history of the internet at the time, maybe even the biggest.
James Flom
It was easily the biggest as far as what it was, which was credit card stealing. It was actually Jon Orbeton who’s, unfortunately, gone from us who was the first one to notice it. He came over and didn't know it was a hack. He just noticed that something was wrong on a server, which turned out to be a laptop.
Robert Hansen
Yeah, it was. Sitting in some little closet somewhere down in LA.
James Flom
It was in Delray. It was one of the first times I worked with the FBI. We went down there. He noticed it one night. We had just hired that security executive team. They were all ex FBI people. We went down there.
Jono and I actually drove all the way down to Marina del Rey. We get to this really fancy-looking office building, and we go up to the top of it. There's a bunch of beige boxes and laptops. And that's what they call DNA center. It was all just literally stacked on top of each other.
Robert Hansen
Old Gateway 2000 machines thing?
James Flom
100%. It was some of the first laptops, and then they'd have a laptop with a beige box sitting on top of it. And it was the jankiest whatever. But that was an interesting hack.
I remember that night, I probably shouldn't because we drank a lot of scotch and beer. Luckily, Jono’s significant other at the time drove us all home. But that was fun.
Robert Hansen
The reason why that night, in particular, was such an important night is it led to all kinds of crazy things happening. You got summarily dismissed at one point. The whole team was thrown up in the air, and the police were involved. It was a massive thing.
It was this multi month-long crazy thing that ended up happening, and they fired and hired you back in the exact same day. All kinds of stuff. So it was a fantastic time, always the Old West.
This is actually one of the reasons why when there's a breach event, oftentimes I'm telling people based on this conversation we had all these years ago, “Why are you trying to chase the bad guys down? Why are you doing this? There's an enforcement arm of our government. It is designed specifically for this task. Why are you spending the time and effort to do this?”
A lot of people just really get caught up in it. They did, too. And I don't think they liked that you were telling them, “Hey, this is a humongous waste of our time and resources trying to find these people.”
James Flom
I figured out they spent like one and a half million dollars trying to find the bad guy. Who cares who the bad guy is? Fix the problem and move on. Because there's a million bad guys. But you're right, it happens all the time.
Robert Hansen
Yeah, exactly. Shortly thereafter, you and I started ha.ckers.org, H A dot C K E R S dot org. More or less, out of your living room. I remember we were trying to get it working on Drupal at first and switch to WordPress eventually.
James Flom
It's hilarious because I found an old firewall online. It had Celeron processors from the mid ‘90s. It was rackmount. I have a great picture of it. I'll send it to you. But that's exactly where it started. We hosted ha.ckers.org off of a firewall.
Robert Hansen
Which pretty much says everything you need to know about that infrastructure.
James Flom
Out of my closet.
Robert Hansen
Out of your closet. Then we upgraded to a friend of ours. His garage.
James Flom
Mauricio.
Robert Hansen
Yeah. We put it in his garage, literally, which is a terrible place to put any server equipment ever because of the heat issues.
James Flom
It was on a shelf above his car. It was ridiculous.
Robert Hansen
Not exactly the best we're capable of. But in a weird way, it was.
James Flom
It was. If you look at the old Slackers, the graphic that we had at the top of that, that was from his garage.
Robert Hansen
slackers.org, for those who have no idea, was the forum we built on top of the blog. So instead of it being H A dot C K E R S dot org, it's S L A dot C K E R S dot org. It was the place to go for web application security experts.
James Flom
If you look at the people that you know today that are web apps experts, they almost all had handles on Slackers.
Robert Hansen
Yeah, I would say so. We actually created an industry almost overnight using that. I'm not totally giving myself credit. Obviously, there was a lot of other people involved. But that really was a nice place for them to all congregate and learn and practice.
I remember one of the things we did on there is we had a full disclosure thing, this little sub forum within the forum, where people just posted, “Hey, I just hacked this thing, and here's how. Here's another one. Here's another one. Here's another one.”
It was more of a race to see how fast we could go through the list of the entire internet and find at least one bug on effectively every Alexa 1000 or million or whatever websites, however far you wanted to go down the list. And it's effectively every single one of them.
It wasn't like 50% or 10% away, it was 100%. Every single one of them we found a vulnerability on. And I think that really changed people's mind about what web application security was capable of and how dangerous we actually were.
James Flom
It did. I could be wrong, but I think things like Burp Suite pretty much had their foundation there. The ideas were built out of it.
Robert Hansen
Yeah, I would say that's probably true. Burp Suite is a tool that hackers use. The audience may have no idea what we're talking about here. It's a tool that hackers use to change their traffic and modify it before it hits the website, change it from, “I'm buying this thing for $1,000.” to, “I'm buying it for no dollars or whatever.” Or other things.
James Flom
Other hilarious stuff happened at Slackers.
Robert Hansen
Yeah, many.
James Flom
Especially the grammar thing.
Robert Hansen
Yeah. Go ahead. It's up to you.
James Flom
Well, it wasn't us. First of all, you wrote something in the blog. And it referenced grandma porn.
Robert Hansen
Grandma prawn, in particular.
James Flom
Whatever. The users of the site, this is also the early days of SEO, decided to promote it until we were the number one site on the internet for that.
Robert Hansen
Yep. It was a nice crossover. In fact, it's funny you mentioned this. A podcast just launched today with myself and Jono Alderson where I'm talking about the overlap between marketing and SEO. And it's incredibly deep.
They're very inter intertwined in all kinds of interesting ways. And that's just another example. We have a whole bunch of hackers who just decided to get me to rank for this. Then there's all these poor jerks trying to find these things and landing on this hacker website totally confused how they ended up there.
James Flom
Because we recorded everything in our logs, every search query that came through, going through a lot of that, it was degenerate as can be. But it also created an entire industry because right after that happened, everybody was looking at the SEO rankings of it. And they built sites around it. So you created an industry of wrinkly, bad stuff.
Robert Hansen
Yeah. Well, that's true. Unfortunately, that is actually true. Then we started SecTheory. Go ahead and explain it since you are now the CEO. I think it's only fair.
James Flom
I still don't remember who came up with the name, by the way. But you were down in LA, and I was still in San Francisco. I was working as a consultant for some rather big companies but hated it. I think you didn't enjoy your job at the time either.
Robert Hansen
No, I did not.
James Flom
Well, the very first job was someone out of Austin. But one of your friends actually got it to go, the contract that they signed, which I'll leave that out of all this. But anyway, it was like what, 20 grand or something?
Robert Hansen
Yeah, or something.
James Flom
It started the company. And after a couple of months, I was able to quit my job. You'd met someone down in Austin that basically offered us free office space.
Robert Hansen
I was on the advisory board of Adometry, which was Click Forensics at the time. So they gave us free office space, which I think they probably strongly regretted after six months or so.
James Flom
It was because we were hosting, well, several other websites besides just SecTheory out of there, ha.ckers.org hosted out of there. They got DDoS-ed and a bunch of other stuff. But we were definitely costing them money.
Robert Hansen
Yeah. But we also helped out, too. So who knows? Maybe we even got them acquired eventually by some of the tips and tricks we gave them. It's entirely possible.
Those three things; how we met, the hacker website, and SecTheory, we're going to dig into this, so don't worry. But that pretty much sums up how we met one another, how we worked with one another and the lifecycle. Each one of those things, I think, were very instructive to how we ended up becoming friends and working together and all kinds of stuff.
Let's go back in time a little bit. When ha.ckers.org was very first coming online, I don't even think it was a blog yet, I think it was literally just a static website. It had no content on it at all that was dynamic. But I had created something called the cross-site scripting cheat sheet, a very early incarnation of it.
What that was a way for hackers to go and they can cut and paste a bad thing, a payload and inject it in websites. If it worked, it would create a little pop up that says it works. I was working at eBay at that time. My boss, I think, looked the other way a lot on whatever I was doing.
He's just like, “Look, Robert’s trying to protect us. Whatever he's doing, he's doing for the benefit of the company” Which is true. It took a while for him to get to that conclusion. But effectively, he realized I was on his team.
What was cool about ha.ckers.org and that cross-site scripting cheat sheet, in particular, is that bad guys all over the world were utilizing it. It wasn't just good guys, it was bad guys. It was everybody. Everybody was using this thing, even back then when it was still just a static website.
At eBay, we would actually see people copy paste the exploit payloads from hackers or try it on eBay and then fail for whatever reason and then go to the next one, copy paste it, fail for whatever reason and just keep iterating.
I could watch them try every single thing from the payload list. Then one day, I got an email from a vendor. I had been watching my logs, and I saw someone had actually been successful. And I'm like, “Okay.” So I grabbed it and put it on the cross-site scripting cheat sheet.
That’s just another example of another thing you can do, we're going to block it. They sent me an email saying, “Hey, RSnake, we have this new exploit, this new way of injecting things you should see.” I'm like, “Oh yeah, it's already on the cheat sheet.” Exactly their payload.
That's when I think they realized that they probably shouldn't be using it because I'm watching them and then maybe who I am because no one knew who I was back then.
Then that was also the time when I realized I probably shouldn't be utilizing it that way because now I'm too intertwined between my company and the hackers website and these contractors who are trying to protect eBay from the outside.
My boss at the time called it asshole.com. Anybody who goes there must be an asshole. I think it was the beginning. That was really the true beginning of ha.ckers.org when I really saw that what we were doing was having actual massive impact to these companies.
I could see it firsthand. I could see my logs. I could see things happening. What was your feeling about ha.ckers.org? What is the aesthetic you got out of it? When did you know it was the thing?
James Flom
It's actually a fairly hard question. Because, at first, I was just hosting it for you just because we were friends. Once I saw the amount of traffic we were having and the discussions on Slackers and the references because there were a lot of references out there to ha.ckers.org, then I knew it.
Then it was really when we started getting attacked. A lot of people don't know that we were on WordPress 1.3.
Robert Hansen
I thought that was 2.2 eventually.
James Flom
Nope. Never went past 1.3.
Robert Hansen
I thought we made it to 2.2. But either way, it doesn't matter because it was horribly vulnerable, insanely vulnerable.
James Flom
You were patching it all the time.
Robert Hansen
Yeah, I was doing a massive amount of work. And you were, too.
James Flom
I finally got fed up with it, and we built The Armored Stack.
Robert Hansen
Yeah, which we'll get to in a minute. Let's not go all the way down that path yet. But I think you're right. I think the point at which it really took off was when people started really attacking it.
The reason they were attacking it is they thought they could get money out of us or they thought they could break in and find out the secrets or whatever or bragging rights.
There’s so many attacks happening. We were getting thousands and thousands a day. I think we were probably more attacked per capita than even the biggest companies on Earth.
James Flom
Yes, easily. I had a call from the CEO of my DSL provider at the time saying, “I love what you guys are doing, but they keep DDoS-ing our network because of you guys.” And he's like, “You need to make that stop.”
Robert Hansen
Which is a polite way of saying, “Get the fuck out.”
James Flom
Yeah, it was Sonic net.
Robert Hansen
They were very polite and nice to us.
James Flom
No, they were super nice. And they actually let us keep going for almost another year after that. But how often do you get called by the CEO of an ISP that says, “Love you, guys. But get out.”
Robert Hansen
Yeah. They were nice. They were actually nice. They were pleasant. They tried to work with us. They tried to stop the issues on their side. How are you going to stop every hacker on Earth, when they are all very determined to get into this website for bragging rights or whatever reason they were trying to get in?
It was interesting. It was fun to watch. The logs were crazy. I ended up eventually writing a book from those logs called Detecting Malice, which is to this day, I think probably the best or maybe even the only real book on internet fraud in that genre. It's free now. You can just download it.
We were hacking everything, really truly everything. And even sometimes by accident. I would say even oftentimes by accident.
James Flom
So many times.
Robert Hansen
Yeah, so many times by accident. One that I thought was really hilarious out of many examples was Amazon because we had written a book on cross-site scripting. Amazon had indexed that book but indexed it in raw text.
The problem is we put raw text up and it has got HTML in it, it will render the HTML. When it rendered the HTML, it actually allowed me to run JavaScript in the context. So I basically could take over Amazon, all of Amazon if I felt like it.
That's not good. But there's all kinds of examples of that. Do you happen to remember any others that you think were interesting?
James Flom
The stupidest one was hacking China on accident. We had a customer. They did mainframe security. They tried to acquire us. They're like, “This is the IP address of our office in China.” I’m like, “Are you sure?” This is before we had figured out we really need to verify things. I started hacking a router.
Robert Hansen
They lied to us also. They made it seem like they own this and this, but they didn't really own that second thing. So by virtue of them not owning it, us going one more upstream to hack the thing that's right above them is hacking something enormous.
James Flom
It was the transpacific link to China. Just hacking all the time. We had complete access to the router with pretty much every subnet China had on it. So there's that. By accident, not on purpose.
Robert Hansen
Yeah. That was actually by accident, strangely. That sounds incredible. How the hell did you guys accidentally hack China? Well, sometimes you're given incorrect information. And you just start doing the normal hacking thing as you normally do. Normal things that you would do, I would say. Not that one would normally do.
James Flom
That’s why I verify a lot of things these days.
Robert Hansen
I imagine. Cisco and some of these other firewall companies, they know you by name. When they want something done, when they want to check their routers or switches or firewalls or whatever, they send it to you. And I don't mean your company. I mean literally you, when they're trying to find these things to see if they're actually vulnerable or not.
That's pretty telling because if you look at these machines overseas, now they're using other operating systems like Huawei or whatever. But for a long time, they were all top layer or Juniper or Cisco or just a handful of these companies.
James Flom
They're still all derivatives of that. They're using Linux for a lot of it, but so is Cisco. So there's not that big of a difference.
Robert Hansen
What do you think makes someone a good hacker versus an average hacker or a bad hacker?
James Flom
Probably mostly perspective. Well, there's two things. One, to be a good hacker, I think you need to learn the system side of things. If you're going to hack the web, you should probably learn JavaScript first.
Robert Hansen
And HTML.
James Flom
And HTML.
Robert Hansen
CSS.
James Flom
Well, one of the interesting things is I don't think you understood exactly how a web server worked when we first started this. You were hacking web pages. But you didn't understand how say, Apache or NGINX worked at the time.
Robert Hansen
I would say that's true and false. I certainly knew how to configure them. But I didn't know what was going on under the hood of why those configuration things did whatever they were doing.
James Flom
Right. And you didn't really have a great understanding of IP at the time.
Robert Hansen
Definitely not.
James Flom
Learning that made you a better hacker.
Robert Hansen
Much better, yeah. DNS, in particular, helped a lot.
James Flom
Yeah. It's learning, not just breaking into things. It's learning why things are the way they are, and then you can break into them better. I think it's one of the things that's missing from a lot of average hackers.
Robert Hansen
I remember when I was very first getting started, day one hacking. Someone was showing me how to debug something over a web server. And they're like, “Oh, you just tone it to this port. And you type get slash and hit enter.”
I’m like, “You do what? What happened? What is going on? What is all the stuff that's happening here?” They tried to explain it to me. And I'm like, “I just don't understand what's going on. You just somehow connected your terminal to a server. How is this occurring?” I couldn't quite wrap my head around it.
As soon as I did it myself and I actually saw what was going on and I could spend the time to actually craft different payloads and get different outputs, I'm like, “Whoa, I cannot believe I am emulating a browser. My hands are a browser right now. I don't need this crazy, complicated piece of software between me and any web server. I would just type it in, and now I'm communicating directly.”
Once I figured that out, then I was able to bypass the super complex nature of browsers and go straight to web servers, which I still hadn't figured out web servers to your point. But web applications, on the other hand, I was all over. That I was destroying right and left.
That was when I think my eyes were fully open to the complexity but also how totally simple certain parts of the web actually are.
James Flom
Completely. I would say, to even take it down further, just understanding at the electronic level how things work is one of the things that's made me a way better hacker. And that goes back to my dad owned a computer repair shop. He had tasked me with repairing just some random computer or something and then you got to switch and some other stuff.
All of a sudden, I had to go like, “Oh, okay. So you're changing voltage on this line to make a bit that's interpreted somewhere.” I was lucky that it happened at a very young age. But it's something that most people are completely clueless.
We were talking the other day, walking my dog, beside calling you. I was talking about my brother and his understanding. He's working on AI models. But he doesn't understand any of the implications or why or how any of it works behind the scenes. And I think that's just super common now.
Robert Hansen
Incredibly. I have this beef with AI/ML in particular. I think there's something that's really desperately lacking from those models, which is forensic logging of what's going on. So this thing, I put this input, I got this output, why? What happened?
That why everyone's super comfortable with like, “Yeah, it's a black box.” What? How was anyone okay with that? How was anyone anywhere like, “That's cool that it just randomly curated this output.” No, no, no. You’ve got to know why that's happening.
To me, it sounds like an insane amount of debugging statements to show me how this thing decided to do this thing. If I can't follow the logic with my mind, that's a massive problem. That means you're probably doing something you probably shouldn't be doing.
James Flom
You're 100% doing something you shouldn't be doing. The last time I talked to my brother, I was like, “You realize someone's going to make an automated so your net pet or whatever has also dated?”
Robert Hansen
Very much so.
James Flom
Just automatically brushes its own teeth or whatever it does to take care of itself. Then it's going to decide, because you have no idea what's going on behind the scenes, that it's going to brush everybody's teeth and take over the world. And that's how the world's going to end with AI brushing your teeth to your death.
Robert Hansen
I don't think it's going to be that way, but I totally agree with the point. I think that there's tons of examples all over the internet. Look at modern programming languages. Old programming languages, they're pretty well-documented. Everything's pretty uniform.
Everyone wanted it to look a very specific way. There weren't a lot of strange hacks or things that built on top of those things. They're all very like, “This is what it is.” When you get to modern languages, they are a mess.
You have like 14 different ways to install things. You have unbelievably and contiguous naming conventions and weird ways to pull things in. There's like 20 different ways to do any function you want, and they all do slightly different things. They're not backwards-compatible. And on and on and on and on.
Literally, a buddy of mine two or three days ago installed an update to his Mac. And he's like, “Why isn't the software working anymore?” I'm like, “I don't know. Did you update your computer?” He’s like, “Actually, I did.” I’m like, “That's why.”
Now it's a three-hour process just to figure out that some library is deprecated in the current version of whatever he just updated to. This is a super common thing with modern programming languages. I think part of the problem is modern programmers. They just don't know that it could be better. They don't remember a time when things weren't like that.
James Flom
100%. Also, to your point of it bringing in things, how many people use node.js?
Robert Hansen
5 million. I don't know.
James Flom
5 billion. But everything's being sucked in from all these libraries that are on the internet. And now you've got a security problem where, if any one of them is compromised in any way-
Robert Hansen
And people intentionally compromise them.
James Flom
Intentionally or unintentionally. That doesn't matter. But you're not in control of what's going on.
Robert Hansen
Yeah, the complexity issue. Old-timey websites, they're pretty static. They didn't really do much. There was a form usually. The form had a couple error functions on it to say, “Hey, you didn't input this correctly.” Then inserted it into a database. And that was it.
There was nothing more complicated than that. As soon as Web 2.0 and XMLHTTPRequest and all these modern web frameworks started being built, there is no one person on Earth who's going to be good at all of those things at the same time.
You can be good at one of those things, I believe it. Maybe even four or five of those things. But the vast array of things you had to know just to run a simple web application, just something that takes some input from a user is almost insurmountable.
If you add in browsers and how they work or network on the other direction hosts and the underlying operating system and web server, there's just no chance that complexity's way too high.
Even if I bleed for a second that you could understand all that stuff, you're probably using third-party APIs and you don't understand those because you don't have access to them.
You don't understand what's going on behind Stripes API when you're trying to swipe your credit card. You have no idea what's happening back there.
James Flom
But I think that part's okay.
Robert Hansen
Do you? I do not think that's okay at all. In fact, I think it's terrifying.
James Flom
Using something like Stripes API, yes, you don't know, but that can be legally limited liability wise.
Robert Hansen
Yes.
James Flom
So, do you care?
Robert Hansen
Yeah, definitely. Because there's a difference between liability and things just not working the way you think they are for an undetermined amount of time that you can't control. Or even if you could control, you won't know that they're broken because things are so obscured in such a weird way.
I've run into systems before where you're going along just fine, everything's good. And you're like, "Our search engine traffic must be down or something and something's weird. We're losing more and more money. Normally we'd be up this time of year."
It turns out that there's just this thing that's been broken for like six months and no one even noticed. Are you going to go try to sue Stripe because they changed some minor thing? Which they by the way told you about and someone didn't read the specs on because it's too complicated. I just don't think so.
I don't think you're going to sue them. I don't think the liability's really gone. Especially when you're talking about voting systems. You looked at Ukrainian voting system, I think it was that they published.
That thing is a mess, an absolute mess. Obama's website, the healthcare.org site is a mess. There's APIs all over the place. No air conditions, no checking to make sure this thing's going to go up or go down, what happens. If it's not online, what happens?
There's no, and then, it's just assuming everything's going to work at all times. I don't think you can chalk it up just to liability anymore. I mean, not safely.
James Flom
Not safely. I don't know.
Robert Hansen
Well, okay. Why do you think hackers.org was so secure? How do you think we survived? I mean, I remember at least a dozen times, at least, probably way more than that, saying out loud someday we'll get hacked. And we never got hacked. I was wrong. I know your stuff doesn't get hacked. I know you're very proud of that, but why?
James Flom
I mean, mostly it started out because I was annoyed. That you were giving me updates all the time and saying, this is broken, or this doesn't work.
Finally, I was like, if I can compartmentalize every little bit of what we do, then that separation can keep us safe. Which is something that's missing even more today than it was back then.
Which, I mean, we started off with flat networks and we started off with people not understanding basic network security or host security. They got it and now they've lost it all again. It's weird. I think one of the advantages I had, I came from Hewlett Packard, so I had used HPUX.
Robert Hansen
Which is an operating system for those who don't know.
James Flom
Yes. It's one of the original Unixes. They had been around since whatever HP started as, 40s or something. By the time I got there in the 90s, they had already built up this infrastructure and had all this stuff in mind. Everything was set, separated, thought out.
We took that into consideration, or when I had that in my brain, I was like, "We're not running HP servers or anything crazy like that. What can I do with what we have?" I'm also the reason that we used free BSD because BSD was more like HPUX than Linux was.
An OpenBSD in our firewall. But yeah, it was just paying attention to just keeping everything separate. Obviously logging, stuff like that, that people just ignore nowadays.
Robert Hansen
I think the way I like to phrase it is nothing trusted anything else in that system. There was not a single thing that trusted any other compartment o this whole system, which is annoying and cumbersome and, yes.
But with a site that's as attacked as that site was running, as you said, an incredibly ancient version of WordPress, you had to. I mean, there was no way to patch WordPress. In fact, WordPress core was having a new vulnerability about every month or so at that point.
Now, it's much more stable. But at the time, you really should not be running WordPress at all on a hacking website. You're just going to get hacked. My browser didn't trust me. The website didn't trust my browser.
The web server didn't trust the application. The host didn't trust the web server. The database didn't trust anyone. The network was completely isolated and blocked off.
And on and on, there was many more layers than I'm describing even. What ended up happening is even when a really very good exploit would come in, it was guaranteed to win on any other site like ours with a different infrastructure. It definitely would've worked. And ours, it just kind of bounced off. Nothing even really got particularly close.
I remember some hackers tried to come after us. Some of the slackers, some of our friends tried to come after us, and they burnt two new exploits trying to hack me effectively to get into the system. They didn't work. They bounced off because of how this whole system was set up.
James Flom
I remember it being three.
Robert Hansen
Oh yeah, sure. Actually, I think you're right. I think it was three.
James Flom
That was by design. I tried to be thoughtful about building something but I was just taking all the experience I had from, having worked years in the industry and applying it to some crappy firewall we turned it into hackers.org.
Robert Hansen
Yeah. Which was definitely not the right kind of architecture for all kinds of reasons, but it was very robust. It worked.
James Flom
It did.
Robert Hansen
I remember one of the other problems that we had was an enormous amount of traffic. For something as small as this tiny little blog, we were getting absolutely destroyed by traffic. An enormous, enormous amount of people were visiting this website.
James Flom
We were on Slashdot a few times. Back then, for people that don't know, it was one of the biggest technology websites.
There was actually a term for it, slashdotted, where your site went down because you got so much traffic from them. Somehow we were able to keep ours up on a freaking, I don't know what is it, $40 a month DSL?
Robert Hansen
Yeah. Very optimized.
James Flom
It was very, very optimized.
Robert Hansen
Yes. And I think there was a lot of questions, Matt Mullenweg who's the inventor of WordPress once touted that one of the biggest hacking websites on earth uses WordPress. I was very quick to say, "Whoa, whoa, we are running an ancient version that I have long ago forked on top of this very, very complicated architecture." It's sort of silly to call it WordPress at this point kind of thing.
But there was a good reason to use WordPress. Once upon a time we were thinking about Droople, we were thinking about building our own. There was MyPhp or something, I forget. Some other CMS that we looked at.
We landed on WordPress purely because it was just simple. There was a very lightweight by comparison. When I was auditing the code on the others, I found all kinds of problems, which is fairly to be expected. There are enormous code bases with a lot going on.
James Flom
And you later found tons of issues with WordPress.
Robert Hansen
Yeah. I sure did. But simple things like, why is the admin page always slash admin? Why shouldn't we name it Kwyjibo or name it Bob or something. Put it somewhere in some of the directory.
All of a sudden all these attacks that require the admin page to be here just don't work. There's all kinds of things like that, that just seemed very silly to me that these things were built so uncustomizable, so unflexible. I just ended up having to fork it just to get i there.
James Flom
I mean, that's an interesting point too. Obfuscation is not security, but it can be convenient in just not having to deal with a lot of other stuff that's happening. And I don't like the fact that people say obfuscation is not security.
Robert Hansen
Yeah. I've never liked that either.
James Flom
If a million attacks get your website don't ever go through because they're automated and they can't find whatever a path. It's not security it's making it more robust infrastructure.
Robert Hansen
I've never liked that theory, that obfuscation isn't security. I mean, it is a kind of security and it totally works. I'm thinking of a number right now. What's the number? Obviously you don't know the number. You could brute force it and try to get it out of me, but it's obfuscated. It's hidden by virtue of the fact that you just don't know it.
James Flom
True. Or look at any Google Docs.
Robert Hansen
Or look at any password.
James Flom
I mean, literally the links on Google Docs are predicated on...
Robert Hansen
Or Dropbox or a million of these platforms. It's complicated URL structure. You can't guess it. You're never going to guess it. You're not going to brute force at any time in your lifetime because it's complicated. It's a very long, weird set of numbers.
You're not going to guess it. I always thought that was extremely silly and bad of our industry to not grok how useful sophistication can be in very targeted situations. Well, I've found it incredibly valuable tool in all kinds of situations in security, both on the defense and offensive side.
We also suffered some fairly large denial service attacks. Real ones, not just because we got hit by traffic. One of them I remember was by some kid somewhere that you ended up finding.
James Flom
Oh yeah. That was a kid in Italy that was dossing us. And luckily I had, once again, dating myself. We had the DSL connection, but I also had a Dial-up connection with Sonic net at the same time. And I ended up using that the Dial-up connection, go 56K modems.
I'm sure there's statute of limitations are running out for this one.
Robert Hansen
Yes. They've run out.
James Flom
Anyways, I hacked his computer and it turned out it was his mom's computer. And I got her email address and I emailed her, it was a Google Translate because I don't know Italian. I shut the computer down but emailed her and she replied back to me, "I'll punish my boy."
Robert Hansen
I mean, that's one way to stop it.
James Flom
It's the only way we had available at the time.
Robert Hansen
We literally got a kid spanked, is how we got this thing together.
James Flom
Some little Italian boy.
Robert Hansen
I mean, I think that's the kind of crazy out of the box sort of hacking that we were having to do kind of constantly to manage this site.
James Flom
It was.
Robert Hansen
Another example that I didn't find out about till much later. I went out to this very weird exclusive conference. There was like 15 people there. It was like a secret service and Google and an evolutionary biologist and some malware people. It was just a very weird group of people.
I have no idea who funded it. I have no idea why we were there. Just very strange. Anyway, I was walking around with this secret service guy, I was just taking a little walk between the sessions. He's like, "Oh, what do you do?" I was kind of explaining what was going on.
He's like, "I think I know you." I'm like, "Oh yeah?" He's like, "Yeah, I'm on the advisory board of this company." And they came to me and they're like, "Hey, this guy tried to hack us. Can you put this guy in jail?" He did some research and figured out what was going on.
Well, what happened is someone on our website had taken one of those things from the crosshead scripting cheat sheet. They had injected it and it worked. They're like, "Okay. Well, we're definitely going to fix that issue." But what they did is, they just removed the word script out of it.
They didn't fix the actual vulnerability. Anytime they saw the word, script, they would just remove the word, script. If you just put SRC and then script and then IEPT at the end, it would remove script and leave script.
I changed the attack payload very slightly so that it would work so it would be an example of the why you shouldn't use that type of filter. Because the crosshead scripting cheat sheet was all about filter evasion. Suddenly this thing worked again and they decided that they looked really, really bad. This is a security company.
They don't want to look bad, they don't want to have these types of vulnerabilities and they did. He took one look at it and he's like, "You can't put this guy in jail. He's not even hacking you. You're just not fixing this vulnerability properly."
Thank god this guy got involved because he could have easily said, "Yeah, I can see you're pulling in code from this website and he's the one who hosts this website and it's over." There was all kinds of things like that one.
James Flom
I mean, that was the whole 90s and early 2000s. People not understanding basically what security was with computers.
Robert Hansen
And all of these exploits that we were playing with, not all, I should not say that. But the vast majority of the ones that we're talking about were things where it was extremely easy to repair and extremely bad that if it was there.
It's easy to show you and it doesn't do anything until you decide to do something with it. And none of these payloads actually do the bad thing, but they easily could.
Sammy for instance, Sammy Kamkar, who maybe it's someday we'll get on the Podcast, he proved that by injecting this bad piece of JavaScript you could take over basically every single MySpace user in under 24 hours of propagating.
And I think he opened a lot of people's eyes because he was able to finally show for the very first time. If you're not careful with these tiny little, the seemingly minor little exploits, it's game over for your company. You'll literally have to shut down to repair this.
God knows how many more he didn't find. That was just the one he did find. Then we had that and we had the slackers trying to hack us. I mean, this is all just us trying to keep this site online. I mean, we had enormous amount of traffic issues. We had hackers coming after us. We had the feds coming after us, potentially in all kinds of weird ways
James Flom
Yeah. That's a legacy that had gone on for a long, long time. I was literally on the no-fly list. Not no-fly list, but watch list. As early as a couple years ago I've just been walking through an airport and I get pulled out of line because of it.
Robert Hansen
Yeah. I've certainly noticed that. Whenever I'm traveling with you, which isn't that common these days, but when it happens, I immediately say, "James, I'll meet you at the gate."
James Flom
And why am I the guy? Why aren't you the guy?
Robert Hansen
I don't know. What did you do?
James Flom
Well, I'm not going to tell you. It's funny because my significant other, we were flying through Zurich and we'd never done international travel together before then.
Robert Hansen
So, she was not aware.
James Flom
I told her. I'm like,, "Shits going to happen." We get off the airplane and there's literally five people standing there waiting to escort me to the next plane for our transfer. We got back roomed after that, everything's searched, whatnot. This is my actual life all the time when I fly. It's ridiculous.
Robert Hansen
Why even fly at this point? Why don't I just take the bus or train?
James Flom
Well, it's hard to take the bus to Switzerland. Just saying.
Robert Hansen
Yeah. That is a little cumbersome.
James Flom
But even here at Austin, I don't even get to the security line and they'll say, "Mr. Flom come over here." I mean, sometimes it's convenient, get to bypass the entire security line.
Robert Hansen
Oh, there's always an upside.
James Flom
But you never know what's going to happen. I mean, the worst one was in freaking Canada where think Canadians are polite and nice, but it was like three hours back roomed.
Robert Hansen
Oh really? Wow.
James Flom
Yeah. That was unpleasant.
Robert Hansen
I didn't hear about that one. But it's all the time. It's literally every time you fly.
James Flom
I talked to some FBI friends. You know who they are?
Robert Hansen
Yes.
James Flom
And it quit for about two years and then right back on. I was somewhere 34 flights in a row I was back roomed. It's beyond ridiculous.
Robert Hansen
That's pretty crazy.
James Flom
I don't know what they think I'm going to do on an airplane.
Robert Hansen
Well, I have some ideas.
James Flom
Well, I have lots of ideas. But that's not the point.
Robert Hansen
I'll get some airplane guys on here at some point talk about airplane security, but that's a whole other day. One of other thing that happened back in the olden days when we kind of first-ish moved to Austin, we started having cigars pretty regularly with a couple security friends.
It was a weekly thing. We go in and we have a couple cigars and just chat with these, these guys and talk about security, talk about what's going on in the world and what's going on in the business. It was a just friendly hangout sort of situation.
Nothing crazy. Much Later on I found out that one of them was a spook and not for the US government.
James Flom
I know.
Robert Hansen
I know you know, but there was a lot of weird things that happened because of that and around that time period. It wasn't just that we were being approached and talked to by, let's say our own government and a bunch of hackers and reporters and all the sort of mess that goes along with the hacking world. We also were directly in the line of sight of these spooks
James Flom
I mean an aside, I don't know if you...
Robert Hansen
Sure. Go ahead.
James Flom
I was visiting Washington D.C and having a few beers with my cousin and was approached by these guys that, I don't know, they look like the most clean cut Gov guys you could ever imagine. And they're like, "Hey, James?" I'm like, "I don't know you."
And they're like, " Oh no, we're going to a party. You want to go to a party ." "No." Your cousin's going to go to the party.
Robert Hansen
But secretly you did want to go to the party.
James Flom
Just to find out.
Robert Hansen
I know it's tempting
James Flom
Because I will fuck around.
Robert Hansen
I know you will. It's super tempting to mess with these people.
James Flom
Well, I won't say exactly what I did, but yeah, I did mess around or maybe I fucked around. You put it however, but yeah.
Robert Hansen
In that guy's case in particular, the one I was just talking about, again, I did not know he was a spook at the time. We were just kind of talking off the cuff. What I usually tell people is there's about a third of the stuff I'll talk about.
A third of the stuff I will talk about if I can prove it. And I'm still kind of working through it. It's still messy in my mind. Maybe I haven't quite done the research yet. I'm still pulling the thread to see how far it goes.
There's a third of stuff I just will never talk about because I know it's too contentious and/or I know I could never prove it because it's just not provable, or at least not with the tools I've got. It stays in the back burner.
But this is an example that comes from that second category where I wasn't ready to talk about it because I'm still working through how you should talk about it. We're talking about spammers at the time. Imagine there's a spectrum of things you could do on one end of the spectrum, there's literally nothing.
You see a spammer and you see them spamming and you do literally nothing. You don't even talk to them. You just walk right past them. On the other end of the spectrum, you have like murdering them. Take them out sort of thing.
This is what I was telling this guy at the time, but this what was going on in my head to a friend in private. But I said, "Well, we have evidence that this actually works." That murdering spammers actually works because there was some guy in Russia who failed to pay his bill, we think.
Some guy hit him with a hammer a whole bunch of times in the head and spam dropped. We saw spam go down. We know murdering works.
James Flom
I remember that exact incident.
Robert Hansen
Yeah, exactly. I had nothing to do with it just for the record. But we know it works. Somewhere along that spectrum is the right amount of things to do to stop spam.
Obviously doing nothing is off the table because spam continues to grow and obviously murdering them is off the table because it's unethical. But there's somewhere on that spectrum that we should be, and probably not where we are now because we still have a ton of spam.
We probably have to move further down the list. Now, I'm not saying we should end up to murder or kidnapping their loved ones. Obviously all this torture stuff down here, we're probably never going to get there.
But we probably shouldn't do nothing and we probably shouldn't just slap them on the wrist only in certain jurisdictions. There's probably something further down the list. That was my premise. I'm still working through still kind of war gaming, how to think about it and how to talk about it. After this guy gets recalled back to his country he stays in touch with me.
He's like, "Hey Robert, I just want to let you know, I actually got in touch with the government and I pitched them your idea." I'm like, "What the hell are you talking about." His government.
I'm like, "What the hell are you talking about?" He's like, "You remember that thing about killing people or doing nothing?" I'm like, "Yeah." He's like, "Well, I pitched killing them." I'm like, "You did what?" He's like, "Well, I think it would work."
And the problem is, they all kind of looked at me like I was crazy, but we're kind of talking through it still. I'm like, "What are you talking about?"
James Flom
Spam assassination squads.
Robert Hansen
Yeah. Right? Which by the way, I have no evidence that that ever ended up happening. I mean, he could have also been lying to me. There's really no way to know, but this is the kind of thing you have to deal with when you're running a site like this for as long as we were.
You just became this incredibly weird target where everyone wants to like seed you with bad ideas or try to get you to do bad things.
James Flom
Yeah. I enjoy that.
Robert Hansen
Yeah. It is strangely enjoyable but also it gets tiresome too. Especially when you're sitting in the airport for three hours missing your flight.
James Flom
That was annoying.
Robert Hansen
Yeah, of course. We also did some crazy research around something called Slowloris which was used to take down a bunch of websites. Basically is code that takes down one of the most common web servers on the planet called Apache.
And with your help and one other guy who helped with some of the threading code. We were basically were able to take down a huge chunk of the websites on the Internet using this tiny little piece of code.
Unlike a lot of code that does something similar, it was very low bandwidth. You didn't have to have this crazy fast connection. You didn't have to have 10,000 machines at your control. You could literally do this from any machine anywhere.
I kind of got the idea of building something like this from a friend of mine, Robert E. Lee who had something called Sockstress which was able to take down TCPIP stack, which is every single thing that runs on the internet basically, from a single machine.
We did this, we built this thing, and it was out there and all of a sudden people were starting to use it against Iran during the Green Revolution when that was all happening.
James Flom
I mean, it was used against almost all of the Middle East or even North Africa, like Egypt, Morocco, Tunisia, they were all taken down by it. A lot of people don't remember the Green Revolution, oddly enough. I've asked like, "Oh, you remember?"
And most people are just like, "What? What happened?" Thousands of people died and protested and they don't know.
Robert Hansen
Yeah. I mean it was such a strange time for me for so many reasons. I even have a hard time kind of wrapping my head around all the things were happening at the time, because a lot was going on and I was focused on work.
I wasn't even thinking about this whole, like, what's going on in the Middle East at the moment. I was just running a company trying to build this code and launch it.
James Flom
I mean, before that part, the funny part is when you first noticed it, it was on a test server that we had in the lab when you noticed that Apache was vulnerable to this.
Robert Hansen
Yeah. But I had been thinking about it for a long time. In fact, I probably came up with this idea originally, originally back in the mid to late 90s. But I had never really thought of how to actually code it up. That was the part, I never really got past the point.
I was in the shower one day, a great mental image, I'm sure. And I was like, "Well, I think I know how to do it now." I finally spent the brain power necessary to actually try it. That's when we started playing around with like, "Well, would this actually work?" And surprise, surprise.
James Flom
No, I remember you coming into my office one day and saying, "Hey, what's wrong with the server?" Or "I think I found something." Whatever it was.
Robert Hansen
Playing around.
James Flom
No, you had written up an entire blog post. You're like, "I'm going to publish it." I'm like, "No, you're not because the first victim is going to be us." And you're like, "Okay."
Robert Hansen
That's true. I forgot all about that.
James Flom
I was trying to figure out how to mitigate it.
Robert Hansen
Yeah, that's right.
James Flom
Which I eventually did. But to me, it was like every hour you're coming in, "Can I publish it now? Can I tell everybody?" Like, "No, I haven't figured out how to fix this yet."
Robert Hansen
Yeah, that's true. I have forgotten all about that. I know I'm annoying sometimes. In the process of doing that. We actually found another issue in Apache in this very specific library that's supposed to stop this exact issue from happening.
We worked around that. And that was kind of the beginning, I think, of the odd phone calls coming in out of nowhere. Where I just get these very disgruntled people like, "Hello Mr. Hansen." I'm like, "Yeah." And like, "Okay, here's the deal. This websites are going down. We need them back up immediately." I'm like, "And your name is?"
James Flom
I mean, the answer should have been how much money did you have?
Robert Hansen
Well, these were very off government sounding people.
James Flom
They have more money.
Robert Hansen
One of the things that was interesting is, it was not supposed to take down things like PayPal or eBay. I remember specifically saying them by name, you're not going to be able to take out eBay. You're not going to be able to take down PayPal because they have a load balancer in front of them so it's going to bounce right off.
But there's a whole bunch of eBay and PayPal that are not behind that load balancer. And all of a sudden PayPal's getting ticked down right and left. And they're like, "Hey, Robert. Oh my gosh, what's going on? Can you please help?" And the unfortunate answer was not really, the tools did not exist, that was the problem.
James Flom
A certain load balancer company spent some money fixing that because it did go through their load balancer initially and they had to build a filter for it.
Robert Hansen
Yeah. I remember. And then eventually it became a big sales thing. People were like, "Oh, we can protect against Slowloris. And for those of you who don't know Slowloris is this like really cute little endangered animal with very venomous teeth and poisonous. Poisonous because it's bacteria. It's not a venom.
Anyway, he'll bite you and you get infected and you die. But people extract their teeth to have these little cute little pets, which is why they're endangered. They're very slow moving, which I thought was really cute. It's this tiny little thing. My code is very tiny. It's very slow, this tiny little thing.
It speaks very slowly to servers, but it does it a lot. It speaks to it hundreds of times in a row, but very, very slowly. In that process, it would take the whole thing down.
But the other one that was around that same timeframe was the great firewall of China research that we were doing. Like, can you send specific payloads across the firewall to get it to shut off the connection for a certain amount of time?
I remember we got it down to the size of a tweet. We could get the entire code into a tweet. I mean, it wasn't super elegant code, but it would fit in a tweet. And I remember, I did a very brief presentation on this many years ago. Only once. And it wasn't recorded.
There's no evidence that this thing even occurred but I remember a guy came up to me afterwards, and he wasn't saying it ironically. He is like, "Are you sure you should be doing it? Do you think you're going to die?" And he meant it like, today. Like, "The Chinese government's going to come after you. What are you doing?"
James Flom
Yes. Neither of us can ever visit Russia or China.
Robert Hansen
Or North Korea or Iran. But those in particular, those are definitely off limits for me. But that China research, I think there was a point at which I decided I was just going to keep doing what I was doing.
It ended up eventually doing some research against the Nenera browser, Redstar operating system, North Korea's operating system and their browser. And I just kind of kept going. I just didn't stop.
I decided at this point it was better to show people what was really going on than shy away from it. And you were there. You clearly had to make the same decision, but probably much earlier than I did.
James Flom
Yeah. But speaking of the story's great firewall in China, I think one of the most interesting research that I did on that part was not involved obviously with web browsers or anything like that.
But it was, we were asked by a client to figure out how spies they had in China could exfiltrate data through it. We decided, or I decided that video games were a great way to transfer information.
Robert Hansen
They're encrypted. Everyone's got them. It doesn't look like weird traffic. If you get caught with one, who cares? You got a video game.
James Flom
You go to an internet café, you don't even have to have it, which was much bigger back then.
Robert Hansen
Exactly. I always like that technique as well. An exfiltrating data is quite important. My code at the time was designed to break the great firewall of China by using its central system against itself. For those you don't know, if you send certain payloads across the firewall and it sees them, it's like, "Hey, hey, you're a bad guy." And it shuts it off.
You basically just don't go to these websites anymore. That's largely been deprecated both by virtue of everyone switching to HTTPS versus HTTP. That's good for security and allows people to finally transit the firewall. But also it makes it irrelevant in the face of things like sesame credit which I talked about on one of the other podcasts.
I think I talked it with Jennifer Richmond, maybe episode two or three or something. I think China has done a really, really good job of making their censorship system not needing to use censorship anymore. They could just use their own populace against themselves.
It's the inform on your neighbor. You don't need a jack booted Stasi anymore. You can just have your neighbor worried about you and not wanting their credit rating to go down. It's pretty crazy.
James Flom
It is.
Robert Hansen
The other big one was Clickjacking. That caused quite a stir as well. If I recall, one of the major problems with Clickjacking, unlike a lot of the other things we did, it wasn't everything's going to be vulnerable to it, even though everything was vulnerable to it.
But it was like, if you're vulnerable to this, you're probably vulnerable to something less complicated than this, and you should probably fix that first.
And then, yes, you're also vulnerable to Clickjacking. I remember people would say to me things like, "Well, you should do something. People don't know who you are in the real world. You should invent something Likejacking." I'm like, "I invented ClickJacking. Likejacking is ClickJacking."
So, Jeremiah Grossman and I collaborated on that research and I remember the biggest thing that caused the most controversies is we had to pull it back at the last minute. I think we were up in New York.
We were about to launch the research and Adobe came to us and like, "Please don't do this." Because in the Adobe Flash Manager, we could turn on your camera and microphone using ClickJacking. We could also get full access to your entire system as well, which effectively is the same thing as access to camera and microphone.
But our idea was why don't we just create a social network based on hacked machines? Everyone just auto create some dorky picture of them sitting there trying to click on something. That's their profile picture, extract all the information from their computer.
It was a fun joke, but we never actually went through with it. But that's the kind of research we were producing. I remember another one that I, I forgot to mention earlier, was a lot of the DNS rebinding research, this is where you became extremely useful.
We had a rack of machines, actually two racks of machines in our office. I would come in with these crazy ideas, pretty regularly.
Like, "James, I want to do this and this and this." And you're like, "It's like a lot of work. I got to sit this thing up and suck this thing up." I'm like, "Yeah. But here's why.” And we'd whiteboard the whole thing out. Eventually I convince you because I knew it would be like way more work for you than it was for me to set it up.
James Flom
Because it was.
Robert Hansen
Always more work for you than it was for me. For me, I just came up with the ideas. I got this idea, but I still need someone to implement the host part of this or create a DNS server or make the firewall turn on and off in weird ways at weird times. All this strange research.
To this day, I still have not found a lab that was anywhere near sophisticated to ours for how small it was. It was tiny. It was just probably 20 machines or something. But the DNS rebinding research was very interesting because it was research that had been done many, many years before by an Italian professor I believe.
He just couldn't figure out how to use it. He's like, "There's this thing you can do, but I just can't figure out what you would do with it." I realized that if you combine some of his research with some of my research, you suddenly could do something really interesting, but I couldn't prove it.
There's no way to do it without building these custom DNS servers and custom firewall rules. Being able to turn things on and off in really weird ways. That's where I think the magic happened. That's why I think your skillset and my skillset despite how totally different they really are, really quite different.
We became extremely potent together. There was no way for me to do a lot of the things I was doing by myself. I just could not get all the components working with. At least without a tremendous amount of work that I wasn't prepared to do.
I probably would've bailed on a huge amount of research that ended up being extremely vital, very interesting research that was later parlayed in all kinds of things. I know was weaponized later on in multiple ways.
For the research that begot that and researchers, their entire career started kind of spanning off of that. Dan Kaminski did a lot of research on DNS afterwards. Again, I could not have done any of that without the network components, the host components.
All that weird, like I need to build a custom thing that does this weird very specific thing. Sorry James, I know how complicated this is or sort of.
James Flom
Sorry. You're not going to have any rest tonight because Robert's coming in and bothering me every hour.
Robert Hansen
I like to think it was every other hour, but you never know. "Are you done yet? Come on James."
James Flom
"Can I just press publish?"
Robert Hansen
I do remember saying, "God damn it James." Probably at least a million times in my life because things would sort of work but they wouldn't quite work." And I'm like, "This isn't doing what I wanted to do and it should do this other thing." You're like, "You didn't say that."
I'm like, "I thought you would understand." Because these five other reasons and you’d always say it'll be done tonight. It was never ever, ever done that night.
I want to talk about something that I was always personally extremely annoyed about. Between you and me and a couple of our guys, we were quite good. I would say some of the best in the world at doing what we were doing, especially for how small we were. I'm sure there's other teams that have more people and therefore could slice and dice things better than we could have or maybe in different areas of security than we didn't touch.
But in terms of where we were in the types of assessments we were focused on, network and web application security testing, I just don't think we had a whole lot of comparable people out there at the time. But we ran afoul of a whole bunch of times that we were simply too good. Really, we were far too good.
You already mentioned the time when we hacked a huge chunk of China. That was a situation where we were supposed to do okay, at that pen test, penetration test. We were supposed to do okay. But we did so much above and beyond that I think the potential acquirers who wanted to buy our company, they just they couldn't wrap their head around how dangerous this small group of people were.
We hacked them the right way. But we also hacked them in a very not right way. I think that that spooked them. Did you ever notice that we were quite often like, I'm going to give you some other examples to jog your memory? There was one example where we were brought in by a company that does flight control software, flight control systems. They were a relatively big company, I would say, medium-sized big company. But they brought us in because they thought we would fail.
Their CEO had basically advised his head of security, “Find some small little company out there who can do it.” With the intention of, “Make sure that we're going to win this one. We really want to be secure.” We had no idea that that was what's going on. You and I were just brought in. This guy definitely had ulterior motives. He definitely wanted to prove how broken things really were for the purpose of making sure that he was not blamed eventually when they got hacked. He wanted to make sure.
He was fairly new to the company. He wanted to do well. He didn't care if it made the executive team looked bad. We went in there and we destroyed them. I mean, you almost couldn't have hacked them worse even if you had written a Hollywood script to do it.
James Flom
We had persistence that they tried to figure out. They couldn't even figure out how we were persisting in their networks.
Robert Hansen
In fact they were so vulnerable to so many different things that I literally would try things I was sure wouldn't work and they worked. Because they designed their system in such a bizarre way. It had grown for so many years over so many different versions of the system.
If you're sending this type of payload, they're like, “Oh, well, you're probably trying to talk to this machine over here.” They would send you to a vulnerable machine instead of just saying, “What are you trying to do here?” It was a very dangerously built system.
James Flom
If I remember right, you were actually popping shells from a web browser. Literally running bash and Netcat on a port from the browser. Because your web server should definitely have Netcat installed on it.
Robert Hansen
Yeah, of course. Exactly. I remember sitting down in this meeting. We did the big reveal. It's what I call The Prestige. If you've ever seen the movie, The Prestige, the three set. Three different parts of a magic trick. I always called this part The Prestige, which is when you finally reveal the trick. How it all worked. Here's how badly you were hacked.
The looks on their faces. They weren't like, “Oh, thank God, we found somebody who could really get in there and do a good job.” They were just a mix of confused and livid. How did this happen? How did this occur? We specifically brought in this tiny little company to fail or to find nothing, or very little. Maybe one or two small flaws that are easy to fix.
But we definitely went way, way, way, way beyond where I think even the guy who brought us in expected us to get. I think that's a problem. We went so far beyond where any expectations of our success could possibly go. That ended up being a detriment to us. We were not invited back.
James Flom
That happened many times.
Robert Hansen
I know. It was my point.
James Flom
I think things have changed in the security landscape. Where people are more accepting of results. Maybe.
Robert Hansen
To some extent, maybe.
James Flom
Yeah. But back then you're like, “You're vulnerable?”
“No.”
“Yeah. I actually have complete access to your system.”
“Oh, that's not that big of a deal.”
Robert Hansen
At least to your customers.
James Flom
No. That was literally the first five years of SecTheory, I think before acceptance set in of some sort in security. It was just denial all the way. That even varies between remote devs all over the world. You have to take cultural considerations into that as well because they have different ways of responding.
Robert Hansen
Yeah. A good example is we did an assessment against a very large firewall web application firewall company. They had a whole bunch of Israeli developers. I remember they gave us months to do this assessment. They weren't going to pay us for it. But we were curious about it. They did not pay us a dime for this. But they are like, “Well, if you guys want to play with it, we'll give you one to play with because we're going to be partners.”
They wanted to make sure everything's kosher. The license expired on whatever day it was, right. Two days before it expired, I suddenly realized how much time we had left. Which is nothing. I was about to leave to do something. I'm like, oh, my God, okay. That's what we're doing tomorrow. That's all we're doing tomorrow. We're just going to test this thing.
By eight hour stint with me and one other web guys, we found 20 or 30 different vulnerabilities in this very popular, very important piece of hardware that everyone in this enormous company is using.
James Flom
It’s literally still sitting in my garage.
Robert Hansen
Oh, really? I’m sure the license has expired. But this is the thing where that should have been like, “Well, thank God we actually had to do this test.” That is not how that went down at all. They were super annoyed at us. They did not like it. They didn't. First of all, they're like, why would anyone ever attack it like this? What do you mean? Why would they attack it like this? What does that mean? They'll attack it. How are they going to attack it? Well, how would they know these things are going to be there?
Well because we know they're there. If they've ever seen one of these devices before, they're going to be able to attack it. On and on. All these very accusatory. Again, I think this is a bit of a cultural thing. But also, they were very unhappy with the results. We're just trying to help. We did exactly what we said we were going to do effectively except it took us way too long to do it.
James Flom
Our main point of contact, I still keep in contact with. Surprisingly, I just ran into him at RSA a couple of years ago. I don't think he was ever the problem. But yeah, the rest of their team was freaked.
Robert Hansen
Yeah, absolutely. There was another example. I'm going to give you a couple. There's another example where there was this bank. They got very badly compromised. They came to us and said, because we had done a penetration test on them before, they're like, “Well, we want you to get in and figure out what happened.” We got in there. It took a little while to figure out what's going on because it was actually complicated. It turned out that it was this demo software that they had that was vulnerable.
I remember distinctly, they were very annoyed with us. “Why didn't you find this?” It's because it's demo software. It's not installed on the machine that we were testing. You didn't have it there so how would we have found it? They were annoyed about that. I mean, if that was the only thing, that would have been fine. But then they got us into a meeting. They're like, “Well, we need to know how bad this compromise was.”
I'm like, well they compromised every single one of your users over this time period. They compromised these 10 subsidiary banks. All of them. Then they probably have access to this, this, this, and this. All these different things down the pipe because they had full access to all these user accounts. It was a pretty lengthy list of things that they could have access to. I wasn't sure if they did have access to, just based on the way the logs were built.
I remember this executive on the other line was just fuming at me. Fuming. Because from his perspective, I just made him look like a complete idiot. You’re scaring everybody and making things look like way worse than they are. I'm like, well if they're scared, that's their problem. I'm just telling what's possible. What could you do and what is at risk at this point. He was not having it. He was insanely… Do you remember this?
James Flom
I remember this. It was actually two different entities. It was the bank, and it was the software that was provided to the bank. I actually thought the bank was fairly cool about it. It was the software provider for the bank that had a problem with it.
Robert Hansen
But they are the ones who ultimately hired us. Why hire us if you don't want to know?
James Flom
No, the bank hired us.
Robert Hansen
No. Are you sure?
James Flom
Yes.
Robert Hansen
Okay. Well, maybe I'm wrong that.
James Flom
What happened was president of the bank actually had to drive, I won't say where. But I went down there with one of our employees. Once we figured it out, we figured out the hack, he's like, yeah. Well what was, I believe, nothing against Ukraine, but I believe it was a Ukrainian developer that had inserted this code into it. He ended up actually contacting the bank security guy.
The bank president, he ended up going out and contacting every other bank he could think of. Our employee at the time, and I did a bunch of research on how many other banks were vulnerable because that one was vulnerable. Because the software was vulnerable. Who used the software? The guy from the bank ended up contacting everybody that we told him that was. He was the good guy.
Robert Hansen
I see. He was the one who was causing the real problems. It wasn't really us. I mean, he's the one who's just making that software look like it was what it was. Which is very vulnerable software that had led to compromises of all those companies. Not every single bank that use the software is vulnerable. Only the ones that had that particular feature turned on.
James Flom
It was the change that that developer had made. But no, the bank was the good guy. The software provider was the bad guy.
Robert Hansen
I see. There was another bank that we did an assessment on. This is very similar situation. But this was even weirder. Where they actually sent somebody on site to physically sit over my shoulder and watch every single keystroke to make sure I wasn't hacking them. Which I was supposed to do.
But I guess this person who's absolutely unqualified to look over my shoulder is somehow going to catch me doing the bad thing, having no context for what I'm doing whatsoever and not stopping the person in the very next room from doing the very opposite thing that I'm doing. This is all quite silly. But I remember I found an issue with the CAPTCHA system, a Completely Automated Public Turing Test to tell computers and humans apart. Which is basically that little squiggly words and letters that you have to type in.
I found an issue with it where I could calculate every single possible CAPTCHA. I didn't have to worry about being stopped by it anymore. I could just bypass it completely effectively. It took quite a while to program it up. I got it all working. It worked like 99.8%, which is basically 100% of the time. They got back in touch with me. They're like, “Why did you waste so much time doing this? You should have been testing these other things.” Later, I found out actually we were the only company that they had test that they had ever had found any vulnerabilities in them ever.
I'm like, well, of course because they never got past that first page. That was the entire point. I had to get past that first page with the robots. With the things that I was using to test with. They were very annoyed. “Why did you waste so much time on this? You shouldn't waste so much time.” Then they said something like, “You could have hired a team out of Romania just to break it by hand. It would have been cheaper. Why did you bother building this?” It occurred to me that they were right but in a not in a way that they thought they were right.
If you just pull down all of these CAPTCHA and had them typed in by these Romanian developers, Romanian CAPTCHA breaking people, or Pakistani or wherever the cheapest labor was at that time. We had a list of where the cheapest labor was for these CAPTCHA breaking crews. It was actually quite popular. I was on the website as a matter of fact. It was quite interesting watching them bid each other down. But wherever the cheapest labor was. They were right. It was cheaper.
But what ended up happening is now I'd have a library of every broken CAPTCHA. I could just replay this thing over and over again. I don't have to break it once per CAPTCHA. It was a very small finite list of possible combinations. Yes, indeed, it would have been maybe $1,000 cheaper to have done it the way he was describing it. But it would have been perfect. That actually proves their CAPTCHA was broken in a completely different way than I was actually trying to break it.
Actually, it was worse, the way he described it. You actually explain how your thing is worse than you're even thinking it is. They were really annoyed with me. Very annoyed. But later, I found out no one had ever even got past that first page. Obviously, no one's going to find any vulnerabilities because everyone's stuck on that first page. You had to do this very specific thing just to get to the point where you're going to do the proper test.
James Flom
Yeah. One of the things about a lot of pen testing is people want performative, they don't want results. They're required by, maybe SOCKS to actually get something done. But if they hire you, and you create more work for them, they don't want that
Robert Hansen
Right. I remember one other time. I was at Microsoft. This should have been a nothing conversation. They just wanted me come in and advise them on some stuff. It's a daylong thing. “Just come in. We want to ask you some questions.” Okay, no problem. Funny enough since I mentioned that whole thing, doing nothing and killing people. That whole spectrum thing. I brought that up. Not in context of go kill people again, just to be super, super clear on this.
I'm just saying maybe you should do more than nothing. Maybe sanctions. Maybe you could pass in some laws. Something more in this direction than literally nothing. Or spanking them. Saying you're not allowed to send email anymore to this address. That's the stuff that they were doing at the time. I remember I had this other conversation with them in the same meeting. Where I was trying to explain to them, they should be looking in certain areas for certain bad guys.
Because I happen to know that certain bad guys were looking at very specific parts of Microsoft and monetizing the living shit out of these things. Just an enormous amount of money being stolen. Maybe not stolen in the way you're thinking. Maybe not stolen in the way that the FBI is involved. But stolen in the sense that you don't know what's going out the door. You should probably fix these things because you're losing out on money for no reason. You don't have to do this.
After it's all said and done, they basically semi-perma banned me from Microsoft campus because of the whole murder people thing. Which again, I did not say. A lot of people taking me out of context.
James Flom
Yeah. Funny point. When you were there, we had to hack you way out of their network so you could connect back.
Robert Hansen
That is true. That was a different time I was there. But yes. That was a different time. We had to hack Microsoft. That wasn't even part of the test. That was just so that I could reach out and do stuff that I needed to do for the test. It was very silly. Hacking Microsoft proper. Microsoft. Not the operating system. The whole company. It's another day in life. I'm not surprised I'm not allowed on the campus anymore. It's funny because I really was ultimately trying to help them.
I feel like that's a common thread throughout all these conversations. We're always just trying to help. We're always just trying to make things a little bit better. It's such a nightmare dealing with the politics and people's misunderstandings. People just don't get what hacking actually is. They're adamant that it must look like this. It must be fit in this box. You must come to us with this type of report and say it this way. I'm sorry that’s just not.
James Flom
The media influence that so much. The opinion.
Robert Hansen
Go ahead. Why? In what way?
James Flom
Because, well (a), like I said earlier, hacking hacker, is probably a bad way to describe what we do. Because it can mean too many things. One of the biggest problems in, I mean not just computer security, but in computers at all, is mis-terminology. Or making terminology too broad.
People get an idea. They heard a word. They heard an idea, concept. Then they try and apply it to what you just told them. That doesn't really work. I mean, hacker could be someone with an axe.
Robert Hansen
It can. If you get through the door then you get whatever you need. Yeah, there's been some really crazy incidents. I remember I have a bunch of screenshots somewhere. I’ll see if I can find them at some point. But it was a two car team broke through the front gate. Then they had a sledgehammer. They went to the side of the building and smashed the window, reached in, opened the door.
It was a steel door but there's a window right next to it. They just opened up, went through, went into interior office, sledge hammer through the door itself, which is a super lightweight door, opened that up. Then there's a sub interior door which also had a little pane of glass next to it. Same sledgehammer, opened it up, went inside. That was the data center room. Then they took a drill, drilled the very specific machine they were after and then left.
The entire thing probably lasted maybe 10 minutes max, in and out. That very specific machine they were after had some extremely sensitive pieces of code on it. You're never getting that back. That's gone. I would say absolutely, those are hackers. They're just a different genre of hackers than people are used to.
James Flom
Right. The media's perception of hackers has formed the opinions of everybody. It's not valid anymore. It should be described in a different way.
Robert Hansen
How should it be described?
James Flom
It's probably not going to be described in a way that's going to be relatable to most people. Which is the problem.
Robert Hansen
Go ahead. Try. Try. How would you describe it?
James Flom
Describe hacking in general?
Robert Hansen
Yeah. If you're talking to a Hollywood producer. Like, “Hey Chris, are you listening?” What would it take to make you write a hacker in a more accurate way?
James Flom
I mean, hacking in and of itself is a mindset. It's not a single thing in any way. You can hack hardware. You can hack through a door. You can hack software.
Robert Hansen
You can hack people.
James Flom
Yeah, hack minds. It's too broad of a term because security has grown so much since we started our careers. It needs to be probably redefined as to what each part of it is.
Robert Hansen
Network security. Web application security.
James Flom
Network security is not even an accurate description of network security.
Robert Hansen
Yeah. I know. What would you do then? How do you describe to Hollywood what they should know? I think that's the problem. We don't even have the right words in our own industry to describe it. I mean, I know what you're saying because I know you and I've been in this industry a long time.
But I couldn't just say, okay, here is what network security looks in a way that's relatable. I mean, it's all going to have to be a pretty GUI on top of it. Some WYSIWYG that makes it look pretty. Otherwise they're just not going to know what you're talking about. Audience can't visualize it. I agree with you. I just don't know how you fix it.
James Flom
I think one of the problems is that because everything's out there, people think they know everything. But they don't. But the internet tells them they do because they look something up. Describing what we do is never going to be some, I'm a hacker.
Robert Hansen
It’s not a five minute conversation.
James Flom
It's not. It is like talk to a quantum physicist. You're not going to know what they do. They did 10 years of research just to get that title. You have to put effort into things. Nobody wants to put effort into things anymore. They just want whatever is now.
When we describe things, it's not that you can describe something to someone that doesn't have prior knowledge of at least the general thing that you're talking about. I think that's one of the biggest problems with security in general. Or compute in general at this point.
Robert Hansen
There was this time this customer came into our office. He sat down. This guy is from overseas. It was just you, me and him in this room. He said he wanted to quiz us on our data security. Really, I think he just really wanted to check out our office. He was really, really curious. He flew all the way over from Denmark.
He asked me, “What would you do if James went bad? What happens? James decides to do something bad, what are you going to do about it?” I'm like, what are you talking about? There is nothing I could do. It's a different area of security completely. I have no control over any aspects of what James is doing. Furthermore, even if you had an expert who you thought was able to stop James, they probably still wouldn't be able to do anything.
Because James is extremely, extremely good at this very specific area of security. There’s nothing. There's literally nothing that can be done. You just have to trust him. It's impossible to stop this. If he wants to hack you, you're done. Then he asked the follow up question. He asked you what you would do if I went rogue. Do you remember your answer?
James Flom
No.
Robert Hansen
It was basically the same answer. I don't know anything about websites. I think that is making your point though. You can't isolate security as one amorphic blob. It doesn't work that way. We really have very different domains of expertise. That makes it hard to communicate in some cases. Because I'm really talking about web stuff. You're like, “Yeah, but there's this networking component.”
I’m like, “Yeah, but networks don't matter in this context.”
You're like, “Of course, they matter. It always matter.” But it's always this weird back and forth, where it's not like we're talking past each other. But we're always trying to find this common ground where these things meet. That's where the magic between you and I, I think has always been. It is, eventually we'll figure out where they land.
Then there's something usually very interesting there. Because you know something I don't know. I know something you don't know. We just land in this weird spot in the middle. There's no expert anywhere who knows both of those things.
James Flom
No. I mean, we're a duo in that. Super Powers. Bam.
Robert Hansen
Yeah. Got to get a ring or something. After all is said and done, I wanted to run through all those examples to illustrate that this isn't a one-time thing. This happened many, many, many times. I'm sure I've missed a bunch of others as well. But why do you think being good at security ended up becoming a detriment?
James Flom
I mean it's just misunderstanding. Detriment in what way?
Robert Hansen
Detriment in the fact that we didn't get follow up contracts. I literally got banned from campuses. There was a lot of things that I did right that ended up becoming a problem.
James Flom
Yeah. Lying. It's just ignorance of what we do. It's the same thing. Goes back to you have to have an understanding, a basic understanding of what we're trying to accomplish. If you don't, people are confused by it. People don't like change. They don't like confusion. They don't like not knowing and understanding.
What we do is we spend years and years and years of our lives figuring it out. They've spent five minutes on the internet trying to understand it. Then you can't take the knowledge or the reasoning that we use to keep them safe. You can't absorb that in a short amount of time.
Robert Hansen
Why do you think security is so hard? Is it just that it's a lot? Or is it that the concepts themselves individually are hard? Or both? What do you think the barrier to entry is for the newbies who are coming in that makes it almost incomprehensibly difficult?
James Flom
I think those are two different things. One is, we think different than most people. It's fairly important. Because people build things not with the thought of them being broken. They build things for whatever useful purpose they want. We look at things and go like, I could break that shit. For newbies coming in, they need to learn that obviously.
Robert Hansen
It’s a mindset issue.
James Flom
It is a mindset issue. But it goes back to where I was saying, you need to understand the technology better from the base up before you get into the hacking part of it. It is not always going to work out that way. You could go either way. But you're going to be a better hacker if you start out with a knowledge base.
Robert Hansen
Know how to program. Have a concept of the history of it. I find myself going back and looking at really old security documents that I either haven't read in many, many years, or maybe never got around to reading. I’m always meant to read that one. I get back there and I'm like, oh, my God, I bet this and this and this are broken.
I found a lot of stuff that way. Just these old, old, old documents. The other thing is, there are documents out there that everyone will tell you to read. Go read the RFC. It's a very common thing to tell people to go do. Go read the RFCs. You get there and that is not how things work at all.
James Flom
RFCs are almost all bullshit.
Robert Hansen
But the strange part is, effectively everybody telling people how things work will point to the RFCs and say, well just read the RFCs. You are like, well that's how it should have worked had they not taken a whole bunch of performance shortcuts and added 28 layers of XYZ on top of this.
Oh, and by the way, there's this new standard that completely blows that whole thing away. Et cetera. Et cetera. There's a lot of hidden complexity in the way the world works, the way the internet works.
James Flom
I think one of the problems with the RFCs is they're not living documents that are constantly changing. If you look at the OSI model…
Robert Hansen
Sorry. I didn't mean to laugh out loud.
James Flom
No, you should. Totally. But I mean, it should be a living document that is updated constantly. It's not. Yes, it was a great reference in 1977.
Robert Hansen
Maybe it was. Maybe it predates my understanding of what good documents should look like.
James Flom
All the RFC should be living documents that are constantly updated.
Robert Hansen
Yeah. They should be more like wikis than they should be full-on documents. Yeah, I totally agree with that. I very frequently, back in the day, would build these things and say, this is a living document. Don't consider complete. This isn't the end of this thing. I'm going to make updates to it whenever I would do that thing. I know that I'm going to come back to this thing over time.
I think more things could be like that. But I also think that people shouldn't treat these things as if they're sacrosanct. These are fallible people writing documents with very limited understanding. RFCs were the beginning. They do all this work to come up with these ideas and how things should work. But they haven't implemented yet. They haven't actually done it yet. Obviously, you're going to make mistakes. Then you're going to get there.
“Oh, actually, it'd be nice if we had these two or three extra features. By the way, we can get a patent if we do this extra thing.” All of a sudden, this thing's way different than you originally got the spec on. I think that's just how things grow. Then through three acquisitions, they got 20 other things going on. They’ve got to make these two things talk to one another. Then it's just way too complicated for any one person to totally understand.
James Flom
Yeah. It was funny. I was reading an RFC a couple of weeks ago. It literally had a comment in there that said, ‘This is a terrible way to describe this. But it's in the RFC. We can't change it now.’ That was it.
Robert Hansen
I mean, the part of the HDTV spec that to this day just has always made me laugh is there's a misspelling in it. It's in the referrer. It is one R instead of two Rs. Your browser, every time you go from one page to another page, it sends referrer headers. Where were you just at before you got to this web page? Every single one of those requests, every time you go from one page to another page, you're sending a misspelling across the internet.
That's just how the RFCs are. They're fallible. They're written by people who… We're way too far gone to fix that spelling mistake. That is 20, 30 years of technical debt to go back and fix that one spelling mistake. We cannot do it. It is way too complicated. Just declare bankruptcy on that spelling mistake. If I can explain that one tiny little issue that everyone's going to understand, imagine how many other things are messed up? That's where we are.
James Flom
The creator of the URL actually went back and said making two slashes was a huge mistake. Because the number of man hours of just clicking on another slash and computational power, there was no point in it. But it's been done trillions of times now.
Robert Hansen
At least. A lot more than that. What bugs you about InfoSec? If you could change something about our industry. Just wave a magic wand and fix it. Or let's say, maybe not magic. Just say you wish people would pay attention to this piece of advice and start adhering to it. What do you think you would change?
James Flom
I think it's really where we were at before. Where you're taking networks and actually dividing them up correctly.
Robert Hansen
Carving up the networks.
James Flom
On top of that, just compartmentalizing everything. Containers try and do that. But they don't do it from a security aspect.
You can't fix people. People are always going to suck at security. There's no way they're going to get good at it. You have to always build technical controls. I think security training is almost pointless. Outside of security engineers, someone you're teaching to do a specific thing, you're not going to get your average employee to do anything right.
Robert Hansen
I have noticed, just in the last 10 years or so, a massive decline in technical capabilities of the people I talked to within various companies. When you and I were getting started, when we talked to an average customer, they actually knew what we're talking about. They may not know exactly what we're doing, but they had a pretty firm grasp on anything we were talking about. I didn't have to spend a whole lot of time explaining any way exploits work.
There was a couple occasions where that's not true, obviously. I'm not saying it was uniform. But I remember at the conferences they would say, how many people know about this exploit? People would raise their hands. Gradually over the years, that went from one or two people to the entire audience knows what you're talking about. That was echoed in all these companies we were talking to. All these companies had relatively good experts.
I don't know if you've seen the same thing. But in the last 10 years, I've seen a massive decrease in capability. Massive. They're saying things that we have known to not be true for 20 plus years. They're saying it confidently like they learned it in school. You know what I mean? This is how it is. I'm like, that is not how it is at all. I don't know what you're talking about.
I think it might be attributed to the fact that, frankly, there's a very limited amount of people who know what they're doing. They're aging out. They're becoming executives. They're moving to other industries. They're retiring. They're moving into other sub-parts of the company like CTO or CIO or something like that. There's a massive brain drain. Incredibly large amount of people egressing out of the industry at the top level. Who's back filling them? Really, it's going to be something like a CISO, Chief Information Security Officer.
That person probably came from somewhere else as well. He probably came from being a CTO or CIO or something less good. Maybe served VP of Engineering or Director of IT or something like that. Or maybe compliance. They're just given that job. There you go. You're it. We figured it out. You're the guy.
Then they're really incentivized to leave as quickly as possible. Like 18 months. They need to get out of there because they're going to get hacked. Because they don't really know what they're doing. If they stay any longer, people are going to blame them. If they leave within the first 18 months or so, they can blame the previous person. Whoever was in there.
Then hiring these new people. They have this massive amount of headcount that they got to backfill. Let's say it is five people, 10 people, they need to backfill. Where are they going to get them? They don't really know anybody. They're not from the security world. They go to local colleges. They find these people who have just graduated and know literally nothing, basically nothing about security. They just know a couple of textbooks.
By the way, we still reference hackers.org as if it's an existing website. I guess, we could always bring it back. But the problem is, there really is no incentive to find this amazing candidate. Frankly, you can't do it anyway. They don't exist, firstly. Then secondly, you probably work for a company on average that is not very interesting. Most security people are not going to work at this not very interesting company. They're going to want to work at a company that's doing penetration testing, or some security consulting of some sort.
It's getting more and more difficult to find people who are willing to take the job as you're getting more and more people at the very bottom of the security rung to enter these companies. I mean, are you seeing this? Or are you feeling what I'm feeling? I think it's just growing. It's not getting better. It's getting worse.
James Flom
It's not because the incentives aren't right. Nobody wants to, I’m not saying nobody, but few people want to put in the effort to become an expert. Becoming an expert is not easy. Maybe it's not even worth it for a lot of people.
Robert Hansen
Sure. It's enormous. It’s a lift.
James Flom
I mean if you're getting a $200,000 a year contract. You're not working hard for it. What incentive do you have behind it? So there either have to be legal remedies, that put pressure on companies to secure things, or economic pressure that makes them need to do it.
Robert Hansen
I think what it ends up coming down to is we need to get security out of the hands of individual companies, they just got to take it away from them completely. And put all of the security in a handful of vendors who are good at it, who can get the right resources in there.
Who really do know how to build things properly, and then back propagate that, all the knowledge and lessons learned back into these companies because they're not going to be able to do it themselves.
James Flom
That would be ideal. I think the biggest problem with that is people making decisions on who that is, don't understand it. Legislators don't know it.
Robert Hansen
That’s absolutely true.
James Flom
Heads of companies don't know it. And if you have a good enough salesperson coming in, and they're selling you bullshit, you're buying bullshit. And that's what I see happening the most.
Robert Hansen
Yes. You and I have had this very interesting way of working together that I think is weirdly antiquated, but I love it. Which is that you and I have always used the Socratic method. This is what it ends up looking like. I'll come up with a thesis usually, almost always coming from me.
James Flom
Then I shit on it.
Robert Hansen
Yes, you'll shit on it. Not always, sometimes you agree, obviously. But when you don't agree, you'll shit on it. And you'll just tell me I'm wrong. You don't bother sugarcoating it. You’ll tell me why you think I'm wrong. And I'll tell you why I think I'm right. And we'll go back and forth and back and forth.
Then we'll start researching it, or if it's not one of those types of things that needs to be researched, it's more explaining things back and forth or coming to a common understanding and sit there and we'll do it. But we do potentially with raised voices, and we get heated.
That does happen. Come on. It definitely happens.
James Flom
Never raised my voice
Robert Hansen
We definitely use strong words with one another because we have a strong point to make.
James Flom
I've been trying so hard not to.
Robert Hansen
You can say whatever you want. You can call me an idiot right now, it's fine.
James Flom
There's probably 100 F bombs I would have dropped by now.
Robert Hansen
You can go ahead, it doesn't matter. But the problem is, if you try to talk to the average employee the way you and I talk to one another. Now I understand, we were sort of the heads of this company, so talking to employees is a different thing.
James Flom
We were equals.
Robert Hansen
We were equals. But if you talk to somebody one or two levels down from you, the same way that executives talk, they think that they're going to get fired. But our intention was never to fire somebody over bad ideas. We just wanted to improve their ideas.
How do you get the best ideas? The fastest way to get there is to come with an argument. So quite regularly, employees would come to us, and they'd have this spiffy idea and I'm like, “That sounds like a terrible idea.” And I beat them down. And I know you did this exact same thing.
I'd say, “Okay, well, here's what I need you to do. You need to reheat the RFC, and understand why what you're saying is not possible, and then come back.” The idea was, and it started working, it took about a year or so but it started working.
What ended up happening is the employees would go back, they'd read it, they know they're not going to get fired, I'm not actually upset. It's just like, “Come on, don't waste my time. Go actually do your research first.” Then they come back, very armed the next time like, “Okay, all right, I've read the thing.
I'm ready to have this conversation.” And then, and it still might be a terrible idea, but at least now, you're starting on a much better footing, you're like, “Okay, you remember when you read that one thing about the blah, blah, blah?” I'm like “Oh, yes.
So if you extrapolate and do this, and that, then it won’t work.” And they're like, “Oh, I get it now. That makes sense that now I see why you were pushing back so heavily.” But I don't think people are willing to do that anymore.
James Flom
I think it's more than that. We were both technical leaders and most companies don't have those. So because we could give advice specific to our domains, it was useful. But that doesn't happen in most companies.
Robert Hansen
You think it's purely just that they're not technical enough to even argue? Do you really believe that? I think there's technical people out there.
James Flom
There's technical people out there, I'm saying leadership, trying to tell someone who is technical has to have at least an understanding of the technology that they're trying to give to their employee. And we were in a unique position where we were domain experts.
So we could hand that off. It’s just not practical in a big company.
Robert Hansen
Yes, I see there's other, for instance, I just had a conversation with John O about SEO, I know that he could do the exact same thing about SEO. It wouldn't be anything to do with security at all, necessarily just marketing, pure play, let's say. And he could do the exact same thing.
Or you could do the same thing with sales. There's a lot of right and wrong ways to do certain types of sales. Trying to sell in New York, compared to Texas, completely different types of sales. You got to be much slower, you got to ask them about their kids. It’s a totally different type of sale.
There's a right and wrong way to do things. And it's provable, you can actually get the metrics on it. But I still don't see people really having these tough conversations with the positive intention, like I'm not trying to beat anybody up when I'm having an argument with them.
I'm trying to get them to come up with the right decision about whether this is true or not. Sometimes I'll fight them just to make sure that they've thought it through. There's this parable about a gate across the road and some guy shows up and he's like, “I want to move this gate.”
The other wiser person says, “Why do you want to move the gates?” he's like “It’s on the way.” He's like, “What's it there for?” He's like, “I have no idea that's why I'm going to get rid of it.” He’s like I'm not going to let you get rid of it until you know what this gate is for.
Once you know it's for then go ahead and get rid of it. The comeback and you tell me why it should be gotten rid of once you know it's what it's there for.” I think that's how I've always seen this whole thing. I'm willing to hear any crazy idea. It doesn't matter how crazy it is.
But I have to know you at least started with the right kind of information first, you're not starting from nothing. If your idea is synthesized from you just having a dream one day. I'm sorry, you're going to have to go do some work before we have this conversation. Okay?
James Flom
I'm not even arguing about that one, it’s the exact same way I think. And one thing I would say is of all our former employees that I'm still in contact with, they're all much better off than they would have been.
Robert Hansen
Absolutely, they're rock stars now, amazing.
James Flom
They're running some of the world's infrastructure.
Robert Hansen
Yes. Literal just top at the top of the game amazing. That would not have been possible if I had let them do what they were doing. If I had just let them go and do whatever they were doing, they would have just languished, they would have maybe gotten slightly better here and there just by pure accident.
But forcing them to think through what their ideas were and actually be good at thinking was critical to their growth I think.
James Flom
It was.
Robert Hansen
They will even say it actually, they will say it out loud.
James Flom
Yes. And obviously put it in the effort to be better.
Robert Hansen
Yes. It’s hard work. I'm not I'm not saying I gave it to them. I'm telling them go fucking work. Figure it out.
James Flom
I'm super proud of a lot of our employees, ex-employees.
Robert Hansen
So this one time I went to Taiwan, I was heavily courted by the Taiwanese military. It was a real wake up call for me in general. I feel like going over there, I was a kid. And when I came back, I was finally an adult, I like to say I finally put on my big boy pants.
Because I finally realized what we were doing was not actually just fun and games for the first time. It seems crazy, obviously I should have known this much, much, much earlier. But I think while maybe intellectually, I knew that that was true. Inside emotionally, I hadn't figured that out.
Then all of a sudden, I was confronted with the Chinese military, a Taiwanese military, all playing their little spy games, which is a crazy story and I'll do it some other time. But the end result was I finally realized how dangerous travel was, I realized how much you and I personally were at risk for reals, not just theoretically.
James Flom
I remember you being freaked out when you came back from it.
Robert Hansen
Oh, yes. Fortunately, I hadn't brought anything with me that mattered. So everything went right in the dumpster kind of deal. But that's the kind of trip, when you realize that you're actually being followed for reels, and people are actually videotaping everything you're doing.
You're confronted by multiple spies, this is real, and they could just bury you, if they felt like it. There's no reason you have to come back from these trips. So I think that was the crevasse that opened for me, that made me really realize we need to be taking this much more seriously.
How we interacted with customers much more seriously, and how we treated our employees much more seriously, how I treated you much more seriously. Everything had to be done with a more buttoned up than even it already was.
I remember a number of times I came into your office, something would happen, I'm like, “Okay, we got to armor up,” you're like, “We're already doing everything.” I'm like, “I know, there's going to be something we're not doing. So let's start doing whatever that is.”
We'd think about it, we'd whiteboard every single thing we could think of that might be slightly better than what we're doing. And we'd implement two or three of them or whatever and get slightly better. I think that's the kind of thing that separated us from a lot of I would say our competitors.
James Flom
At the time. Obviously way more now. Ridiculously more.
Robert Hansen
So that leads me to my next question. So you were overseas, at one point working for a foreign military in a different foreign country training them, which is a little odd, but that's where they want to do their training. And you encountered your own version of the spy game, would you kind of elucidate some of that?
James Flom
The foreign government that we were, not training, but the one we were in, was spying on us. And it was obvious. We’re in a hotel, and there were cameras and listening devices and whatnot and it was obvious they weren't even good at that. But yes, I was there for six months.
I was teaching them how to oddly find satellites, satellite receivers, so they could target some insurgents in their own country. And a lot of just basic security training around that to hacking cell towers and stuff. I don't know where you want me to go with this.
Robert Hansen
Well, I think the thing that stuck out in my mind was the cell phones were constantly under attack.
James Flom
Yes. They were but it goes two ways. It's funny because I was using the country that we were hosted in infrastructure as a training tool on how to hack because I was hacking that.
Robert Hansen
Wow, I didn't know that. I'm sure they didn't love that. Or maybe they can learn something from it.
James Flom
Maybe. But anyway, we built a lot of devices while in the country, which was difficult to get actual equipment into the country to do that. I use burner cell phones when I ever travel outside the United States, I use burner cell phones all the time. I think I've currently nine active cell phones, which is ridiculous.
Robert Hansen
You're like a drug dealer.
James Flom
Better call somebody.
Robert Hansen
But that led you into building something that you had been talking about for a long time. I don't know where you are with the project but it might be worth talking about. Something to identify if you're being followed, I think that's worth talking about.
James Flom
That wasn't actually where it came from. It was from that meeting in LA that we had with one of your clients that you were working for. And we had a long discussion about a fairly famous person that had security issues. And I had thoughts about the rest of it before then.
But I was like, what are all the different things that could identify a stalker? Or a combatant? or whoever. Which leads me down the bad path of privacy and all the issues around that. But then I started putting it together.
Robert Hansen
Of course.
James Flom
If I don't do it, somebody else did it. Right?
Robert Hansen
I would never say that about anything you work on. But yes, it's possible someone else would do it.
James Flom
Anyway, basically any RF signal, I'll take in and try making a unique identifier about it. But then I started expanding that to image, figuring out, someone walking this way, that way, who they are, grabbing license plates, taking images of drones, and explain…
Robert Hansen
Using all kinds of different spectrums.
James Flom
Yeas, so garage door opener. A lot of cars now have Wi-Fi built into them. So you can do a lot of identifying information about freaking anything at this point because everything's wired, or wireless, whatever.
So I started building out a sensor network with the original idea of just protecting someone, like knowing if someone bad is in your area, or if you're being observed. Things like that.
Robert Hansen
That was certainly something that both drug dealers and celebrities have in common.
James Flom
It is but there's more than that.
Robert Hansen
Yes, sure, of course. Some people have evil stalker ex-girlfriends.
James Flom
Or some people want to be noticed. And this might be a way to do it. But I can’t go in the specifics of it, but it does a lot of stuff. And once I realized that if I built a database that can take in all this data and then correlate it, there's a lot of interesting information you can get. I've added stupid things like ornithology to it.
When I walk my dogs, I look at birds. I'm like, “What kind of bird is that? When are they here?” So it's a sensor network. So I'm just taking in information.
Robert Hansen
So are birds real? Have we figured them out?
James Flom
I'm pretty sure.
Robert Hansen
So I think people listening this are going to say wow, this sounds super paranoid, and, what the hell are you guys talking about? And let's walk back a little bit back to the SecTheory days, just because I think this is a useful example.
You one day came into my office and said you found a carnivore box on our network. Can you explain what happened there?
James Flom
So it's actually hilarious. We were working on a case with a customer and the FBI asked if they could put a box on our network. And we did, they're using a flash exploit, to try and reveal the real because the user was using a VPN to make the threats against our client.
We put that on our network, isolated. And at the same time, I decided to start looking around, I'm obviously a paranoid person. So I was like, I'm isolating the FBI box, I started looking at the network. So at the same time, we're hosting a box to reveal the IP using a flash exploit. There was also a carnivore box sitting right next to us.
Robert Hansen
What do you think was doing? What do you think that was for?
James Flom
It was for slackers.
Robert Hansen
Yes. But what for? What were they doing with it?
James Flom
I'm sure they were trying to grab all the information we had because you specifically said at one point that we're not encrypting anything,
Robert Hansen
That’s right. It's critical. It's actually very important to the way a lot of the site worked.
James Flom
Yes, so we're not encrypting anything and then the FBI put a box on the same, a SPAN port on the same network. So they were copying all the data. So everything that ever happened on slackers.org is definitely in the FBI’s hands.
Robert Hansen
Or could be anyway. The reason I didn't ever want to encrypt things is because it actually broke certain exploits. The way browsers used to work, not so much these days. But you couldn't guarantee that it wasn't going to pull from HTTP. And if we only support HPS, the encrypted version of it, it would break a bunch of stuff.
So there was a method to the madness. But you and I were always safe because we had an encrypted backdoor to get into these things. So everything we ever sent was encrypted. It was really just everybody else.
James Flom
It was. That was the great part about it. It led to the armored stack, the separation.
Robert Hansen
That's right. So partly your trip overseas, and some of the people you met while you were over there. I remember you telling me “Hey, there's this thing you should probably do, which is go get your FOIA.
Go FOIA yourself, or FOIA.” Which basically is asking a Freedom of Information Act. I would like to know everything the government knows about me, or this government agency knows about me, or whatever. You can ask about any question you want.
So I said, “Okay, that sounds interesting. I should probably know how that works.” So I submitted two FOIAs, one to the CIA and one of the FBI based on this conversation with you. Because one of your friends had actually submitted one and had actually gotten some data back, which I thought, okay, cool. I’ll do the same thing.
So I got the response from the CIA, basically, immediately. They said, it was a Glomar response, which basically means that they are going to not admit or deny whether they have something or not, or whatever.
But it was written in a very particular way, the first paragraph was like, here's the Glomar. Second paragraph was, “We don't have to respond to you because of these six very specific national security reasons or whatever. Like very specific not like global like this applies to everybody.
Specifically we want to do it this in your case.” And the last paragraph, it's all just one sheet of paper was, “You can appeal this, but I'm the guy you are going to appeal it to and basically just don't bother.” One of those. So I got the point.
The FBI on the other hand, they called me, they call me up on the phone like, “Hi, I'm Agent so and so.” It’s a woman. I'm like, “Oh, hi.” And she's like, “Well, why are you trying to get your FOIA request today?” And I explained exactly why I was, because I don't know how you guys are hacking me.
I would like to know. And I know that you guys have in this case, it's a carnivore box. And I'd like to know what else I don't know. And I'm sure she didn't love that answer. But it was honest. I don’t lying to the feds. But anyway, she took it, she's like, “There's a danger here in that, if this gets out of my position, I'm not going to be able to guarantee its safety.”
I'm like, “You mean, the physical drive? I wasn't quite sure what you meant, the thing you put in the mail? The pieces of paper are?” What are you talking about? And I'm like, “Are you saying there's agents inside the post office, and they're going to see it?”
She kind of paused and she's like, “Well, I'm not, not saying that.” like try to dissuade me. And I'm like, “Okay, I think I'm going to go for it. I think I'm going to do it anyway.” So another couple of days passes, maybe in the local field office calls me like, “Hey, Robert, just want to let this thing crossed my desk.
It is now off my desk. I completed it immediately. It is now off. By the way, why were you interested in this thing?” I'm like, “Remember that carnivore box you guys put in our network?” And of course, he's like, “I don't know what you're talking about.” One of those.
I'm like, “You're probably the guy who did it. What are you talking about? What are you even talking about?”
James Flom
The most hilarious part was we knew there was a carnivore boxer because it had a web interface.
Robert Hansen
Yes, it said carnivore on it.
James Flom
Try, at least a little bit.
Robert Hansen
So I think many years passed at this point, years went by, and I was messaging him about every year like, “What's going on with this thing? Are you guys ever going to respond to this thing?” And just, it's yours kept going on and on.
I finally got some response back, they're like, “Well, there's three different types of cases, there's four.” I think there's small, medium, large and extra-large. So think of small is anything less than I think they said less than 10 pages or something like that.
You can imagine, like large would be small, medium, large, and extra-large. Extra-large would be like Hoffa or something or it'd be like JFK assassination, or one of those crazy huge cases. Yours is a large. And I'm like, “Holy crap well, how big is this document?”
Then I was picturing, they might have every single page of every single thing that had ever been submitted on Slackers through that carnivore box, maybe, who knows. So another year or so passes, and they're finally done, they actually send this thing to me.
It was only 560 something pages of content. So about two reams of paper worth of content, of which they redacted about 500, and something pages of it. So just right there, gone. So that only left 50 pages, I think 50 even pages.
Of those 50 pages, every single word of every single answer of the boilerplate template that they put on every single case was all 100% redacted, including things like my name. I'm not even allowed to know my own name. And I know what was going on there.
I know that they just know that they don't want to answer.
James Flom
It’s a giant FU.
Robert Hansen
Yes it was.. But it was an interesting FU. I actually learned things in the process. I just think that there's something weird and magical and fun about this whole entire time and environment where we probably ran afoul of every dragnet spying operation that had ever been performed by the US government, and probably all of its allies.
Part of what took them years to respond is it probably took them that long to find somebody who didn't have an enormous amount of work on their desk, who was cleared for every single one of those programs enough to know whether I was capable of figuring out things I shouldn't be able to figure out from one or two random pages in some FOIA response. That's kind of insane.
James Flom
It’s a brokenness that should be fixed in the government, obviously.
Robert Hansen
Yes, it's crazy. So at the tail end of us working together, was during the Great Bust of 2008. The subprime mortgage crisis bust. I remember standing looking over Wall Street, and I was in one of our customers, big bank, it was in their office, it's big, beautiful office, all glass.
I was sitting there with a customer, and he's like, they're going out of business, they're going out of business. They might be okay, they're going out of business, they are probably going to go out of business.
I'm like, “What's going to happen with you?” And they're like, “Well, I can tell you right now, we're not going to be buying any more services for a while. It's like 100% austerity.”
James Flom
They went out of business.
Robert Hansen
The problem is, we had about six months of reserve cash, which is maybe a little bit more, maybe eight months or something. But we didn't have a year or more. And what we really needed was a year or more.
So that was big, that was very hard on me, extremely hard on me because we had worked so hard. Both professionally and personally, there was a lot of risk. It was a very hard company to run for all kinds of reasons. Literally, everything was coming at us.
Aside from people literally shooting at us, we had everything coming at us. Certainly guys with guns, just no one actually shooting at us. We had to come out of that thing with enough cash to survive an enormous economic decline that we could not have easily seen from our position, at least not how bad it would have been.
So I ended up having to fire myself. I literally went to a mirror, I told myself, “You're fired.” I needed to do it ceremoniously. So I could get it out of my system. And I went through a pretty bad depression about it.
But you ended up running the company. Ended up taking over partly because you were super important for running, we had Armored Stack running, the other company called Falling Rock Networks. And we had customers at the time.
So you were far more critical to the day to day operations for the existing customers we had, which was recurring revenue, and they weren't going anywhere, at least for a while. So we ended up not exactly shattering the business, but certainly downsizing significantly.
James Flom
We laid off everybody except for our admin.
Robert Hansen
And you. So did you have any lessons learned out of that? If you were to tell other business owners, especially people starting in the security world, I think I would tell them just don't start a consulting company, because that is a hard business.
James Flom
100%, it is super hard. Yes, building products that have recurring revenue is way better than trying to build something that is an ideology. Fixing other people's problems, as we've talked about earlier, that some people don’t even want to fix, it's a lot of stress, it's just not worth it.
So I don't think I would ever start a consulting company again. It just doesn't make sense, financially, it doesn't make sense from also a stress point on yourself. I'm sure running any small business is hard, outside our industry, but specifically like people that have active dislike of you.
At the same time, if you look at economic downturns, what's the first thing that's going to go? Things that don't have immediate financial value to the company. So security is especially hard at that.
Robert Hansen
Yes, agreed. I think one of the things I didn't like about running a consultancy is if you're not selling, then you don't have a pipeline. But if you're not working on a customer, you don't have revenue coming in. So what is it going to be?
Are you going to build your pipeline or are you going to make money, are you going to build your pipeline, or are you going to make money? Because those things are not the same thing and you can't do both at the same time, at least not well.
Running a company and being as public as I was, I knew, people wanted us, they didn't want our employees. So it was always like “We’ll be there too.” But really, we need our employees to do most of the work otherwise, we're going to be totally sunk with too many tasks.
So it was always either sell or work and you can't do both and you can never really take any time off because you're always trying to get one of those done. And I think that sunk into my psyche. I think what ended up happening is I became a sales guy for the first time in my life, I'd never really been a sales guy.
I was treating everybody very transactionally. You can either buy my product, or you can't. And if you can't, why am I spending time talking to you? I really became a kind of a shitty person. I look back on myself, I'm like, ah, I really didn't like how I treated people back then.
I'm glad I went through the downside of it, just so I could examine the kind of person I was being and really didn't like that. I don't think we ever really talked about this.
James Flom
No.
Robert Hansen
But I feel like, I spent a lot of time looking at people for what they can offer or provide me instead of looking at them as people with interests and desires and cool things going on. And I think one of the cool things about this show, it just shows exactly the opposite.
I've totally swung the exact opposite direction, I'm super interested in people, I spend more time thinking about people than I do anything related to work at all these days. I'm in my spare cycles in my brain. Because I think people are extremely interesting.
I had not given them credit for being as interesting and complex as I think they actually are. And after that, not immediately afterwards, certainly, but took a while for me to figure it out. A lot of reading and thinking and whatever.
I came out of it and I realized, I'm just going to decide to be nicer to everybody. Doesn't matter what they look like, doesn't matter what they sound like, doesn't matter what economic background they have, whether they can buy our products, whether they're in security or not, I don't care.
Just try to be nice to everybody. I started getting friends all over the place, like an enormous amount of friends, compared to where I was before and I was very well known before, I was not like not well known person in our little industry, not that anyone outside our industry would know who I am.
But now the amount of friends I have just dwarfs that back then. And I credit that all to having a little hindsight going. I don't like the fact that I was treating people like that.
James Flom
Introspection is good.
Robert Hansen
It is. But did you notice anything about me back then? Did you notice that I was like that or treated people that way? Or have I been basically the same in your mind the entire time?
James Flom
There was a big change. Because you were doing all the sales for SecTheory. I wasn't. And I'm not that person to this day that does sales. I could see your attitude change. I didn't look at it as a bad thing.
Robert Hansen
We’re making money
James Flom
Robert is bringing in business otherwise this company would not exist, because I never could have started it that way. I've noticed a lot of changes in you through your other relationships, and how they've gone up and down.
Robert Hansen
I look at all of those things as positive in the end. Even the parts of my life I look back on I'm like I was shitty. But it's still a great learning experience. And I would not trade it for anything. I'm glad I went through those times that I know that I was not a nice person to whomever I was talking to.
I’d go to parties, and I would intentionally stopped talking to somebody just to talk to the person over here because I heard them say some technical term, and I'm like, there might be some chance that they're… just really, really not treating people with the kind of respect that they deserve.
I completely stopped doing that. I think it's important, I think people can treat people that way. And you find yourself listening, and you're doing that, I strongly encourage you to rethink things. Because even if, let's say I was in sales right now, I know I'd make way more amount of sales than the way it was back then. Way more. Just a nicer person. People like nice people.
James Flom
So I'm pretty much fucked on that.
Robert Hansen
Well, I wouldn't say James, but yeah, of course. No, I worry about you, honestly. And I want you to do extremely well. But maybe consulting isn't right for you. So that leads me to my next question. What's next for James? What are you going to do next? Now you're thinking about moving.
James Flom
I'm thinking about moving. I've almost got the entire house packed. So definitely getting out of there. Loading up the motorhome. So I have a big stupid 36 foot motorhome.
Robert Hansen
It is stupid.
James Flom
It is so stupid at this point. I'm just going to turn it into a moving truck because… I don't know, honestly there's some things in security I want to do. There's a whole lot of things I don't want to do. I don't want to manage hardware anymore. I don't want to build infrastructure of any sort,
Robert Hansen
I don't blame you, operational responsibility is a nightmare, then you're really bound to a specific location too, you really have to stay wherever you're at.
James Flom
You're. I love the idea of controlling everything, because paranoid me needs to. But at the same time, I'm like, what are the things that I need to control that are important? I can build those out in a farm in the middle of nowhere. Connectivity these days is good enough.
Maybe Starlink or something, I can buy some shitty proxy service or whatever, and still have my connectivity through it. But I don't know, I'm trying to find something interesting to do. I have two interesting projects. One being the sensor arrays. And another was a phone I built. Because I don't trust phones.
Robert Hansen
Yes, I know you don’t.
James Flom
So I built my own phone. So there's those, and they probably have value. I would need other people to help out on those. Or at least one of them, I can get the phone, I already did that.
Robert Hansen
Product research is quite expensive. Even if you have a mostly working prototype, it just takes money. Pricing and packaging, building skews, the whole thing. Having built cell phones in the past for a client, is quite an endeavor.
Very, very complicated and lots of moving parts and then you run afoul of Google's ecosystem and their contracts and all kinds of crazy stuff. So it's really, really tricky.
James Flom
It is. I could build boats at this point.
Robert Hansen
All right. So how do people find you online James, or do you even want to be found? Just going to go and live in the woods somewhere? Send carrier pigeons in.
James Flom
There's a reason my handle is ID and it's been that way for a long time.
Robert Hansen
I can respect that. All right. Well, thank you so much for coming on the show, James. I really appreciate it.
James Flom
Thank you for having me. I just came here to bullshit with you.
Robert Hansen
We did that.
No Transcripts Are Available Yet
