MICROSOFT, AMAZON & TECHNOLOGIES TO SCALE A BUSINESS

Robert Hansen

Were you there when SQL Slammer came about?

Matt Domo

I can't remember that one to be honest.

Robert Hansen

It was a vulnerability.

Matt Domo

A vulnerability, yeah. I think so.

Robert Hansen

There was a guy named Dave Litchfield who came up with his vulnerability and it was bad, but on the grand scheme of things, it wasn't that bad.

Although it was bad enough that everybody was forced to patch. By his estimation, before SQL Slammer came out less than one out of 10 of the servers he investigated, interrogated during his tests were up to date.

Afterwards, more like one out of 10 weren't up-to-date. In a very strange way whatever happened during that project, it actually kind of kicked off the, we need updated software revolution. It's kind of interesting little tidbit about security history there.

Matt Domo

Security became a big deal at Microsoft because people complained about Microsoft and the security practices. I'll say it wasn't the forefront of everything, but to their defense, I mean, hundreds of millions of users. People are going to find things.

You only have so many people, so there's a large surface area. There's bad guys trying to do bad things all the time. In their defense, it's going to happen then it became a serious issue because it was damaging customers.

So, legions of people were focused on fixing security and making it a core part of the development process. That's when things started to get better.

Robert Hansen

Yeah. I think I was even working as a contractor right around the same time as they announced they were going to have this trusted computing initiative and all of a sudden it was hundreds of millions of dollars poured just into security all of a sudden.

Basically overnight you saw this massive increase in security, new ways of thinking about security, new security research coming out of that division. Was pretty remarkable considering what a big kind of old company Microsoft was at that point.

It was already decades past its prime in terms of brand new innovation. Kind of interesting. So, with the SaaS product, this was really the very first time you were starting to move to more business in the Cloud.

You could actually have business in a box. You could kind of almost sort of construct a company out of the tinker toys that were available on the internet. Can you talk a little bit about that?

Matt Domo

So, the product was called B-Central and it was one of our five important strategic bets in Microsoft at the time. The leader was Satya Nadella, who's now the CEO of Microsoft. He was my boss. I led architecture and a core platform development. It was all around getting the small business online.

It was geared towards small businesses and helping them operate online. It was built by products we had, plus acquisitions we had. One was to build an help quick tools and then host their website. Then we had an acquisition link exchange, years gone by. They were the founders of email marketing. We were the founders of Spam.

Robert Hansen

Spam, there you go.

Matt Domo

It was all about email marketing campaigns. Then we had banner ads and then we had online accounting. We had accounting and then we had CRM. We would keep adding things around the back office and the front office elements of running a small business. We learned a lot of things, one of which is, was a little ahead of its time.

Robert Hansen

Of course it was. Probably two decades ahead of its time.

Matt Domo

The other thing is, it was different than anything else Microsoft had done shrink wrap software to that standpoint. You had Waterfall and you had to change your development model because you're controlling everything.

You had to move quickly because people wanted capabilities and you didn't have the CD printing process and the fear of making a mistake because then you'd have to resend out media and we didn't know how to do it.

We were kind of figuring it out as we went. That brought in a whole genesis of things to come about how to iterate, I'd say wagel. So, different iterative development models and how to host things at scale.

I mean, we had a lot of users and it broke some of our products, core Microsoft products. There were a lot of benefits from there.

Then we also figured out that medium-sized business at the time was actually the core market. Not as much small businesses, because candidly the internet hadn't got. It's not as pervasive as it is today back then.

But medium-sized businesses had the wherewithal, the money predominantly, and the appetite to kind of grow that as a business. We went out and looked at great planes and great planes was accounting and a few other packages.

We went and acquired Great Plain Software, which ultimately went on to become Microsoft Dynamics. So, that kind of came into B-Central. That's kind of the lineology, if you will of what happened.

Robert Hansen

Microsoft made some really big bets around that time frame, like live.com, which still exists to the day. But having a centralized form of authentication, basically every big company on Earth was racing to get that deployed.

Even eBay, everybody wanted it, Google wanted it, Yahoo wanted it. Everybody was trying to come up with a federated identity model where as long as you have access to LinkedIn or whatever, you suddenly have access to everything else.

But I remember when I was at realtor.com the day that they chose to withdraw the single sign on functionality within live.com and it all hell broke loose. This is like all of a sudden lots of users can't log in.

They wanted to log in through their Hotmail account or whatever and they couldn't. Microsoft really was sort of this central point of a lot of this crazy technology that a lot of people just didn't know it even existed per se.

Matt Domo

Or they took for granted.

Robert Hansen

Yeah. They took for granted. All of a sudden when it's gone, which is one of the reasons I'm always very wary of Cloud-based SaaS applications, especially when it's run by companies like Google because they have a history of just yanking out access. Dozens of companies, sub-companies just yanked out, just pulled the plug.

Microsoft, that's the one time I remember them really pulling the plug on something that was pretty vital that impacted anybody I was aware of. But the other one that I saw that was interesting was the acquisition of Hotmail. I think it was. That was an interesting one. Were you around at that time?

Matt Domo

I was, yes.

Robert Hansen

I remember all the engineers I was talking to were all laughing about how it was like three times the amount of machines to get it working and it was trying to port everything over from, I think it was FreeBSD onto a Windows platform.

Matt Domo

Back then Microsoft was kind of anti, not Microsoft software. They were interested in buying customers. It was about buying customers and buying markets predominantly.

One of the first things they tried to do is bring them into the Borg or the Microsoft eyes them which really meant we're going to rewrite it on Microsoft technology. "Did you really need to do that?" Not quite sure. I think they're still learning that. But scalability was definitely a challenge with Hotmail.

Robert Hansen

Absolutely.

Matt Domo

And a few other things too.

Robert Hansen

I have this fun project that I had been running for quite a while actually called Smartphone Exec. The entire idea is, could you run a business from a smartphone? Just purely just a smartphone.

I don't mean with a kicker with a laptop sitting off to the side or something. I mean, just a cell phone. The short answer is no. Not easily anyway. But it was a really, really fascinating just deep dive into mobile technology that I had not bothered to do up to that point.

I got pretty far along. I got to the point where I was basically using my cell phone for everything. I had a little foldable laptop. There's a number of photos of people taken of me, of me working on my foldable keyboard with this little tiny little iPhone kind of thing.

But for me, one of the reasons I don't like the mobile environment as far as running a company is I also do some programming work. Just setting up the websites and that kind of stuff. Even very simple programming work just to make sure, this thing talks or this thing.

It turns out having a tiny little form factor is really difficult when you need a contact switch between what somebody's saying about how to program the thing and how to program the thing and you got to switch back and forth.

I always like the idea of something Microsoft had created called Continuum. The entire context of Continuum was wouldn't it be great if I could walk up with my cell phone, put it down, and all of a sudden I have my full desktop environment with the same files.

Even running the same programs all of a sudden poured it over and it's all running on a nice big screen where I can do that contact switching and I have a full big keyboard and a side mouse that's actually more functional and that kind of thing.

I was very bummed to see that Microsoft killed that project. The way I think about it is, Microsoft had a desktop operating system and a mobile operating system that were both open. You could install whatever you wanted on them, you had reasonably good access to the file system, you could pretty much do whatever you want on it.

Apple has a very open desktop, but a very closed phone. There's switching between the two is actually much more complicated because you don't have easy access to the file system.

I can't drill down to things as easily on an iPhone. It tries to kind of stop you from doing that. There are ways to do it, but it's more tricky. Then you have the Android operating system, which is extremely closed, but the phone, which is extremely open.

Matt Domo

Different spectrum.

Robert Hansen

Yeah. They're all different. Microsoft of the three was the best suited to actually do this. I felt like they were going to do it. I mean, not just because of continuous, actually before this I thought they were going to do it because if you look at the Metro operating system, which was weird and clunky, and they never quite explained why it existed.

They're like, "Oh, it's for mobile." But everyone's like, "Why? I don't care. It's weird and clunky." But if you watched it over the period of time, gradually got smaller and smaller and it then it fit into the start.

When you click the start button, there was this little window that popped up and that looked an awful lot like Metro and acted like Metro and you could launch metro apps from it. Which looked a lot like the same form factor as a cell phone.

I figured that's what was happening there. The mobile operating system was going to be combined and that would be your start button and all that metro stuff would be driven from there, except on desktop where it would know, "Hey, you're on desktop, go ahead and launch the full version."

What the hell happened? Why can't I just walk up to a device, throw my phone down and it just knows what I was doing?

Matt Domo

Well, I think there's a myriad of things there. It's a multi-step thing. If you will go as far as you want to go.

Robert Hansen

Sure. Let's do it.

Matt Domo

First off, Microsoft, they had the mobile market initially. They had Windows CE. We were running around with pocket PCs and they had it. It was a stylus. But they had it, the synchronization with it was weak.

Synchronizing, you had to tether it to your desktop and synchronizing it. It was kind of weak. The user experience, they wanted to make it windows-like, but how do you do a stylus on Windows? It was a little ahead of its time. What made it a little different?

First off, you either have to be, first you have to have cheapest, and somewhere in between is quality. They had first, definitely weren't cheapest because they charged everybody for the Windows operating system, all the OEMs, which made the price point pretty high.

It wasn't the easiest to use. It was better than anything else out there. Not the easiest to use. Apple came out with something that was revolutionary. A pretty simple thing. I touch, it's pretty simple and elegant.

Robert Hansen

Even my cat can do it.

Matt Domo

Even your cat could do it. That started to get people to think a little differently. Now, we're talking about the outside of the house.

Robert Hansen

That sounds like an ad right there.

Matt Domo

Yeah. That's the outside of the house. What you see, it was simple, it was elegant, it was innovative. The inside of the house. Microsoft was very tech forward as we know. It was all about pushing the edge technology wise.

The API design for developer was really, really hard. I mean, windows was hard to develop for, but it was the only game in town for a long time. It was like, "Let's make the Windows experience the same on the device." Well, the device has different runtime characteristics.

All the things you talk about for compute, network and storage. It's just different. It has to be different. You don't have to worry about layers of device drivers and all that stuff, but network is unreliable. If I'm going to sync with your device, how do I make that reliable?

The programming interfaces for the developer to write applications was hard. The differences between the desktop, it is elegant, like what you're saying but in practice to write that code, there's a lot of tricks that you have to do.

You have to write an interface that runs over all the different types of phones that are going to run the Windows operating system. Because of all that friction, they didn't win the hearts and minds of developers. Google figured out, simple and easy, we can debate whether the Android operating system's easy. We can debate that.

But they got easy, one was cheap, one was logged down, they both knew how to get developers and they had their stores. That attracted people to the platform. Microsoft, they were late to the game with enticing developers about how their ecosystem, they could make money.

It was all about enticing developers to port their app to run on the device but the incentives were skewed still towards Microsoft. Now let's lay it out. Probably it was an innovative interface not simple to program against. There's some friction and therefore cost for the developer that's writing the program to do.

There are some idiosyncrasies that are different than anybody else. I mean, you could look at a program today if you run on the iPhone, some parts of it, it'll take 27 lines of code on an iPhone and maybe 500 on an Android.

Microsoft might be larger than that with its programming interface. So, what's the incentive? Then the idiosyncrasies of the difference of the hardware itself, what you get on the device versus what you get on a laptop, it is different and it's just hard to make that simple. That's my long-winded answer for you.

Robert Hansen

Well, I could go even deeper. Google for instance, they really lock down access. I can't create an Android device that does have the Google apps and access to the App Store unless all things are locked down to Google, including search.

The problem with that is, I can't have a sort of a side loader with all my own. I can't build Continuum because they're going to say, "Well, you can't do all this other stuff."

Matt Domo

You got to use our stuff.

Robert Hansen

You got to use our stuff.

Matt Domo

Microsoft, same way.

Robert Hansen

But Microsoft's problem is, I think you are right. I think their biggest problem is they really never had the developer store, they never had that ecosystem. The place where I can go download all of the Windows apps.

I mean, it came later, but it was not as robust. I mean, I was searching through it maybe a handful of years ago and I'm like, "Wow, there's, you know, dozens of apps, hundreds of thousands of apps." This is very sparse.

The Metro App store in particular, which would've been the mobile device. That's everything. That's all 12 apps. Obviously I think it grew, but not by the leaps and bounds that you would have to kind of expect.

I need Uber on there. I need the ability to contact people over Signal. I mean, there's all these apps that are missing from that environment. I think that made it much, much worse. Or even totally untenable from the perspective of Microsoft. Why are we just pouring money into this thing?

Matt Domo

And they always tried to take it from Windows to here, windows everywhere. Did it have to be Windows everywhere?

Robert Hansen

Well, apparently not. Because now you have sort of Cygwim like things built directly. You have full Ubuntu built directly into the operating system.

You can run a Ubuntu apps directly. Just have to enable some flags and all of a sudden you're running it. Clearly Microsoft isn't married to the operating system, nor should it have ever been. It's really more about the users. Watching Microsoft, I'm sure far after you've left has been an interesting thing.

A friend of mine was in charge of deciding, or part of the decision about moving away from using Edge the native code Edge and moving more towards Chromium.

I told them, "Do not do that. You're going to lose your autonomy. You're always going to be months, maybe much more than months behind Google. Just don't do it. I realize you're cutting some headcount and that's nice or whatever, but you're basically wasting your last ditch effort. You're never ever going to get this back unless you fully start from scratch kind of thing."

They did not agree and now we have even less diversity in the user agents out there. It's been interesting to watch. But that's all about ad revenue. So, from their perspective, it's not about Edge, it's about Bing. How do we get people using Bing.

We don't really care if the code is written by Google, we just need to make sure that the search box is Bing. Which goes back to Ad revenue. So that's been interesting in watching Cortana come up.

So, where do you think go things are going for Microsoft? What do you think their sort of next big hits going to be? Because it seems they're getting their lunch Eden on the consumer side, but I have a feeling there's a whole bunch more under the hood coming on the Azure and business side.

Matt Domo

I think you've hit one of it. They're stepping away from the consumer side. I mean, the Xbox, but the Xbox is starting to fall behind Sony and the PlayStation.

They are taking a different tack now than Sony and the PlayStation. I think that will be an anchor, but I don't know if they'll overtake PlayStation. They had it for a while and then they didn't.

Robert Hansen

I'm actually friends with Shamus Blackley, maybe I'll bring him in here one of these days.

Matt Domo

That'd be good. That'd be a really good discussion.

Robert Hansen

Yeah. He's an interesting guy. He's a physicist.

Matt Domo

I think you're right. The consumer, it's just not their play. I think they're going to really make a heyday is the Cloud. And there's a couple reasons for that.

One, right or wrong, good or bad Windows is the desktop of business. Figuring out how to marry the desktop of business to making it easy to use Cloud business applications like Office 360 dynamics, all that and weave it in a simple experience.

Then the different pace of services that are going to come out in Azure is a difference maker. They have the enterprise an outstanding footprint in the enterprise too.

I think as they figure out how to come up with an expanded suite of services beyond Office 365 and Dynamics, and figure out how to make Windows lighter weight and cater to the developer a little bit, and then make that experience for the business apps and Azure kind of work together really, really well.

Back to your analogy of how it didn't work, I think there'll be innovation there. They're incentivizing people tremendously, especially in the business space. Not so much the startup community, but in the business space to really invest in Azure and the on-ramp for it and things of that nature.

That's going to continue to grow there. But I think the innovation is the breadth of services they're going to offer. And I think part of it, they'll invent, part of it they'll acquire.

Robert Hansen

I keep thinking there's probably some weird marriage between, as much as I would dislike this person like Google and Microsoft where you have the Ubuntu underlying underpinnings operating system running.

You could probably dovetail that and make it work with something like a Chrome OS or Android or whatever and suddenly now you can run Android apps inside the Microsoft ecosystem.

I remember I was talking to one of the developers over there and he's like, "There's no plans to do anything like that." And I'm like, "Yeah, yet." But when someone figures out that it's not a huge leap to go from Ubuntu to Android, you might think about that.

Matt Domo

It'll be interesting development. I just wonder if Microsoft's just been burned so many times on the device, whether they're just going to walk away. We lost that one.

Robert Hansen

I mean, more like as a bridge between the two. So, now you could finally start running your apps on Windows environments. You could pull up your Android device, put it down, and suddenly those same apps are running in a Windows environment.

Matt Domo

Yeah. I totally get it. That's actually a pretty innovative idea.

Robert Hansen

Thank you, sir.

Matt Domo

I wonder if some of them are thinking like that.

Robert Hansen

Well, you can tell Satya that I told them so. It sounds good. Then you went on to AWS. Tell me about that. What'd you do over there?

Matt Domo

I started the database division of Amazon Web Services. The first product was SimpleDB. It was the first database as a service product. We were trying to figure out, we knew it was also a little bit ahead of its time.

To scale, Amazon had figured out that relational databases at a certain size, it's the index maintenance of maintaining a relational database that just kills you. It's the index maintenance and the writing and things of that nature.

You wind up de normalizing and just writing flat out big, large, smaller number of big vertical tables if we were going take cross-section of it.

Then the other part is, we needed back to the cap theorem. You can have consistency or availability in the face of a network partition because they happen all the time, but you can't have both, which were you betting on? We betted on eventual consistency.

You're going to have a more of a column nurse store, the first one to do that and it's going to have eventual consistency. Eventually there, you're going to see what you need so you just pull. We did that then a whole bunch of other things happened that nobody ever knew.

We wrote the first self-healing distributed system based on gossip that scaled beyond 100 servers. We actually got it to scale to a massive size.

Robert Hansen

How would you describe the reason you decided to build this thing in the Cloud? Why bother doing it in the first place?

Matt Domo

When we were working on, we were debating what we wanted to call it. Cloud computing or Utility computing. We had storage and we had an operating system, which was EC2. We needed the third leg of the stools.

We needed a database that's kind of the core of everything that you build and an application. We knew we had to build a database product and we were looking at what it took to run amazon.com kind of rat holed on previously. So, we said, "Hey, we need a database product."

From there SimpleDB was born. But also, like I said, trying to describe, there's a lot of complexities. We made the API really simple. We made the developer experience really, really simple. But the complexities underneath, which is the core of the cloud, are actually quite complex. That's why we went after it to go do it.

Robert Hansen

You felt like you were in a unique position to do that because EC2 was coming online at the time?

Matt Domo

S3 was the first one to come online. Then EC two was about a year earlier, maybe a year and a half earlier. It was growing. But everybody's like, we need a database, we need a database, we need a database. We're working towards doing it. SimpleDB was born.

Robert Hansen

What was the next one? Was that Redshift?

Matt Domo

RDS. Relational database service. Like I said, it was a columnar database, what we built. But it was still a little foreign to people. We just assumed. Because hey, anybody at internet scale that's right, and the internet app they're going to figure out you have to denormalize and do all those things. That was foreign to a lot of people.

Robert Hansen

Especially newbie devs.

Matt Domo

They wanted a relational database experience on it. That was when RDS was born.

Robert Hansen

Eventually Elasticsearch and Redshift came along. Is that after your time, or were you involved?

Matt Domo

Redshift. Figured out that we needed Redshift in the design of business model, my time. But it was built after I left.

Robert Hansen

I've used a lot of those products. Extremely high performance. We always debated, should we do it ourselves? I'm sure a lot of companies do this. It's like, we could do this ourselves. We lose the support but we also gain a lot of work. We think we can do it better. But then we gained a lot of work so we never did. We never ended up building. We never built anything ourselves.

It was all entirely the cloud based version that you use to make Redshift and Elasticsearch. It works. It absolutely runs a business for sure. One of my favorite things about Elasticsearch is that it runs and can actually search JSON. Similar to what you built all those years ago with XML Search, allows you to search within JSON. Which is really, very useful. Extremely powerful.

Matt Domo

Really powerful. Because JSON is now the lingua franca of the internet.

Robert Hansen

It really is. It certainly turned that way. Much more than XML anyway. I want to change the topic just a little bit. Since now you are at AWS in this point in your career. I think this is an interesting thing. I found this. It was an internal memo that was from a Google employee who was previously an Amazon employee. I don't know if you've read this thing but it is a work of art.

Matt Domo

I may have. Every cover. It may pop.

Robert Hansen

I'm not going to read the whole thing. I'm just going to read two sections of it because I think they're both hilarious. First of all, he's talking about how Google could be better. It could be like Amazon. It's not like Amazon today. But it could be that way. I'm just going to read these two quotes. I'm not even sure they're in order.

But just so you get the drift. ‘He didn't -referring to Bezos- and doesn't care even a tiny bit about the well-being of the teams nor about what technologies they use. Nor, in fact, any detail whatsoever about how they go about their business unless they happen to be screwing up. But Bezos realized long before the vast majority of Amazonians that Amazon needs to be a platform.’ First of all, what's your reaction to that?

Matt Domo

100% true.

Robert Hansen

I like the idea of everything being a platform. Because that means that you can pick it up and move it around. You can scale it. It’s foreign enough to other applications that they don't have to worry about it. It is its own thing and needs to be dealt with by itself.

Matt Domo

But the number one thing about a platform is they're very sticky. Once you get it right and you're on the platform, switching costs are high. When we were building AWS, one of the biggest things we were concerned about was switching costs. If they came on to us and hit at us, how hard would it be to move?

How simple we could make it to get on to AWS. But how hard was it going to be for somebody to move off of AWS? When you're a platform vendor, that's the ballgame. You want to get them there and keep them there.

Robert Hansen

Yeah. For instance, it's really easy to move data between instances or between S3. It's basically free. But you move it outside of AWS, it's extremely expensive. Extremely expensive. Moving data in, inexpensive. Moving data out, very expensive.

Matt Domo

I wonder why that is.

Robert Hansen

Yeah, exactly. It's funny because I'm dealing with a very large glacier array right now. It's dirt cheap. You want to put a lot of data in our data center? Great. It'll cost you nothing. Alright.

Here's the second quote. His BIG MANDATE went something along these lines. ‘Number one, all teams will henceforth expose their data and functionality through service interfaces. Two, the teams must communicate with each other through these interfaces. Three, there will be no other form of inter-process communications allowed. No direct linking. No direct reads or edit other teams’ data store. No shared memory model. No backdoors whatsoever. The only communication allowed is via service interface calls over the network.

Four, it doesn't matter what technology they use, HTTP, CORBA, Pub/Sub, custom protocols. Doesn't matter. Bezos doesn't care. Number five, all service interfaces without exception must be designed from the ground up to be externalizable. That is to say the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. Number six, anyone who doesn't do this will be fired. Number seven, thank you. Have a nice day. Ha-ha. You 150 odd Amazon folks here will of course realize immediately that number seven was a little joke I threw in because Bezos most definitely does not give a shit about your day.’

How does that strike you?

Matt Domo

I remember this whole thing. It's accurate. Everything was built as a service in that day. If you go back to Amazon retail, pre-AWS, there was the order service. There was the cart service. There was the catalog service. So on and so forth through amazon.com.

Robert Hansen

How culturally did that play out? Did people like that about it internally? Did they enjoy that technology stack?

Matt Domo

As a developer, there were pros and cons. They liked the decentralized ownership model. I own my service. I scale my service. I design my service. The APIs became a dependency because there was no arbiter of who controls what version of an API that is out there.

If the order management service changed the orders API, cart now had to do that. It could slow cart down. One piece that was missing is, I'm talking about, you had to have rules, a schema and the negotiation around it about how you manage versioning. But the whole mandate of having it be an interface is 100% true.

Robert Hansen

Got you. How about the part about Bezos not caring? Is that a real thing?

Matt Domo

Jeff didn't get into technology. He would hire people around him that got into the technology. I would even say people on the S team, his directs, weren't too heavy in the technology. They would have people they trusted like any other senior person, that they would go off and do. Those people would inform them. They might have what's called a shadow. That shadow was their technology advisor. Jeff, you really need to pay attention to a, b, and c because this is important.

That's how he would glean insight into it. But he really didn't force mandates. As a CEO, he probably shouldn't. That's not his role. He had a CTO, Werner, who did and did not mandate stuff. But he did patrol around and influence things. But by and large, back to the decentralized teams management piece, what he did influences what I described. Let's decentralize, give people control, allow them to move really fast and try to build as little dependencies as possible between teams. Which I thought was really smart.

Robert Hansen

Yeah. It seems like it. Certainly Google thought it was too. Otherwise, they wouldn't have started building things like GCP and building a lot of the similar technologies. I mean, it is smart. It's absolutely the right way to do it. Microsoft agreed with Azure. I think that exploded the whole market in many ways.

Matt Domo

Yeah. It was a big bet.

Robert Hansen

The first mover advantage. You talked about this a little bit. Where do you think AWS really had that or secured it? Or is there a big enough moat? Do you think they are bound for the history books? Or do you think that they will come out ahead? Who's going to win this big cloud computing war do you think?

Matt Domo

Well, you asked a multipoint question. They absolutely wanted first mover advantage. We talked about it all the time. Got to be first. Got to be first. Got to be first. We're willing to make tradeoffs to subterfuge how ready, how big etc., to be ready. But we're absolutely at all costs going to be first with everything. The premise was, people always remember first.

I actually thought that was a stroke of genius. Because people do remember first. They got the tinkerers. It incentivize the developer. They're like, Amazon's big. They've got this brand. They're making this available. They've run it on Amazon. I'll play with it. It's pretty simple. They keep coming out with things on a regular basis. There's more additive things.

That changed the paradigm of computing in my opinion. It is not only being first but the regular pace of innovation. Continuing on a good cadence. Releasing new things.

Robert Hansen

If they've built the three legs of the stool, what's next? What do you think they need to build that's not part of that stool that needs to be on there?

Matt Domo

We keep getting more value added services. It's more than just compute networking. Back to your thing to answer that. Have they built first mover advantage? Absolutely. Has that propelled them to be the market leader? Absolutely. Is it an insurmountable moat? I don't know. It really depends. Over Google, IBM, and Oracle? Absolutely. They are just way too far behind. We can debate. Their approach is way alien to the developer. It wasn't simple. Wasn't easy. All of that. That outstanding, there's decent enough money there for them to continue.

The next one is going to be Microsoft. We talked about that earlier in the presentation. How can they tie all that together? Will that create a formidable thing there? That's going to play itself out in the next few years. What's next in cloud? It's all around the value added services to build the next generation of applications. We're now getting services to do OCR for AI.

Robert Hansen

That's exactly the example I was going to come up with.

Matt Domo

Yeah. 80% of internet traffic right now was video. All the kids on Tik Tok or whatever else it is. But you need AI and that optical recognition to start doing innovation around that. Well now we're starting to get services around that. AR and VR. We can debate the metaverse. I'm not thinking it's going to be a big deal. But that's just one man's opinion. But I think AR and VR will be. I think that's the bridge to whatever the metaverse ultimately becomes.

Well, there's now services to do that. Cloud augmented reality pushing the processing here and between the edge to do that. I think the distributed nature of computing with edge and what you push into the data center, is going to blossom a whole new set of applications that come out.

Robert Hansen

That certainly makes a lot of sense. That burst compute. You don't really need this incredibly heavy process to run all the time. You need them to run in batches. Just run it whenever you have time to run it. I want the cheapest option. Oh, it turns out it runs at four o'clock in the afternoon. Fine. Just run it. I just need it done. That's perfect. Because I think that one of the things that's underutilized about EC2 is the ability to run things in the cheapest way you possibly can.

There's entire services around this. I just want it to run at some point. Well, what if we need to pull it back? Well, then the batch doesn't complete. You lost this amount of money. You just run it again when it has time again. Fine. As long as your code is resilient enough, you can definitely make a lot of money, save a lot of money in the process of doing that.

Matt Domo

I think things like edge. Number one to pay attention to is quantum. Quantum computing has a lot of potential uses. We can all look about cryptographic keys. That's the one obvious use case everybody can wrap their head around. An uncrackable key or the converse. I've got a quantum guy sitting there trying to crack every key. That's one. The paradigm issues from metallurgy to electronic design to medicine, modeling, all of that. We're way early days on it.

But the applicable things that you can do with quantum, once they make the programming model not have to be of an astrophysicist to program, quantum is a big one to come out. AWS is starting to come out with quantum services. It’s the value added services and the continued value added services that are beyond the three legged stool. The pace at which those come out is going to be there.

Robert Hansen

I totally agree. Back to the AI thing. I really think it's not just OCR. There's so many applications for AI. It's impossible to enumerate even just a small fraction of them. But anything from, I just need to know some signal from this crazy amount of noise. I'm just going to pump in an enormous amount of data. I have this algorithm. Runs across. It's like, oh, great, there's some things I should possibly look at.

There's no way that makes sense to run on a desktop environment or in a personal compute environment. You're talking about something you're only going to have to run once or maybe once a year. It's a very rare event. Why bother spinning up all this physical infrastructure? You have the cloud right there. It's already well suited for this task. Serverless compute. Yeah.

The idea that I can run these micro services that only run for a split second. I need to run this thing. I just need the data back. I don't care about the operating system. I don't care about the patch levels. I don't care about the networking stack. I don't want it to boot. I just want to run something and come back with the data. That's hugely valuable.

Matt Domo

The AI piece. There's something. Generative AI. What happens if you don't have to write much code anymore? What happens if AI was watching what you're doing? We have Grammarly today. I use Grammarly because I'm a horrible typer. Well, developers in general are not the greatest typers.

What happens if generative AI came out and it was pretty good at writing code? That revolutionizes things quite a bit, don't you think, about speed and agility in the development process. We are ways away because you have to get the things there. But in the next three to five years, could it become really realistic? Absolutely. Talk about changing the game a little bit. That one's there.

Robert Hansen

We've seen things go from mainframes. Centralized, big, shared compute to local desktop environments out to services, call it SAAS software, back to, now it's running on mobile. Then it's all out in the cloud. Now we're back to, maybe we should be able to just have more compute on your personal compute device and actually run AI right here. Why are you sending this off into the cloud? Because a lot of people don't want the sensitive information to leave their device.

Where do you think we land? Do you think it's always going to be hybrid? Do you think we're going to land on one model? Where it's the mobile phone or desktop environments. A thin client compared to this very complex infrastructure. Or do you think it's going to be a mix? Where do you think it's going to land?

Matt Domo

That's a multi-part two. I think computing is becoming distributed. Let's work back from the device. You're going to have rich devices, or phones. Or a sensor that's working in an oil field that's tracking viscosity of oil through a pipe. Or that's tracking the traversing of a vehicle down a river to medical equipment that's tracking our health. To cameras. To connected vehicles. To kids watching pixelated video that's no longer pixelated. You're going to have a spectrum of devices.

Why hasn't that materialized right now? Because of connectivity. What's the statistics? Every three years the raw number of connected devices doubles. We slowed down because of the pandemic plan. But why? Because of the chip shortage. That's going to accelerate. Now we also have 5G. The premise of 5G, before, I think what the statistics was about one million or a couple million. I'll have to go back to it. But you could have the potential for 3 billion devices in the New York metropolitan area really soon.

You couldn't do that on 4G. Now with 5G, you can. It has a broader spectrum as it goes out into rural areas. What does that open up? It opens up a lot of use cases on devices. Great. Some devices are going to be rich. Some devices are going to be not rich. They're all going to need connectivity.

Now, you're going to need something in the middle like a mesh network. That's where edge computing comes in. Because we're not going to be able to send it because they're going to want to respond to the experience. We're not going to be able to send it all as fast as they can build data centers.

You can't send it all back to a centralized cloud data center to process everything just because of the bandwidth problem. There's only so much bandwidth they can produce. That necessitates putting some frontline processing. You talked about AI. Do some initial OCR on a video that you're seeing then send that heavy data for processing back to a centralized cloud. That centralized cloud is in a region that satisfies the privacy laws that you have.

I think that's the model that computing is going to be. You're going to have a wide spectrum of different device types. You're going to have mesh in different metropolitan areas of different devices. That's materializing now. Then you're going to have cloud or physical data centers in those regions to satisfy the privacy laws.

Robert Hansen

One thing I have not seen yet any of these device manufacturers go down the path of it. Saying you can run your own. We’ll create Apple's infrastructure in a box. You can just throw it in your network. It'll be yours. Fully encrypted. You're the only one with the keys to it. Drop it in the mud puddle, it's dead. That kind of thing.

It seems like there could be a market for, yes, it's in the cloud. But it's your cloud. It's your personal cloud. Because I think a lot of people are starting to get a little nervous about how much data is being mined by these very large companies. It'd be nice to have an alternative. Some alternative.

Matt Domo

Well, the IEEE just released a paper in the last two days about making identity and your privacy of your data decentralized so that the end user controls through a paradigm. They control their identity and their personal information is shared with a processor. That's the mode that is going to happen. You talked about the Microsoft Live paradigm, two factor auth. There's really a small number of identity providers out there.

They all want it because they know all the information about you. They can sell insidious ads. That's really all it's for. What happens if you could turn the paradigm over? You as the end user could choose. I control my identity. I can control what I share with you. It's my choice to do that.

The IEEE has a whole working group on that. Just released their first paper on doing that. Now there's going to be some wars on that with the obvious players because they're not as incentivized to see something like that work. But that is the direction that it's going to go.

Robert Hansen

This is actually where some local storage and local AI is actually not a terrible idea. Because you can actually push just a bulk amount of ads down. The person's phone can say, hey, this is the kind of person they are. I'll show them these ads. But the people who sent me the ads have no idea if I'm ever going to see them or not. It's never known to anybody really other than me because I'm the one who saw it. If I decide to click on the ad and go to somewhere, obviously, the advertiser who sent me the ad gets to know about it. But that's it. Or maybe a centralized authority gets to know that an ad was clicked, so to charge so and so a right amount of money. But that it would literally be the only time.

I think there is some interesting hybrid models out there. I've heard of this idea before. Another friend of mine talked about it many, many, many years ago. He's like, I would like to go to a website and say, I'm okay giving up my email address but I'm not okay giving my address. Or I’ll be okay telling you these parts of my demographics. Or I'm okay telling you these interests, but not my demographics. But if you want it, I'll sell it to you. I'll sell it to you for some amount of money. Like $1.

If you really, really want it, you can pay me $1 through some transactional interface. I don't even notice it's really happening. It is an interesting model. I think a lot of people would like that. It's like, well, I want none of this. But if I'm a wealthy person. I really don't want my privacy invaded. Maybe it's 20 bucks, 100 bucks, 1000 bucks. Who cares? It costs exactly what they're willing to pay for it. The market ends up normalizing itself. Interesting idea. Eventually you went on to be the CTO of Parallels. Tell me all about that.

Matt Domo

Yeah. Parallels at the time, we were famous for running Windows on the Mac. Mac virtualization.

Robert Hansen

Was this a stealth project from Microsoft? You're like, go there and help build it.

Matt Domo

No. That's what they were famous for. But the stealth project, that was the cash cow. That's where we made margins. But the bigger one was server-side containers. We built the first commercial version of containers technology, called Virtuoso. It was built for the hosting market. It was all about trying to get density for hosters. I was trying to push us to make it a little more simple because I knew the problem coming from Amazon.

That's where the ballgame was won on EC2. It is all about density and doing it cost effectively. We could go on and on and on about that. How many virtual instances you packed on the box. They could run higher because VMs, you virtualize the hardware. In a container, you virtualize the operating system. You clone that. It was figuring out how to make that more reliable, etc. Try to make it simpler. That's why I went there. I’m like, I saw this as an angle to really go.

Robert Hansen

Out of curiosity. This has got to be before OpenStack came around?

Matt Domo

It was before OpenStack came around.

Robert Hansen

How did they see OpenStack?

Matt Domo

OpenStack was driven by Rackspace. It was one of their customers because of the hosting part.

Robert Hansen

Was that an existential threat or how did you guys perceive that?

Matt Domo

When OpenStack first came out, it was something we wanted to monitor but viewed it as more of just something to monitor. Not something to take super serious. The challenge with Parallels was the founders were tired. They wanted to sell. We got into the paradigm of, well, virtualization has been commoditized.

There's all the different open sources. They're winning. VMware and Microsoft Hyper-V. Then you have all the three different flavors of open source. It's commoditized. We're not going to win even though we had a different way to go about it. We ultimately sold.

Robert Hansen

I remember working at Savvis after they bought Cable & Wireless America which bought Exodus which bought Digital Island. That's how I got involved in that. It is a long chain of companies. They were very forward thinking. They eventually got bought by CenturyLink. CenturyLink’s entire back-end is effectively the same stuff I built years and years and years ago. I know this is going to be extremely obvious to you but maybe not everyone listening.

The entire concept was, we have a whole bunch of machines. But we don't necessarily use all of these machines all at the same time. Or we might burst up or down. I used to work at eBay for instance. Oftentimes, the systems are completely overloaded. Everyone's crashing the system trying to get stuff at auction. But those are certain hours of the day. Then there's a whole other chunk of the day when there's a lot less going on.

Why not run the batch job? Well, why do we need a whole separate set of machines for that? Why not run it on the same machines? Just spin down the machines that are doing the front-end stuff and then spin up the ones that do the back-end batch jobs. You can greatly reduce the amount of cost and load. From your perspective, how did things like Docker come around? How did this evolve and change the industry? How did you see people talking about it before and after?

Matt Domo

At first, it was the density problem you're talking about. A server is like a jar. How many rocks, pebbles and sand can I get in my jar? That's what the discussion was about. Because CIOs and CTOs are all like, save me money, save me money, save me money. How did Docker come about? Well, once they started getting all those on. I'm a heavily packed jar. Now I've got to move stuff because the jar breaks.

Now I've got to have some level of movement because breaks have to have some fault tolerance. I need orchestration, right. I need something to move me from one jar to another. Then the other thing they started to look at where Docker came out is about the application. When you start making orchestration, you had to do some unnatural things to the application to get it to work.

Where Docker came in was, I'm going to make it really simple to take just an image of your app. With some configuration, I've got this orchestration thing, then I'm going to be able to clone your app and run it in multiple places. Make it really simple to go that route. Even multiple data centers. When they came out with that way, the foundations had to be built in the order we came up with. The pain had to be felt by the adopters of once they did it.

But again, where these platforms really take off is when you attract the developer. We talked about the phone. We've talked about the cloud. Now we're talking about this. When you won the minds of developers, people want productivity out of the developer. They want simple, easy, frictionless infrastructure because it's cost. Once you tied all those things together, magic starts happening. That's where Docker came in, turned it upside down and brought it all together. That's how it won.

Robert Hansen

Docker plus Kubernetes. I think those two, those the peas and carrots, they go. The orchestration you're talking about. We've moved into an interesting place now where we have not just virtualization in the server sense. We have virtualization literally even in the browser. Virtualization is everywhere now. This thing that we thought was just going to be about putting marbles in jars is now about isolation.

It's about protecting data from moving one place to another. If something crashes, it only takes itself out. It doesn't take out the entire thing. This has grown a lot since the early days. One thing I thought was interesting is, a friend of mine was building an app. Every single component was built in a Google environment using effectively workers and microservices. I was asking him, well explain your database schema. Explain this. Explain that. What kind of operating systems. He is like, “I don't have any of those things. I don't know.”

It was foreign to him. Like, I don't really think about things like that. I don't have any of those things. He literally does not have user roles. There's no place to log in with regard to his server environment. All of that's changing. I know that there's ways to build workers inside. Like Cloudflare for instance, where you effectively can run an entire extremely complex, dynamic application as long as you're okay not having a back-end database. Entirely on workers.

Where do you think that was all going? Do you have any opinions on the future of the front-end version of cloud based processing and virtualization?

Matt Domo

Well, you mean from the browser on down?

Robert Hansen

Everywhere. All over it. Do you think I'm going to have it in my phone where I can just swipe, be in Android and swipe, begin? How far is this going to go?

Matt Domo

I don't think we'll get there. I think the paradigm that you described that was interesting is having an isolation. Some level of isolation. That really depends on how many different applications are going to run on the end device, how rich they are, and how to isolate them. That's an interesting paradigm. That's what's going to dictate it. I really haven't thought deeply enough about it to really give you a coherent answer.

Robert Hansen

Well, it's fair enough. I'm pretty interested to see how far these front-end proxy servers like Cloudflare, Akamai, Imperva, cloud based web application firewalls and CDNs can take this idea. Why do you really need to think about databases the way you've thought about them? Just think about them as JSON blobs. We'll take care of the rest.

Matt Domo

Well, do you mind if I interrupt? I actually think the role of the CDN is back to the distributed part and the edge computing. Now I've got one, the top of the pyramid. You always think of things as cakes. The icing is the number of devices that I have. We talked about that. Then the next layer down is that mesh of processing. Now, what's a content distribution network get great at? Distributing things with low latency repeatedly.

If I'm in Austin right now, you can't build a lot of data centers in Austin. Why? Because it's power. You're going to have a lot of micro data centers all over Austin. Hotels are going to have them. Every little Chick-fil-A around here's got a little cabinet running in it. How do you distribute the application and the APIs that are the bridge between the devices running in my Chick-fil-A or out my metropolitan area to how it gets coalesced, processed and sent back through the cloud data center?

I need a CDN-ish thing. Because they're really good at distributing that stuff. I distribute my APIs. I distribute the AI processing. I distribute all the ML processing. The lightweight ML processing. Not the stuff that does the heavy stuff. CDNs like that are really good at distributing that out to a different paradigm. The mesh that will become the edge endpoints could be distributed and processed through a CDN. I think that's a cogent model that could emerge.

Robert Hansen

I keep thinking that they're eating the world. Eventually, they're just going to say, why don't you just also run all your apps inside Cloudflare, or Akamai? Why even allow AWS to exist? Just start eating that market share piece by piece until eventually one of two things happens. Either Amazon has to buy them or they have to start competing in that world.

Start building their own CDNs. To some extent, I think they're already in that business. With S3, they're basically doing very similar things. Big JSON blobs living in the cloud. They already have the concept of load balancing. They're right there. They just haven't made it productized. All the tinker toys are there. They just haven't put them together.

Matt Domo

Yeah. If you can build the way to distribute back to the developer, make the developers’ life easier. Distribute the APIs, give them tools to manage it, give them tools to operate it, give it tools to administer it, and distribute it in a way. But I don't think the cloud is going to go away. Why? There are some heavy duty processing. It always exists. There's only so much you can do.

We have the privacy thing which is just going to continue to expand. We're going to have more and more privacy laws. You're going to distribute to provide that snappy experience, low latency, highly reliable. It's going to have more and more processing. To a point it's going to hit a wall then it's going to have to send us somewhere.

The CDN is going to be exactly that. The distribution of that area. It'll become more and more powerful. But I still think the role of the cloud data center or the hyper scalar big data center, for those that want to hold and control their own, is going to be around for a long time.

Robert Hansen

Yeah. I keep thinking that there's a role for data centers in a box. Where you just say, hey, I just need you to roll up with a big tractor trailer. Drop it off. It's got its own generator in there. It works. It does everything. Oh, I need five of them. Fine. Just click them all together.

Matt Domo

They’ve got it now. I went to a conference here in Austin called Data Cloud. I was invited to speak about edge computing because I write articles about edge. Because I actually think that's an untapped thing that's going to change. There are things that that unlocks we haven't even thought about yet. What I took away that I didn't realize is, many of the largest data center providers are doing precisely what you just described.

I want a couple racks here. I want a couple racks there. I want five racks. I want this. I want it in this thing. I want to put it on the roof of my hotel. Or I want to put it in the back office of the Chick-fil-A. Or I want to put it in my oil pump and everywhere in between. But I want it. The reason why it's very appealing to them is it's lightweight, it doesn't consume a lot of power, the heating and cooling is a low. I can expand more readily across, like our Austin metropolitan area, rather than sending everything to a data center. Which we've already described. There's limited real estate, limited power, etc. They're all working on it. It's really interesting.

Robert Hansen

One of the things I've been thinking about for a while is, let's say you're driving a Humvee or something out in Afghanistan. Let's say something starts going weird in your vehicle. The car is starting to shake. It’s seriously something wrong with this vehicle. We can all feel it is happening. What is your option? Well, your option is send this enormous data burst of every single thing that could possibly be wrong with this vehicle over a satellite link, which just sounds like an enormous waste of bandwidth power processing all this stuff, hope that it gets there.

Then hope that someone figures out what's wrong with it then sends back some telemetry back down to, maybe by that point, I'm back to my base. There's just so much wrong with that. It just seems like it's untenable. Not to mention the fact that the alternative is even worse. Okay, well, we're just going to store all this data local to the device local to this Humvee. But meanwhile, it gets attacked. Now you have an exact map of exactly where this Humvee has been over whatever life span that thing got its data packet.

It seems like there's a role for this. This middle ground. If you think about it, just like a cell phone or anything else. There's this middle ground where it's like, you don't need the entire data packet. You really just need the one or two sensors that are obviously saying through a local AI on the inside of the Humvee, hey, we're acting weird. You send that tiny data packet over the cellular connection or satellite.

The big mama bird says, “Hey, I've looked at all Humvees everywhere throughout all history. I can tell it's probably one of three problems. I'd like these three sensors to report back over this time slice.” It sends it back to Humvee which takes a while. The Humvee says, “Okay, here's the three.” It's like uh-huh, your transaxle is going out. Or you got a low flat left tire. You’ve got to go fix that. It's a much faster diagnosis.

Meanwhile, all the data aboard just gets encrypted. So if you go and blow it up with a mine or whatever, and extract the device, you just got a big encrypted blob, and it's not worth anything until you take it all the way back to the base.

I think there's room for like AI, plus compression, plus using the cloud for these very large datasets when it makes sense, shipping large drives of data, like the Amazon Glacier model. Where you're going to do these large scale batch job analysis, training analysis over long periods of time across every single thing that's ever gone wrong with every device.

So you get the best of all of the worlds, where you get the local immediate reaction that you need, plus this long tail of the enormous amount of data that would not make any sense to transmit over the public internet like that. So what do you think?

Matt Domo

I think that's a class of device that's out there, if they can have distributed model, I actually think that is a class of device. The industrial Internet is the same way, if I have these big machines in an industrial factory, they're not going to want send every diagnostic of every sensor on a big machine.

But they'll want to send a subset and I'm going to have some processing there. Furthermore, the latency of sending it, even if we had a distributed close, there is like you described, the latency of I need to react to this so quickly that I can't send everything back.

Even if it's something in the same city, I can't afford that latency to do it. I need some level of processing right where I'm at to do it. I think there's a class, you call that the Humvee example. I think there's others for sure.

Robert Hansen

Yes, even consumer class versions of these same things. Like a Tesla, let's say, there's really no reason that I should be phoning home all the time, about every single piece of data that you could possibly ever collect on a Tesla, that's just wildly way too much data.

Matt Domo

Because they want you to pay your subscription if you don't pay, they’ll lock you out of your car.

Robert Hansen

But fundamentally, think about what your car is doing all day long, especially a self-driving car, it's basically got LIDAR or camera around all of the corners of your car all day long. And it's terabytes of data, you're not going to be transmitting that over cellular, you're not.

But to be honest, it's not that crazy to be transmitting terabytes of data, or subsections of that terabytes of data when it comes home, and it's just sitting there in the garage, that's not insane. You could theoretically do that, it would be annoying, maybe here and there.

You can pick and choose and make it a little bit smarter. But I would want my Tesla or whatever device I was driving in to say, “Hey, I'm seeing something wrong with this car, it's acting, I push the throttle and it's not going or I push the throttle and I take it off and it's still going or whatever.

There’s something happening with his vehicle that's outside the normal parameters.” So after all this you temporarily retired.

Matt Domo

I went to Rackspace and then another company.

Robert Hansen

You went to Rackspace. I didn't realize that. So you actually were familiar with OpenStack up close and personal.

Matt Domo

Yes.

Robert Hansen

What happened? Rackspace kind of imploded. Was that on your watch? Or was that much later?

Matt Domo

It was much later. What happened? Late to the game a little bit, have mindshare, figuring out, is it a distro like another Linux? Or is it a thing all of its own? How do you win on this? If you're going to build a bunch of open source clouds, how are you going to win the developer?

You got to make it frictionless. And that's the portability. Remember, you described, make it easy that the AWS Roach Motel, it's easy to get in, but you can't get out. Now we're building all these, sorry for that analogy, I'm sure amazon is going to be mad at me.

Now I'm building out all these OpenStack sites. It was changing so fast. The design of OpenStack was just moving so fast. Like when we're engineering that and launched the public cloud, we had determined that if we fell behind more than two weeks of the main line, we would never catch up.

We're deploying on 10s of 1000s of servers. So we had to build this whole automation framework to ensure we never fell behind in our entire pipeline from the time we downloaded it, till we changed it, till we got back in.

Robert Hansen

You had a point there where you actually had to shut down people's computers to update them right?

Matt Domo

Yes.

Robert Hansen

Yes. How did that go?

Matt Domo

I was the bad guy and said zero downtime deployments. Had to do that. But back to the paradigm of what you asked, everybody's building an OpenStack thing, but how do they differentiate, but then how do you out Amazon out and out Azure?

You build a bunch of and expand quickly and have the similar API, and similar experience across all these different things, plus some special sauce that makes HP or Dell or all these other things valuable on it. Never could quite figure that out. I think that is what hurt OpenStack.

Robert Hansen

This is like the fourth or fifth time I've heard this story, just a different technology, but the same story. Like “We used Open Source, we build this thing. But eventually they update that thing and now we have to update that thing.

So we're always beholden to whoever is actually maintaining the open source thing. And you're never really getting ahead of it because you're always struggling just to do what you're currently doing and update. So you can never really get ahead and eventually you fall behind.

Or if you don't fall behind, you're just struggling constantly to keep up. So you can never really innovate on top of the Open Source project that you're talking about.” I think a lot of people get very upset by this implication, but would you recommend running a company on Open Source based on that knowledge?

Matt Domo

Yes, I would recommend and have run multiple companies on Open Source. Here's a couple qualifiers for you. How much is the project changing for the paradigm you just described? Like how much maintenance am I going to take on by taking on that technology?

Robert Hansen

Let's say something like Chrome or an operating system like Ubuntu or something like OpenStack, like very large projects, would you recommend that?

Matt Domo

Again, it depends on how fast, how frequently it's changing. Ubuntu fine, OpenStack, like we talked about, that's a very large and complex thing that changed really quickly and really frequently. Ubuntu doesn't change that quickly and frequently.

Why? Because it's later in its lifecycle. OpenStack wasn't. So how frequently is it going to change? The second thing is, do you have an idea how long it's going to be around? You get a lot of developers excited about Open Source projects because it's new, it's cool, and I want to be involved.

Then you figure out at some point, they die down. That usually happens in the 12 to 18 month timeframe. After 12 to 18 months, you're going to figure out whether that open source project has legs or not. Do you want to take that on? You brought up Kubernetes.

I was on a company and we started on Kubernetes point seven and there was the Docker Swarm, and then there was Apaches orchestration thing, and they were all going neck and neck. And everybody thought, Google will win because it's Google.

Well Google's not the most famous success for Open Source either. But in the early days, they were iterating quickly, and there was a lot of churn. And it wasn't so clear it was going to be around, we made a bet. And we analyzed it, and we stuck with Kubernetes as our orchestration thing.

But it took a while to vet and we looked around the energy around it. Like I said, that's the whole ball of wax, how fast moving, is it? How many releases are happening in what succession? How many developers are on it? And what's its lifecycle and how long has it been around?

What's the energy? Look, a huge portion of the internet is built on Open Source.

Robert Hansen

Of course, your phone, if you're using Android, if you’re Android or Apple. Apple's based on FreeBSD. So let's phrase it in a different way. Where would you not build a company? If you looked at an Open Source project, and like, this is the hallmark of a company that should not be built on top of this product.

Basically, I want to hear the opposite version of what you just said. If you're like, this project meets these requirements, therefore it’s something that's got no energy behind it, or it's something that's bound to be replaced by something else.

Or there's just way too much movement in, it's going to change too quickly. We're never going to be able to keep up.

Matt Domo

Are you looking for qualifiers like that or specific product examples?

Robert Hansen

That would be great if you had them. But just generally if you look at Open Source project, you're like, that is a hallmark of a particular project that would be dangerous to build a company on top of

Matt Domo

Here’s another one, UI. So we're now going through, in Python, for example, there's oodles of Python apps, oodles. Do I go with a thin framework? Or do I go with a more thick framework? And people are going back and forth? Why?

The thick frameworks been around a lot longer, there's a lot more developers to hire, but it's got some words, doesn't perform as well, it doesn't scale as well, et cetera. So people are like, “Hey because of that, I need more control, so I’ve got to write more code, I need to thinner framework.”

There's how many five now? Which one is going to win? Back to those criteria. So the breadth of choice, another one is how do you hire for it? If you're going to go look, and you pick up a specialized one that's kind of nascent, you're making a bet, how readily are you able to hire someone to operate it, maintain it, et cetera.

Because that places a risk on your entire organization, that specialization. So that's another one to take into account, whether it's a DevOps engineer or developer or whatever, however you want to look at it, they're all equally important.

But if you put single points of failure like that in there, that's a key consideration because if they go do it, and they leave with all the myriad of reasons we have attrition. Now, what do you do?

Robert Hansen

I like that one. That actually dovetails very closely with something I've been working on with regards to M&A. So it turns out, you can kind of tell from the public Internet, if this company is going to be a good fit. Because you can say, they're running a bunch of whatever, Drupal stuff, we're looking around your organization, like who knows Drupal?

Literally nobody? Well, that's not going to work very well, or at minimum, we're going to have to really incentivize that Drupal team to stay aboard because we have nobody who can handle this internally, we're going to have to hire for it.

So we got to treat those guys in particular extremely well. I totally get and agree with what you're saying. What about licensing like, GNU?

Matt Domo

That's another one is, how important is the licensing to you or not? It becomes more important for bigger companies, I think, or ones where regulatory compliance are important et cetera because they're bigger targets. Smaller companies aren't as worried about that.

It's just the name of the game. But for bigger companies, like you talked about the different licensing agreements that does play a role.

Robert Hansen

Or if you want to get acquired by one of those big companies, then it becomes very important. Okay, so why did you end up leaving?

Matt Domo

One was the travel. I had hundreds of employees in five cities and I was raising young kids and one of them had health issues. My wife said, "We don't need you at home.” So that was the big driver. So went to a smaller company, and it worked out.

Robert Hansen

Then you eventually retired and moved to Mexico.

Matt Domo

No, I retired, and I built while I was still working, my wife and I built a vacation home in Mexico. But when I retired, my wife's like now is a perfect time to go get your permanent residency because there's benefits of being a property owner down there.

We're both eventually going to get our permanent residency. I went down there, at month one of retirement, I thought it was going to be the three weeks and I wound up to staying two months. So went down to Mexico? Oh, darn. But my wife was amazing about it.

She took care of our teenagers while I did that.

Robert Hansen

She didn't come down to visit you?

Matt Domo

She did come down. Is that after a while after they got out of school, I went in April, and they came down in June and the immigration people “Oh, it's coming in two days. It's coming in two days.” They ran out of blanks and all the cards had to be made in Mexico City.

They ran out of blanks. So they couldn't stamp new cards. And then they only had one machine in the whole country to make permanent residents cards. Who would have thought this or they were just yanking my chain.

I don't know whichever one it was.

Robert Hansen

You just forgot to grease the wheels.

Matt Domo

But it just perpetually moved along. But it was cool. I got to spend time at my beach house.

Robert Hansen

Yes, so what do you think of Mexico living there?

Matt Domo

I think it's fine. Look, do those things that you read about in the papers happen? Yes. Have I seen it in my area? No. I'm not in a super touristy area. I'm in a local beach near a big metropolitan area, but not in it. It's not highly trafficked.

I'm pretty low key, like I don't come down wearing Prada, my wife is not wearing Louis. We’re not driving big cars there, don't wear flashy jewelry. In general, don't draw attention to yourself. Have a good time. Just relax and you'll by and large be fine.

The Mexican people are generally family oriented. They want to help you, if you try to speak Spanish, they're going to help you. They generally like to have fun, so if you like joke around a little bit everything, they generally will engage with you.

They want to be treated with respect.

Robert Hansen

Oh, who doesn't?

Matt Domo

Yes, exactly. and I think people get a bad rap now. Are there bad elements there? Yes, there's bad elements here in Austin, Texas too. So that's my view.

Robert Hansen

Yes, for sure. And after a while, you'll get a sense of where they are and work how to avoid them. So have you encountered anything noteworthy or it's been all just vacation for you?

Matt Domo

Yes.

Robert Hansen

Good. All right. So then you decided to get back into the game. You're like, I'm bored sitting on the beach, which, by the way, is a very common affliction. A lot of people are like, “When I get enough money. I'm just going to go retire on the beach.”

Well, that lasts a while and then just like, “I’ve got to get back out there. I’m going to get bored. I just can't do this.” Was that your experience?

Matt Domo

Yes. It's a great question. So it was like month three of my residency, I could get used to it. I'm not marching to somebody else's drum. This is great. Month six, I'm just going to do a little bit of stuff to keep my head going because I need to head candy. It's how I'm wired. I like to do things.

I like to golf every once in a while, but I don't want to spend every day on the 19th hole, saucing it up, and a lot of people do. It's just not my thing. So a friend pulled me into a consulting arrangement and we helped a company. And he's like, “Hey, I could need your help.”

I helped create the strategic technology plan for this pretty large company that was born out of a collection of acquisitions and really enjoyed it and it was part time. I didn't have to work mega hours. And it was pretty flexible.

Robert Hansen

You didn’t have to travel.

Matt Domo

I didn't have to, I traveled a little bit, but not great. And keep in mind, this is right before the pandemic. I was like, hey, I think I could do this. So in January of 2020, I hung my shingle out.

Robert Hansen

The perfect timing.

Matt Domo

The perfect timing, actually worked out. I started marketing myself to do this consulting and it worked out. So FifthVantage was born.

Robert Hansen

FifthVantage. What's the name mean?

Matt Domo

Five star advantage. Company tagline is we offer expertise, our expertise is your advantage. So it's really about, we have the expertise, we build Amazon, we built big products at Microsoft, we built big products at Rackspace.

We’ve taken them from their ground up to a very large size. Along the way we've learned what to do, but more importantly, what no one talks about, we learned what not to do. And that's even more important. So it's about sharing that expertise.

Robert Hansen

So you mentioned to me over the phone, that you do a lot of like business transformations like taking people from X to Y, what does that look like?

Matt Domo

One, for example, we helped a cyber company, they used to sell a million dollars’ worth of their software and rack of hardware to every one of their customers. They knew they weren't going to be able to scale distribution wise to accomplish all their revenue goals with that much of an investment upfront.

So helped them redesign the entire product to SAS. So how to redesign it, how to architect for cloud native, what they write versus what's in AWS, how to reorient their processes, so all about digital engineering, how to iterate a new product as quickly as possible, all your releases et cetera.

How to structure the organization for that rapid delivery, how to merge from the old to the new. And they were successful with over recommendations after a year and went public. Helped other companies with their growth paradigm. I'm helping a company now the technical advisor for there in Africa.

Wonderful, great guys, super smart, super humble, love the mission. They're digitizing the supply chain in Africa. So most of commerce happens in the big cities in Africa and it's through the smaller retailers, the big suppliers, think Coke, Pepsi, they don't have a way to get into those stores.

So they have figured out a paradigm to get the smaller retailers from the point of sale into a micro distribution center locally to the big suppliers to float Coke and Pepsi for example into those. So been with them with everything, like how to scale the org, who you hire, what's your processes, how you design AWS versus your own. It's all that kind of stuff.

Robert Hansen

Is this the playbook you were talking about? Talk about that. How do you bring someone through that process?

Matt Dom

Yes, I have a playbook of kind of what works and what doesn't. We talked about your Open Source question. Like, how do you pick technologies? So one of them is, it's a data driven exercise that always is.

Typically where people do it as a developer, super passionate, and they like it, and they write code, and people weren't paying attention, and all of a sudden becomes production, critical, and it's in there.

But it's not about a data driven exercise, take the time upfront, like write out your objective criteria of your decision making, then get hands on a prototype and then make your decision. So that's one kind of thing is a decision making matrix is one kind of thing.

Another thing is the processes that you have to take to try to get your code out faster. I call it the code devalue metric. From the time a developer writes their line of code till the time it's out in production. What is that time?

Where are you spending all that time in that pipeline? Work backwards from there to the time it's out in production. And then how do you reduce through automation processes, tools, people, a methodical way to reduce that timeframe.

So that's really about best practices, and all the things I talked about. The other thing is writing the balance of what you'd leverage in AWS versus you write in your own code. Have a playbook, we talked about platforms. How important is that technology that you own in case it fails? Versus whether you write in your own code.

What's the cost? That's the evil thing no one ever talks about. It's a math problem. You have to have a spreadsheet that you write the math, the calculus to map out the costs. How much time and people does it cost you to write your own software to do this versus what it goes on AWS?

How does that change in one year, three years and five years? What's the calculus around that? And things of that nature. So play books around choosing technologies, operating technologies, structuring an org, the processes you make, and around making decisions. That's really the playbook have built up over the last 30 years.

Robert Hansen

Out of curiosity, how do you feel about technical debt? First of all, should you have it? Or should you try to get rid of it right away? Or should you have a plan for maybe a year, two years down the line, like you're going to have to refactor your code, get to version two, just plan to have to rewrite every single inch of this code?

Because that's not necessarily a bad thing, because, as we all know, the second time you write something, it's usually better. How do you telegraph how to deal with technical debt?

Matt Domo

First off, everybody has junk in the trunk, that's what I call it. There's no software project that doesn't have it. The question is, how did you get there? So you can solve that problem by planning your strategic investments. Where are you investing in and why? How often you invest in it?

What's the cadence that you invest in it? I'm writing a blog about one topic. One topic is A from over a certain period, let's call it a quarter, how much do I invest in technical debt versus features? You're always going to have technical debt where blows up on you if it’s always all about the features.

In general, things are going smoothly, you have some debt, you have features, but things are by and large, operating well, and you have clear line of sight for 12 months, you don't think you have to make big investments, should be 70% features, 30% debt.

You're taking on features, and you're also taking on some of the debt. So you're balancing it. If it's 90% features and 10% debt that rooster is always going to come to roost at some point. And the pendulum is going to have to swing, where it's going to be no features for a while or 10% features and I'm going to take on debt.

Robert Hansen

Just to be clear you what you mean is 70% new features and 30% dealing with technical debt.

Matt Domo

Dealing with technical debt.

Robert Hansen

Gotcha.

Matt Domo

It’s a paradigm, all debt, or no debt, all features, no features. And what I'm saying is by and large a healthy roadmap, all things being equal, we're operating well have clear line of sight on the abilities, have clear line of sight on scalability.

I do have a clear pipeline on features, 70% features, 30% of working on debt is about the right equation. Then you adjust based on the market needs of the business. And like you said, if you see something that's going to blow up in 12 months, you probably have to work backwards and start working on that now, and it may have taken on more debt now.

Because it's going to take me a while to work on it. That's how I look at it.

Robert Hansen

I really like that. I keep thinking I like the pendulum as well because I feel like every once in a while, your team just wants to buckle down and just work on all these backlog of stuff. They really enjoy, it’s like, I don't have to think about a single feature for a while, I can just go and just get busy get in there and actually start unwinding a lot of this bad code.

Because it's often better done in this very large bulk thing, it just feels better to get it all like sorted and dusted the whole place and vacuumed and everything looks perfect and pristine. Okay, now we can start building new features. But the business has no appetite for that.

I can never find a situation where everyone's just okay, taking a breather for a while.

Matt Domo

Yes, that I agree. That’s where it takes the leadership that developed, not just, when I say leadership, the thought leaders that are developers, as well as the leaders that are people managers need to band together and paint the story to the business.

This is what matters to the business metrics, this is what matters to the customer and this is why it's important to look at it and have the discussion. It's about a value discussion at that stage. But you always have to pay some level of debt.

Robert Hansen

I got asked this question last week, when I was giving a presentation, I'd love to hear your answer to it. So this guy asked, “We have all these things going wrong in the company,” all these vulnerabilities in their cases, but they're talking about. “How do I get the board to care?

How do I telegraph this message in a way that can trickle all the way through the executive team and actually make it to the board and the board goes yes, it's worth spending time to slow down and fix this stuff.” Which by the way, is just technical debt, if you think about just another version of it.

What is your answer? How do you Telegraph that message all the way through the food chain up to the board?

Matt Domo

I don't know if there's a one size fits all answer on that one. So let me predecease that. I think you have, and there's reasons for that. The more layers of people you go through, the more opinions you have. And the more opinions they have, and louder voices and everything, the more the story changes.

That's why we have coconut phones. I think the thing is, what's the damage to the brand? The most important thing people will care about in the board is what's the damage to the brand? What's the damage to customers, what's the damage to our success?

If you say, look, this security thing, if we find out let's say the security thing comes over and they take control over the data, the Humvee, they know where we are, and knowing where we are bad things can happen. And it's easy to do it and it can happen in X amount.

Help them understand this is the damage to the business. This is how easily or not it could happen and this is the ramifications of what it costs to fix it. Now you can facilitate a dialogue. It always is a dialogue. And they may come back and say show me more, proved it to me.

Like I said, the more knots on the coconut phone, the more things will change. But I think it's really starting to work from the value of that way. And that's what I see work. I read a statistic that by 2025 they're going to have a board member on many companies oriented just towards security.

Because just the volume of cyber predators doing things is becoming that important to the board. So we'll see if that happens.

Robert Hansen

I would be shocked if that happens. But I don't think it's a terrible idea, especially because security people tend to think about the adversarial model in general, which does not just include adversaries in the ascension sense. What could impact the value of this company negatively?

That's a myriad of things, especially if you have more of a business focus. My answer was similar to yours, actually. I'm glad to say we're on the same page. I basically said, I think what you need to do is value the assets, understand what the total expected value of loss is and times likelihood, potential likelihood.

Sum up that risk and say, here's the total amount of value that is currently at risk across the company. I'd like to improve the value of the company. And it's going to be hard for them to say, “I don't want more value in the company,” because ultimately, you're just taking on liability.

I got in a very interesting conversation online a couple days ago. Actually, I was kind of trolling it, I was involved, they tagged me in it. I was watching it, but it wasn't actually interacting. But the current thinking, and I like this thinking, is that companies aren't worth as much as they think they are.

They really aren't.

Matt Domo

Especially now.

Robert Hansen

Especially now. But there's this glut of technical debt, and IT hygiene issues and security issues. And you forgot to pay XYZ license, and just all this garbage, it's sort of hanging out there.

And you can insure against some of it, I wouldn't recommend triggering the insurance for a wide variety of reasons, but you can insure against some of it, you can mitigate some of it, but ultimately, your company just isn't worth as much as you think it is.

So the best you can really do is just hope the acquirer doesn't notice that. So if you can get a couple of spare cycles to clean some of that stuff up and get rid of that technical debt, or security issues, or whatever, I think that that improves the value.

I think that's how you communicate it to CFOs. And once CFOs care, I think that gets up to the board level.

Matt Domo

That's another angle as well. The CFO.

Robert Hansen

How do you feel about the delta between traditional employees and 1099s? If you said, you're running a company now, it's your company, do you think you should have mostly contractors, do you think?

If you're a small enough company, should you have all contractors, so you can burst up and down as needed? How do you feel about that?

Matt Domo

That's a great question. In my business, most are 1099. And it's mostly because my folks are mostly ex execs like myself, and we're like, sometimes want to do something, sometimes I don't. So building that flex in the pipe was more important to the business.

So that's one thing. Then the next element of it is, as you build the pipeline of your deal flow, and you figure it out, in my opinion, it's better to have some contractors, you have a core, my core, my company's me, and a couple other people. But the rest, you have a core figuring out the heavy lifting, and the some of the other parts of your deal flow, that's where you get the flex in the pipeline.

Then once you get like a repeatable locked in thing that's going and you can keep them busy. That's the key thing. repeatably busy and growing. That's when I think you bring on full timers.

Robert Hansen

Gotcha. There's a lot of laws that basically say, especially California, they have a lot of laws about this. If someone acts like an employee, and they aren't in control of their day, they are an employee. So it's kind of this weird, it's like, I would like to make you a contractor, I would like you to have the burst capacity and you decide you don't want to work for a couple of weeks or whatever.

You don't have to take vacation, you just don't work, whatever. And you know if I want to bring you back, I bring you back if I don't because I think that's annoying, I don't, That flexibility, I think works for a lot of people.

But I think it'll be interesting to see how the gig economy and in my world, especially web application security, there is a negative unemployment rate. Which is to say that every single person working in security has at least one job, if not more than one job.

Matt Domo

I was hearing that not just security, developers too.

Robert Hansen

Yes. I think the world has just got to be prepared for a lot more 1099s, a lot more people who have two or three things going on all the time. Things are more expensive in many ways. So that makes sense. People want more money just to afford the basics.

But also frankly, nine to five, a lot of these people have a lot of time on their hands and why not take on another job and I've got all of my evenings and weekends free, I'm a single person, why shouldn't I?

Matt Domo

You bring up an interesting point, I was just talking to somebody in the recruiting space today actually for lunch. They were telling me employers are starting to get, I won't say who they are, well known high growth, very visible people in Austin.

They're CEOs like, “I'm going to be pulling people back in office because my developers are stealing from me when they work remotely,” and they're like, why? And they're like, “Because they've got three laptops, I can only see mine, but they're coding for two other companies at the same time they're coding for me, but I'm the one paying them and giving them the benefits.”

So there is pro and con of the gig economy like you're describing. I see both sides of the coin. I think we're about to figure it out over the next 18 months, we'll see how it goes.

Robert Hansen

Yes, but that's a rough one. Because now you're having to monitor people in their homes, or get them to physically come in which they hate. Not everybody, some people like that. But a lot of developers just like, “Why, I can be doing this on the beach. Why do I need to physically come into office for to do this?”

Matt Domo

Well, depending on how the world's economy goes, will it matter anyway? We're starting to see the layoffs, the pull backs. What I see in the market is we have the growth companies that are continuing to grow and invest.

Then we have people that are just kind of scaling back and working incrementally, and nobody in between, I'm not seeing anybody in between. That's foretelling things are slowing down. So if you're slowing down, will the gig economist have that many choices anyways?

So will it just naturally flush itself out? I don't know, we're going to find out in the next 18 months.

Robert Hansen

Yes. Minimum I think the people doing that are going to have to be a lot more sensitive to telegraphing to their major places they're making money, wherever that is, their major job, let's say, I don't know how to phrase that exactly, their primary source of income.

I think this will be interesting to watch how this all plays out, I'm a little concerned that what it means is employers are going to feel like they have an even greater reason to overreach and get into people's laptops.

So for instance, some of the conversations I've been having, and I'll get somebody on here who focuses on these types of issues. They have a duty of care issue, which means they are physically required to pick up the phone and call the police when they figure it out.

Because they're monitoring somebody's laptop or whatever, they're beating their spouse or they're doing drugs or whatever. Now they're required to pick up the phone and call because they have evidence of a crime. So the more draconian you get, actually the more liability the company takes on to know these things.

Matt Domo

It's actually a valid point.

Robert Hansen

Yes. I'm worried that companies are not, first of all, they have no business in people's homes.

Matt Domo

I agree.

Robert Hansen

But secondly, you're just taking on a lot of risk. Granted, it's nice to catch the bad guy every once in a while for sure, everyone wants the bad guy off the street, but the costs associated with it are just enormous and who wants to work in that kind of environment. So your HR morale issue, just to a level that I think is just not worth it.

Matt Domo

That'd be a big discussion. That's a very deep and broad conversation for sure.

Robert Hansen

Kind of crazy, huh? So you're doing some side stuff as well. You have another job? You're writing for some…

Matt Domo

Yes, I write for Newsweek and Fast Company. I write about cloud, innovation, digital transformation and entrepreneur leadership. So that's been fun. I had never written before. I have learned, I've gotten good feedback. In fact, in the last few weeks, the editors have both said, your articles have high readership, we'd like more. So went through and built out a list for articles.

Robert Hansen

So you are going to get a team to write this for you.

Matt Domo

No. I have a theory on that. What I have found is you lose your voice when that happens. You wind up writing it yourself anyway. I'd rather just crunch it out and do it. So yes, I have that going.

Robert Hansen

So how do people find you? We've talked about a lot of different things. Do you have social media?

Matt Domo

Yes. Thank you for asking. I'm on LinkedIn, Matt Domo, you could search on me on LinkedIn. We have a company page on LinkedIn FifthVantage. We also have a website fifthvantage.com or info@fifthvantage.com.

Those are the main way. Lots of people reach out on LinkedIn. I talked to people all over the place. I'm actually pretty open to that. So feel free to drop me a line. I'm happy to talk and help and have a conversation like this.

Robert Hansen

That's great. I just think it's really funny that you have two jobs after all this. You are not alone, my friend.

Matt Domo

I guess I'm a true gigist.

Robert Hansen

That’s absolutely right. Matt, this has been great. I really appreciate it.

Matt Domo

Hey, thank you for the opportunity. I really enjoyed it. Thank you. We talked about things I haven't thought about in a while, but it was good.

Robert Hansen

Appreciate it, man.

Matt Domo

Thank you.