SUPER-REALISM, DEEPFAKES & METAVERSE IN AR VR

April 6, 2022

S01 - E09

RSnake discusses augmented reality and virtual reality with Alex and Tim Porter of Mod Tech Labs. The conversation includes technical issues with XR including user level issues such as superrealism. We also discussed some of more positive and encouraging use cases for AR/VR. Errata: RSnake was thinking the Unibomber (Ted Kaczynski) with regard to Project MKUltra when he asked Tim Porter about Timothy McVeigh.

GUEST(S):

Alex Porter, Tim Porter

VIDEO TRANSCRIPT

We're seeing a lot of interesting cool use cases come out of the woodworks, and we come from the XR space. So we actually ran our own augmented and virtual reality studio for five years prior to launching this company.

Robert Hansen

My understanding was that was a big problem in the industry, that it's hard to do what you're trying to do. In a typical world, if I want to get a 3D model, I have to build up a 3D model. And that is very time-intensive and slow and ugly and error-prone and a very specialized skill.

What you guys can do is take photos of an existing thing that exists in the real world like a rock, let's say, and just take a bunch of photos of it. Now you have a rock, and you don't have to do any extra stuff.

Tim Porter

100%. Yeah, that's the biggest thing when you're talking about any sort of digital medium. Content creation is extremely expensive. And that was one of the big things that we ran in as we were running Underminer Studios, which was the last company we were at.

We would work for major corporations, people like Intel, Microsoft, KPMG, and things like that. They would come to us and they'd be like, “Hey, can you guys do this really cool project?” As we continued doing bigger and cooler projects, not necessarily did the budgets get bigger.

We got more marketing potential at one point in time, almost about six months. We were the actual YouTube ad for Intel, us talking and everything like that. It's all top-up, and it was great.

Obviously, the biggest thing when you're talking about project is bigger but the budget isn't bigger that means that you're starting to cut in. So we started making more and more tools. And that goes to my background was building automated tools and things like that.

Robert Hansen

Can we talk about XR? Can you talk about the difference between AR and VR? Because not everyone who's listening is going to be up on all these terms, so I think it's useful. Just a brief primer. A lot of acronyms.

Tim Porter

Do you want me to do it or you want to do it, Alex?

Robert Hansen

Yeah, sure. Whatever. Yeah, you do it.

Alex Porter

Go for it.

Tim Porter

Sure. The scale of immersion is the way that you think of the difference between AR, VR, and everything in between. XR is this combination term of all of these new styles or ways that people can interact with 3D and digital content.

AR is typically, and you see this with your cell phone, like a sticker. It's like Pokémon Go over the world. So you end up getting the understanding, very little understanding of the real world space. Basically, you're being invited to the space. While on the flip side, you have VR.

VR is entirely immersive. You live in the space. You're entirely immersed into the space. Then you end up getting places in between the two of them, where you start questioning, what is reality? You get MR, which is merged or mixed reality depending on which side of the coin, if you're on a Microsoft or Intel side of the coin.

The environment knows everything about you, and you know everything about the environment. So you could have a character that runs behind a screen, in front of a screen. You could play with it, throw things at it. And it would react and interact as if it lived with you and as if you are part of that world at the same time.

Robert Hansen

One of the things I thought was very cool about your tech, and I had not seen up until that point was, this idea of 3D volumetric photogrammetry as a movie, not as a standalone object. Because what I think most people are thinking of when they think of building up a 3D model is, “I build up a model, and then I can spin around the model or whatever.”

What if the model isn't a model? What if it's an action? What if it's more like a train that's moving with all its intricacies of its engine or whatever? Or what if it's a child playing or something, where it's very dynamic? A lot of things happening.

There's ways to build up models. You put little balls on people's wrists as they run around or whatever, and you build up a stick frame and overlay your existing model. But this is much more accurate because it actually looks like them. It's got all of their pores, their hair, and all of that stuff.

I think the cool part is that you can make it an actual movie. You can actually rewind to the part where they jump like, “Okay, how do we model this exact part of this movie and make that thing the focal point or actually rotate the camera to a better angle and actually capture them in a different position than the camera wasn't naturally built to be in that position?”

Alex Porter

Volumetric video is really fascinating. It’s definitely its still early days. It also could be called video gravimetry. We look at it as video gravimetry. You're taking videos from all around of a performance. This is an on-demand style performance. So you're capturing an action, a movement, a performance, and then playing it back.

We are primarily that on-demand versus real time in this performance level. But what it really gets you is that human realism, which is really hard to achieve with any type of modeling or any type of artistic human form.

You get micro and macro expressions, the flushing and the lines of the human face that we are all very perceptive and understand clearly. You could even tell on someone who's had a lot of Botox or potentially some other facial work. Their face doesn't quite react normal. And we are very attuned to that with virtual beings as well.

Robert Hansen

I'll have you know, I have no Botox whatsoever. So this is all natural. From your perspective, I know this is a super cheeseball question. But how far do you think we are away from things like the holodeck? Because all of this is really going down to you maybe have to wear something in your head, which is pretty clunky.

How far do you think it is going to be until we have something maybe that we can't necessarily touch? I'm willing to give that one up. It looks realistic, from my perspective. In all the ways, it passes the uncanny valley. I can't tell that I'm in this virtual world, except for the fact that I know there's an accent.

Tim Porter

Yeah. If you're talking about being able to walk through a space like that, we're talking about two years. If you're able to stand outside of space and look inward, that's right now.

There are a couple of different places, and they're called caves, which are doing VR, except it's projected on the surface. And there's all of this holographic projection that can happen. These entire experiences are to the naked eye, and even to many professionals, indistinguishable from the real world.

Robert Hansen

A good friend of mine was involved in the Harry Potter performance or whatever. The exhibition, I guess, is the word I'm looking for. I know that they built in a lot of the projection mapping stuff into that based on my recommendation, actually, like, “You should do this because I think it's going to turn out really good.” Apparently, they did.

The idea is, you actually are mapping something that's truly 3D with something that's 2D on top of that thing. Do you think that's where everything's going? Or do you think that where we're heading you have to actually build up a 3D thing and project on top of it? Or do you think we're actually going to get to the point where something can detect where my eyes are and where I'm looking and decide that this is the thing that I should be seeing? Or where is that tech going?

Tim Porter

We have tech, at this point, that can detect where people are looking and things like that. The biggest thing is being able to project into a three-dimensional space and then deal with beaming technology so that when you look into that area, all of the different angles are played at the same time.

Some of the advancements that we're seeing in NeRFs, if anybody watched Nvidia with their conference that came out, they did what's called Instant NeRF, which is a massive advancement on this about two-year-old technology. It allows for the segmentation of any of the different planes of three-dimensional existence.

When you're talking about being able to reproject these objects into full three dimensions, you have the segmentations already all ready to go than being able to play them back to where as you're walking around in this projector is happening.

They've been quite successful, but they fail in all the ways that you would expect. They have to be connected to the internet, they have to be able to get data across, they have to be able to have GPS tracking, they have to be able to visualize a whole bunch of different types of data with very limited input capabilities.

I think most of the advancements to create realism are going to be less on the visual quality and things like that. And they're going to go more towards, how do we interact with the systems? Because when you start having bad interactions, you lose that whole immersion thing.

If you remember Nintendo, they always had really great and amazing games. People were massively immersed in them. Now, the visual quality never matched what PlayStation could do. People didn’t care. They really did enjoy and were further immersed, but it was a better interactive experience.

Robert Hansen

Have you guys seen there's some videos out there where they have holographic concerts? Hatsune Miku in Japan, which is actually just a vocal pack. It's a vocal pack that they are trying to sell. But as part of their marketing, I guess they decided to build this 3D model. Have you seen this? Do you know what I’m talking about?

Alex Porter

Virtual influencers.

Robert Hansen

Yeah. They've actually managed to create a fully three-dimensional looking, from the audience's perspective, virtual character who's singing songs that no person has ever sung. Ever. They just record whatever notes that they want and whatever lyrics they want. And suddenly, you have a performance that is sold out in Japan. This is not a little thing.

Have you seen anything like that? Do you think that that is the way things are going? Or are we're going to have more virtual performances, less real people doing things? I get in this conversation all the time. I really feel like people are not understanding that this is becoming like a freight train.

There's all these virtual performances that are right around the corner that's really going to degrade the real performances, both in terms of quality but also just cost. I think it's going to drive down cost. I think the insurance, just dealing with someone 20 years, they had a great career, everything's fine. And all of a sudden, they go up on stage and slap a comedian.

Everything is fine, and you can't predict that that's going to happen. So why not just take the actor or the musician or the performer in general out of the equation? And still make something that's palatable and interesting for the audience or maybe even more visually interesting, because they can do things that humans can't do, and de-risk the whole thing. From an entrepreneurial perspective, that seems way better.

Tim Porter

100%. You remember Dave Grohl? When he broke his leg, that cost millions of dollars. That's a massive cost. But as far as technology goes, you're 100% on there. Nvidia released a face animation based on voice, so it should start to talk. It starts doing the animation that's on that.

As well, they have their metahumans, which I challenge anybody from a distance to be able to tell the difference between the two of them. They've also allowed for people to scan people into it. It's a metahuman, so it's a pre-created creature that can talk and walk and do all this fun stuff.

Then now they're letting you be able to scan someone's head and then put that on it, and then it actually does all of this stuff.

Robert Hansen

I'm taking the human out of it completely fine, if I'm running that company. I use Kevin Spacey. Great face, everyone likes him, everyone knows him. And all of a sudden, it turns out he's doing some stuff he shouldn't be doing behind the scenes.

Not that I don't like Kevin Spacey for his acting skills. But I don't want that risk, if I'm an entrepreneur, if I'm a studio. Why not just go for the virtual route, especially if it can be visually close to a human?

Alex Porter

Well, I don't think we're there with a visually close to a human yet. I don't think we're close enough to completely cut humans out of this scenario yet. Obviously, there's deep fakes. But we even saw this during COVID. Actual physical production was not happening. And so what took place in that, primarily, was actual on-location productions.

People were filming from home, we all saw the quality of those. Questionable. And then we saw a ton of uptick in animation. There were actually things that were slated to be real-life, real-action that were actually transitioned into animation. Specifically, because they could not get people together to do the filming.

I think there's always going to be a little bit of everything. I'm a big fan of the idea that we're going to have something more along the lines of a combination of mediums. We're going to have stylized, we're going to have traditional animation. We're going to have real-world human digitized. There's going to be a massive swath of things that are in these digital spaces.

Robert Hansen

Alex Porter

Tim Porter

Oh yeah.

Robert Hansen

All right. Chris, I feel like you owe me a beer out there. All right. I've been asking a lot of people about this. So I'm glad I got some authoritative. Let's talk about the limits of specifically the glasses, in particular, because this seems like this is really where things are stagnating right now in terms of people's adoption of it.

Everyone's got these scripts, the Microsoft HoloLens or whatever version, the Galaxy. They're literally putting a phone on their face, which just blows me away.

Tim Porter

Several other people, too. Basically.

Robert Hansen

What do you think that the issues are? For instance, capturing and rendering I think is straight in your wheelhouse. Where are the issues there?

Tim Porter

I go first? When you're talking about being able to get data, easy enough. You have a chicken and egg situation when you're dealing with XR headsets. You have ones that are tethered, and you have ones that are not tethered. You have these all.

The problem that you would expect is, as you've gone away from being plugged into a computer, you start losing horsepower. And so you end up having to optimize things as things are going wrong.

Robert Hansen

In a way, work on optimizations.

Tim Porter

Oh, exactly. If you have a headset, it’s the same amount of power as the cellphone that's in your pocket. For it to go ahead and render the experience, first, it needs to go ahead and see where your head is in three-dimensional space. You're moving your head up, down, left, and right. You can walk forward, and you can walk backwards.

As well, it's also rendering the image twice. So you have a left eye and right eye and then rendering that to a single image and then place it to CPUs so that you can showcase the eye. And then the very last thing is having the assets that actually live in the scene.

This is wacky.

Tim Porter

I was working on a project, and this was for a new cancer drug that was going to come out. Don't ask why they were asking an XR company to go out and do that. But it was entertainment, training, combination thing. I'm like, “Sure, okay.”

The script was here out in space, and then this asteroid flies by. You're like, “Wow, that's so wonderful.” Then you're going to fly into Earth. Immediately, I'm looking at that. There's like 12,13 laws that you don't break in XR to make people not sick.

I was like, “Okay, we're going to make this nice and calm. Then we're just going to take them down to Earth and put them in.” Then I get on a call with the producer. And he was like, “It gave me an Earl Grey. You know what I want? I want a beer with a whiskey back. Make it happen.”

I was like, “But you're going to make somebody sick.” He was like, “No, I don't think you understand whiskey back.” I was like, “All right, dude.” And so I do everything I can. I take it. I literally do anamorphic lenses, lens flare. It looks like JJ Abrams just all over this whole thing.

The story we got back was, “Could you put it back to the way it was?” I was like, “No. This is not how this works. What happened?” He was like, “Our PM went home sick.” And I was like, “Could you describe what happened?” “She said she felt like she had food poisoning.” “That was simulation sickness. We talked about this.” Never want a whiskey back, people. Don't do that.

Robert Hansen

Yeah, okay. What are some of the strategies in removing that? I certainly get it. Even just trying to play normal 3D video games, first-person shooters, I already get that. One of the reasons I think that is the case is because they can't predict how far I am away from the screen. They always projected the same size, whatever that size is.

It is like looking through a window. You're not always looking through the window and seeing the world the way it is, as you move forward and backwards. The world is changing. But it's a window that never changes. And it makes me very ill to watch that, especially because I'm not moving with it, my inner ear isn't moving, etc. What are your thoughts about how some tricks are?

Alex Porter

Sure. Yeah. I actually helped write some of the standards that were around this. We’re part of CTA, which is Consumer Technology Association. I'm the chair for the XR group. We wrote many of the different standards, things like, never try to break the vestibular feedback loop.

It's just like you're saying your inner ear doesn't feed with the same kind of experience. So as somebody walks forward and walks back and looks left and right, their vision needs to match up. Many of the different laws and groups and things that come out of movies help.

If you're going to cut, you want to cut and fade to black. You bring people up. You don't want to do things like moving the chair underneath them unless the chair underneath them physically moves in real-life. Because as soon as you move the headset, then you break that experience.

There's also things like oversaturation of color. If you have way too bright colors or changes in color that are too much, the intensity can make a lot of people very sick.

The other thing, and this is something that I can help, is sound. If you have sound, especially ambisonic sound, you can make it to where it guides people to, “Okay, I want somebody to turn to the right. Let's start making a sound that was over on that side.” And then they can guide themselves towards going that direction.

The last thing you want is something jumping over your shoulder and you've already lost, like we talked about, most of your field of view because of these headsets. Then you're quickly moving, and that makes people sick. So a lot of nothing, really jolty movement, things like that.

Robert Hansen

One thing I find as these video games get better and better is when they have little bugs, it gets creepier and creepier. Once upon a time, a bug would happen. You're like, “Oh, my sprite moved halfway across.” or whatever.

Now, it’s like people half-embedded in tables. It's very creepy-looking and pretty realistic, on top of it. You're like, “That's going to give me a nightmare. They're trying to talk to me as they're embedded in the table. Whoa.”

Tim Porter

What’s going on? Alex, you want to talk about the Slack channel we have?

Alex Porter

The Slack channel?

Tim Porter

The Slack channel with broken things.

Alex Porter

Oh. Yes, we have a Slack channel of broken things of all the terrible, odd, horrific things that we've created in 3D that just did not work, for one reason or another or many reasons.

The other thing I was going to say is you can also retrain your brain. Using VR specifically is what I'm talking about here. I actually have a lot of motion sickness, I have my entire life. And I was really very concerned about being the tester for our projects.

One of the early projects that we did in our VR days is actually a system specifically geared toward fears and phobias. And so it was a great, aided, levelized exposure therapy style. You're exposed directly to the thing you're afraid of, and you do it in steps to overcome this fear or phobia.

Level one was very cartoony. Level two was a little bit more realistic, and then it went up all the way to level five. Level five was photo realism. That's when we started really getting into photogrammetry, volumetric video and started creating the foundations for what became Mod Tech Labs.

Going through and actually repetitively, frequently repeating this training of being in this space, having this fear of heights, having that sickness in the VR simulation, I was actually able to overcome that.

Robert Hansen

Either I've seen this before myself using your exact tech or someone build an exact version of this because I was at a conference and they said, “Do you have any phobias?” I’m like, “Okay. Well, let's play this game.” I gave him heights. I actually do have a bit of fear of heights. So I'm like, “Okay, let's try it.” I put the goggles on.

I was suddenly on the top of a building, but I was pretty far away from the edge. This is their lowest setting. I'm like, “Well, that's actually pretty far down. I actually get the sensation.” They are like, “We'll move you up to level three or whatever.”

Now I'm pretty close to the edge. Whoa, that's actually pretty intense already. They'll put you on the edge where your feet can dangle off the edge or whatever. And then they deliver you do a safe space afterwards like this meadow or whatever. So you feel like you're nice and relaxed. It's wide open spaces, no agoraphobia or any other phobias you might have.

I think that actually has a lot of promise for all kinds of phobias, like phobias of snakes, or spiders, or heights, or claustrophobia, all kinds of things.

Tim Porter

Almost three fourths of phobias are actually self-treatable as well. So you could send somebody through an experience like this fear of spiders, fear of heights, things like that. You wouldn't need a doctor to go ahead and actually deal with that.

Alex Porter

What is the number one phobia?

Robert Hansen

Speaking in front of people, which actually falls exactly in your wheelhouse. If you could create an audience of people I know, which would be even better, that I would expect to see in the audience and a bunch of randoms that you can throw in there, that makes it extremely hard for me to feel like that's not real.

Tim Porter

Half of it, they always tell you, “Envision people naked.” I'm like, “If it's all the people you know, that just makes it worse.”

Robert Hansen

That's a different setting.

Tim Porter

What are they doing?

Robert Hansen

Robert Hansen

That's exactly what I want. It seems like there should be infinite focal length cameras set up side by side, so you get the 3D effect or ideally that ball thing I was talking about. You look in every direction, and you can look at infinite depth.

As long as it's tracking your eye, it knows you're trying to look and knows what direction you are looking. So you would have that full experience. And you can have this Marvel movie with all this crazy stuff happening all around you without that weird sense of it being a video and not really being there.

Alex Porter

Alex Porter

We’ve both done work on this. Tim, specifically, with CTAs, Consumer Technology Association, the folks that do CES. They're also a lobbying body, on the other side. They create standards, education, and do regulatory, specifically on behalf of technologists.

We're in there with large corporations. We've been working with them for the last four years really making sure that we are helping to facilitate opportunity for education and good use cases and ethical boundaries and appropriate levels of technology and interference by corporations, etc.

Tim, specifically, led the charge on the rules around XR for limited mobility that was published last year. That's an open, free document through the CTA website. And it is from a developer's perspective. We're talking about all these pieces and parts. Half of it is the experience, is the hardware itself. And the other half is really the content.

You have to be thinking about both sides of that. How realistic are we going with both pieces? How realistic can we get with the visuals themselves based on the actual hardware and then the content? So myself, I'm leading right now. We're actually close to wrapping diversity and inclusion for XR. And so again, creating standards, best practices.

We've talked about things that, honestly, hadn't really even occurred to me. Marital status, for instance. If you're thinking about someone who has married versus not married versus pregnant versus with children, there's this whole other piece. People get discriminated against for that piece of the world and their life, etc..

If you're portraying someone in a specific way or they want to be portrayed in a different way, those things all have to be considered.

Robert Hansen

I think if anyone's listening, and they don't see that this is coming, I think one example that we can all look to and go, “Aha!” is Snapchat filters. Because that is creating a lot of body dysmorphia. People are saying, “Oh, my eyes are too this. My forehead is too blah.” or whatever.

It's leading people down a path of trying to figure out, “Well, my body really should look like this doll or whatever. I should be more muscular.” Whatever the thing is. And they're actually uploading it. So even if you don't use those Snapchat filters, you're starting to see, “Oh, that's what everyone looks like. That's what a nudie looks like. And that's what people should look like.”

If that's already happening, and these people are not wearing VR headsets, they have no zero super realism issues going on. They just think that that's what real life looks like. I think we're headed down a pretty dangerous path, if we're not careful.

Tim Porter

Robert Hansen

I think those apps are never ever going to work because the person who is one of those things is not going to sign up for it. But I think if you took that same principle and you put it inside of a video game or whatever where you're suddenly this other thing and you have to experience life in their shoes, it's going to have the same effect.

You're going to go, “Wait, wait, wait. Why are you treating me like this? I'm a nice person.” Because you're only seeing yourself from yourself. You look in a mirror, and then you look at your skin or whatever. Or people are treating you in a certain way and saying certain things about you like, “Oh, geez, that's what it's like.”

You'll have a very unique experience. So the dedicated apps, I'm not so pro on just because I just don't think they're going to work. But I think that concept has a lot of merit.

Tim Porter

It’s a lot of merit. I think, obviously, you could let that swing even further the other direction. What was it, Minority Report, where they locked them up and made them watch all the horrors and atrocities in their shoes for a 10,000-year situation? Yeah, government control could really make this fun.

Robert Hansen

Yeah. Well, we'll get more into that too in a bit. But then it also gets into more creepy areas as well. Child predation, for instance. You could put yourself in the body of a child who is being predated upon. And a murder victim, where you could actually experience what it's like to be murdered by this person who would normally be a murderer.

You can actually start trying to deprogram people by making them experience what it would be like to be their victim.

Tim Porter

Yeah. Well, and that feeds directly into what happens in your brain when you start facing a fear phobia, that same exact concept. Basically, as you're feeding in towards your vestibular feedback loop and your amygdala becomes more and more desensitized for everyone, that's your fight and flight response.

Then at a certain point, you end up getting oxytocin. If you think about as a human, if you run away from a big predator and then you made it, you're like, “Yeah!” you get that feeling. You're like, “Bruh, that rush.” First, it’s that dump of adrenaline. And then it's the oxytocin. You survived, guess what you can do. You can do bigger, better things.

People will not even just be like, “Oh, that was scary.” They're going to be like, “I want to be murdered. I want to be murdered by this horrible person.” Humans are fun.

Alex Porter

That went sideways.

Tim Porter

Did it really though? I felt led.

Robert Hansen

I did, but they carried out though.

Alex Porter

I am familiar with some apps that actually are doing just that in the sex trafficking arena, where they're specifically doing it as a first person. You are the victim being trafficked. But it's also about recognizing the signs and the opportunities that these predators have.

In part, it's training and education. In general, it would be targeted at airline staff, travel station folks, people that would come across these people in transit, hotel staff and hopefully, be able to put a stop to it ideally before they are out off the grid.

There's an element of training, there's an element of empathy. And then there's really, I think, just general education. I don't know all the signs and symbols of what to look for in that interaction. But there are ways to learn.

The learning from VR is actually a much deeper, more concentrated, longer-lasting type of learning because it's interactive, because it's immersive, because it's first-person, whether you're learning to crank a wrench on an engine or you're learning to recognize human movement symbols for sex trafficking.

There's a wide variety of things that you can learn that really imprint themselves more effectively on your psyche and your being.

Robert Hansen

Probably.

Robert Hansen

To me, AR is the killer of the two. If I'm going to pick one horse to ride on this race, AR seems to be the one that's just so universally, incredibly useful for almost innumerable reasons. What are some of the AR apps that you guys are seeing that are interesting? Anything that stands out?

Yeah. Actually we've seen a couple of use cases that have come through for geological study where they would end up using our app to go ahead and do the initial scan. And then, with augmented reality, the individual could actually look at what the object was.

So if you have a geologist, the geologist isn't going to go out in the field, but you could send out some young buck with a camera and take all the photos and then upload the servers and then that end user can end up seeing that object.

So it's there and there's lots of parts and pieces that are there. I think the bigger thing is the mindset. The people in the verticals that have the money are completely diverted from solutions that are like that. So the two bigs, of course, are going to be, as always, oil and gas and entertainment. And those are the ones that you want to see adoption. Oil and gas runs into obvious major issue that a lot of their people are aging out. And that's on the geology side, that's on the capture side. That's everything all the way up and down.

So how do we make sure that when people are drilling and doing things that they're doing in an ecological way and going out there with scanning tech and making sure that it's cheap enough and easy enough that they can't just say, "No, they were too difficult." And then, it goes out to some geologists and then as well some ecologists to make sure that they're not killing sea turtles. That sounds like a great thing to do.

Robert Hansen

Unless you like killing sea turtles. But yes.

Tim Porter

I kick them at least once a week.

Robert Hansen

So a friend of mine, Dan Kaminski, RIP, he just recently passed invented a technology where you can actually change the visual spectrum so you can actually see in color or see some version of color for color blind people. Which is just a phenomenal app. It's called DanKam, for Kaminski.

I think that type of app has a lot of promise as well for an innumerable amount of people. And once it was Google Glass. Google Glass was just not the right tech for all kinds of reasons. It was very bright, it was ugly, it was Google. It was very invasive because it was always taking video.

I mean, I actually told multiple people who had them, like, "You are not invited over to my house if you're wearing those things. Sorry. I don't know what you're going to be taking photos of." It's one thing if someone has a camera in their pocket, it's a whole other thing if they're waving it around at all times where it's recording. That's a very different experience.

Being on camera right now, does that feel different? I bet it does feel a little different than your average every day, which is sort of walking around town.

Tim Porter

We know you pretty well. So this is a pretty typical sit down and talk and usually have some maybe some gin or something like that with you, right?

Robert Hansen

Yeah, exactly. But it is quite off-putting, I think, to have something like that happening. But yet I think that that's where everything's going. I don't think it'll look like that. I don't think it'll look like that at all for a number of reasons. It's just very, very ugly. But I think Dan was onto something.

That tech I think is very, very useful for a wide variety of people. And because it can be overlaid in contact lenses, let's say, or just traditional glasses that just look better on people. I think that that's something we could see definitely in the next handful of years.

And I'm surprised Apple hasn't already come up with something to compete with, various different platforms.

Alex Porter

They're waiting for a mature market.

Robert Hansen

I think they will make the mature market, is my bet.

Tim Porter

They do tend to mature technology. You're correct. They did very well with the LIDAR technology and now they've patents that they have. Actually, I was privy to a headset that looked very realistic. It was something that's not quite out there yet. But it looked like a regular pair of glasses. And then, you would have a processing device in your pocket.

So it's not like a HoloLens or anything like that, but I was very blown away that I was like, "Oh, this is something I wouldn't mind wearing out."

Robert Hansen

So one thing the marriage of VR and AR I think was interesting is the Avatar set, Avatar Two, when they're filming, they've already decided that some characters are going to be running through a forest. And they have this sort of iPad-looking thing and this big room with all these sensors all over the room.

So know exactly where this iPad is and at all times. And various different camera aperture and lenses or whatever, joysticks sitting on either side where they can sort of say, I want it to look like this and pan in Zoom or whatever. And they walk around this 3D world that it is VR on in the sense that they're looking at the virtual world as they're walking around it.

But it is AR in the sense, they get to decide where they're going to be in this three-dimensional world in the real world, in actual real 3D space. Which I thought was very interesting that exists first of all. But also I think that that is where everything's going. It's like almost the opposite of motion capture, right? The motion has already been decided.

It's more where the camera's going to be and you have the exact right angle on it. Have you heard of anything like that before?

Tim Porter

I mean, it's called Virtual Production. And I was used heavily on a number of different movies. So when I was working at Sony Fisher Hammer's works we ended up doing cleanup as part of my training on Cloudy with the Chance of Meatballs number two. And part of the initial training was actually for the first Cloudy. And there were using a big old JVC camera where they pushed through video into the headset. And you could actually do that as over the shoulder camera.

As everybody knows or may not know, Cloudy was an entirely cartooned event. So somebody was running physically through a fully digital scene and they looked in through the eyepiece and it was the digital world that they were able to see. But this combination technology was also heavily used on Lion King. And then everybody, if anybody's seen any BTS or behind the scenes for Mandalorian, that's when it really came out.

Robert Hansen

And that was interesting because the stage itself became... It was almost like projection mapping on the stage as they were acting. Was pretty interesting.

Tim Porter

That's a big thing. You can do combination between LED walls. You can do full projection mapping. You can do everything in between. But that's some of the heavy stuff that we're doing with movie studios, is virtual production because you can't do it in post. That was always the joke. "We'll fix it in post." There is no post anymore.

It's all happening. You have to do it in pre. You have to have things scanned, you have to have the world scanned. You have to have 3D assets. And our technology as being heavily used in a number of different groups and studios.

Alex Porter

Virtual production is really dynamic. So not only do you get to choose where you are in space, where the camera angle is, you can alSo with these digital sets, actually influence the scene. So you say, "Oh, I've decided it should be dusk, not dawn." You can literally change the lighting. You can change the entire scene behind you. "Oh, I want this side of the canyon, not that side of the canyon."

And as long as you have a high quality scan of an on location shoot that you've done, number one, you don't have to port your entire camera crew and actors there. You can actually bring the set to the stage and do the production there.

So it really is this culmination of the physical and the digital coming together in a way that is not only going to make it more cost effective to actually create these productions, but also it helps the actors and the folks that are on set be more in the moment and actually really engage with the scene even further.

Green screen is going to be a thing of the past. The amount of cleanup post-production and things that have to happen to make green screen effective really is going the way of the dodo. And getting replaced by these digital screens.

Robert Hansen

Yeah. For all kinds of reasons. I've tried to work on green screens a couple times. And it's funny how one thing I posted on Facebook one time was a photo of me in front of a green screen and I'm like, "Oh, nothing bad could happen to this photo. So I'm just going to place it right here." getting everyone to play along.

So a lot of people did jump in and they were trying to do stuff with it. And in every single one of them, there was a bit of a tinge. It was very difficult for them. And I didn't, I didn't do any work to soup it up to make it better, but at the same time, that is just an enormous amount of cleanup, as you said.

So I think you're right. I think this is all going to this virtual world sort of design and or these walls that are 3D printed walls/projection mapping/LEDs, because I think that is just way cheaper, way better.

Tim Porter

That's just a new aid set design.

Robert Hansen

Robert Hansen

So you would get the fastest route would take you across 72 lanes of traffic, you needed to take an unprotected left. And you're like, "Why Ways?"

Tim Porter

And it was uphill. You're talking about the one in L.A, right? That thing is horrible.

Alex Porter

It was horrifying. L.A was actually a great city for Ways except for when it did a Ways left on us and we're like, "This is so bad." But I mean, this is an aggregate of a lot of data. This is an aggregate of wayfinding.

I know someone who did something like this with Google Glass and they set up a LinkedIn hit and they were trying to figure out all the investors that were in the room.

So as they turned, then they would get photos on one side eye, and they would go, "Okay, these are the people that I actually need to talk to." It didn't really help them much, but it a really interesting story. They're like, "Oh, I got to talk to them." But it was interesting.

Robert Hansen

It is interesting. So I saw another thing that I thought was really cool about AR and VR in particular. It allows people to explore trolley problems that normally you wouldn't get to. So we always talk about trolley problems. Would you throw this person on the train track to save these four other kids? And usually it's some fat man that you're going to be throwing in this trolley problem.

So one version of it is you just have the switch. You switch and kills the fat guy. And then, another version, you actually have to push him off a bridge and then he'll land on the train tracks and it'll crush him, but stop the train and the kids will be saved.

And despite the fact that they have the exact same effect, one person dies, four people live people have a lot of problems with the second one. They just don't want to do it. That's a little bit too intimate, I guess. Pushing a button's way, way, way easier. I think AR and VR allows you to actually start exploring these things a little bit more carefully.

Like, what is it about that? And you can really tune it to the exact thing that people are allergic to or do or don't want to do about these different versions of these trolley problems. I think that's really an area that no one has truly explored and could be interesting.

Tim Porter

There's an interesting one I've seen before. I don't know if you know about this one. I think a Stanford study where they were trying to figure out if somebody would hurt another person. And they were told if they pressed the button over and over and over again, it would end up hurting them more and more.

And there was no reason why they were told. It just can tell them that it was getting progressively worse. And many people, by being very pressured by the other, the people who were running the study, they would continue to turn up the power. Even to lethal doses just because they were pushed.

And once again, we've talked about governmental hands and fingers into things and nefarious and nefarious actions. I think you're 100% right. Nobody's really touched it on that side. And then of course, my fear is misuse and abuse

Robert Hansen

Of course. But also it's very useful for things like, "Are you going to press the button and the nuclear bunker? Are you actually going to go through with your job?" It's a lot easier to actually experience all the sight and the sounds and the touches and whatever which I realized they tried to do with actual live simulations.

But you could do it 20 times a day when they're off duty or just hanging out in weird scenarios that are actually very difficult. Like all of a sudden there's a gas leak, what do you do? These weird scenarios where they're having to take corrective action that. You might actually have to kill the guy next to you, for reels. Well, you're probably not practicing that every day. Well, you could in this VR sense.

Alex Porter

Human are pack animals, right? So I do think that it has that influence of like, who is around you? What is the scene being set? Are you saving 20 people and killing one person? But I do think there's a lot of value in that continued practice in an immersive way, right?

Even going back to literally just like retraining your brain for stressful things that are coming. So for me, particularly, I would go back to like having kids. Giving birth is like a very daunting thing. And you don't know what it's going to be like until you're doing it.

So you can practice, you can visualize, you can do everything. I mean, I did all the things to sort of prepare myself for it. It still didn't happen the way that I expected it to, but it was still an inevitable, like happening, right? So I think the more you can kind of, you know, prepare yourself in different manners, the better off you're going to be. So I think there's a lot of value there.

Tim Porter

And reduction of PTSD and things like that. So if you pre-train appropriately enough, when you're talking about police, military or EMS they've seen that it reduces the chances of those individuals having PTSD. That's another pretty useful thing.

Robert Hansen

So let's talk about that. So one of the projects that we sort of toyed with at one point a long time ago was the idea of using the 3D volumetric, capturing ideally in slow motion. Which is actually an area that it's like this little sub-niche of this whole thing to watch somebody, let's say, taking a gun out of a holster and pointing it and pulling the trigger.

And the idea would be are they pulling it too quick? Are they putting their finger on the trigger ahead of time? Are they actually looking at the sites? Are they dropping the muzzle right before they shoot on all these sort of shooter problems that you could extrapolate and turn into military applications?

And the utility for something like that on the battlefield, when you start extrapolating to all of these little sort of micro movements, you're like, "Where did this person go wrong? Why did they keep missing? What's happening there?" It's actually really difficult to diagnose.

All you can see is that they missed. But why did they miss? It's sort of like that golf swing where it's the perfect golf swing, but for some reason it always just keeps going left. What's going on? Well, turns out that they right at the last second this thing is happening. And you can slow the video down if they're in a stable enough location.

But it would be even better if you could spin the camera around, it's like, "Ah, look, it's their ankle that's twisting here." Or whatever is causing this minor issue to occur. Where do you see that sort of going, especially in the military space? Where do you see that going?

Alex Porter

I mean, I look at that all as athletic performance. Whether it's a gun or a golf club, your body movements and your mental state of course all point into your end performance. Very early on we identified that there were immediate use cases for VR training specifically in this realm.

And again, it's that training piece is really important, but that feedback and the analytics part is also really important. Understanding not only how to mentally prepare yourself for these different scenes and scenarios, but also how to physically prep and overcome some of those shortcomings that you might have with your ankle twisting or your finger being on the trigger too early or whatever the case is.

We have looked at that more in depth on the sports training side, softball and a few other golf as well. And how we can help sort of create solutions in that realm. What it often boils down to is that you need cameras that have a high frame rate because the movement is quick.

You need to be able to capture all of those minute movements be able to play them back and analyze them effectively. And then of course, depending on how you are actually analyzing whether it's a human or whether it's a machine, "Well are you training it? What is your data sets look like? What are you really trying to get out of this?"

Robert Hansen

One of the things I think is wrong with a lot of training is that there's a human doing the training. Humans are terrible. We're really, really, really bad at training. And what other humans are bad about is receiving bad human tracing. if someone's yelling at you over your shoulder telling, "Ah, you're doing this all wrong and I've done it 100 times, let me just show you how it's done."

Tim Porter

"Well, that flashlight higher son."

Robert Hansen

Whatever. We've all had very bad people we just can't learn from. We just can't do it. But computers on the other hand, I mean, they're totally emotionless. They're just saying, "Hey, this is what the problem is, if you want to correct it, here's how to correct it."

So I can do it a thousand times wrong. I'm like, "Oh, I really want to do it right." And the computer's not getting frustrated. The computer's just like, "Okay, you're doing the thing wrong again." And I just keep doing it and do it, and then finally I do it right? And it's like, "Yap, you did it right." And it's like, "Wow, that was the time I did it right. That's amazing."

No one's time is being abused. It's just me and a computer just hanging out doing the thing we need to do. And I think for military applications in particular, that thing, allowing a soldier to fail a thousand times until they just get it right. I think that could take a lot of soldiers who would normally just not be suitable for whatever job and make them suitable. Can make their brain work, can make their body work in whatever way it would.

It could be a sports team. It could be diving, who knows, right. It could be anything. I think that there's a lot of promise in that sort of paying very, very close attention to every micro movement. I actually saw something recently in basketball where they were trying to do photogrammetry. I wouldn't call it real time because I'm sure it was delayed.

But they were moving the camera around and sort of going through their players and so you could see the shot from different angles or whatever. And that's fine for post-game analysis, but I think it's even better for training where you're, "I'm trying to hit this shot, I'm just trying to do this weird thing." And it's always weird because it's off angle I'm trying to shoot in a weird way or whatever."

It's not something that's going to happen very often on a game. Certainly not often enough for that to make sense for that photogrammetry thing to actually be useful because it's never going to happen or very rarely going to happen. But in training, I've got hours and hours and hours just to practice this one shot. "Show me how to do it right. And no coach.

I know you've got 50 other people to go pay attention to and media stuff to go handle and whatever, but I just need to get this one shot right." So you practice it over and over, the computer just gently tells you, "Nope, you're off by this. You moved your ankle again." I think there's a lot of value to that.

Tim Porter

It's a ton. Actually, we had looked at police training. I don't know if anyone's seen actual police training. It's walls on screen. Everybody remember going to Chuck and Cheeses and you shoot? It's that. That's exactly what they have.

And being able to take that to another level where you're talking about scanning people, putting people in, and you can replay that scenario. Not necessarily the same way over and over and over again, but things can change.

So we end up putting a white paper together with Intel that was about being able to go ahead and set up very easy police and military training where you could move around things like foam blocks. You could some rover around the room, you could drop these QR code-like objects into the room. And then the machine would go ahead and set that up.

So say you're practicing something like say a school shooting. Trying to protect kids from that. Well, you need the layout. So you could put up these foam blocks and it looks like this. Here's the stairs. Everything like that. You put on the headset, boom, you're immersed entirely in that environment.

And then, you could run these simulations over and over again. He comes out this side, he comes out of that side, they come from that. There's five people here, there's 12, and you can run these completely contrived situations that you never could even possibly see in the real world.

But it prepares for so much more and I see these as acceleration.

Robert Hansen

But even better is than the computer says, "Here's what you just did wrong."

Tim Porter

Exactly.

Robert Hansen

And then you might have an instructor who can go and coach you or whatever. But they have gone home and now you're just working through the night, just doing your exercises until you got good at it. I think there's a lot of value to that part in particular.

It reminds me of, I was talking to a friend of mine, I was like, "We built this app." And he was talking about, it's like, "Oh, well how do I get this thing set up?" I'm like, "Oh, just give us a call." He's like, "I have to call you to do this. Can I just go to a website or something?" I'm like, "Well, it's new. It's beta software." He's like, "I'm sorry. I just don't want to deal with sales people."

There was another example of this where, I'm not sure, I think this actually predates COVID. But they were trying to do like Teledoc stuff. And they're like, well, it turns out if the Teledoc asks you questions, people are way more likely to answer correctly. They're not going to lie to that computer.

One of the tough parts with that entire sort of ecosystem is that clinicians are the ones that are typically expected to train people on using their wheelchairs. To be the oversight. So there's limited bandwidth. So you're talking 20-ish hours of training that you get in a power chair.

And if you've had a severe accident and you are now quadriplegic, this might include time learning to sit up again, not even learning to actually operate something that accelerates and directionally changes based on a single movement of your head or your chin or your hand, depending on your body function.

So this is an exact direct symbol in a different arena, right? It's not, it's about maximizing resources and allowing people to access this training and do it to the best of their ability as much as they possibly can.

Tim Porter

That thing that you said was actually, some patience well beyond human capability was the reason why IDrive was actually created. I had done a ride along with a family friend of ours, his name's Kevin. And he actually worked for a competitor of Quantum.

So the ride along that I did was for a seating. And this gentleman had been in a massive car wreck and his entire spine was fused. I saw a man attempt to sit in a chair and after his hour period that he was allowed to be in it, wasn't capable of doing nothing, they're like, "Oh, that was your hour." And I said something needed to happen

Robert Hansen

Yeah, of course.

Alex Porter

Human dignity is what mobility is about.

Tim Porter

It's taking away someone's legs.

Robert Hansen

Yap. So let's change the topic a little bit and talk about deep fakes. Just change it up. So what I think deep fakes really comes down to is no longer being able to trust our senses. Now that might actually be a great thing because now we might be in a situation where we're starting to question everything.

A lot of people are like, "Well, I don't want to be in a world where I have to question everything. I just want to be able to believe my pundit." It's like, "Well now you can't even believe that because that can easily be hijacked." What we're really talking about is identity theft.

A different sort of identity theft. They don't have to have your driver's license anymore. They can just look like you. They can be on Zoom theoretically if the models worked well enough and were dynamic enough or they could just pre-record it and just send him a video or whatever. And here's your favorite pundit saying this terrible thing. There was a guy, I'm sure you saw Chris, whom Metaphysic ai did the Tom Cruise deep fakes.

So I personally, I was talking with Charlie Burgoyne about this. I personally did not think that that was passing the uncanny valley personally. Maybe I've just got a very specific eye for this thing or something, but it was very close. I could see where it was going. For sure.

Where do you think that's going to be in another five years? Do you think it's just going to be something everyone has on their Snapchat filter and they can just play anything they want? How far away are we do you think?

Tim Porter

I think with the, It looks like them situation, yes. I think the thing that you detected, and this is how the uncanny valley works, it has to do with motion. That's the biggest part of the uncanny valley. It can look just like a person and if he froze and you saw a single frame, you might have a difficulty understanding that that was or wasn't him.

Why not? I mean, digital me is worth a lot more than physical.

Robert Hansen

That's true.

Alex Porter

I mean, I think there's a lot of opportunity for iteration in this space. And I do think that we are starting to see a lot more movement in protection of consumer rights. Particularly, if you're looking at the DMA, the things that are happening. The Digital Media Act in the EU.

We're starting to see movement toward consumer-friendly, not tech-friendly. So Big Tech, this is actually geared particularly at Big Tech in the US. They are looking at specific revenue rates that's when it's really going to hit.

So we're talking your gatekeeper. So Google, Facebook et cetera. They're going to be the ones that are subject to this. They have to have open-ended ecosystems. The EU says that everyone who buys something on Google Play should be able to play it on your Apple device and vice versa.

So open ecosystems, this is already a much more friendly opportunity for us as consumers. And then, we're seeing, you know, that they can't just scoop up every tiny company and then take over the entire space and do a lot of antitrust stuff.

I think this is directly in line with the movement that should be happening in order to stop some of the movement around deep fakes around media spoofing and all the things that are happening that are duping people. And I personally want to own the rights to my digital self and the rights to my data. And I believe that we're going to see a lot more movement in that direction.

Tim Porter

This and the ethics around XR and technology around that was the reason why we actually went into CTA because the last thing you want is an uninformed lawmaker.

Robert Hansen

I literally just had this conversation with the salient guys last week. So I'm with you on that one. So I think one very interesting thing that is in this space is just, there's a lot of photos of people online now.

There you go.

Alex Porter

That being said, we work in digital, right? Literally everything we do, and the way that we frame our work is around enabling others to do content creation. And this includes people, right? At some point.

That's not really our focus at this juncture, but we see opportunities and we will put in the ethical boundaries as makers. I do think that part of the responsibility falls to the people that are making these enabling tools, the people that are making this content.

If you're crossing an ethical boundary and you're taking someone's digital presence, digital identity, and using it for inappropriate purposes, without their consent, you should be held liable for that.

But a big enough space where you can't feel anything. There's nothing to touch really. And then you wake up and you're in this complete void where there's no sound. You can't feel your skin, you can't really feel anything. You're just free floating body.

Tim Porter

It's a full disassociation.

Robert Hansen

Full disassociation. And the idea is if you're a religious, even a little bit, you'll think you've died. So they'll leave you in there for like a day or whatever. And then they basically say one word, "Confess." And now you think you're in purgatory, and you're like, "Oh, I better like get it all out." Every single last detail of anything you can think of that you've ever done that's bad.

You start spilling your guts. So they just gather it all up and then when they're done, they're like, "Great." And they just let you out on the street with zero bumps or bruises on you. And you're completely fine.

I mean, I think if people don't think super realism is a thing, I think they're just not being creative enough. I think that is the power of just a little bit of trickery of your eyes, your senses, just get a little bit numbed or feel a little bit different than you might expect. And you might truly believe that you've died or that you're in some place, or that you can jump off buildings because you're the guy in the Matrix or whatever.

Abraham Lincoln and Mark Zuckerberg. They got in their way back machines.

Alex Porter

Colluded in their time machine. Holy moly. Yea, no. That's actually fascinating to me that people are so detached from reality. That being said, this is my world. So I'm like, "Oh yeah, you didn't know that that was like actually factually not a thing at all? It's bizarre to me.

I look at the Metaverse really as just like a bucket of technology, to be honest. It really is about the culmination of human experience that is fantasy and reality smooshed together in one sort of virtual experience. I personally look at virtual as any device that we're interacting with.

So it could be augmented reality, could be virtual reality could be a headset, a computer, a phone. It doesn't really matter. It could be a billboard. There's ways that we consume digital content every single day. And those are all virtual reality to some extent, or augmented reality.

So the way that I think this is really going now is that, the magic of the Metaverse, it's this decentralized, wondrous land where everything is interconnected that is like so far from real at the moment. Everything is built in silos. We have our individual singular platforms that do not play nicely with other platforms. We have our Facebook Horizons and our Sandbox and our Fortnite. All these are variations of the Metaverse.

Some of them are more work oriented, some of them are more play oriented. We're going to see a convergence. There are some technical infrastructure pieces that need to happen in order for those things to all work well together. And we actually are working specifically on a solution right now for a file type that is like a vector for 3D.

Alex Porter

Thank you for having us.

Tim Porter

Been wonderful.

No Transcripts Are Available Yet

Comments

Share Your ThoughtsBe the first to write a comment.

THE RSNAKE

STORE

Show your support by getting yourself a new t-shirt, hoodie or any of our products available in the store!

VISIT OUR STORE