Hi. Okay, I am Andrew Duchowski and I'll be presenting our recent work on eye tracking in 360 videos
with the focus on accessibility standing in for Chris Hughes. He's really our
developer of our...you know what, I'll play the slides...were we are...
He's really our lead developer in our LEAD-ME COST Action Workgroup.
And so here in Vienna, what I'll talk about is a little bit of the background and motivation.
accessible, immersive video. And then what we do, eye tracking
in 360 video to test various usability type
applications and accessibility type developments. Okay. So background and motivation. Here is
what standard 2D subtitles are
used to
percent, or what we are used to seeing when they are presented. So here, in this case
on TV, on the 2D display, and here they were showing color and a box, a specific kind of font.
So you've seen all that before. and
this kind of thing can show up on most devices these days. And even this is actually taken from, I think the Opera House in Barcelona, where you even have something called surtitles where the subtitles are above the stage, or even in the seat back in front of you.
So the point of this is that we need this kind of accessibility in modern
extended reality applications, VR being one of them, augmented reality as well.
Yet another form is sign language which I will quickly skip because we're not really implementing this or looking at this in our particular workgroup.
Instead, we're doing immersive video, which is 360.
So this goes back to a BBC project that Chris was involved in building the immersive accessibility player
and looking at the media there and trying to get
subtitles to show up in VR.
And the ImAc player and recorder was part of that design. So that was back then it was user centered with
user requirements. They eventually built the platform but the trouble is that you'll see that this development kind of was short lived, or maybe limited in its extent, because
too much was uncovered in the sense, too many variables, too many type of possibilities, and it was really hard to pin down what worked.
But anyway, what we want to do is, do this for VR glasses specifically, and one of the concerns is limited resolution. And so where do you put the subtitles. Do you put it
in the regular kind of like 2D display at the bottom. or what? You have to remember that the user's head moves around here.
And so we get a 360 view, and so it's much different from the standard TV 2D display.
The resolution is not much of a of an issue, except that you don't want to put subtitles there, because it would distort the text.
The other issue that it shows up, or what was uncovered in the ImAc
player was that
this person speaking here is not speaking. In fact, he's just standing there, but the speaker is off to the side somewhere where we don't see them at the moment. And so here, how do you indicate
where they are? They use an arrow to test.
So yeah, he's not speaking.
Speaker is over on the left. Go look there! Meanwhile the other technical challenge here is, how do you present
video and 3D. You could do various geometric mappings, using cube map, or
pick, we rectangular what the difference here is
is a distortion, and so the plot below shows you how much distortion there is.
and where, and so most of it happens at the polls, as it were. That's why you see often the globe is kind of stretched.
There's a video here, but it doesn't want to play in in the player.
which just shows you
which shows you what the distortion looks like. So if it's done properly.
the distortion
isn't there because these lines look like they're
like lines, and they're not distorted
like curves like you would expect here in the image.
Alright, other types of mappings include standard cube. Back, then, you see, the distortions are kind of around the corners of the cube.
and then also equi-angular cube map seems like one of the better options, but it's a little bit more exotic, and
I don't know.
not quite as sort of prevalent as the equi rectangular projection that we use a lot of 360 cameras use, and so they give you that pretty much for for free.
So here you see what the what the mapping looks like, where the distortions are.
and if you're mostly in the center field of view. Then you're kind of okay. And so it seems to work very well.
Now user testing for accessibility. The trouble here is,
first of all, people might not be used to the technology. So, in fact, in some of our testing that Marta may tell you about
when asked which type of subtitle. Do you want head
or a headlocked, or fixed, people say what's VR? And so there, that's an interesting kind of technological challenge in itself. Content is super important
because how do you actually get? Excuse me, objective results as opposed to subjective kinds of things.
So this is what they learned in the W3C group is, they looked at various
VR approaches, and here's what they came up with.
Lots of things! Are the subtitles fixed in the scene,
or headlocked, so do they, are they linked to where your head direction is pointing.
Turns out that they didn't get a very good answer to all this, even though they were able. and Chris was instrumental in this, and implementing this. So this is a technical challenge.
On the right. The little cone is the head, direction and head position on the the plane. There is where the actual part of the video shows up where we can draw subtitles.
So
in VR world in 360 video we have location of elements using polar coordinates.
polar angle and azimuth.
And then what you do in computer graphics is you build a scene graph. So the scene means the world. The world includes a camera. It includes the captions.
the video.
the sphere and the fake camera is where the where the head is.
So how does it move? Is it fixed and so forth?
Chris came to this because they, I think they asked him to do this with a
with in beyond, he said, oh, yeah, sure, I'll be happy to do that. It's pretty easy. And then they gave him this video where the speakers were on the side in an airplane seat, and right away you can see that things didn't quite work.
It wasn't quite obvious who was speaking when and so forth, and so
fixed subtitles. He had to basically reinvent them, using something based on particle systems. This is a screenshot from
Star Trek II The Wrath of Kahn. I think a 1982 or 85 movie where each particle here
has a position inside this flame.
So each of these sort of strands in the video. Here is a is a particle, and it's got a position of velocity. So speed, direction, color.
age, lifetime. Eventually the particles die, and they are removed from the scene.
There's also transparency and things like that. So Chris thought, hey, why not use the same computer graphics technology to represent subtitles as particles.
And that's what he did. And of course, with that you've got an emitter so you can throw these captions, as he calls them, or subtitles anywhere you want.
Each one has a certain lifetime or lifespan. They can have their own color, font shape. box, transparency whatever.
And so here he is with all these options you've got this ton of things to pick from. Here on the right you can see the menu. It's quite involved
guide modes, responsive captions, time codes, all sorts of things. Trouble is, how do you actually test this? How do you actually establish what is useful
out of all these myriad of choices.
Here's what they look like, and some of the things that that particle systems give you. You can use physics engine to prevent collisions. You can stack them so that they don't overlap each other.
You can do the different colors. You can make sure they're inside the scene. This is Chris's living room, by the way.
There he is. You can basically see him in the middle picture. There he's standing up, and he's talking to his kids and all the various things you can do
to make these display in VR. We still don't know, though. which is the best of all these, what's the most effective? What's the most accessible.
And so. like Henry Ford, if he was asked what they wanted, they would have said, Just give me the old style 2D subtitles. so that I can see them. So they would not want all these new fangled things.
So here's how we test them in 360. The technology that we've conceived of here, and that Chris implemented.
And we use an eye tracker to find out what you're looking at.
The HTC Vive Pro Eye is such. an eye-tracked, endowed VR headset that we use, and we've got several of these, and so
Dr. Krejtz has used one in Warsaw. Marta has taken it to Slovenia,
Spain, and Chris has one in the Uk. I've got one here in in the States, so we all share the the source code. And we we all test this and run this
when it works. 120 Hertz ey tracking that's the sampling rate.
It comes with this sort of software development toolkit SDK: from SRanipal which gives us the gaze information vectors.
Here's what it looks like. You get a position
for where the gaze ray emanates at each eyeball, the direction, and then the intersection
on the screen, if you want it, or you can calculate it.
So
the geometry is there, and it works very fairly well.
We get
the gaze, origin, direction, pupil diameter, eye openness, and whether your eye is in the sensor or visible by the sensor.
I am.
We get all this in Unity. We can output these files and put them in. And so we've got.
Chris has created this
system architecture where it all relies on a threaded kind of recorder. So we record the data, the screen, eye movement data, save it to a file, and then later you can read it all in
and play it back via this
player that he's also developed. So that is the technology. I think
Marta will come in and show you an example of this
video that we have, or several videos. And the key concern here is that we've got all these options, and we don't quite know which is the best. How do you test this?
Technology itself is the learning step that users have to put the helmet on to learn how to use it.
And then the eye tracker that we get
that will give us objective evidence of what people are doing. Are they actually reading the subtitles or not. This was actually not known before people would say, oh, sure, I read the subtitles.
but we actually have data that shows quite the contrary.
So I will leave you at that. That is the technology that we've built up, Chris mainly.
And then Marta, Dr. Krejtz
has gone and obtained really interesting data, and I think you'll see more during this LEAD-ME COST Action
workshop.
Thank you so much on behalf of myself. Dr. Krejtz, Marta, and of course. the indubitable Chris Hughes in Salford.
Thank you. See, you later.