A lot of the implications for ray tracing are on the dev-side of things. It’s a bit hard to explain without going into technical details.
Essentially, getting light to look “right” is very very hard. To do it, devs employ a lot of different techniques. One of those older techniques is baking the light on static objects, essentially pre-rendering where light goes and how it bounces. This has been done for a long time, e.g. even in Half-Life, the lights are baked for static geometry. So in a way, we have been using ray-tracing in games for a long time. however, it isn’t real time ray-tracing, as the information gets stored in light map textures, so there is no performance impact other than storing the texture in RAM/VRAM and drawing the texture together with others.
The inherit problem of that technique is that it only really works for static geometry. If you move your light or any objects in the scene, your lightmaps will no longer match. To solve this, there are mixed modes which use real-time lights, dynamic light maps, and other tricks. However, these are often subject to problems and/or the limitations of using real-time lights. Real-time light problems are: You can only do a limited number before getting a serious performance impact, especially if the lights produce shadows. Soft shadows, shadows in big areas, and very detailed shadows are extremely hard to do as well without some advanced tricks. Also, ambient occlusion and global illumination is not something you can just give lights (there is screen-space GI and AO, but they don’t look good in all circumstances, and you have limited control. There are also some other techniques some engines did for real time GI.).
Also there is the problem of baked light affecting dynamic objects, such as characters. This has been solved by baking so called “light probes”. These are invisible spheres that store the light data and the closest data then gets applied to the characters and other dynamic objects. This again has a some problem, as it’s hard to apply multiple light probes to the same object, so lighting might be off. Also, light direction is not accurate, which causes normal maps to look very flat in this light, and local shadows do not work using light probes. The same is done for reflections using reflection probes which are static. These are 360° “screen shots” essentially storing the reflection at that point in space. This however costs DiskSpace/RAM/VRAM, and it will not hold any information for moving objects (that’s why sometimes you can’t see yourself in the mirror in games). Also, the reflections sometimes look “out of place” or distorted when the reflection probe is too far from the reflecting surface (again, these cost VRAM and RAM so you don’t want to place them in front of every single reflective surface). It costs a lot of time to find the right balance. For the rest, usually screen space reflections are used, as any other real-time reflection is extremely costly as you essentially render the whole scene again for each local reflection. Screen space reflection is an advanced technique that works very well for stuff like reflective floors, but you will quickly see its downsides on very mirrored surfaces as it lacks information that is not on the screen. Some games like Hitman for example use the mix of those techniques extremely well.
Coming back to lighting, there are now better techniques used for example by unreal and some other engines (and now unity in experimental). The light gets stored in more predictable data structures, such as 3d textures. This way, you can store the direction of all light in each cell. The light then gets applied to the objects passing through those cells. This looks pretty good, and the runtime cost is fairly low, but the storage cost of such textures is a tradeoff of texture resolution and fidelity. These textures cost a lot of VRAM to store and without using advanced techniques and tricks, have their own limits (e.g. for scene size). It also costs a lot of time to create each time you change the scene, and it also doesn’t eliminate all problems mentioned above, like reflections, moving lights, etc.
Specifically, there is the problem of character lighting itself. Using light probes on characters usually looks pretty bad, as it removes a lot of detail of advanced skin shaders. Even with the above mentioned techniques, character lighting is still extremely hard to do. There is also some other problems, like ambient shadow in already shadowed areas, and light balancing for character versus scene lighting.
For that reason, most AAA games use separate light rigs for characters. Essentially floating lights that ONLY affect the character and move with them. When the mixing with the scene lights is done right, the rig adapts to the current situation in terms of light direction, color, and intensity. If you look in most AAA games, you can often see situations where rim-light comes from a direction where there is no actual light source. However, this way, the devs and artists have full control over lighting the characters. Essentially like a real movie production would have, but without the limitation of the real world.
Now, ray-tracing as you know it right now is not quite there yet, but eventually, ray tracing is the solution to a lot of the problems mentioned above. Things like polygon density, light count, global illumination, ambient occlusion, light direction, reflections, and much more are simply “there” for you to use. Now this doesn’t mean that it will automatically make everything look great, but with the overwhelming amount of different tricks that have to be used for current gen games to make the look good, it opens a whole new world of possibilities.
Also, something that will not directly influence the final game, it will eventually simplify things for devs so that more time can be invested into other things.
At this current usage of ray-tracing, it’s more like a gimmick, because devs will still focus most resources on the current ways to use light. This is because most people don’t have cards with sufficient ray-tracing capabilities. So for the moment, I agree that the performance hit is not worth it. However, eventually it might become the default way to draw games. While we are not quite there, in terms of performance, I think that things might become a lot more consistent and predictable eventually for raytracing.
YES, thank you! You saved me a lot of writing haha
This is spot on and the real advantage of ray tracing - when it becomes the norm it’ll look better, provide effects that are extremely difficult or impossible and do so with minimal dev pain.
Awesome and great explanation for a layperson. Because the industry has been faking lighting for so long and lighting is quite important, the industry has become incredibly good at it. But it also takes a lot of development time that could be spent on adding more content or features. There’s a reason the opinion about ray tracing is extremely positive within the game development industry. But also nobody’s expecting it to become the norm over night, and the period with hybrid support for both raytracing and legacy lighting is only just starting.
Worth mentioning that we’re also about halfway on the average time for these big features to hit significant saturation, like with PhysX. It’s pretty common for a GPU (and sometimes CPU/Chip set) to take 3-4 generations to trickle down enough through new products and used product sales to have decent enough depth/usage. At this point depending on how Apple is handling ray tracing, they might slow down the transition away from rasterized.
A lot of the implications for ray tracing are on the dev-side of things. It’s a bit hard to explain without going into technical details.
Essentially, getting light to look “right” is very very hard. To do it, devs employ a lot of different techniques. One of those older techniques is baking the light on static objects, essentially pre-rendering where light goes and how it bounces. This has been done for a long time, e.g. even in Half-Life, the lights are baked for static geometry. So in a way, we have been using ray-tracing in games for a long time. however, it isn’t real time ray-tracing, as the information gets stored in light map textures, so there is no performance impact other than storing the texture in RAM/VRAM and drawing the texture together with others.
The inherit problem of that technique is that it only really works for static geometry. If you move your light or any objects in the scene, your lightmaps will no longer match. To solve this, there are mixed modes which use real-time lights, dynamic light maps, and other tricks. However, these are often subject to problems and/or the limitations of using real-time lights. Real-time light problems are: You can only do a limited number before getting a serious performance impact, especially if the lights produce shadows. Soft shadows, shadows in big areas, and very detailed shadows are extremely hard to do as well without some advanced tricks. Also, ambient occlusion and global illumination is not something you can just give lights (there is screen-space GI and AO, but they don’t look good in all circumstances, and you have limited control. There are also some other techniques some engines did for real time GI.).
Also there is the problem of baked light affecting dynamic objects, such as characters. This has been solved by baking so called “light probes”. These are invisible spheres that store the light data and the closest data then gets applied to the characters and other dynamic objects. This again has a some problem, as it’s hard to apply multiple light probes to the same object, so lighting might be off. Also, light direction is not accurate, which causes normal maps to look very flat in this light, and local shadows do not work using light probes. The same is done for reflections using reflection probes which are static. These are 360° “screen shots” essentially storing the reflection at that point in space. This however costs DiskSpace/RAM/VRAM, and it will not hold any information for moving objects (that’s why sometimes you can’t see yourself in the mirror in games). Also, the reflections sometimes look “out of place” or distorted when the reflection probe is too far from the reflecting surface (again, these cost VRAM and RAM so you don’t want to place them in front of every single reflective surface). It costs a lot of time to find the right balance. For the rest, usually screen space reflections are used, as any other real-time reflection is extremely costly as you essentially render the whole scene again for each local reflection. Screen space reflection is an advanced technique that works very well for stuff like reflective floors, but you will quickly see its downsides on very mirrored surfaces as it lacks information that is not on the screen. Some games like Hitman for example use the mix of those techniques extremely well.
Coming back to lighting, there are now better techniques used for example by unreal and some other engines (and now unity in experimental). The light gets stored in more predictable data structures, such as 3d textures. This way, you can store the direction of all light in each cell. The light then gets applied to the objects passing through those cells. This looks pretty good, and the runtime cost is fairly low, but the storage cost of such textures is a tradeoff of texture resolution and fidelity. These textures cost a lot of VRAM to store and without using advanced techniques and tricks, have their own limits (e.g. for scene size). It also costs a lot of time to create each time you change the scene, and it also doesn’t eliminate all problems mentioned above, like reflections, moving lights, etc.
Specifically, there is the problem of character lighting itself. Using light probes on characters usually looks pretty bad, as it removes a lot of detail of advanced skin shaders. Even with the above mentioned techniques, character lighting is still extremely hard to do. There is also some other problems, like ambient shadow in already shadowed areas, and light balancing for character versus scene lighting.
For that reason, most AAA games use separate light rigs for characters. Essentially floating lights that ONLY affect the character and move with them. When the mixing with the scene lights is done right, the rig adapts to the current situation in terms of light direction, color, and intensity. If you look in most AAA games, you can often see situations where rim-light comes from a direction where there is no actual light source. However, this way, the devs and artists have full control over lighting the characters. Essentially like a real movie production would have, but without the limitation of the real world.
Now, ray-tracing as you know it right now is not quite there yet, but eventually, ray tracing is the solution to a lot of the problems mentioned above. Things like polygon density, light count, global illumination, ambient occlusion, light direction, reflections, and much more are simply “there” for you to use. Now this doesn’t mean that it will automatically make everything look great, but with the overwhelming amount of different tricks that have to be used for current gen games to make the look good, it opens a whole new world of possibilities.
Also, something that will not directly influence the final game, it will eventually simplify things for devs so that more time can be invested into other things.
At this current usage of ray-tracing, it’s more like a gimmick, because devs will still focus most resources on the current ways to use light. This is because most people don’t have cards with sufficient ray-tracing capabilities. So for the moment, I agree that the performance hit is not worth it. However, eventually it might become the default way to draw games. While we are not quite there, in terms of performance, I think that things might become a lot more consistent and predictable eventually for raytracing.
YES, thank you! You saved me a lot of writing haha
This is spot on and the real advantage of ray tracing - when it becomes the norm it’ll look better, provide effects that are extremely difficult or impossible and do so with minimal dev pain.
Awesome and great explanation for a layperson. Because the industry has been faking lighting for so long and lighting is quite important, the industry has become incredibly good at it. But it also takes a lot of development time that could be spent on adding more content or features. There’s a reason the opinion about ray tracing is extremely positive within the game development industry. But also nobody’s expecting it to become the norm over night, and the period with hybrid support for both raytracing and legacy lighting is only just starting.
Worth mentioning that we’re also about halfway on the average time for these big features to hit significant saturation, like with PhysX. It’s pretty common for a GPU (and sometimes CPU/Chip set) to take 3-4 generations to trickle down enough through new products and used product sales to have decent enough depth/usage. At this point depending on how Apple is handling ray tracing, they might slow down the transition away from rasterized.