Faking shadow detail in 3DS and Photoshop

[Zarf]

As I will be gone most of the summer, and since I've decided that not doing this is just greedy of myself, I've decided to write a tutorial on how to fake shadow details in 3D Studio Max for models in Oblivion.

The way models are handled in Oblivion is simple. Faces that are pointing toward the sun are brighter, while faces that are not are darker.

As you can see in the above image, the faces that are pointing at the light are brighter than the faces pointing away. These are not shadows. This is called shading, and the following image shows what a scene without shading looks like.

Yikes. That looks like a cartoon, doesn't it? The following images shows shadows. The planes are all casting them. However, Oblivion's engine does not support models casting shadows. At least, not very good ones.

Here's a simulated example of what Oblivion's lighting system would do to a typical Redoran Hut. Of course, this is highly exaggerated, to get the point across.

So how do we fix this problem?

Here's a better idea of what it would look like with Oblivion's lighting system. Don't get met wrong: Oblivion's lighting system is fantastic. It doesn't cast ideal shadows, which would take up far too much processing power.

We must fake the shadows we want. This will be our end result once we fake the shadows. As you can see, it looks like the outer ridges of the hut look like they are casting shadows on the inside of the hut, which is exactly what we want. This tutorial will show you how to create those shadows, how to fake them, and how to implement them.

In order to fake shadows, we must take shadow information and flatten them out into a texture map. This is an easy step. First, go to the "Create" panel. It's highlighted in this screenshot:

Now, move to the Lights panel, and select Skylight, as highlighted in Step 2 of the below image. Place your skylight anywhere in the scene, and go to the modify panel (The panel with the arrow pointing to it in Step 3).

Your scene should look a little like this.

If you were to render it out, you would see those nice, soft shadows. Skylights are great for this. However, there is one major drawback: It takes FOREVER to render! No wonder they didn't implement this in Oblivion.

Now then. Let's set up a model to fake textures on, shall we?

This is just a test model. It was created in three different parts, and will use two different texture maps. This is not a UVW tutorial, so we won't get in to that. Let's assume you already have your model UVW mapped, and maybe even textured.

Apply a white material to the entire model. The easiest way to do this is to select the entire model, hit the "M" key on your keyboard (This opens up the material editor), make a white material, and hit the "Assign material to selection" box.

Now, select one part of your model. Hit the "0" (Hit 0, not O) key. This opens up the "Render to texture" dialog.

The above picture shows you what to change. Select "Use Existing Channel" so the renderer will divide the model depending on your UVW coordinates. Then, click the circled "Add" button, which brings up the dialog that has the lower arrow pointing to it. Click "ShadowsMap", then the "Add Elements" button. Then, press "2048" to get a huge texture size. If the map you are using is only 512.512, you should hit the 512 button. If 1024x10124, hit the 1024 button, and so on.

Here's where Photoshop comes in. You might be able to do this with Paint Shop Pro, but I don't know how to use that program. No tutorial for you!!

Save the resulting image to your desktop or somewhere. Open it in Photoshop, and it should look something like this. Maybe without the black background.

Congratulations! The hard part is behind you! All that is left is to get the texture for the building you were using before you realized there were no shadows on the model. This is the texture I was using:

Now, take your shadow detail map, and put it on top of the base texture.

You should now have two layers: the base texture and the shadow texture. Here's where the magic really happens:

Get it? I made the shadow texture layer "multiply" instead or "normal". This is what you get.

Check it out in 3DS Max. Are the shadows apparent enough? Need to add detail? Use the Burn and Dodge tools in Photoshop to add details. This is the unedited image in 3DS Max. There is a skylight, but it's not casting shadows. That evens out the lighting. The only shadows present are our textured shadows.

There's room for improvement. Say we want our shadows to be darker. This is easy, as long as you still have it open in Photoshop.

Select the shadow detail layer, and go to Image->Adjust->Brightness and Contrast. Here, you can change how bright or dark a layer is. Make it darker and put the contrast up to about 30. Save that map, and apply it in 3DS Max.

The shadows are now much more apparent, and even. Continue this process for the other portions of your model, and enjoy a professional-looking fully shadowed model ingame!

[Razorwing]

Cool, thanks for writing it Zarf!
I'd love to mention your tut in a news item on the front page. As I'm not familiar with the subject matter, could I trouble you to give me a nice 1-2 senctence "soundbyte" to sum up your excellent tut? TIA!

[Zarf]

I'll give it a try. :-D

Do your custom meshes look like they are being lit from all sides? This tutorial shows you how to get rid of that problem, and even how to add details to your model with no performance hit!

[Razorwing]

Originally posted by Zarf
I'll give it a try. :-D

Do your custom meshes look like they are being lit from all sides? This tutorial shows you how to get rid of that problem, and even how to add details to your model with no performance hit!

Perfect, I've posted it! :check: