Dota 2
276 ratings
Textures Guide for 3d artists - Part 1 - A good base from baking without 2d knowledge
By T_Vidotto
For the Part 2 click on this link
for the Part 3 click this link
On this tutorial we are going to use a model I made on the first months of 2014 for the Dota 2 workshop called Spirit of Anger

This tutorial is focused on baking informations from the highpoly model in order to make your life easier when painting the textures and having a consistent texture quality

You can find the ingame item at:
Steam Workshop link
Dota 2 wiki[dota2.gamepedia.com]
Dota 2 store link
Presentation video
   
Award
Favorite
Favorited
Unfavorite
What is "Baking" and why I need to know to create 3d assets


Baking
"Baking" is the act of pre-calculate some informations that are going to be stored on the texture, aiming to reduce the amount of processing to render that model.
In some production like movies or architectural represetantions, it is normal to spend minutes or maybe hours on just one image. For a game you need to speed up the process to be able to render at least 30 images each second, and getting more processing power is not the answer since you need to address that to an average user gear. In other words you need to make the asset light enough to an average* computer be able to render in realtime.
*Average depends on the public you are adressing

Advantages
  • It is necessary for a game asset if you want more details then a simple flat texture or a hand painted texture
  • It can help you on the texture painting by guiding your strokes and reducing the 2d knowledge required
Disadvantages
  • Takes more time to start texturing because you will need the model UV unwrapped
  • It may have low quality textures depending on your UV skills and engine limitations
  • Not everyone likes UVs*
* I love it, is like playing tetris but with more complex pieces

HighPoly and Lowpoly

On this kind of workflow were you need to "bake", it is necessary to have a HighPoly model to generate the informations over the LowPoly

Below is a comparison between a Highpoly model and the LowPoly model with the textures applied

If you ignore the color, they look pretty much the same right? The most visible difference is in the silhouette because of the reduced number of polygons, that is why the silhouette on the concept is really important. But that will be convered on another tutorial

And this is the clean LowPoly

At the end of this Tutorial, you will be able to get this clean lowpoly and build the textures to look like the one above

Usually the Highpoly is made in 3d sculpting programs like ZBrush or Mudbox and it can involve other programs like maya, 3d max, blender...
The workflow of creating the highpoly can vary a lot depending on the artist, some starts from a rough shape, some starts from a sphere/cube and some does almost everything on a more traditional 3d program (maya, max, blender..) and just add some details later.

The Lowpoly is normally created after the Highpoly and follows its topology. While the highpoly's triangles organization are not important, on the LowPoly they are extremelly important, specially if the model is going to be animated

*I am not going to cover the Highpoly or the Lowpoly production here, It will be on another tutorial
Programs used on this tutorial or to create the content
  • Adobe Photoshop
  • Pixologic ZBrush
  • Xnormal (free)
  • 3D Coat
  • Marmoset Toolbag 2
Preparing the bake on XNormal
Before start baking you need to have model with all the polygons not exceeding 4 sides each, called N-Gons. If you do so XNormal will give you a warning message and stop the calculation
Your model is all made of triangles, using quads helps visualize the edge flow and if you need to smooth your mesh the engine will know a proper way to subdivide it. That is why when you count your polycount, you count in triangles and not in polygons

In the end all your models are going to be all triangles inside the game engine, so it is not necessary to worry in keeping everything in quads. I use a lot of triangles in my workflow

You also will need your lowpoly model Unwrapped, which means that you need to project all your faces on a 2d texture. Deformations will appear and it is up to you to hide then or use in places that are not that visible.
*Again, another subject for another tutorial

UV Layout of the model


UV Layout with mirrored parts to save space and have more resolution


Looking the UV Layout now months later, I can see parts where it could be a lot better, but it is part of the evolving process, learning with your mistakes and making it better on the next project

The texture limitations in Dota 2 are sometimes hard to use, it makes the game playable for more people but makes the artist life more challenging. Mirroring the UVs is one of the many solutions you can come up to get a good resolution and still be inside the pre-defined requirements for each asset

to Cage or not to Cage?

The baking process will get your highpoly model and project on the lowpoly. The size or influence of this projection can be determined on XNomal or by creating a Cage.

The cage is a copy of your lowpoly model but with its vertices offseted on the outside convering all the highpoly

Using a Cage will give you more control over the projecting but it will increase the work load. Simple objects doesn't need but on characters I strongly suggest to use one. That way you can keep the higher polygon density (more important parts), like the face, with less deformation and you can fix small problems by tweaking the vertices position
Baking
Enough introduction, lets bake the textures You will need to fill the highpoly and lowpoly paths on XNormal


For those not used to XNormal, this is the interface. And no, is not an winamp skin

This is what you expect to see while it calculates.
I normally follow the cores calculating to look for issues on the first bakes. In my opinion, is the best way to check all your texture.
If you just look at the end with 100% rendered, you may miss some details.


The first bake
Adding the high/low* models and hitting bake with the default option (just the normal map)

*Notice that im am baking just the body of the model, that is why the result is not filling the UV space

As you may noticed, we experience some issues on the texture like this one.
We will need to tweak the numbers to make it work, or use a cage.
You can keep baking with different values on the "Maximum frontal ray distance" and "Maximum rear ray distance" (at the Low definition meshes) and use Photoshop to merge then, but we are going to use the cage to avoid that.

When you know the bake is fine?

This is the result you want to achieve, notice that the normalmap looks fine without errors


I am using 8 pixels of edge padding (expansion of the texture to avoid showing the seams) and the Y axis inverted


You can invert the Y on Photoshop later by inverting the green channel (select the green channel and press ctrl + shift + i)

Why inverting the Y?
that is because source engine work better with that configuration, it will work without inverting it but it may cause some seams showing on mirrored models

Bake is working, now lets bake the other maps

I use to bake:
  • Normal Map
  • Occlusion
  • Bent Normals

And later i will produce:
  • Cavity map
  • Lightbake *
  • Vertical Gradient
  • Vertical Saturation *
* on another tutorial



Other baked maps:

Occlusion


Bent Normal
Organizing the Bakes in Photoshop
Occlusion
Normally I start with the occlusion and put it right over my Value Block group.

And then I set the layer blending of the GROUP to multiply.

Multiply means that it will get the grayscale version of your image and make the whites transparent. So if you have a 50% gray pixel, that pixel will be 50% transparent. A full white one will be 100% transparent.

Here is the occlusion in multiply. I added an orange background to show the transparency



Bent Normals
We are just using the green channel of this one
The bent normals is an unique normal map which is based on the world coordinates, the green one represents the Y axis (vertical one) and you can use it to fake an light from the top of the model.

On this one I set the group blending as SoftLight and I add a layer with a flat gray at 128 128 128 as screen blending
This layer is to control the darker parts of the green channel of the bent normals.
With the opacity of this layer in 100% you will kill all the dark parts, you can slide it down till you think it is ok. On this example I am using it at 80%


Vertical Gradient
this one will be made not on XNormal. There is a lot of ways to do this one, but the easiest one is using a maya plugin called Kodde Ramp[www.kostas.se] created by Kostas Gialitakis[www.kostas.se]

It can be dowloaded here[www.kostas.se]

Remember everytime you use some free amazing script like this on the Dota 2 workshop, to drop an small message to the Valve developers saying you think this guys should be listed as one of the community contributors for that 5% extra share.

a small tip, the default values for the texture will be a gradient from 100% white to 100% black. Tweak a little the black, i normally put 0.1 instead of 0.0. That way i can mask the final texture and delete the extra bits

That will be the result from the script

But if you are doing like me and baking not everything on a single pass, you will probably don't want your texture black on every non used space. That is why I suggested on not using a full black. You can use the magic wand to select the full black areas to delete

Before hitting delete, there is a workflow that I got stuck with it to expand your texture later.
  1. select the black with magic wand (load the uvs over it if necessary just to check)
  2. expand your selection by one pixel (select > modify > expand)
  3. delete
  4. defringe it by one pixel (layer > matting > defringe)
  5. expand using another great script called "Dilate many times" (download here)[docs.unity3d.com]
  6. merge the new layers and dilate again if you think it is not enough




I usually put this layer over the occlusion but you don't need to do exactly the same, actually is really good if you try to experiment by yourself. I do that all the time and each texture is a little different.


Cavity map

Time for another script, this one was shared me by the great artist Andrea "Motenai" Orioli[www.andrea-orioli.com]
download link here[www.dropbox.com]


The cavity map will help with some sharpness on your texture and is great to guide your color masks later

To use this one you will need to add a background of the flat normal color ( 128 128 256 ) to your normal map (you need to duplicate because the script will destroy it)

After duplicating and merging the background with it, go to Photoshop, open the Actions window and over Utilities hit Cavity map





The script will give you 2 layers called "light" and "dark" which are great for controlling the intensity.

I group then over a "Cavity" group with an "Overlay" blending

Those 2 textures come too strong, so it is a good idea to tone the opacity down later. Right now is on 100% but I normally using it around 30%
Final Result
Now it is time to tone down some of the maps
I reduced the Cavity to 30% opacity, the Bent to 78% and Occlusion to 73%

the nice thing of this workflow is that is not destructive, you can change the opacity later on the coloring stage.

there is still one really important pass called Light Bake, but it is not that quick so I will create one tutorial just for that.

Here is our texture created without any 2d skills. Of course it is not finished but painting the colors and values on this one is going to be easier




Suggestions / Critiques

As usual, suggestions and critiques are always welcome. I hope this tutorial is useful and I get some good suggestions for the next one
Conclusion
This tutorial can help you on the texture phase if you don't feel confortable with your 2d skills by giving you a good reference of the model and its volumes.

It also can help on a team production where the animator is waiting the model to start rigging/skinning. Providing him this base will help a lot on understanding the deformation of the mesh. In other words, he doesn't need to wait the finished texture to do the animations.

Download
The files used on this tutorial can be downloaded from my dropbox here[www.dropbox.com]

Full Set Video Presentation
The model used on this tutorial is part of this Dota 2 Set
Contact
For more of my work you guys can find at:
Portfolio
[www.tvidotto.com]
Facebook page
[www.facebook.com]
Faq
Do I need to use Xnormal to bake?
No, you don't, but it is free and works amazingly. You can bake on other programs like Maya or 3d Max

Do I need an powerfull computer?
No, if it plays dota is enough. I work on a notebook (a good one but still a notebook)
32 Comments
Maxxy #FixTF2 Oct 8, 2014 @ 11:05am 
this tutorial is awesome, thank you so much!
Smeagol Sep 27, 2014 @ 6:27pm 
Impressive
Super Novice Sep 27, 2014 @ 7:33am 
nice
SaLAm4nD3R Sep 26, 2014 @ 9:02pm 
Thank you very much for your tips and a great job , congratz!!!
Fernando Sep 26, 2014 @ 8:01pm 
good job
ophee Sep 26, 2014 @ 6:32am 
impressive
Matbodo Sep 25, 2014 @ 8:38pm 
Askum, i have 2nd hand 80 kg full body armor, kevlar and helmet for sale. PM me if you intredasting :) harga boleh nego sampai jadi. Call H/P "0175262614" Juan
背锅侠冰羽 Sep 25, 2014 @ 7:33pm 
nice
おかね Sep 25, 2014 @ 7:12am 
why not use 3ds maX ?
Nope Sep 24, 2014 @ 11:33am 
Very nice job! but sad thing i cant run maya :-(