Houdini PDG - TOPs and automation

PDG (Procedural Dependency Graph) is a recently added feature in Houdini. It is meant for building networks (not just chains) of tasks, where each task might have some sort of dependency on others and Houdini then knows how to execute the graph in an optimal way – by executing some operations in parallel or distributing them on multiple machines. It works by processing “work items“. Each work item is passed down the network from node to node and can hold attributes (similar to geometry attributes – strings, numbers etc.). A task in a PDG network can be, for example – cooking a Houdini network and getting it’s result or doing some Houdini unrelated tasks – like executing Python scripts, running  command line applications or even spawning even more work items or modifying their attributes.

In order to create any pipeline with PDG, you must have a way to generate work items and each work item must hold all necessary information. Your PDG pipeline can be just a chain of dependant operations, but the real magic happens if you can execute several things in parallel  – like generating a bunch of game assets or simulating 50 different explosions. If that is the case, you must build your projects in a specific – PDG friendly way.
When executing PDG network, separate Houdini instance gets spawned for processing each work item. Imagine opening HIP file and just having access to attributes stored in one work item.  Based on these attributes, your HIP file should be able to do just that one part of whole work independantly, without doing anything extra – for example generate just one specific asset and nothing more. You do that by referencing PDG work item attributes in your setup. For example, if your pipeline should generate different rocks for a game engine, your node setup must be able to generate any single rock just by receving some PDG work item attributes – like random seed, rock type, size. You can reference work item attributes with @attribname syntax in almost any node parameter. There are also built in work item attributes – such as index, frame, input, output and name.  They are automatically generated. You can access them by writing @pdg_attribname.

Speaking about interacting with Houdini from PDG – you actually can’t just tell it to execute some Houdini network. What you can do is three things:

  1. Cook an exported HDA (Houdini Digital Asset) and get the resulting geometry which you can save or pass (as intermediate file or keep in memory) to other PDG operation along the work item.
  2. Execute (Invoke) Compiled Block inside a HIP file and get it’s resulting geometry like in the first case.
  3. Execute a ROP node (Usually a render operation or geometry export operation). In this case, depending on which ROP node it is executing and how it is set up, it might cook entire Houdini network if it is fed into the ROP node.

In my case I chose to use option number 3 to keep things a bit simpler.

In my pipeline, I base everything on input geometry (blockout) and object names. Therefore I must generate work item for every input mesh I want to process and store it’s name as work items attribute. Then in my SOP setup, I can extract that one piece of geometry and continue working just with that. Because I’m using my own naming convention, the name also contains some additional information that I can extract and use in my setup – like indication if object is a wall rock piece, ceiling rock piece or a small boulder. I also use the same name attribute to generate names for all output textures and final 3d mesh.

My TOP network for generating rocks turned out to be extremely simple – just three nodes:

  1. Geometry Import – to generate work items based on my input geometry.
  2. ROP Fetch – to execute FBX Rop node which in turn cooks my SOP rock generation network.
  3. Python Script – to execute external sbsrender.exe tool which renders my final textures made in Substance Designer.

I’m using Geometry Import TOP node to generate my working items. I’m setting it to iterate over a packed primitive object – this way each primitive represents whole object (single rock piece). I prepared this packed data specially for this – by using the Pack SOP node and using name as a packing attribute. This way each rock piece is now represented by a single primitive inside the packed object. Then I can use the data extraction feature to copy primitive attributes to work item. This way I get the rock piece name as a work item attribute.

Then in my SOP context, I can reference the work item attribute via @name syntax and use it to extract only one rock piece that current work item represents.

Note: It might look a bit confusing, because I’m comparing primitive attribute @name with work item attribute @name. But I guess in this case Houdini is smart enough to understand what is what. To make things more clear, you could just rename the work item attribute to something else – like “workitem_name” and then use comparison expression @name=`@workitem_name`.

After extracting one rock piece to work with, you can forget about PDG. You have just one rock piece with it’s name and that’s it – the rest of your SOP network is built for processing just this one thing. This way, PDG can effectively execute as many parallel work items as your computer memory allows or even among multiple computers if they are available and are configured properly.

Fix ROP pre-render python script:

export_obj = hou.node("../../EXPORT/")
merge_sop = hou.node("../../EXPORT/object_merge1")

#Set object transforms
piece_node = hou.node("/obj/Input_blockout/ROCK_LOW_BAKED")
geo = piece_node.geometry()

name = geo.stringAttribValue("name")
name = name.replace("/","_")
name = name.replace(".","_")

xform = geo.floatListAttribValue("xform")
mat = hou.Matrix4(xform)

export_obj.setWorldTransform(mat)
export_obj.setName(name)

rop_node = hou.pwd()
path = hou.hscriptExpandString("$HIP") + "/unity/" + name + ".fbx"
rop_node.parm("sopoutput").set(path)

As I mentioned, the SOP network is actually called from the ROP Fetch TOP node, which tries to execute a ROP node. In my case this is a Filmbox FBX ROP node, which writes geometry out to a FBX file. You must specify a path to a OBJ node to use as a scene root object. For that I have a special OBJ node with a single Object Merge SOP node inside, which references my finished low poly rock. By having this chain, when PDG tries to exectute the FBX ROP node, it cooks the OBJ node which in turn cooks my rock generation SOP network.

To save each rock piece to a different file name, I use the Pre-render python script feature insinde the FBX Rop node. Another thing I do here is setting the name and  transformation values for the OBJ node. As I mentioned earlier, transformations values got baked in geometry when loading FBX via the File node. To get them back in exported FBX, you need to have them as OBJ level node transformation values. Of course you also need to remove them from the geometry itself, before that. Setting the name of the OBJ node is important too, because FBX ROP uses this name as object name when exporting fbx – and you probably want this name later down the pipeline when using the model in game engine or something like that.

Rendering substance textures

Last step of the pipeline is the baking of substance textures.

For that to happen, we first need to generate all the geometry, bake all the utility textures and save them to disk. Then we can use PDG Python Script node and go through all work items and generate command line parameters for the sbsrender utility, which will take input textures from given paths and generate a bunch of output textures in given paths. We do all that just by knowing the work item name attribute – form that we can generate all the input and output names based on our naming convention.

Then we just execute the sbsrender and wait for it to finish.

import hou
import os, sys
import subprocess

sbsRender = r"C:\Program Files\Allegorithmic\Substance Automation Toolkit\sbsrender.exe"
hip_path = hou.hscriptExpandString("$HIP")
os.chdir(hip_path)

substance_name = r"substance\cave_rocks_B.sbsar"
output_path = r"demo2\substance"

piece_name = pdg.strData(work_item, "name")
piece_path_name = piece_name.replace("/","_").lower()
if '.' in piece_path_name:
    out_path_name = piece_path_name.replace(".","_")
else:
    out_path_name = piece_path_name

out_rez = 10

if "wall_block" in piece_path_name:
    out_rez = 12
elif "top_block" in piece_path_name:
    out_rez = 11
elif "boulder" in piece_path_name:
    out_rez = 10

output_size_param = "\$outputsize@{0},{0}".format(out_rez)
tex_base = r"unity\textures"

commandArgs = ('--engine d3d10pc ' + 
               '--input "{0}" '.format(substance_name) +
               'render ' +
               '--output-path "{0}" '.format(output_path) + 
               '--output-name {0}'.format(out_path_name) + '_{outputNodeName} ' +
               '--set-value "{0}" '.format(output_size_param) + 
               '--set-entry Normal@"{0}" '.format(tex_base + "\\\\" + piece_path_name + "_normal.png") +               
               '--set-entry Bentnormal@"{0}" '.format(tex_base + "\\\\" + piece_path_name + "_bentnormal.png") +               
               '--set-entry Position@"{0}" '.format(tex_base + "\\\\" + piece_path_name + "_position.exr") +
               '--set-entry WorldNormal@"{0}" '.format(tex_base + "\\\\" + piece_path_name + "_worldnormal.png") +
               '--set-entry AO@"{0}" '.format(tex_base + "\\\\" + piece_path_name + "_ao.png") +
               '--set-entry Curvature@"{0}"'.format(tex_base + "\\\\" + piece_path_name + "_curvature.png"))
               
result = subprocess.check_output('"{0}" {1}'.format(sbsRender, commandArgs))

That’s it. PDG will wait until this python script finishes for each work item.

Other stuff

As I mentioned earlier, I’m also generating the stalagmites on the ground – each as a separate unique object. Here my setup is very similar – I have another TOP network for that, which is almost exactly the same as one for generating rocks. Only In this case I’m creating my work items by iterating over stalagmite seed points and use different naming convention for everything.

For the Collector demo I used Houdini to generate other assets too – like ground sand mesh, destroyed columns and vector fields for particle effects. But because these are single objects, there was no point in using PDG for that and I used simple manual approach where I bake and export everything manually.

Last thing I wanted to add is that because Houdini is so awesome, I can have everything inside a single HIP file – all my subnetworks that do different things. It’s very very handy to have everything together.

Unity Import Automation

This is unrelated to Houdini, but importing everything in Unity and setting up all the materials and mesh parameters, can also be a tedious task – especially if your pipeline is generating large amount of assets. What you can do is to write a custom asset import script called AssetPostprocessor
that gets executed in Unity Editor whenever an asset is being imported or re-imported. In this script, you can easily add your custom asset processing logic – for example, I added automatic material generation based on imported objects name and automatic texture assignment to that material. I also added a feature that checks if texture name contains the string “_normal” and if it does, marks the texture types as normal map. For fbx models, I disable loading animation and creating a rig component for all the static assets.
This way you an create fully automated import pipeline as long as you stick to a predefined naming convention.

CaveAssetPostprocess.cs (Unity editor script file sample):
using System.IO;
using UnityEditor;
using UnityEditor.Presets;
using UnityEngine;

public class CaveAssetPostprocess : AssetPostprocessor
{
    void OnPreprocessModel() 		
    {
       if(assetPath.Contains("Cave/Cave_") || assetPath.Contains("Cave/stalagmite_"))
       {
            ModelImporter importer = assetImporter as ModelImporter;
            importer.generateSecondaryUV = true;
            importer.animationType = ModelImporterAnimationType.None;
		    importer.importAnimation = false;
       } 
    }

    void OnPreprocessTexture()
    {
        if (assetPath.ToLower().Contains("_normal.") || assetPath.ToLower().Contains("_bentnormal."))
        {
            TextureImporter textureImporter = (TextureImporter)assetImporter;
            textureImporter.textureType = TextureImporterType.NormalMap;
            textureImporter.convertToNormalmap = false;
            textureImporter.maxTextureSize = 4096;
        }

    }

    Material OnAssignMaterialModel(Material material, Renderer renderer)
    {
        //Automatic material creation of cave pieces
        if (assetPath.Contains("Cave/Cave_") && assetPath.Contains(".fbx"))
        {
            string pieceName = assetPath.Substring(assetPath.LastIndexOf("/") + 1);
            pieceName = pieceName.Substring(0, pieceName.Length - 4);
            //Debug.Log("Importing cave piece!! " + pieceName);

            string mtlName = pieceName + ".mat";
            string[] mtls = AssetDatabase.FindAssets("mtlName", new[] { "Assets/_SCENE/Props/Cave/Materials" });

            Material pieceMtl = null;
            if (mtls.Length == 0)
            {
                pieceMtl = new Material(Shader.Find("HDRP/LitTessellation"));
                pieceMtl.name = pieceName;
                AssetDatabase.CreateAsset(pieceMtl, "Assets/_SCENE/Props/Cave/Materials/" + pieceName + ".mat");
            }
            else
                pieceMtl = (Material)AssetDatabase.LoadAssetAtPath(mtls[0], typeof(Material));

            if (pieceMtl != null)
            {
                Texture2D baseColor = (Texture2D)AssetDatabase.LoadAssetAtPath("Assets/_SCENE/Props/Cave/Textures/" + pieceName + "_basecolor.png", typeof(Texture2D));
                Texture2D normalMap = (Texture2D)AssetDatabase.LoadAssetAtPath("Assets/_SCENE/Props/Cave/Textures/" + pieceName + "_normal.png", typeof(Texture2D));
                Texture2D bentNormal = (Texture2D)AssetDatabase.LoadAssetAtPath("Assets/_SCENE/Props/Cave/Textures/" + pieceName + "_Bentnormal.png", typeof(Texture2D));
                Texture2D masks = (Texture2D)AssetDatabase.LoadAssetAtPath("Assets/_SCENE/Props/Cave/Textures/" + pieceName + "_masks.png", typeof(Texture2D));

                pieceMtl.SetTexture("_BaseColorMap", baseColor);
                pieceMtl.SetTexture("_NormalMap", normalMap);
                pieceMtl.SetTexture("_BentNormalMap", bentNormal);
                pieceMtl.SetTexture("_MaskMap", masks);

                pieceMtl.EnableKeyword("_NORMALMAP_TANGENT_SPACE");
                pieceMtl.EnableKeyword("_NORMALMAP");
                pieceMtl.EnableKeyword("_BENTNORMALMAP");
                pieceMtl.EnableKeyword("_MASKMAP");

                pieceMtl.shader = Shader.Find("HDRP/Lit");

                return pieceMtl;
            }
            else return material;
        }

        if (assetPath.Contains("Cave/stalagmite_") && assetPath.Contains(".fbx"))
        {
            string pieceName = assetPath.Substring(assetPath.LastIndexOf("/") + 1);
            pieceName = pieceName.Substring(0, pieceName.Length - 4);

            string mtlName = pieceName + ".mat";
            string[] mtls = AssetDatabase.FindAssets("mtlName", new[] { "Assets/_SCENE/Props/Cave/Materials" });

            Material pieceMtl = null;
            if (mtls.Length == 0)
            {
                pieceMtl = new Material(Shader.Find("HDRP/LitTessellation"));
                pieceMtl.name = pieceName;
                AssetDatabase.CreateAsset(pieceMtl, "Assets/_SCENE/Props/Cave/Materials/" + pieceName + ".mat");
            }
            else
                pieceMtl = (Material)AssetDatabase.LoadAssetAtPath(mtls[0], typeof(Material));

            if (pieceMtl != null)
            {
                Texture2D baseColor = (Texture2D)AssetDatabase.LoadAssetAtPath("Assets/_SCENE/Props/Cave/Textures/stalagmites/" + pieceName + "_basecolor.png", typeof(Texture2D));
                Texture2D normalMap = (Texture2D)AssetDatabase.LoadAssetAtPath("Assets/_SCENE/Props/Cave/Textures/stalagmites/" + pieceName + "_normal.png", typeof(Texture2D));
                Texture2D masks = (Texture2D)AssetDatabase.LoadAssetAtPath("Assets/_SCENE/Props/Cave/Textures/stalagmites/" + pieceName + "_masks.png", typeof(Texture2D));

                pieceMtl.SetTexture("_BaseColorMap", baseColor);
                pieceMtl.SetTexture("_NormalMap", normalMap);
                pieceMtl.SetTexture("_MaskMap", masks);

                pieceMtl.EnableKeyword("_NORMALMAP_TANGENT_SPACE");
                pieceMtl.EnableKeyword("_NORMALMAP");
                pieceMtl.EnableKeyword("_MASKMAP");

                pieceMtl.shader = Shader.Find("HDRP/Lit");

                return pieceMtl;
            }

        }

        return material;

    }

}

THE END

I hope this was useful to someone. This was not meant as an exact step-by-step tutorial on how to create something, but more like general overview about all the different steps  required to put together a pipeline like that and reasoning behind those decisions. I spent a lot of time going through all of this by trial and error. I hope this allows for you to save some time.

Don’t hesistate if you have some questions or suggestions – I would be happy to answer them.

Post A Comment