Smarter Project Structure

It is very hard to come up with project folder structure that works. Especially if you are using multiple content creation applications. You often want to use the same assets in different places. Usually you end up with a huge mess – files from different apps mixed together with temporary files and hard to use relative paths ( like ../../../scene/textures/bricks.jpg) or even worse – absolute paths that break as soon as you move project files around.
Another issue is when it comes time to clean things up for archiving – everything is mixed together. It’s hard to know which files, sequences and temporary files are safe to remove (that can be easily re-generated if you need to work on the project in the future).

I try to solve all of these issues with clever use of symlinks (symbolic links). It’s a feature that is common in Linux/Unix/Mac based systems (but less known on Windows), where you can create “virtual” folders or files that actually point to other folders or files that are located elswhere. Other way to think about it – contents of your folder can be accessed from different places (each place being a folder with it’s own path). Like shortcuts, but invisible to user – creating illusion of actual folders or files existing in that location.

Imagine having a folder in your project D:\MyProject\Textures that actually shows content of completely different folder C:\AssetLibrary\Textures\Best.
This allows using relative paths to textures in your project (D:\MyProject\scene.blend referencing Textures\Brick.jpg).
This will work although in reality Brick.jpg is located in C:\AssetLibrary\Textures\Best.It’s pretty cool, right?

To simplify everything, I assume the following workflow:

• You build Assets (Props, Characters, Environments etc.) and export (publish) final usable results (fbx, usd, abc, blend etc.) files in shared assets folder.
• Each asset work files live in separate folder
  • Each asset or shot has a separate working folder (like a mini project) for each content creation app with a different internal folder structure suited for that app.
• You should never reference files between work folders of different assets or shots directly
• You only link or import thse finished assets into your shots or other assets scenes through shared folder.
  • Each asset or shot work folder contains links to shared folders – as if they are local in the work folder.
  • Each asset or shot work folder also contains links to shared render and temp folders where you can store rendered sequences or differenct caches

It might be easier to understand by looking at a very simplified sample project folder structure.

Notice how folders inside Build/Asset_a , Build/Asset_B and Shots/Shot_101 point to the same folders under shared Assets/3D/Fbx and Render folders.
It means if Asset_a exports a FBX file to Asset_A/Fbx folder, the FBX file will be visible also in Asset_B/Fbx folder as well as Shot_101/Fbx folder. This is because exported FBX file actally is in Assets/3D/Fbx folder. All other Fbx folders are just symlinks to this folder.

Here is a more complete explanation of possible folder structure. Of course it can be completely different based on your work style, but what’s important is the main idea of separating everything in mini-project folders that share assets through symlinks to shared folders.

ProjectRoot
	Assets  [Assets shared between shots and apps]
			[ Don't use directly. Linked in work folders ]
		2D
			Artworks
		3D
			Fbx
				Ext_Models [link to External models library - i.e. photogrammetry folder on a server]
			Bgeo
			Obj
			Alembic
			USD
			C4D
			Blend
		Tex
			Ext [link to External texture library - i.e. texture library on a server]
		Lib
			Python_Libs

	Build [ Work folders for buildig assets, scenes, etc. ]
		Asset_Name_A
			[Houdini] - Houdini project
				Link: Assets/Tex
				Link: Assets/3d/Fbx,Bgeo,Obj,Alembic,USD
				Link: Render
				Link: Temp/Assets/Asset_Name_A Flip Geo Sim
			[Blender] - Blender project
				Link: Assets/Tex
				Link: Fbx,Obj,Alembic,USD
			[C4D] - C4D project
				Link: Assets/Tex
				Link: Fbx,Obj,Alembic,USD
			[Other] - Other 3d dcc app projects if needed (Zbrush, Marvellous etc.)
			Textures - Textuiring work (PSD files, Substance project etc.)
				Link: Shared/Tex
				
		Asset_Name_B
			...
			
	Shots [ Work folders for shots/scenes. Similar structure to asset work folders]
		Shot_XXYY 
			[Blender]
				Link: Assets/3d/fbx, obj, bgeo, alembic, Assets/Tex..
			[Houdini]
				Link: Assets/3d/fbx, obj, bgeo, alembic, Assets/Tex..
				Link: Temp/ShotX
				Link: Temp/Shared
			[Maya]
			[C4D]
			
			Comp [ Compositing projects - links folders from Render, 2D Assets, Preivs, Out]
				Link: Shared/2D/Artworks (Images needed for compositing)
				Link: Render/Shots/Shot_XXYY
				Link: Render/Previs/Shot_XXYY
				Link: Out (Project main output folder (deliver, images, previs, review)

	Render [Rendered Image sequences. Can remove when archiving to save space]
		Assets
			Asset_Name_A
			Asset_Name_B
		Previs
			Shot_SEQXXYY
		Shots
			Shot_SEQXXYY
	
	Out [ Valuable project outputs - easy to find in one place ]		
		Previs [all previs videos]
		Review [all "final" videos]
		Images [stills, screenshots, making of pics etc]
		Deliver - place for final deliveries
		
	Temp [ All temporary files - caches etc. Remove when archiving or saving space ]
		 [ Don't use directly. Linked in work folders ]
		Assets
			Asset_Name
				Flip
				Geo
				Sim
		Shots
			Shot_XYY
				Flip
				Geo
				Sim

There are actually several link types:

• symbolic links – this is like a shortcut. A redirect to another path. It does not point to data on disk directly, but to another path. Can also point to another symbolic link – creating a chain of links.
• hard links – points to data on disk directly. But data can only be on the same volume – that is you can’t make a hardlink to data on another disk drive. If you move data to another location (on the same volume) – all hardlinks pointing to it will still work. It’s like an actual file or folder.
• directory junctions – similar to hard links, but uses different implementation to allow making links to data in other volumes.

In our case we are more interested in symbolic links as they are more flexible and less potentially destructive.

While fully supported, symlinks are a bit of a hidden feature in Windows. To use symlinks on Windows, you must have NTFS filesystem and enable developer mode in windows settings (Update & Security -> For developers -> Developer Mode : ON). Unfortunately there is no support for creating symlinks in Windows Explorer and you must use console command mklink link target.
Here is how you use it:
mklink c:\Folder\file.txt  D:\Stuff\some_file.txt – creates file symbolic link c:\Folder\file.txt that points to a file D:\Stuff\some_file.txt
mklink /d D:\ProjectA\Assets\Textures  F:\Lib\Assets\Tex – creates folder symbolic link

It is also important to mention that you can create symlinks with relative paths – this is handy if you want to move your project to another location, so all relative symlinks will still point to correct folders inside your project structure.

As you can imagine, creating this folder structure by hand can be quite time consuming – especially if you need to create all symlinks by hand using console commands.

Fortunately, If you create your symlinks with relative paths, they will still work if you copy your project to another location. This way you could create template project and just copy it to new location whenever you need to make a new project. But still – whenever you need to add new asset or shot, you would have to manually copy a sub-template folder structure for an asset or shot. Quite cumbersome and error prone.

Better solution would be to have a python script that does all of this automatically. This way you just have to write the python code once and never waste time on making folders by hand. Also it is easier to make changes or add new features to your project structure – even mid project. Once we are there, we can add more and more features – start integrating our script inside 3D applications and at the end we have ourselves a small pipeline toolset.

This is exactly what I did. I started out by making a script to automate folder structure creation. Then I added a GUI project manager. It allows configuring which 3D apps you want to use for the project and will create work folders only for those apps.
After that I added integrations for Houdini and Blender – so we can run these apps with pre-configured environment for our project and have a project manager GUI available from inside the 3d application. I was planning to add support for Maya and C4D also, but these days I’m not using these apps so frequently anymore so there is not much motivation at this point ( although it should be quite easy ).

I will not go into technical details – you can get the source from GitHub and use it yourself. The only thing I wanted to mention is that year 2021 is the year when all DCC apps finally support Python 3 and that makes everything so much simpler. It is a good time to dive into world of Python if you have avoided it in the past!

I built the Project Manager GUI using PySide2 (Using Qt Designer). If run as a standalone app, it allows basic project creation, asset and shot creation (creation of folder structure and making needed links), as well as general project configuration – including linking external asset libraries under shared assets folder.

You can also configure and launch 3D applications from the last tab (currently just Blender and Houdini is supported). When doing that, it preconfigures application environment to the active project and adds pipeline python scripts to python path – opening integrated Project Manager window inside 3D application.

Once Project Manager is open from inside 3d application, some additional options become available – like ability to create and open scene files for assets and shots. While doing this, it also configures environment (like setting JOB environment variable in Houdini to asset/houdini root folder).

Just one more thing – Houdini has an awesome feature called Python Panels. This allows running your custom Python PySide/PyQt application as a panel inside Houdini UI with basically two lines of code. With Blender it was not so easy, as Blender does not have a built in PySide/PyQt support. But with a little help of the internet, I managed to do it anyway.

Please let me know if this topic is interesting to you.