The Problem
It is very hard to come up with project folder structure that works. Especially if you are using multiple content creation applications. You often want to use the same assets in different places. Usually you end up with a huge mess – files from different apps mixed together with temporary files and hard to use relative paths ( like ../../../scene/textures/bricks.jpg
) or even worse – absolute paths that break as soon as you move project files around.
Another issue is when it comes time to clean things up for archiving – everything is mixed together. It’s hard to know which files, sequences and temporary files are safe to remove (that can be easily re-generated if you need to work on the project in the future).
I try to solve all of these issues with clever use of symlinks (symbolic links). It’s a feature that is common in Linux/Unix/Mac based systems (but less known on Windows), where you can create “virtual” folders or files that actually point to other folders or files that are located elswhere. Other way to think about it – contents of your folder can be accessed from different places (each place being a folder with it’s own path). Like shortcuts, but invisible to user – creating illusion of actual folders or files existing in that location.
Imagine having a folder in your project D:\MyProject\Textures
that actually shows content of completely different folder C:\AssetLibrary\Textures\Best
.
This allows using relative paths to textures in your project (D:\MyProject\scene.blend
referencing Textures\Brick.jpg
).
This will work although in reality Brick.jpg is located in C:\AssetLibrary\Textures\Best.
It’s pretty cool, right?
Project Structure
To simplify everything, I assume the following workflow:
- You build Assets (Props, Characters, Environments etc.) and export (publish) final usable results (fbx, usd, abc, blend etc.) files in shared assets folder.
- Each asset work files live in separate folder
- Each asset or shot has a separate working folder (like a mini project) for each content creation app with a different internal folder structure suited for that app.
- You should never reference files between work folders of different assets or shots directly
- You only link or import thse finished assets into your shots or other assets scenes through shared folder.
- Each asset or shot work folder contains links to shared folders – as if they are local in the work folder.
- Each asset or shot work folder also contains links to shared render and temp folders where you can store rendered sequences or differenct caches
Sample Project Folder Structure
It might be easier to understand by looking at a very simplified sample project folder structure.
Notice how folders inside Build/Asset_a
, Build/Asset_B
and Shots/Shot_101
point to the same folders under shared Assets/3D/Fbx
and Render
folders.
It means if Asset_a
exports a FBX file to Asset_A/Fbx
folder, the FBX file will be visible also in Asset_B/Fbx
folder as well as Shot_101/Fbx
folder. This is because exported FBX file actally is in Assets/3D/Fbx folder
. All other Fbx folders are just symlinks to this folder.
Here is a more complete explanation of possible folder structure. Of course it can be completely different based on your work style, but what’s important is the main idea of separating everything in mini-project folders that share assets through symlinks to shared folders.
ProjectRoot Assets [Assets shared between shots and apps] [ Don't use directly. Linked in work folders ] 2D Artworks 3D Fbx Ext_Models [link to External models library - i.e. photogrammetry folder on a server] Bgeo Obj Alembic USD C4D Blend Tex Ext [link to External texture library - i.e. texture library on a server] Lib Python_Libs Build [ Work folders for buildig assets, scenes, etc. ] Asset_Name_A [Houdini] - Houdini project Link: Assets/Tex Link: Assets/3d/Fbx,Bgeo,Obj,Alembic,USD Link: Render Link: Temp/Assets/Asset_Name_A Flip Geo Sim [Blender] - Blender project Link: Assets/Tex Link: Fbx,Obj,Alembic,USD [C4D] - C4D project Link: Assets/Tex Link: Fbx,Obj,Alembic,USD [Other] - Other 3d dcc app projects if needed (Zbrush, Marvellous etc.) Textures - Textuiring work (PSD files, Substance project etc.) Link: Shared/Tex Asset_Name_B ... Shots [ Work folders for shots/scenes. Similar structure to asset work folders] Shot_XXYY [Blender] Link: Assets/3d/fbx, obj, bgeo, alembic, Assets/Tex.. [Houdini] Link: Assets/3d/fbx, obj, bgeo, alembic, Assets/Tex.. Link: Temp/ShotX Link: Temp/Shared [Maya] [C4D] Comp [ Compositing projects - links folders from Render, 2D Assets, Preivs, Out] Link: Shared/2D/Artworks (Images needed for compositing) Link: Render/Shots/Shot_XXYY Link: Render/Previs/Shot_XXYY Link: Out (Project main output folder (deliver, images, previs, review) Render [Rendered Image sequences. Can remove when archiving to save space] Assets Asset_Name_A Asset_Name_B Previs Shot_SEQXXYY Shots Shot_SEQXXYY Out [ Valuable project outputs - easy to find in one place ] Previs [all previs videos] Review [all "final" videos] Images [stills, screenshots, making of pics etc] Deliver - place for final deliveries Temp [ All temporary files - caches etc. Remove when archiving or saving space ] [ Don't use directly. Linked in work folders ] Assets Asset_Name Flip Geo Sim Shots Shot_XYY Flip Geo Sim
Symlinks on Windows
There are actually several link types:
- symbolic links – this is like a shortcut. A redirect to another path. It does not point to data on disk directly, but to another path. Can also point to another symbolic link – creating a chain of links.
- hard links – points to data on disk directly. But data can only be on the same volume – that is you can’t make a hardlink to data on another disk drive. If you move data to another location (on the same volume) – all hardlinks pointing to it will still work. It’s like an actual file or folder.
- directory junctions – similar to hard links, but uses different implementation to allow making links to data in other volumes.
In our case we are more interested in symbolic links as they are more flexible and less potentially destructive.
While fully supported, symlinks are a bit of a hidden feature in Windows. To use symlinks on Windows, you must have NTFS filesystem and enable developer mode in windows settings (Update & Security -> For developers -> Developer Mode : ON). Unfortunately there is no support for creating symlinks in Windows Explorer and you must use console command mklink link target
.
Here is how you use it:
mklink c:\Folder\file.txt D:\Stuff\some_file.txt
– creates file symbolic link c:\Folder\file.txt
that points to a file D:\Stuff\some_file.txt
mklink /d D:\ProjectA\Assets\Textures F:\Lib\Assets\Tex
– creates folder symbolic link
It is also important to mention that you can create symlinks with relative paths – this is handy if you want to move your project to another location, so all relative symlinks will still point to correct folders inside your project structure.
Automation with Python
As you can imagine, creating this folder structure by hand can be quite time consuming – especially if you need to create all symlinks by hand using console commands.
Fortunately, If you create your symlinks with relative paths, they will still work if you copy your project to another location. This way you could create template project and just copy it to new location whenever you need to make a new project. But still – whenever you need to add new asset or shot, you would have to manually copy a sub-template folder structure for an asset or shot. Quite cumbersome and error prone.
Better solution would be to have a python script that does all of this automatically. This way you just have to write the python code once and never waste time on making folders by hand. Also it is easier to make changes or add new features to your project structure – even mid project. Once we are there, we can add more and more features – start integrating our script inside 3D applications and at the end we have ourselves a small pipeline toolset.
RE Project Manager
This is exactly what I did. I started out by making a script to automate folder structure creation. Then I added a GUI project manager. It allows configuring which 3D apps you want to use for the project and will create work folders only for those apps.
After that I added integrations for Houdini and Blender – so we can run these apps with pre-configured environment for our project and have a project manager GUI available from inside the 3d application. I was planning to add support for Maya and C4D also, but these days I’m not using these apps so frequently anymore so there is not much motivation at this point ( although it should be quite easy ).
I will not go into technical details – you can get the source from GitHub and use it yourself. The only thing I wanted to mention is that year 2021 is the year when all DCC apps finally support Python 3 and that makes everything so much simpler. It is a good time to dive into world of Python if you have avoided it in the past!
GITHub Project:
I built the Project Manager GUI using PySide2 (Using Qt Designer). If run as a standalone app, it allows basic project creation, asset and shot creation (creation of folder structure and making needed links), as well as general project configuration – including linking external asset libraries under shared assets folder.
You can also configure and launch 3D applications from the last tab (currently just Blender and Houdini is supported). When doing that, it preconfigures application environment to the active project and adds pipeline python scripts to python path – opening integrated Project Manager window inside 3D application.
Once Project Manager is open from inside 3d application, some additional options become available – like ability to create and open scene files for assets and shots. While doing this, it also configures environment (like setting JOB environment variable in Houdini to asset/houdini root folder).
Just one more thing – Houdini has an awesome feature called Python Panels. This allows running your custom Python PySide/PyQt application as a panel inside Houdini UI with basically two lines of code. With Blender it was not so easy, as Blender does not have a built in PySide/PyQt support. But with a little help of the internet, I managed to do it anyway.
Please let me know if this topic is interesting to you.