If I understand correctly, Flamenco has the ability to load all linked libraries in all the slave machines recursively and without the need to pack your blend file or alter it in any way, which I find simply amazing (though I haven't tried it yet). Do you plan to implement such a feature as well? To give you some context, in my daily job (as well as most if not all movie-making pipelines), packing a file at render time is not an option because it would break the asset editing process. Being able to edit linked assets, materials and lights after rendering is crucial, since rendering is precisely the only way to tell what should be edited (at least for materials and lighting). If I pack a file, render an image and then perform some changes on the assets, these changes won't propagate in other files of a larger project as they should, since nothing is linked anymore.
Support Crowdrender!
To see this working, head to your live site.
Search
Dec 10, 2019
· Edited: Dec 10, 2019Non-destructive recursive library syncing
Non-destructive recursive library syncing
3 comments
0
Hi there! Well the truth is we are looking at how best to solve this. Solutions like flamenco are interesting, they tend to be aimed at studios that have large projects to work on. I have heard from many people that F is hard to setup and requires a lot of know how.
Our add-on is meant to be drop dead simple to get running. So it has not adopted this very hands on approach for syncing assets.
Packing is not really what we want to have artists do though, we've talked with other artists and we've gotten mixed responses. Some have told us that they use a NAS/shared folder setup where each node has access to the shared location. In this setup, there's no need to relocate assets since, at render time, all the assets are referred from the NAS/shared folder.
If you can afford to have a dedicated machine or NAS, then in my opinion, this is the most flexible and easiest to debug option. It is not the fastest however, and depends on your network infrastructure as to how much it will lag behind having all the files local on each hard drive.
There are a lot of things we have to weigh up when considering the solution we will roll out. Lots of studios use NAS systems so they don't need this, smaller studios will require something to help with this problem, but then they're not likely to use Flamenco as it requires a tech person on site to setup (again that's me repeating hearsay, I don't know if its still true).
We have not yet decided how we'll deal with this. Though we're leaning towards actively managing the files as flamenco does. We're going to be releasing a full beta next year and part of that will be having the ability to sync computers at remote locations to your master. This obviously requires assets to be relocated and so there's no way out of that, unless studios and other users already have a VPN setup so that the remote sites are part of the VPN.
That said the approach in that case of having a VPN, requires that all computers now run the VPN client and there is a VPN server, so there is more cost with this option and more complexity, now you need your tech expert on site to be able to configure and manage the VPN and the NAS!
We're aiming to build into CR the ability to do this, but we're still designing it because we don't want to mess with the current studio practice of NAS usage. That requires some clever design on our part as we don't want a learning curve with Crowdrender for existing setups, we should be able to just drop in and go.
Hope that helps :)
James
This is already, in many ways, a solved problem for smaller setups with a limited asset library.
If the assets are stored in OneDrive, DropBox, Creative Cloud, etc. and all render nodes are already synced to the relevant cloud provider, then no explicit sync is required. Simply specify the root path to the sync folder for each node and all assets will then be relative to that.
The same solution would work equally well if the shared location is a NAS.
All that's needed is a simple check of hashes to validate that everything is indeed up to date.
Hi Kevin, good point, and yeah, used this approach before, works well. A NAS is a good idea if you're serious about building a home or office render farm, it makes using external assets much, much easier. Using A synced cloud storage folder works just as well though watch out for issues with the path to the shared folder when using services like dropbox etc. The path used on your client machine will be saved into the blend file and then naively used on all the render nodes, so the path to the sync folder has to be exactly the same on each node.