V0.7.0 - our render farm software's next feature update
Some big upgrades to our render farm software 🎉
As you can see from the image below, we've been busy in the month of November, getting our render farm software's latest feature update ready for an alpha release to our studio members (click here if you'd like to support us by becoming a subscriber.
The following is a quick list of the updates that V0.7.0 will bring and why this update is one of the most important we've ever done.
First, speed, in a recent test we did, on a SINGLE computer, V0.7.0 rendered 1000 frames of a test scene a full 42 minutes FASTER than Blender on its own 😱🫨 This is our 🎉NEW🎉speed boost feature in action!
Next, we fixed an absolute🤬🤬of an issue that was causing some longer render jobs to stop returning frames to the client machine. After a LOT of testing and a huge effort to redesign how data is transferred between computers, we can now render thousands of frames on exactly the same project that could barely get past 400 frames before progress on the render seemed to stop.
Finally, we have done a MASSIVE update internally to prepare the way for you guys being able to share your hardware with each other, peer to peer style. This update has been redesigned so that it now uses a different architecture for communicating between computers. Though V0.7.0 won't be able to let you share GPUs/CPUs directly with your friends yet, this is the ground work.
Why this update is so important ❗
If you've been following our project since the beginning, which is roughly 2017 when we first launched our website, then you'll be aware that our goal is to build a system capable of allowing direct, peer to peer sharing of hardware for rendering.
That goal is alive and well. But, there are many steps to get there. V0.7.0 is perhaps the most important so far because it introduces the main architectural concept that we're building our CRESS (Compute Resource Sharing System) on.
That is event driven architecture.
V0.7.0 uses an event bus to replace the old architecture 🏗️
The current architecture of our addon evolved from a basic idea of a simple client, server architecture. Each computer in CrowdRender is both a client and a server so that you don't have to specifically tell each machine which role it is playing. You only need to connect, sync and render, the details about who is what kind of thing are abstracted away and managed for you. This worked well enough for many years, but there was always a limitation on how fast we could develop new features, and some that we'd likely not be able to do at all, at least not well, and we'd be putting at risk our ability to consistently update and maintain the addon.
The biggest issue we faced with the current architecture though, is that it is monolithic. In monolithic architectures, different parts of the code tend to depend on details of many other parts, especially those they interact with. This is a problem when you want to change things, suddenly you find that making one small change means you have to carefully update many other parts of the code, or things break.
One of the biggest benefits of using an event driven architecture is that interactions between different parts of the system are now done using events rather than through function or method calls. In non technical speak, this allows the code in one module to not care what is going on elsewhere, it no longer depends on the internal details of other modules. This decoupling means we can make changes more easily, maintenance is easier, building new features is surprisingly easier, things get built to a better standard, all good stuff basically.
Preparing the way for CRESS 🌱
CRESS will eventually replace the core of our addon, it will manage connections between computers automatically, synchronisation of data, distribution of rendering work and collection and aggregation of rendered tiles and frames. The biggest benefit of CRESS is that it supports sharing computers directly over the internet, peer to peer style, similar to peer to peer file sharing programs.
The V0.7.0 update is super important because now the core of our addon has been re-written to use our implementation of event driven architecture, the event bus. This means that we've been able to extensively test the same concept that we're using in CRESS. Its solved many design issues we've had and has made upgrades far easier.
V0.7.0 will ship with this new architecture meaning we're now back on track to start offering more upgrades and eventually ship CRESS so you can start collaborating with each other, sharing hardware through the internet for rendering!
Fixing a really nasty and random 🦟bug🪳and how that got us here
In 2023 we had a couple of reports that our addon had stopped rendering before the job had finished. At first we couldn't reproduce the issue with our systems and the set of blend projects we use for testing. Then we found a combination of settings that produced what we thought was affecting these users.
For projects where a LOT of render passes were enabled, we could cause a very similar issue to happen where finished frames were not being sent to the client machine. To the user, this would look like the render had stopped. Though in our case, each computer continued to render, though, the finished frames just sat uselessly on each computer and never arrived on the master/client machine where you could see them, and Blender could mark them as done, allowing it to move onto the next frame.
This kicked off an investigation which turned into one of the biggest refactors we've ever done. Our apologies if we've been a bit quiet and haven't put out many updates, it has been quite the journey to get to this point.
Anyway...
We found that the code which transfers the frames/tiles from workers to your client or master computer was freezing. It was odd as it could render thousands of frames of the same project (which you can download from Blender's demo pages, we used the BMW scene) with the default settings, which is what we'd been using for testing.
It was doubly odd that the point at which the failure happened was different each time. The conditions seemed to be that if a certain amount of data was being consistently pushed through the system, it would eventually hang.
We looked at repairing the module that implements our file transfer, but, after a lot of work debugging the system and seeing its limitations, we decided that the module should be replaced. This was the excuse we'd been waiting for to try to create our first module using the event bus. But, like someone pulling a thread on a sweater, it soon became clear that as we modified one module to use the event bus, it became clear that it would be better to just refactor the majority of the core services. Time will tell if we were right on the call.
The client, network file transfer and worker modules were all substantially re-designed to take advantage of the new architecture. The details of why we did this can be summarised in the benefits.
The benefits of Event Driven Architecture in our system ⚡
The new design allows us make changes to one feature with very little to, in some cases, no impact on others. In the pre-0.7.0 architecture, each module was highly dependent on many of the other modules, which meant changing things was a slow and painstaking process, error prone, and also quite restrictive.
Taking logging as an example (see the image above), it's now trivial to create a logging service that can log all the events running through our system in a few lines. To do this before, calls to a logging system would need to be scattered throughout each module, and disabling or modifying them is a chore in comparison.
New methods of distributing render jobs can be created with ease, and all sorts of actions can be triggered based on any event in the system. For example, say you want to run your finished frames through a compositor when you're done, its now trivial to setup a service that can call on a compositing program once a frame is finished. That program doesn't even have to be Blender, it could be any program that has the ability to be manipulated from the command line.
Debugging and maintenance is now a lot easier. Since the changes are usually confined to what are now very small components of our system, the number of changes we have to review and test is dramatically smaller. This means the limited amount of human analysis we have isn't spread across lines and lines of code across multiple files.
There are some drawbacks, but they've so far not really been noticeable. The main drawbacks are;
Latency is higher,- events take a bit more time to process when compared to one component making a function or method call to another. So far though, the latency hasn't prevented us rendering faster than the previous architecture.
Debugging is more complex - normally, when debugging code where one function calls another, development tools can follow each step of the program. Events are not as easy! An event can be published in one component and then acted on in any number of other areas and the debugging tools don't follow the event, only function calls.
So far though, despite some frustration and a big learning curve, the drawbacks have been 'affordable' in that they're worth it!
Faster than Cycles on one machine !!! ⚡🚀... sometimes!
We gathered render data from 1000 frames, rendered consecutively in one job, first with V0.7.0, then Cycles and finally V0.6.7 for a comparison on how the new event bus powered addon works. What we found was pretty impressive! V0.7.0 was much faster than Cycles for the same render job, and, we were only using one computer to do all this!
Caveats, you'll be wanting a lot of RAM 🖥️
This was somewhat unexpected! Also, we want to be upfront and say that in our tests, the speed boost feature that has given us these results depends greatly on having enough ram. Speed boost can use up a LOT of system memory. We've seen easily double that compared to using the previous build (V0.6.7 at the time of writing this article).
Because of the ravenous ram appetite, we've seen other tests show that speed boost can slow a render down relative to cycles and the previous builds. This happens where there isn't enough system RAM. Once your computer can't fit all the working processes in memory, performance falls off a cliff. So its best to budget double the system RAM that your OS's process monitoring tool reports for the process that is rendering.
When budgeting how much RAM you expect to use, please don't go off what you see in Blender (see image below). The stats you see printed when rendering only provide the memory used by the render engine, Blender is also using memory to store the scene for editing purposes. When you render, Blender will consume significantly more RAM than you see reported in the image editor. In our tests, though Blender reports a peak of 327MB, our process monitoring tool showed an actual memory consumption of nearly 1GB, some three times the reported value.
A trick for hybrid rendering using the addon 🪄
Those of you who've used Cycles may know of its hybrid mode where the CPU and GPU render together. In our experience this has mixed results, sometimes its quicker, sometimes not.
We often render in our own hybrid mode using a trick to render with both the CPU and GPU independently with results that usually are better than Cycles' own hybrid mode.
The trick is simple enough, you use two render nodes on the same computer. To explain this better, first realise that when the addon starts, a node called "Local" is loaded into your list by default. This node represents the computer you are working on, often we call this the 'master' or 'client'. In our tests, we setup this node to use only the CPU, we also reduce the number of threads it can use to 20 out of 32 so the system has pleanty of processing power for getting the next frame or tile ready to render.
The tricky part is creating a second node that will actually run on the master/client machine. To do this we add a new node, then give it the same name and IP address as the computer we're working on. Usually if some form of DNS is running on your network, or you use windows, just giving the new node the same name as its host or computer name is sufficient as the addon will lookup the ip address for you (you can stil type it in manually if you want).
The second node (you can see it in the image above, called "RYZEN-NINJA") is configured to only use the GPU. When rendering, one node uses the CPU, the other uses the GPU. Since they are both rendering different tiles or frames, this can often be faster than Cycle's own hybrid rendering, though not always, it does depend on the job, and your hardware. We're lucky to have a very capable Ryzen 3950X with 32 threads, and a now aging but still quick RTX2070 Super.
The results 📈
Where did the extra performance come from??
The chart about shows the speed difference of V0.7.0 to rendering without the addon using Cycles. As you can see, it finished significantly ahead, shaving 42 minutes off the time of rendering the same test without CrowdRender. Most of this is thanks to Speed boost being enabled, and some thanks to the hybrid rendering trick described above.
The time saving per frame over the fastest Blender could complete the job on its own (which was with Cycles using only the GPU) was 15.6 seconds per frame down to 13.56 seconds per frame. That is a saving of 13% BEFORE you start adding other computers 💪 just make sure you have pleanty of system ram 🖥️
The technique we're using is also quite new to us, so we may be able to improve on this as we refine our code. For now, you get a decent speed up over Blender's native performance, even if you don't have more than one computer!
How to get V0.7.0 💾⏬
Eventually V0.7.X will be released and available from our downloads page, right now, its available to our studeo level subscribers who have access to our slack channel. We're refining the eventual stable release as V0.7.0 is currently in alpha, with beta planne in the next month and stable release available for pro subscribers in Jan 2025.
If you'd like to get in on the alpha and beta phases of testing, there's a couple of options.
If you're an existing subscriber, we can help you upgrade (you can't currently upgrade automatically, just email us at [email protected] and we'll upgrade your account) to the studio tier, you'll get access to developer chat as well as being able to try the latest builds.
If you're not currently a subscriber, then you can simply click here and you'll be presented with options to subscribe, choose the studio tier and then choose whether you would like to go montly or yearly. Subscriptions have no lock in period, or cancellation fees, you can sub for as long or as short as you like.
A major improvement 💪
To sum up all the above, in our opinion, this update represents a significant step forward in terms of not just peformance but also potential. The code we're working with feels far more like its helping us make a great tool, rather than getting in the way. That alone gives us confidence that the future of this tool is looking really bright.
What you can expect from here is that we'll be working with our studio tier subscribers to finalise a release candidate. After that there will be regular updates to this, and CRESS will become the substantial new project we're working on. We'll be gaining confidence with the new event driven architecture, learning its pros and limitations intimately as we build a system worthy of providing peer to peer sharing of computer power for 3d enthusiasts and studios everywhere 😀
Comments