I have two machines, Master and slave. It can distribute the render as expected and I can set the slave to either Cuda or CPU. For the master it does not seem to matter whether I set it to Cuda or CPU it will only render on the cpu? Also, the combined render is much slower then an individual render ?
Mark Greenhalgh
3 days ago
Question
Is there any way to distribute a render between machines to give each machine a complete frame to render, rather than all machines rendering a portion of each frame? I've noticed on simpler scenes, it's almost as quick rendering on one machine, as opposed to four due to the caching stage taking a high proportion of the overall render time. This would suggest that each machine starting a frame at the same time would reduce the render time by four, but rendering the same frame on each machine would reduce the render time by 10%. Does this apply more to single images, as the cache is reduced with an animation? Are there potential savings to be made in terms of time in simple scenes - for instance an animation with a Toon Shader? Are these type of animations best rendered just on one machine, rather than distributed? Thanks for your time.
Hi Arnar, have you tried using the manual load balancer? This might help with the slow combined render. You can choose how much of the image's screen area each machine will do. Try to balance this with your best guess and see if the render time improves.
V016 (which I assume you are using) has a quirk that if you first render using just one machine, then add a second, the second machine will not render very much. It was supposed to be a fail safe for animation renders where if the machine that gets added second is slow, this technique would avoid giving that slow machine too much too render. But it seems this has just caused too many problems with users seeing slow renders anyway since the expectation is the rendering will be faster using our addon. We're going to change this.
As for the CUDA issue, that is a bit strange, our addon should ignore the user preferences settings and use the settings from the node menu. If you set the local/master machine to use CUDA in the little node menu (which has a gear/cog icon to the right of the node's name), then it should obey that setting. However, if there is an issue with finding the device, then it falls back to CPU.
It would be great to know a bit more about your system to see what's going wrong here. If you can, please e0mail us and we can get in touch?
info at crowdrender dot com dot au
With the slow render its hard to tell since only one machine will render with CUDA, but it did seem to start to balance the load with the current situation, one CPU , one Cuda. I did set the local machine to use Cuda . I will email , pretty excited to get this working. :)
As for the cuda issue , I think I have that fixed , for some reason it was not saved in the settings. If I only turn on CUDA in preferences (not saving settings)..it will render with CUDA in normal blender. With Crowdrender , it will only use the CUDA if I save the CUDA settings from preferences. EDIT: still some problems with Cuda ...let me investigate.
One more question ,now the load balancing seems way off , I would rather count on the auto load balancing , especially since I will be adding more computers. Can I make it recalibrate the load balancing by doing a few renders? It does not seem to do that , it hands way too much to the second lesser computer. Maybe there should be a setting where you zero the load balancing and do a fixed number of renders and then it should be set ...in terms of user experience ? At this point I dont know if its monitoring the load and calibrating it if that makes sense? As for setting the load manually , that seems reasonable for two computers but I would be unsure of what I am doing with multiple computers.
Hi Arnar, Admittedly the balancing aspect of our system is not great. It certainly doesn't earn us much praise. We're designing a new management system that will hopefully change this. Until then my suggestions are that the manual load balancer works well for single frame renders which you can't afford to do too many 'calibration' renders, which you can do at lower sample counts to get the current system to balance well before turning up the samples again.
The automatic system is better for animation renders where you set it going and then leave until the next day or longer since animation renders usually have hundreds if not thousands of frames meaning that the small penalty of the slow renders at the start is not so much of a problem.
@James But will it distribute the load along the way if I set it to do multiple renders ?
I am using a 1080ti one my main computer , gtx560 on the second one, load set to 84-16 and the speed now syncs. 1920*1080 - 500 samples and I am sitting at 47 sec with a simple cube. Disabling Crowdrender alltogether and only using the 1080 and I get 57 sec.
Further testing on my simple box, seems to give me 10 sec , so that is quite stable. Still having problems with opening for example the BMW test , will test further. Also getting more computers later today for testing.
I will make this my log of getting this to work :) So...creating a project, simple cube , rendering on 2 computers , setting the load manually seems to work. Now...I open an existing project and it will not sync , restarted both Blenders ,Conn, does nothing, Resync does nothing. Tried save , revert...nothing. Opened another existing project , and now it works ? No idea as to what the problem is there. PS: simple cube works with a load distribution of 84/16 but when I apply that same setting for another project its way off and I am waiting for the second computer. So it seems like the load distribution calc needs some work as I would prefer it calculating it for me. I can imagine that its a lot of tweaking to do it manually for multiple computers. In fact if its not set perfectly you can end up with a longer render , especially since my second computer has a crap GPU. As it stands I get a faster render on one computer , probably due to the load balancing. Tweaking that to perfection gets me a faster render. Also , seeing the progressive render would be very nice a lot of the time that is where you spot problems , cancel fix, restart. But I guess that is not possible . Also, I am getting this render problem. Second computer renders purple ? Probably does not transfer the environment hdri. Tried "automatically pack into blend" resaved, resynced. and now it works perfectly.
Load balancing seems to be a little of scale when going from 100 samples to a 1000. I think it would be cool to be able to set up all the nodes , do a "load balancing render" and its set from there?
Now running one computer 1080ti , second computer 980ti , with tweaked load settings to even things out. I get 1:42 BMW benchmark with both computers rendering and 2:28 with just the one 1080ti. I would love to try this with 1080ti in both computers but I dont have one ;) In terms of load balancing , when I have the 1080ti set to 55 it finishes when the 980ti is at 67% If I then set the 1080ti to 56 in an attempt to sync them the 980ti finishes when the 1080ti is at 67%. So, maybe the load balancing needs a finer setting or something ?
Then in other Projects it seems like the second computer is faster , even with a 50/50 load? Been working on this all day , bit confused at this point. Still having problems with some projects with the second computer not syncing from my main computer, looking at the second computer it always says synced in Blender even though its not working. Nothing I do makes it come online. Opening up an older save of this same project and it says uploading 1% and just stays there. So in conclusion , works very well, when it works , seems unstable in terms of when it works ?
Tomorrow I will hook up more computers , lets see.
@Arnar Helgi Adalsteinsson
"In terms of load balancing , when I have the 1080ti set to 55 it finishes when the 980ti is at 67% If I then set the 1080ti to 56 in an attempt to sync them the 980ti finishes when the 1080ti is at 67%. So, maybe the load balancing needs a finer setting or something ?"
The load balancer swaps the part of the image that each node does for each frame. Which means each node will render a different part of the screen for each render, this can cause the times to vary, what you should do to test properly is to let it run for maybe 20 frames or so. This should give you a better sample and more realistic idea of how well the two cards are performing.
"Still having problems with some projects with the second computer not syncing from my main computer, looking at the second computer it always says synced in Blender even though its not working. Nothing I do makes it come online. Opening up an older save of this same project and it says uploading 1% and just stays there. So in conclusion , works very well, when it works , seems unstable in terms of when it works ?"
I think I know that this .... but would have to see the issue for myself. In any case, 017 will be released soon and from what you have told me I think it will help with this particular issue. The fix for this, at the moment, seems to be to shut down blender on your master/client machine, then also look in your task/activity/process monitor tool for any blender processes that are still running, kill them, then restart blender.
Apologies for such a terrible work around, this is what we're fixing in 0.1.7
So I was rendering an animation now, needed to change a few things so I did a resync , it said resync failed. I closed both Blender instances , found nothing to kill in the task manager. Now when I open the animation project nothing happens with the linked computer , cannot connect, cannot resync. I did at some point remove some sculpting brushes that it says it cannot pack in the project , not sure how I can get rid of that or if that is causing a problem. At this point I have quite a few projects that just will not work ,no matter what I do,some even open up saying rendering 0% and stay like that, weirdly sometimes a previous save of the same thing will work . It seems that if at any point a project fails , like this animation, it will not work again when I open it. Also get this message when I open the animation project now .
No problems , I am all in at this point . :) I did try shutting it down ...only on the master , then only on the slave , then both ...but. I did not look for any stray processes , will check that.
Btw, Is there any benefit in rendering eevee with this setup ?
Eevee is an interesting one, for viewport rendering, none at all. Thats because we don't interact with blender during viewport rendering. But if we did, its likely you would not see much difference in fps. Eevee is really best suited to game engine like scenarios, so having a powerful GPU should suffice.
As for using eevee as a rendering engine, that might be a different story. We have successfully rendered on multiple machines using eevee, however, I think you would have to really push the limits of this engine to see benefits to distributing it. It is so darn quick after all when compared to cycles that it may not be worthwhile. But it should work, so maybe you can test it and see if there is any benefit for you?
@James I will test it , for some reason my animation takes 1-2 minutes per frame , obviously not bad but should be faster for Eevee I think. But yes , will test.
Just saw the image you posted, nice! I like it, but the purple part is usually due to a missing texture, or in some cases HDRI if you are using the background to light the scene. Each node needs those textures and so far we do not have an automated mechanism for collecting all the external assets and distributing them. So this means you will need to pack them into the blend file.
If you could try packing all your assets into the blend file and let me know if that fixes the purple image, that would be great!
Yes, exactly right , I actually mentioned that in the post. Packing, resyncing and everything is nice:)
Today I will be testing 4 computer. 1. Main computer 1080ti 2 Second computer 980ti 3. Third computer 2x 970 4. Fourth Computer 1060 My testing last night ended with me kind of giving up on the inconsistencies of when It works. Some projects just stop wanting to sync, others will open with it saying 0% rendering. Others will fail resync. I will be trying to take this as far as I am able , lets see.
So far it seems impossible to make this work for me . At some point one computer got sync but that is a far as I could take this , most of the time its either unsynced or cannot connect.
Just now I got it to sync , tried rendering , image was obviously not synced , tried resync... Sync failed.
Hi Arnar, try saving the blend file, then re-open it with the node connected :)
I got it to render ..on two nodes , 3rd node is stuck at rendering 0% and main computer insists on rendering on CPU . 3rd computer is probably also rendering on CPU , cant operate it at the moment. EDIT: yes , nr 3 is rendering on CPU. Stopped the render , now node 1 -2 are ready and 3 is synced , not sure about the difference? Is there a procedure I should follow to make this more likely to work , like always saving and resyncing before trying to render ? Most of my resyncs fail so I have hesitated to use that . Also, any thoughts on why some nodes seem to refuse Cuda render ? EDIT: I got the main computer to render on CUDA by going into preferences , disabling CUDA, reanabling and doing the same for CRowdrender main computer. 3rd computer does not want to play ball. Not sure if it means anything but nr 3 says synced, while the other ones say Ready. So , its Semi working at the moment, god knows what happens if I open another project , seems very hit and miss :) As it stands , two main problems . 1. 3rd computer will not start CUDA render. 2 . Main computer will not CUDA until I disable/enable. PS...the speed is insane!
Remove one node as its not working... save , reopen , now I am back to Main computer on CPu and a fourth of the image not rendering. Can it not know that one of the nodes has been removed ?
If you are still on manual load balancing that can make this an issue, try going into the load balancer editor and seeing if you can rebalance all the nodes, or try setting it to automatic? There is a bug in the manual mode where the screen area is not adjusted back to 1 after a node is removed.
I see, that worked. Only working with a simple cube , rendering on 1080ti is 57 sec. Rendering on the main and 2 nodes x*970 and a 980ti gives me 22 sec. Now to see if any regular projects work . ;)
Suggestion , its strange to set the manual load when it autocalculates the other values , especially when you are working with multiple nodes , you have to lower one value to raise the other and then maybe it raises the third, bit of a mess. Maybe it should allow one to set the values and then auto calculate numbers if they are off when you hit ok.
Good suggestion, yeah this is a hard thing to get right. We've got to make sure that the numbers will always add to 1.0 otherwise, well, you've already seen what happens xD
@James Indeed :)
In conclusion from testing pretty much all day , when it works , its awesome ! some of the time its flawless, change something ,hit resync , bam , all good !
Most of the time it does not work...Why ? I dont know , it seems quite random. Sometimes there is just no activity on the nodes , connecting , resyncing either does nothing or fails, saving , reverting does nothing. Sometimes it even says render and nothing happens and it goes back to saying sync. Basically I have had all possible versions of not working. Having said that , node 3 now works ...I fully expect it to stop working :) Weirdly , sometimes, reopening the project does not help , its pretty much dead in the water and i need to use another save and try again. I dont know if I can be of any assistance , send a log or something , I would love to have this work properly. Or do you have any ideas to narrow down the problems I am having ?
I dont know if this tells you anything, but I was testing with an plane and a box, add materials , resync , works. Then I added a sphere , resync, resync failed and now its dead , nodes dont respond to anything. Saving, reverting ,reopening does nothing . If I reopen the project its the same , dead. Must be something wrong with the project itself at the node side. And I am going to sleep.
Hi Arnar,
ok, the glitching around connection and associated trouble is probably part of a known issue where the client/master gets into a bad state. Mostly this is due to opening a file whilst having nodes connected, we've been working on the next release to fix this.
What you can do about this? Basically, if you encounter this, shut down your local blender session, check your task/activity/process monitor and shut down any blender processes that don't close and restart blender. This should allow you to solve the issues where nodes are 'dead' as you put it.
Also, not every thing you can do in blender is supported. Actually only a small number of edits will be successfully synced.
Its probably helpful to consider what use cases we currently support well, and what we don't.
So, tweaking composition, colour, camera settings and render settings prior to a big final render, works well. We support editing transforms of objects and edits to existing shader node settings (but don't support adding or removing links or nodes), support also covers camera transform and data and render layers, plus passes and most if not all the render settings you'll need.
There's basic support for duplication, delete and undo/redo, but adding things, as far as I can remember, is limited to just cubes for now.
Everything not on that list either will cause a sync fail or will go unoticed by our system causing you to see differences between the tiles, which you've already come across with the nice image of a keyboard.
So, what are we doing about this? We're completely rebuilding the sync system from the ground up. When we started, back in 2016, we really had a poor understanding of the basic problem. At the end of last year we went back to basics and took some time out of building and releasing to understand the problem better and redesign how nodes can stay synced in real time. We're now in the mid stages of a complete redesign with most of the conceptual work complete and we'll be working more and more to a detailed design with hopefully some real code being written before mid year, 2019.
CR does work well as you said, when you don't go past the limits of what we're currently able to support. So if you're texturing, modelling or animating, those changes are likely going to cause problems. I'd strongly recommend using CR to tweak things listed above once most of your work is done, so either don't connect to your nodes until you're ready to start rendering, or disable the addon. CR works really well when you have finished most of the work and are ready to start preparing for the final render.
@James I see, maybe there is a user experience "issue" as it were, or me expecting it to work differently. You kind of expect when starting a project that CR will just sit there until needed, in that when you then need to render , you save , with packing,hit sync and you can start rendering. In terms of workflow , for me , I am using the render way before I am planning to do a final render to test issues , materials etc , so I imagined CR would be standby to render when I need throughout the project as long as the save, pack, resync procedure is followed. Am I right that there are 2 types of sync ? The one you talk about where it automatically syncs limited edits and tweaks and then the resync function that would sync everything ? I expect the resync to load the project as is to the nodes as long as its saved and I am good to go ? The auto sync being more applicable when working on minor tweaks for the final render. And the manual resync would be more applicable when just doing test renders in a project in progress. Am I incorrect ? Is your approach that the addon is mostly for final tweaks and not for intermediate renders? ps. I can see a start button when I open Blender , but its there just for a second and then CR is engaged. Maybe the start button could engage the addon when needed ? So I can have all nodes setup and just sitting there waiting to be engaged.
pps. I have tried shutting down Blender and I never see anything in the task manager to shutdown? Would these tasks be marked as a Blender task or something else ? Again, LOVE it when it works and looking forward to the next version. Rendered the BMW in just under a minute last night which is pretty amazing.
Hi Arnar,
Would it work well for you to just use CR with the nodes not connected, then when you want to render, connect. There are no sync issues when you are not connected to any nodes. You can leave the addon enabled and running, it will not cause issues that way. Then when you connect, you can tweak the things I mentioned and render.
I use it this way and it works particularly well. Connecting to nodes really doesn't make sense unless you are intending to render using them. If you are only going to be editing, then there's no point connecting to your nodes anyway.
Hope that helps
Apart from any syncing, just to get it to connect is a big problem. Sometimes only one connects , sometimes none, its pretty random. Having said that , start a new project , node computers shut off. Make a simple project, default cube Turn on computers , they appear in the list , try connect , nothing happens. I tried removing and re-adding Blender 1 , not sure what is the difference between showing the camera and settings for the other 2.
Tried shutting down , searching task manager for tasks, none found. Reopen , same thing.
Now I open a new project and this is how it opens . Only one node seen. I Hit conn and it hooks up, works to render.
Adding another node. Failed to connect.
I try resync , and it just sits like this.
Try again, new project . Node 1 Works. add more and it goes random.
And this is just with the default cube. In my case I cannot get it to work reliably at all and its just so random that I cannot pin it down. I would be happy with just at certain points, enable the nodes ,hit render. But , no such luck
Then I open up the bmw that I have been testing and it works with all nodes, no idea why ? Not sure what to try ? Should I try to remove blender and do a reinstall or something ?