Render manager - jobs stuck 'waiting for nodes'

Render Manager Issues
#1

We are having a little trouble with our farm recently. Had been perfect since setting up, but 6 month-ish later we are now getting jobs stuck in the queue. There are machines in our farm available to rendering, but the job stays stuck on ‘waiting for nodes’ and the nodes are saying ‘searching for task’.
Wierdly, if we send the exact same job to the queue again, it will start rendering straight away, but the first submission just sits there doing nothing. There are no errors showing at all with these hung jobs.

The 2 possible ideas I have, but Im not a pro in these things, are either short interupts in our network/internet connection. Or maybe the anti-virus our company uses. It is the ‘endpoint’ suite. I include this as we had major trouble with a protection program and backburner in the past.

many thanks in advance

EDIT
Through some testing we have found that a job will start rendering if we send it twice. The first one sits waiting for nodes for eternity, the 2nd starts like normal.

#2

Please check this for the possible causes: https://www.pulze.io/faq/my-render-manager-jobs-are-waiting-for-nodes-but-my-render-nodes-are-idle-and-ready-to-work.-what-can-i-do

If nothing in the faq helps then please install this nightly build that will solve this problem: https://public.pulze.io/pulze-downloads/pulze_service/2.1.4/pulze_214_win.zip

#3

Sorry, I forgot to add in my original post that the jobs had no errors posted at all.
Everytime I have checked the Admin tab, a computer has been assigned as the distributer. Although I guess this doesnt rule out the nodes themselves losing network connection.
Thankyou for the nightly I will try this and post back.
Many thanks

#4

This seems to be working at the moment. Can I ask what the patch actually addressed? I wondered if it was something we were doing wrong at our end

#5

The patch in version 2.1.14 should fix several job distribution and job logic bugs.
For example if you had 5 single frame jobs where for one of the nodes made an error on the first one, it would create a jam and none of the jobs below will start.
From the information that you provided it was definitely the Render Managers fault.

#6

Just to bump this but we are having the same issue. We are using the latest versions of Scene Manger and Redner manager and sometimes the jobs don’t pick-up with the message waiting for nodes. It seems random when this happens. The latest example if I sent two cameras rendering over lunch and only one of them picked up and rendered. The other was tuck waiting for nodes. There is no error in the jobs.

Is there anything else that could be causing this? The idea of sending a high number of renders over night or weekend and them not rendering is a major issues, let alone an animation.

When it works Pulze is great but obviously we are quite worried about this.

#7

Hi @visualisers

If this happens again could you please send us some logs (based on this guide: https://www.pulze.io/faq/how-do-i-collect-my-render-manager-logs-and-send-them-to-pulze-support) to support@pulze.io and we will have a look asap!

#8

I have the same problem with this today and yesterday even with a two-step job, a tiled render and the assemble, where the render finished normally but the assemble was waitting for nodes. Then again we added some jobs and they never got picked up by any machine. Distributor was ok, no errors on the jobs. I managed to “force” the render to start by selecting a specific machine to render this image by assigning exclusive node to the job. As soon as i gave the exclusive node the job started.

#9

Same on our end this week.
Had to force render by assigning exclusive node to the job. Otherwise no node would take it on board.

#10

Had an issue similar to this last night:
Submitted a job to our farm, job went though no problems, no errors, waiting for nodes because there where other jobs rendering.
Login in the morning to find the jobs still waiting for nodes despite the farm being empty, restart pulze to find my jobs had vanished. remote connected to another PC on our network, confirming the jobs did not exist. On resubmission the job begins rendering fine.

Here are the logs: the max file name is “1351 DXD Nantucket 05.max”