WUs stuck in pending

Message boards : Number crunching : WUs stuck in pending
Message board moderation

To post messages, you must log in.

AuthorMessage
corris
Avatar

Send message
Joined: 3 Sep 15
Posts: 6
Credit: 18,509,242
RAC: 0
Message 1919 - Posted: 29 Dec 2017, 12:43:32 UTC
Last modified: 29 Dec 2017, 12:44:05 UTC

Hi

All WUs for the past 12 hours or so are stuck in pending

Server Status says all running ok

But, validator seems to have gone on vocation, or is it just me?



Meanwhile Seasons Greeting one and all
ID: 1919 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Eric Driver
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 8 Jul 11
Posts: 944
Credit: 102,324,051
RAC: 68,107
Message 1920 - Posted: 29 Dec 2017, 17:08:20 UTC - in response to Message 1919.  

This is probably related to a problem we had yesterday with the transitioner.

I see you have a bunch of WUs, but I cant tell which one is bad, or if all are bad. The most recent ones in your queue say "aborted via GUI" and it appears the transitioner is not generating a new result for those. Are those the ones you are referring to, or is there also a successful result that is having the problem? Can you give me the WU ID of a bad one? And also the ID of a good one for comparison purposes (if you still have it).
ID: 1920 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
corris
Avatar

Send message
Joined: 3 Sep 15
Posts: 6
Credit: 18,509,242
RAC: 0
Message 1921 - Posted: 29 Dec 2017, 18:46:41 UTC - in response to Message 1920.  

Hi Eric


wu_sf3_DS-16x265_Grp126104of256000_1 as an eg. There are 45 stuck in pending having completed processing from approx midnight last (UTC) with status "Completed, waiting for validation"






Pl ignore all those that say aborted
ID: 1921 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Eric Driver
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 8 Jul 11
Posts: 944
Credit: 102,324,051
RAC: 68,107
Message 1922 - Posted: 29 Dec 2017, 23:14:31 UTC - in response to Message 1921.  

So the transitioner is still having problems. The reason those are not validating is that the transitioner is not changing the validation state. The reason for that is there is a huge transitioner backlog. I will not bore you with the details and I don't completely understand the root cause yet. But what's important is it appears that the transitioner is getting caught up and will eventually get to processing those WUs.
ID: 1922 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
corris
Avatar

Send message
Joined: 3 Sep 15
Posts: 6
Credit: 18,509,242
RAC: 0
Message 1923 - Posted: 30 Dec 2017, 7:46:43 UTC - in response to Message 1922.  

OK Eric

Thanks

Pended WUs decreased from 45 to 30

I'll wait till all fixed


Thanks
ID: 1923 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Eric Driver
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 8 Jul 11
Posts: 944
Credit: 102,324,051
RAC: 68,107
Message 1924 - Posted: 30 Dec 2017, 17:24:31 UTC - in response to Message 1923.  

The transitioner should be completely caught up in the next several hours. Part of that catch up was to transition 60k+ WUs to the need-validation state. Now the validator is playing catch up as it has a backlog of about 25k WUs.

So keep me posted when your WUs are completely validated.

Thanks!
ID: 1924 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
corris
Avatar

Send message
Joined: 3 Sep 15
Posts: 6
Credit: 18,509,242
RAC: 0
Message 1933 - Posted: 2 Jan 2018, 12:29:16 UTC - in response to Message 1924.  

All the old WUs stuck in pending have cleared


Those still pending are all of today's date
ID: 1933 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Number crunching : WUs stuck in pending


Main page · Your account · Message boards


Copyright © 2019 Arizona State University