Use of HIGH PRIORITY

Message boards : Number crunching : Use of HIGH PRIORITY
Message board moderation

To post messages, you must log in.

AuthorMessage
Rasputin42

Send message
Joined: 5 Nov 11
Posts: 25
Credit: 1,107,815
RAC: 0
Message 353 - Posted: 20 Nov 2011, 0:02:05 UTC

Hi,
Please restrict the use of HIGH PRIORITY mode for tasks, that are not in danger of missing the deadline.
There are a number of projects, that are doing this(including yours).
I understand, that there are some circumstances, where it is necessary to do so.
However, i would like to run some other projects and your project, going into HIGH PRIORITY mode, almost all the time, prevents these from running.
Please consider, that you are not the only project.
ID: 353 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dagorath
Avatar

Send message
Joined: 2 Sep 11
Posts: 57
Credit: 1,274,345
RAC: 0
Message 354 - Posted: 20 Nov 2011, 1:06:02 UTC - in response to Message 353.  

The server does not and cannot designate tasks as high priority. It doesn't happen at this project or any other project. The BOINC client running on your computer, not the server, decides whether tasks run at high priority or normal priority.

Projects that you have given a very low resource share will tend to run at high priority when they should not. If you have projects that fit that description then try increasing their project share slightly.

BOINC FAQ Service
Official BOINC wiki
Installing BOINC on Linux
ID: 354 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Rasputin42

Send message
Joined: 5 Nov 11
Posts: 25
Credit: 1,107,815
RAC: 0
Message 355 - Posted: 20 Nov 2011, 1:31:16 UTC

Thanks for the reply.
Why does SETI never go into HIGH PRIORITY mode, if " Boinc" decides to do so or not?
I find your explanation to be not reproducible and invalid.
ID: 355 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Eric Driver
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 8 Jul 11
Posts: 945
Credit: 103,971,248
RAC: 68,717
Message 356 - Posted: 20 Nov 2011, 3:27:17 UTC - in response to Message 355.  

Hi Rasputin,

Sorry for your troubles, but I believe Dagorath is right, I have no control over that.

On my client, where I run 2 projects (this one and a private test project), I notice that sometimes a WU goes into "high priority" mode when it nears the deadline. After setting the grace period to 1 week, this behavior seems to have gotten better. If you have a bunch of WUs queued up, they may all be hitting the deadline and causing your problem. Other than that, I'm not sure what else could cause this.

Eric
ID: 356 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dagorath
Avatar

Send message
Joined: 2 Sep 11
Posts: 57
Credit: 1,274,345
RAC: 0
Message 357 - Posted: 20 Nov 2011, 7:24:08 UTC - in response to Message 355.  
Last modified: 20 Nov 2011, 7:29:39 UTC

I got to thinking about your previous statement

Please restrict the use of HIGH PRIORITY mode for tasks, that are not in danger of missing the deadline.


How can the server reliably know whether any given task is in danger of missing the deadline? It can't. Only your client can know and that is why your client, not the server, makes the decision to run a task at high priority.

If SETI does not run at high priority then perhaps it does not have a very low resource share. There could be other reasons why SETI doesn't run at high priority, the reason I gave isn't the *only* reason.

If you would be so kind as to post the following information, perhaps someone can provide a better explanation:

1) your "Connect about every __ days" setting
2) your "Additional work buffer __ days" setting
3) a list of your projects and the resource shares assigned to each project
BOINC FAQ Service
Official BOINC wiki
Installing BOINC on Linux
ID: 357 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ChertseyAl

Send message
Joined: 19 Aug 11
Posts: 45
Credit: 1,014,069
RAC: 0
Message 359 - Posted: 20 Nov 2011, 10:52:47 UTC - in response to Message 356.  

Other than that, I'm not sure what else could cause this.


In my case the cause is that the initial estimate of time to completion is way too high 99% of the time. All my WUs come in with estimatates of 60-120 hours and often go into panic mode. Once they've crunched a bit, the estimated remaining run time drops to a sensible time and they drop out of hi pri (apart from a couple of WUs that really do look likely to run for 4 or 5 days).

Al.
ID: 359 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dagorath
Avatar

Send message
Joined: 2 Sep 11
Posts: 57
Credit: 1,274,345
RAC: 0
Message 365 - Posted: 20 Nov 2011, 22:31:27 UTC - in response to Message 359.  
Last modified: 20 Nov 2011, 22:37:54 UTC

Other than that, I'm not sure what else could cause this.


In my case the cause is that the initial estimate of time to completion is way too high 99% of the time. All my WUs come in with estimatates of 60-120 hours and often go into panic mode. Once they've crunched a bit, the estimated remaining run time drops to a sensible time and they drop out of hi pri (apart from a couple of WUs that really do look likely to run for 4 or 5 days).

Al.


That is exactly what *was* happening on both of my computers too. Then I stumbled upon a post from John Macleod VII that explained that the scheduler tends to run projects that have a very low resource share at high priority. Sure enough, I had given NumberFields and a few other projects very low resource shares and all those projects were *always* running at high priority. I increased the share for those projects a little and now they run at low priority unless, as you said, they get close to the deadline.

Yes, I know it doesn't make sense that a very low project share will induce high priority but it can (JM7 explained precisely why it can) and it did on my computers. Perhaps that is the problem on your computer(s) too.

@Eric,

I don't think the client knows about the grace period so I don't think extending that to 7 days had much effect.
BOINC FAQ Service
Official BOINC wiki
Installing BOINC on Linux
ID: 365 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Eric Driver
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 8 Jul 11
Posts: 945
Credit: 103,971,248
RAC: 68,717
Message 367 - Posted: 21 Nov 2011, 0:33:04 UTC - in response to Message 365.  


@Eric,

I don't think the client knows about the grace period so I don't think extending that to 7 days had much effect.


Sorry, I meant the "delay bound", which I increased to 7 days many weeks ago. Unless I am mistaken, that is the deadline that most users see on their client.

The "grace period" is actually set to 48 hours. So users really have 9 days to return results.

Eric
ID: 367 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 28 Oct 11
Posts: 134
Credit: 110,238,564
RAC: 29,214
Message 388 - Posted: 22 Nov 2011, 18:42:43 UTC

On the subject of High Priority: I agree with Dagorath - this is entirely determined by the client, and nothing to do with server/project settings at all.

'Runing in High Priority' simply means that some task, somewhere, is in danger of missing a deadline unless the running order is shifted around a bit. SETI has, size for size, perhaps the most relaxed deadlines of any BOINC project I've seen, so their tasks very rarely need the boost: but the 7-day turnround here will activate it more often.

And it's simple to explain the Resource Share link. If you have two projects, both with the same resource share, tasks will take it in turns, running for one hour each. So if Eric says the average task will spend 10 hours actually running, it'll be finished after 20 hours of wall-time (assuming single-core CPU). But if you give the other project RS 100, and this project RS 1, then the other project will run for 100 hours each time this project runs one hour. That 10-hour task would stretch to 1,000 hours (actually, 1,010 hours): that's well beyond deadline, so 'high priority' would be invoked long before the task was complete.

Having said that, there is a looming problem with 'high priority' because of the CreditNew server code in use here. I'm seeing it a lot with the new v6.13.12 test version of BOINC.

We know that some NF@h tasks run really long - two or three days. When that happens, BOINC immediately increases the DCF (Duration Correction Factor), which increases the estimated runtime for all other tasks from the project - a precautionary measure, in case all tasks are going to behave the same way.

But BOINC only reduces DCF gradually as normal-length tasks run through, so it's likely still to be high next time you need to fetch new work. My DCF was about 7.8 when I made this work fetch:

05-Nov-2011 09:51:49 [NumberFields@home] Requesting new tasks for CPU
05-Nov-2011 09:51:49 [NumberFields@home] [sched_op] CPU work request: 365384.35 seconds; 0.00 CPUs
05-Nov-2011 09:51:49 [NumberFields@home] [sched_op] NVIDIA work request: 0.00 seconds; 0.00 CPUs
05-Nov-2011 09:51:52 [NumberFields@home] Scheduler request completed: got 43 new tasks
05-Nov-2011 09:51:52 [NumberFields@home] [sched_op] Server version 613
05-Nov-2011 09:51:52 [NumberFields@home] Project requested delay of 21 seconds
05-Nov-2011 09:51:52 [NumberFields@home] [sched_op] estimated total CPU task duration: 2773276 seconds

In the old (server code) days, the server would have taken note of the DCF value that the client reported - the amount of work allocated would have been reduced, so that the server and client estimates of the amount allocated matched reasonably closely.

In this case, I'm sure the server allocated 365,384 seconds of work (or something close to that) - but it did so assuming a DCF of 1.0000

But the higher DCF in effect at the client led to the much higher estimate of 2,773,276 seconds for the same work. And the high (local) estimate leads in turn to a (local) decision to run in high priority - because the client thinks it's stuffed.

I've asked the BOINC developers to re-instate the server code DCF safeguard against over-allocations like this, but I didn't get a reply. It might help if this project (and other projects with what I've called "non-deterministic tasks" - tasks wehere the runtime can't be predicted in advance) could re-inforce the need for server code which guards against over-allocation.
ID: 388 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
frankhagen

Send message
Joined: 19 Aug 11
Posts: 76
Credit: 2,002,860
RAC: 0
Message 390 - Posted: 22 Nov 2011, 19:11:33 UTC - in response to Message 388.  
Last modified: 22 Nov 2011, 19:11:57 UTC

I've asked the BOINC developers to re-instate the server code DCF safeguard against over-allocations like this, but I didn't get a reply. It might help if this project (and other projects with what I've called "non-deterministic tasks" - tasks wehere the runtime can't be predicted in advance) could re-inforce the need for server code which guards against over-allocation.


pin-pointed!

but you might concur to what i learned over the years - they will not learn a lesson, but keep frickeling around. they do not fix the ancient porblems which are still around, but come up with new fields like MT-apps, VM-integration, open-cl support and so on.

so it's getting worse and worser - and for my measures, this is way beyond FUBAR in the meantime.
ID: 390 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dagorath
Avatar

Send message
Joined: 2 Sep 11
Posts: 57
Credit: 1,274,345
RAC: 0
Message 391 - Posted: 23 Nov 2011, 1:37:56 UTC - in response to Message 390.  

I crunch GPU tasks, CPU tasks and I run T4T@home tasks in a VM. BOINC runs just fine here, no problems, no worries, just sheer joy.

Wanna know why?

Because I know BOINC isn't perfect and it never will be perfect and instead of continually bleating like a sheep about it I use recommended workarounds and I use the recommended version of BOINC not some version Jesus Christ used before he became a carpenter.

If you don't use the recommended workarounds and insist on running fossil-ware then you're never going to find joy with BOINC. It really is that simple. And if you think you can do a better job developing BOINC then fork a branch and show DA how it should be done. It's all open source and just crying out for you to save it from DA's evil clutches.

BOINC FAQ Service
Official BOINC wiki
Installing BOINC on Linux
ID: 391 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
frankhagen

Send message
Joined: 19 Aug 11
Posts: 76
Credit: 2,002,860
RAC: 0
Message 392 - Posted: 23 Nov 2011, 15:40:01 UTC - in response to Message 391.  

If you don't use the recommended workarounds and insist on running fossil-ware then you're never going to find joy with BOINC.


sounds interesting - then you can probably tell me how to upgrade to boinc 6.xx on a windows-dc. i got 2 hosts where i'm stuck with 5.10.45...
ID: 392 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dagorath
Avatar

Send message
Joined: 2 Sep 11
Posts: 57
Credit: 1,274,345
RAC: 0
Message 397 - Posted: 24 Nov 2011, 2:35:19 UTC - in response to Message 392.  

I'll answer your question when you answer my questions which I posted in response to your brilliant accusation.

BOINC FAQ Service
Official BOINC wiki
Installing BOINC on Linux
ID: 397 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Number crunching : Use of HIGH PRIORITY


Main page · Your account · Message boards


Copyright © 2019 Arizona State University