New optimized versions of the GetDecics app have been discussed in other threads, but I thought it best to start a new thread.

First, a thank you to Alperen for optimizing the current apps. Yes, he used Claude, but that's not necessarily a bad thing. I don't have the experience required or the time to learn the AVX tricks that were used here. So without Alperen and Claude, this probably never would have happened.

I have spent many hours going through the new code. The original code was basically left untouched, and the change was in the addition of a new subroutine that pre-filters the polynomials prior to the normal processing chain. This leaves just the one subroutine that needs validation. From a theoretical perspective, the method is sound. The method can be decomposed into 2 parts:

1. The first part was a simple trick (I wish I had seen this myself!) that allows us to deal with the polynomial discriminant modulo a small set of primes. This eliminates the need for the multi-precision integers that were growing to ~1000 bits, and gives the first speed improvement. This was simple enough I could independently code it up myself using Pari's built-in functions. In some offline tests I was seeing a speed improvement of between 2 and 3, just for this simple change.

2. The second improvement was in the coding of the resultant modulo q. This was essentially done in parallel using AVX2, which allows resultants to be computed for several primes simultaneously. This is the part that is very tricky and hard for me to follow. I welcome any expert in this kind of optimization to look at the code. Alperen has made it public here:
https://github.com/alplix/passiflora

So the method in general is sound, but the second part was hard for me to validate. For this reason, I will use sufficient testing to validate the new apps. The idea is to test each new app version on ~1000 old cases which can be taken as truth. This will be done using the "testDecics" app that was recently created. When an app is deemed good, it will be transitioned into an official app.

I will be creating a new app called fastGetDecics where the new validated apps will go. This will still allow me to keep the results separate. It will also allow me to create new WUs for the fast apps that will not put a burden on the server, while retaining older style WUs for the legacy apps. This will give me time to test and migrate the fast apps in pieces rather than all at once. App versions for the original GetDecics will be deprecated once the corresponding fast version has been deemed trustworthy.

So the guy who was just banned from the BOINC forums is now writing apps for this project?

Interesting.

So the guy who was just banned from the BOINC forums is now writing apps for this project?

Interesting.

Did you actually read the news post???
If it works and produces accurate and valid results, then why not use it??????

---

Grant
Darwin NT, Australia.

This was essentially done in parallel using AVX2, ...

Because this project uses a fixed value for the Credit granted, and I'm pretty sure the with present Credit system the Credit is set per Task and not per application, i suspect it will cause issues with credit received for work done.


With the standard BOINC Credit system, the amount of Credit is based on the amount of work done- using the project's stock application the amount of Credit will vary depending on the amount of time taken to process the Task (eg (made up numbers) 2 hrs gets 500. 1 hr gets 250. 4 hrs gets 1,000); but the awarded Credit per Task for the stock application and a higher performing optimised application is the same (eg 2 hrs stock but 1 hr optimised- both still get 500. 1 hr stock but 30 min optimised- both still get 250. 4 hrs stock but 2 hrs optimised- both still get 1,000.)
It still gets the same amount of Credit as the stock application for a given Task- but as the optimised application takes less time to do the same amount of work, it will get more Credit per hour.


While AVX2 has been around for a long time, there would still be quite a few systems without it- if you're going to continue to support non AVX2 systems it will result in those systems getting much less Credit per hour than they are currently getting if you reduce the Credit per Task (as been done for the present Test Tasks).

If all work is done using the new applications, no problems, although it will block older non AVX2 systems from participating.
But if there is a mix of AVX2 and non-AVX2, the lack of Credit for non-AVX2 produced results will discourage those from continuing.

---

Grant
Darwin NT, Australia.

Eric, thanks for the progress update. I think with the new fast app, the remaining work batches can be completed much faster, maybe around 20X faster based on my limited study on both CPUs and GPU that I currently owned.

As for credit, there is going to be some discussions if the current fixed credit should remain the same or not. There are two camps of boinc volunteers: those who go after credits and those who care for science but honestly having reasonable credit makes this whole boinc volunteer computing more fun.

For me, I'm leaning towards the principle of "Equal work for equal pay". On the hardware side, CPU and GPU continue to see architecture and speed improvement (and other things like memory speed, PCIE bandwidth, etc), software is written to utilize new instruction sets (e.g. old SSE4 vs AVX, AVX2, to new AVX-512) and also with the help of AI coding as a tool, software can be re-written quickly to optimize for speed. Like you said understanding the code or algorithm to ensure the method is logically sound is a different ball game altogether. Here the "equal work" means a completion of a work unit. A recent example is Einstein@Home project where initially using a datacenter/professional gpu card with good FP64 & high memory bandwidth completed the same work unit faster (and more power efficient) than the consumer grade gpu card (other than the enthusiast high end consumer card) while both still earn the same credit, "equal pay". Overtime optimized Einstein apps were developed and run faster (2X to 10X, depending on the hardware) than the stock default apps. Both still earn the same credit till today. Seems like having a credit inflation, lol.

Anyway, just my thought but I'm sure others here may have different opinions.

This was essentially done in parallel using AVX2, ...

Because this project uses a fixed value for the Credit granted, and I'm pretty sure the with present Credit system the Credit is set per Task and not per application, i suspect it will cause issues with credit received for work done.


With the standard BOINC Credit system, the amount of Credit is based on the amount of work done- using the project's stock application the amount of Credit will vary depending on the amount of time taken to process the Task (eg (made up numbers) 2 hrs gets 500. 1 hr gets 250. 4 hrs gets 1,000); but the awarded Credit per Task for the stock application and a higher performing optimised application is the same (eg 2 hrs stock but 1 hr optimised- both still get 500. 1 hr stock but 30 min optimised- both still get 250. 4 hrs stock but 2 hrs optimised- both still get 1,000.)
It still gets the same amount of Credit as the stock application for a given Task- but as the optimised application takes less time to do the same amount of work, it will get more Credit per hour.


While AVX2 has been around for a long time, there would still be quite a few systems without it- if you're going to continue to support non AVX2 systems it will result in those systems getting much less Credit per hour than they are currently getting if you reduce the Credit per Task (as been done for the present Test Tasks).

If all work is done using the new applications, no problems, although it will block older non AVX2 systems from participating.
But if there is a mix of AVX2 and non-AVX2, the lack of Credit for non-AVX2 produced results will discourage those from continuing.

AVX2 came out over a decade ago, which should apply to most Intel/AMD systems on this project. So I think your concerns apply more to non x86_64 systems, such as MacOS and the new ARM builds.

As a reminder - credit is fixed and is based on average run times. So if one batch of data averages 1 hour/WU and gets X credit, then a new batch which average 1.5 hours/WU will get 1.5*X credit. All timing is done offline using 100 WUs on the same system under the same conditions, to get the most accurate timing. I plan to do the same thing with the new fast apps, so that 1 hour of compute time pays the same credit whether it's with the fast app or the legacy app. That seems to be the fairest thing to do, especially during the transition period.

But as you point out, this creates a predicament for the non-AVX2 systems. I'm not sure if I will do this, but maybe give the non-AVX2 systems a bump up in their credits??

But as you point out, this creates a predicament for the non-AVX2 systems. I'm not sure if I will do this, but maybe give the non-AVX2 systems a bump up in their credits??

Honestly with the fast app introduction, contributions from those non-avx2 systems will be very insignificant in a grand scheme of things. Sorry not meant to push those contributors away. I leave it to you.

If my data is representative (small sample size on CPU and GPU), a 20X improvement (maybe 10x worst case??) in speed will cut the remaining ETA time by:
Batch progress current_ETA Fastapp_ETA
DS13x12 4.0 % 667.8 days → 33.4 days
DS14x12 4.0 % 720.0 days → 36 days
DS15x12 4.0 % 631.6 days → 31.6 days
DS16x10 94.5 % 16.3 days → < 1 day
DS16x11 ?
DS16x12 ?

Am I hallucinating that we will finish this project very soon?

BTW, what is the average speed improvement seen from the ~1000 tasks that were just completed recently? Not sure if you have the old run time for the ~1000 tasks or perhaps the overall average run time of the DS13x12 tasks?

...so that 1 hour of compute time pays the same credit whether it's with the fast app or the legacy app. That seems to be the fairest thing to do, especially during the transition period.

I agree, that is the best thing to do- you get the same amount of Credit for doing the same amount of work, regardless of the hardware, OS, application being used. If you are running an OS, hardware, application system that can process more Tasks per hour, then you will get more Credit per hour- that's always been one of the points of the Credit systems.

But with it based on the new optimised application, that will result in a massive drop in Credit for non-AVX2 based systems as they won't be processing the Tasks with greatly reduced Credit nearly as quickly.
Either that, or keep using the current Application for determining Credit, and the systems that can run the new applications when they are released will get a huge boost in their RAC while the others will still keep their present RAC.

But as you point out, this creates a predicament for the non-AVX2 systems. I'm not sure if I will do this, but maybe give the non-AVX2 systems a bump up in their credits??

Or leave them as is, and the systems using the new optimised application get paid a much lower amount- if it is even possible to do.
But from past experience it's the sort of thing that really tends to annoy people- getting differing amounts of Credit for the same Task, depending on who processed it. Especially so when someone that took 5 hours to process something is getting more Credit than someone that took less than an hour.



If using the new applications for determining Credit it might be best to put a note on the projects main page (not than many people tend to read them...)

explaining that certain systems now have a very highly optimised application, and the amount of Credit that systems that don't have these applications will now receive will be much, much less than it was in the past.

or

explaining that certain systems now have a very highly optimised application and that other systems are no longer supported.





My personal vote-
Maintain the Credit per Task as it is now- based on the existing application -systems that can't use the new applications will still maintain roughly the same RAC they do now, all others will get a huge boost.
And it saves you the effort of implementing differing amounts of Credit depending on the application that processed it- if it is even at all possible.

Of course this will probably result in a surge in new systems joining the project as people hear about the amount of RAC it produces.
Will the current servers be able to cope with the increase in work returned with the new applications alone, let alone a boost in crunchers joining the project?

---

Grant
Darwin NT, Australia.

Well done Alperen

PS:

So the guy who was just banned from the BOINC forums is now writing apps for this project?

Interesting.

Are not you Skillz a guy who use Claude Code to take over boinctasks-pp application and do all changes using Claude Code except writing one readme text change without AI assistant?

https://github.com/lblanchardiii/boinctasks-pp/commits/main/

Commit 46d461b
lblanchardiii & claude
on Jul 27
Add .gitignore rules and AUTHORS for the Linux port
Records eFMer as author of the original BoincTasks, the port's authorship,
and what the port changed beyond translating the code. Keeps build output
and release artefacts out of the tree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---

I crunch for Ukraine

That's great news!

Will it mean that the MacOS/ARM apps will also see an update. Not the AVX of course, but maybe there are also optimizations possible.

Sjmielh

Am I hallucinating that we will finish this project very soon?

BTW, what is the average speed improvement seen from the ~1000 tasks that were just completed recently? Not sure if you have the old run time for the ~1000 tasks or perhaps the overall average run time of the DS13x12 tasks?

No, you are not hallucinating. The search over S={2,5} will complete at least 10x quicker. Maybe within a year now. Of course there are other cases to do, so the project is not necessarily going away.

I can't tell you what the average of the 1000 cases was, but it was at least a 10x improvement. Over 900 of the tasks were for the x86_64_pc-linux-gnu platform, so that one will be migrated soon to being an official fast app version.

Will the current servers be able to cope with the increase in work returned with the new applications alone, let alone a boost in crunchers joining the project?

That's why I need to create a new set of WUs for the faster apps. If I didn't change the WUs, then the server would have to process >10x the current rate of WUs, which would bring the server to its knees. It's an easy enough change - each task now gets 10x the number of congruences to process, thereby slowing it down by 10x.

That's great news!

Will it mean that the MacOS/ARM apps will also see an update. Not the AVX of course, but maybe there are also optimizations possible.

Sjmielh

Yes, there will be MacOS/ARM versions which will probably be between 2 and 3 times faster. The AVX gives x86_64 another ~4x speed up.

Will the current servers be able to cope with the increase in work returned with the new applications alone, let alone a boost in crunchers joining the project?

That's why I need to create a new set of WUs for the faster apps. If I didn't change the WUs, then the server would have to process >10x the current rate of WUs, which would bring the server to its knees. It's an easy enough change - each task now gets 10x the number of congruences to process, thereby slowing it down by 10x.

Ah, i wasn't aware of that.
So there isn't going to be a big cut in processing times for the current Tasks.

Ok, that makes things more complicated as the work done by the new and current apps is actually different- they're not processing the same Tasks.
It will be a case of setting the amount of Credit for an Optimised application task to result in a strongly similar RAC to the current Tasks.


While the Optimised application Tasks should produce significantly more Credit per Task, as more work is actually being done, but if you are to keep the existing application to process Tasks then the new one will need to result in a similar amount of Credit per hour.
A task that requires 10 times the processing should get 10 times the Credit- but if it gets 10 times the Credit for a similar processing time to the existing Tasks and applications, then no-one will want to continue processing the current Tasks and applications- they'd need 10 times the Credit as well (even though they're doing 10 times less work).

A big issue at Seti when they had different applications (and a big variability in processing time for a given amount of Credit for some application's Tasks) was people tended to migrate to processing those Tasks that gave them the highest RAC. That's when FLOPS counting (and eventually Credit New) were developed so that on a given system, the Credit per hour for the different applications was as close as possible to equal so that people were happy to process either as the reward/acknowledgment for their time was similar.
If you had a more powerful system, you got more done per hour, so you had a higher RAC, but the Credit awarded for a given Task would be the same across different systems (A given task took 4 days on a phone, 30sec on a GPU- but got the same Credit).


I think getting this right will take most of your effort.

---

Grant
Darwin NT, Australia.

There is no need to overcomplicate the issue of credits. As long as the Gridcoin whitelist rules are followed, any method and any amount will suffice.

---

So the guy who was just banned from the BOINC forums is now writing apps for this project?

Interesting.

Bullies like you (and Ian) fancy yourselves the owners of the BOINC ecosystem, using your connections to get users banned—that’s all there is to it. You use Claude for every single line of code you write, yet you have the nerve to criticize me. To make matters worse, despite all your "vibe coding," the results are abysmal and a complete failure.

Learn how to craft prompts and keep pace with the modern era. You’ve been contributing to this project for years—you could have optimized this code yourself. But you chose not to, and now you’re attacking those who did.

You’re just jealous.

---

... any method and any amount will suffice.

No, it won't, as history has shown.

The entire point of Credit in BOINC was to acknowledge the resources and time participants provide to a project, and to allow for valid comparisons between Hardware, OS, applications and between projects.
Numberfields has gone with a fixed amount of Credit for it's Tasks, and as the time taken to process them has decreased or increased with the differing batches, the amount of Credit has also been varied to keep the RAC for a given system relatively consistent with the RAC that system would have had with the initial work supplied.


Any increase in the amount of work done should reflect that in the Credit provided- however if the older applications and Tasks are going to remain in use, then the amount of Credit given to results produced by the new Taks and applications needs to take in to account the Credit produced by those still running legacy applications and Tasks, so that neither one nor the other ends up being blocked or just dumped due to the disparity in RAC they produce.


For reference-

A BOINC project gives you credit for the computations your computers perform for it. BOINC's unit of credit, the Cobblestone (named after Jeff Cobb of SETI@home), is 1/200 day of CPU time on a reference computer that does 1,000 MFLOPS based on the Whetstone benchmark.
 GigaFLOPs = RAC/200
 TeraFLOPS = RAC/200,000
 (Remember that a 1 GigaFLOP machine, running full time, produces 200 units of credit in 1 day).
1 GFLOP * 86,400sec = 200 Cobblestones.

---

Grant
Darwin NT, Australia.

It would be a bit of a downer for users if credit would remain the same although so much more work is being done. Is the power usage similar? If it's higher then the stock app then it'd be a hard sell for the same credit. Shoot, at least a small bump.

4 of the 10 top 'users' are pools where the user does not have access to the app selection and get what they get. Those 3 pools alone have 43% of the project credit. To benefit from the increased app performance, the project might need to favor the new apps when sending out work. I think Moo and Asteroids have something to send out the plan_class versions that perform the best on a given host as they have SSE/AVX/FMA and OCL/CUDA versions of the same app.

@Eric
If you have tested the credits on a host with an average CPU that supports AVX, then the results are fine.

In our project, you can see that AVX2 is 40% faster than AVX, while AVX-512 is three times faster. More work done in less time means more credits. Just look at SETI, Collatz, and some other projects from the past that used highly optimized applications. CPUs without AVX support are rarely used.

The important thing is to maximize the amount of work done in a short period of time.

@Eric
If you have tested the credits on a host with an average CPU that supports AVX, then the results are fine.

In our project, you can see that AVX2 is 40% faster than AVX, while AVX-512 is three times faster. More work done in less time means more credits. Just look at SETI, Collatz, and some other projects from the past that used highly optimized applications. CPUs without AVX support are rarely used.

The important thing is to maximize the amount of work done in a short period of time.

Yes
On Asteroids@home and NFS@home, there's an AVX512 application; it's faster, and the performance gains are very noticeable.
The only downside is that it causes the processors to overheat.