server maintenance

Message boards : News : server maintenance
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Eric Driver
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 8 Jul 11
Posts: 1323
Credit: 411,285,298
RAC: 247,587
Message 3632 - Posted: 19 Jan 2024, 23:40:14 UTC

Just a heads up, the server was upgraded to the latest version of Ubuntu this morning.

The database seems to be crashing every so often with an error about exceeding max connections. Bear with me as we look into this.
ID: 3632 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Eric Driver
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 8 Jul 11
Posts: 1323
Credit: 411,285,298
RAC: 247,587
Message 3633 - Posted: 20 Jan 2024, 16:13:18 UTC - in response to Message 3632.  

I believe all the kinks have been worked out now. But please let me know if you notice anything suspicious.
ID: 3633 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
pschoefer

Send message
Joined: 10 Oct 15
Posts: 5
Credit: 38,148,839
RAC: 16
Message 3634 - Posted: 21 Jan 2024, 6:57:30 UTC
Last modified: 21 Jan 2024, 7:02:53 UTC

It looks like the validator also went crazy at some point, everyone who was reporting tasks during that time was awarded several times their usual daily throughput between 19 Jan 14:00 UTC and 20 Jan 05:00 UTC. I cannot find any tasks with suspiciously high credit (or extremely short run times), so most likely the credit for each task was awarded several times. It has been back to normal since.

The only persistent "problems" I can see right now are PHP warnings on the login page:
Deprecated: urldecode(): Passing null to parameter #1 ($string) of type string is deprecated in /home/boincadm/projects/NumberFields/html/user/login_form.php on line 26
and on the result detail pages (e.g., https://numberfields.asu.edu/NumberFields/result.php?resultid=212105635):
Deprecated: Implicit conversion from float-string "7664.013119" to int loses precision in /home/boincadm/projects/NumberFields/html/inc/util.inc on line 413
Deprecated: Implicit conversion from float-string "7574.109" to int loses precision in /home/boincadm/projects/NumberFields/html/inc/util.inc on line 413
ID: 3634 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Eric Driver
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 8 Jul 11
Posts: 1323
Credit: 411,285,298
RAC: 247,587
Message 3635 - Posted: 21 Jan 2024, 16:11:38 UTC - in response to Message 3634.  

It looks like the validator also went crazy at some point, everyone who was reporting tasks during that time was awarded several times their usual daily throughput between 19 Jan 14:00 UTC and 20 Jan 05:00 UTC. I cannot find any tasks with suspiciously high credit (or extremely short run times), so most likely the credit for each task was awarded several times. It has been back to normal since.

The only persistent "problems" I can see right now are PHP warnings on the login page:
Deprecated: urldecode(): Passing null to parameter #1 ($string) of type string is deprecated in /home/boincadm/projects/NumberFields/html/user/login_form.php on line 26
and on the result detail pages (e.g., https://numberfields.asu.edu/NumberFields/result.php?resultid=212105635):
Deprecated: Implicit conversion from float-string "7664.013119" to int loses precision in /home/boincadm/projects/NumberFields/html/inc/util.inc on line 413
Deprecated: Implicit conversion from float-string "7574.109" to int loses precision in /home/boincadm/projects/NumberFields/html/inc/util.inc on line 413


Thanks for reporting those php warnings.

In regards to the multiple validations, that actually makes sense. The root cause of all the problems was an upgrade to python3. For some reason this caused the startup script to not create the lock files on the daemons and as a result every 5 minutes it would spawn new copies of the daemons. These daemons use database connections and so eventually we would exceed the max number of connections. In addition, the duplicate daemons were stepping all over themselves and causing all sorts of errors in the logs. Apparently the multiple validators each gave credits for the same task.
ID: 3635 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : News : server maintenance


Main page · Your account · Message boards


Copyright © 2024 Arizona State University