You may have noticed, we were down for about 24 hours.

Turned out to be nothing serious. For those interested in the details, read on...
Someone in the department (not me) did a remote reboot of the server and it did not come back up. They had to send someone into the server room to manually boot it. Then there was a network problem that took a while to diagnose. I was told the connection was changed to static so they had to change some network settings to make it work.

Glad to hear it was nothing major.

---

Grant
Darwin NT, Australia.

Thanks for the update, Eric. Happy it was nothing serious with the actual server hardware.

Thanks for the info. Glad it wasn't another Gerasim ;)

Thanks for the info. Glad it wasn't another Gerasim ;)

I do daily backups of the database, so no worries there.

Thanks for the info. Glad it wasn't another Gerasim ;)

I do daily backups of the database, so no worries there.

Smart.

Well, the web site is back.
But uploads, downloads & Scheduler requests are still slow/timing out and taking a couple of tries to get through.
And posting takes forever to finally happen.

Some network/server issues?
AI bot attack?

---

Grant
Darwin NT, Australia.

And for the last couple of hours uploads, downloads and Scheduler responses have returned to normal.

---

Grant
Darwin NT, Australia.

And then the Web site goes MIA again, along with stalled uploads, downloads etc. ANs then it's back again.

Things are not well here.

---

Grant
Darwin NT, Australia.

And then the Web site goes MIA again, along with stalled uploads, downloads etc. ANs then it's back again.

Things are not well here.

I too noticed the periodic lag in connecting. I looked at the apache2 access logs and found a ton of requests from "Lightpanda", mostly requesting result pages from random users. I googled this and found that is was most likely an AI web scraper trolling for information. Gemini helped my modify the apache config to block all requests from Lightpanda. Let's hope this fixes the problem.

Still taking a significant time to process standard transactions - moving from page to page in the website, and uploading/reporting completed tasks.

I think this has been a concern for several BOINC projects for a few months now. There's been a lot of activity in the Berkeley BOINC community, with a lot of work being contributed by new volunteer code developers, using AI programming tools to tidy up the code base, and hardening it against these sort of pointless AI data scrapes.

BOINC isn't yet ready for an update release for either the server code or the client: but it might be worth you reporting your observations back to David Anderson in Berkeley. He might even be able to give you an estimate for the timeline of the hardened code.

I still notice some sluggishness, but looking at the access logs it appears to be legit - many downloads, uploads, and general web access; all from different IP addresses. There may be some backlog with client requests due to the previous bot problem blocking traffic. Hopefully this will clear up with time.

I still notice some sluggishness,

Likewise, but at least the web site and forums are back and functioning.
And while uploads etc are still taking a while, or more than one attempt, they are quicker when they do go through and it's not taking more than a second attempt for them to get through.

Edit- posted too soon, forum is now back to a crawl.
And the web site is unreachable on and off again.

ERR_TIMED_OUT

%*@*(%# AI.

---

Grant
Darwin NT, Australia.

Had to block a few more web scrapers like YisouSpider and Yandex. Things seem to be better at the moment. Let's hope it stays that way.

Things seem to be better at the moment. Let's hope it stays that way.

Yep, things are good at present, and have been for a while now. Fingers crossed that's the worst of it.

---

Grant
Darwin NT, Australia.

Things seem to be better at the moment. Let's hope it stays that way.

Yep, things are good at present, and have been for a while now. Fingers crossed that's the worst of it.

I can see that they are still trying to access the site, but they are being sent a 403 "access forbidden" response. They were trying to scrape information from the various database tables: hosts, users, results, etc. Not necessarily malicious but most likely trying to obtain information for training their LLMs.

The root cause of the slowness was the db queries. The top command showed mariadb at the top followed by a bunch of apache2 threads. Things should be good for now until another bot shows up.

Web site taking ages to load, forums slow, and getting upload timeouts again.

---

Grant
Darwin NT, Australia.

Web site taking ages to load, forums slow, and getting upload timeouts again.

I noticed that too. I didn't see anything in the apache access logs suggesting another bot attack, but I restarted the apache2, php-fpm, and mariadb services just for good measure. As of this writing things seem to be back to normal.