Server problems this past weekend

Message boards : News : Server problems this past weekend
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Eric Driver
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 8 Jul 11
Posts: 1144
Credit: 183,994,315
RAC: 162,967
Message 2918 - Posted: 16 Nov 2020, 17:21:24 UTC

We had what we thought was a hard drive crash - the drive went into a read-only state and the server crashed. It would not even allow a remote reboot. We finally got someone in there this morning to do a hard shutdown.

The system came back and appears to be working now. But we will continue to keep an eye on it.

If you are like me, you will have a bunch of WUs that need uploading. Please have patience as everyone will be uploading at the same time, so there might be delays.
ID: 2918 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 28 Oct 11
Posts: 164
Credit: 145,919,268
RAC: 53,617
Message 2919 - Posted: 16 Nov 2020, 17:57:57 UTC - in response to Message 2918.  

Welcome back. I've sent one batch up, and all seemed well - I'll do the others in stages.

Did you see the specific error message on uploads?

16/11/2020 14:43:17 | NumberFields@home | [error] Error reported by file upload server: can't open log file '../log_mimas/file_upload_handler.log' (errno: 9)
Might be helpful.
ID: 2919 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Eric Driver
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 8 Jul 11
Posts: 1144
Credit: 183,994,315
RAC: 162,967
Message 2920 - Posted: 16 Nov 2020, 18:18:54 UTC - in response to Message 2919.  

I don't see any errors in the logs on the server side. I looked at my client event log and see the same error about not being able to open file_upload_handler.log. The time stamp on those error messages are during the time the server was being brought back up. I am guessing it was during the file system check when the drive is made read only. But if you are still seeing these kinds of errors, let me know and I will dig deeper.
ID: 2920 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 28 Oct 11
Posts: 164
Credit: 145,919,268
RAC: 53,617
Message 2921 - Posted: 16 Nov 2020, 18:36:25 UTC - in response to Message 2920.  

No, it goes right back to the start of the problem:

14-Nov-2020 20:29:27 [NumberFields@home] Error reported by file upload server: can't open log file '../log_mimas/file_upload_handler.log' (errno: 9)
is the earliest on this machine.

I think it can happen if/when you run short on disk space?
ID: 2921 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : News : Server problems this past weekend


Main page · Your account · Message boards


Copyright © 2021 Arizona State University