Posts by Richard Haselgrove

1) Message boards : Number crunching : I cant send finished wus (Message 2018)
Posted 11 Apr 2018 by Richard Haselgrove
Post:
And we can see that WU (23257062 - task 25321670) was reported and validated by host 95038 at 2:10:51 UTC on 10 Apr 2018. We can't see the user name ('anonymous'), to match the 'hidden' setting for your computers. But you might be able to see some host log activity around that time.
2) Message boards : Number crunching : I cant send finished wus (Message 2016)
Posted 11 Apr 2018 by Richard Haselgrove
Post:
Here is an absolutely off-the-wall observation, which may be completely unrelated - but it has some similarities with this situation. I present it as food for thought, nothing more.

Like HerrJeh, I have multiple machines - all are at my home, and share an internet connection. They have the same public IP addresses, but different private IP addresses behind the router. I'm connected to a different BOINC project, which like this one runs every BOINC function from a single server.

That other project distributes long-running (up to 18 hour) workunits, but regards them as time-critical - it likes to have them returned within 24 hours. So I don't want to get a cache of tasks in advance, and in fact the project often has no tasks immediately available on demand. All of which is a long way of explaining why I find myself repeatedly clicking the 'Update' button in an attempt to get new work before the current task completes.

What I am observing is that I can issue repeated requests every 30 seconds from one machine, and connect to the server every time. But if I try to connect from a different machine (same LAN, same IP) in between, the second machine can't connect. If I stop updating the first machine for a minute or two, the second machine can connect, and goes on connecting for as long as is needed and allowed by their 30-second backoff interval. The nature of the server contact required doesn't affect the connection failures: I've just had a machine which couldn't connect to upload results, while I was requesting new work on a different machine.

It isn't simply congestion at the server port: that would be more random. This observation is strictly about multiple connection attempts, closely spaced in time, from different computers sharing the same public IP address. It feels more like a server OS-level problem than a BOINC problem, and it's been consistent for weeks, if not months.

Their server is running

11/04/2018 11:03:50 | | [http] [ID#1] Received header from server: Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_auth_gssapi/1.3.1 mod_auth_kerb/5.4 mod_fcgid/2.3.9 PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5
My machines are all Windows 7/64 and run recent versions of BOINC (mostly v7.9 test builds). I haven't yet explored the http logs for further clues, but I'll give it a try when I have time.
3) Message boards : Number crunching : I cant send finished wus (Message 2009)
Posted 10 Apr 2018 by Richard Haselgrove
Post:
hosts that could return Decic tasks but not Septic tasks.
I noticed that in the logs, and couldn't understand it. Both types are uploading from my machines with no distinction.

All communications - specifically including uploading - are handled by BOINC (the same BOINC in both cases), so there should be no difference. The build tools and libraries used in your applications should make no difference either - the comms happen after your app has already shut down. I looked at the destination of each type:

<upload_url>http://numberfields.asu.edu/NumberFields_cgi/file_upload_handler/</upload_url> (septics)
<upload_url>http://numberfields.asu.edu/NumberFields_cgi/file_upload_handler/</upload_url> (sf4_DS-13x271)

and my eye can't spot a difference. Is there a difference in the file size offered for upload?
4) Message boards : Number crunching : I cant send finished wus (Message 1999)
Posted 4 Apr 2018 by Richard Haselgrove
Post:
Maybe others are also unable to return or get new work?
I've not noticed any problems here, across multiple Windows machines. Here's the http_debug for a successful upload, for comparison. I don't think xfer_debug would add much extra information (though no harm in selecting it): these problems usually occur during the header negotiation.


04/04/2018 09:26:18 | NumberFields@home | [http] [ID#15238] Sent header to server: Accept: */*
04/04/2018 09:26:18 | NumberFields@home | [http] [ID#15238] Sent header to server: Accept-Encoding: deflate, gzip
04/04/2018 09:26:18 | NumberFields@home | [http] [ID#15238] Sent header to server: Content-Type: application/x-www-form-urlencoded
04/04/2018 09:26:18 | NumberFields@home | [http] [ID#15238] Sent header to server: Accept-Language: en_GB
04/04/2018 09:26:18 | NumberFields@home | [http] [ID#15238] Sent header to server: Content-Length: 721
04/04/2018 09:26:18 | NumberFields@home | [http] [ID#15238] Sent header to server:
04/04/2018 09:26:18 | NumberFields@home | [http] [ID#15238] Sent header to server: Expect: 100-continue
04/04/2018 09:26:18 | NumberFields@home | [http] [ID#15238] Sent header to server:
01/01/1970 00:00:00 | NumberFields@home |
04/04/2018 09:26:18 | NumberFields@home | [http] [ID#15238] Info: We are completely uploaded and fine
04/04/2018 09:26:18 | NumberFields@home | [http] [ID#15238] Received header from server: HTTP/1.1 301 Moved Permanently
04/04/2018 09:26:18 | NumberFields@home | [http] [ID#15238] Received header from server: Date: Wed, 04 Apr 2018 08:26:29 GMT
04/04/2018 09:26:18 | NumberFields@home | [http] [ID#15238] Received header from server: Server: Apache/2.4.18 (Ubuntu)
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: the ioctl callback returned 0
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Received header from server: Location: https://numberfields.asu.edu/NumberFields_cgi/file_upload_handler/
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Received header from server: Content-Length: 360
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Received header from server: Content-Type: text/html; charset=iso-8859-1
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Received header from server:
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Received header from server: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Received header from server: <html><head>
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Received header from server: <title>301 Moved Permanently</title>
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Received header from server: </head><body>
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Received header from server: <h1>Moved Permanently</h1>
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Received header from server: <p>The document has moved <a href="https://numberfields.asu.edu/NumberFields_cgi/file_upload_handler/">here</a>.</p>
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Received header from server: <hr>
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Received header from server: <address>Apache/2.4.18 (Ubuntu) Server at numberfields.asu.edu Port 80</address>
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Received header from server: </body></html>
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: Ignoring the response-body
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: Connection #21336 to host numberfields.asu.edu left intact
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: Issue another request to this URL: 'https://numberfields.asu.edu/NumberFields_cgi/file_upload_handler/'
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: Found bundle for host numberfields.asu.edu: 0x4180640 [can pipeline]
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: Trying 129.219.51.76...
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: Connected to numberfields.asu.edu (129.219.51.76) port 443 (#21337)
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: ALPN, offering http/1.1
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: successfully set certificate verify locations:
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: CAfile: D:\BOINC\ca-bundle.crt
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: CApath: none
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: TLSv1.2 (OUT), TLS header, Certificate Status (22):
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: TLSv1.2 (OUT), TLS handshake, Client hello (1):
04/04/2018 09:26:19 | NumberFields@home | [http] [ID#15238] Info: TLSv1.2 (IN), TLS handshake, Server hello (2):
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: TLSv1.2 (IN), TLS handshake, Certificate (11):
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: TLSv1.2 (IN), TLS handshake, Server key exchange (12):
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: TLSv1.2 (IN), TLS handshake, Server finished (14):
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: TLSv1.2 (OUT), TLS change cipher, Client hello (1):
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: TLSv1.2 (OUT), TLS handshake, Finished (20):
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: TLSv1.2 (IN), TLS change cipher, Client hello (1):
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: TLSv1.2 (IN), TLS handshake, Finished (20):
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: ALPN, server accepted to use http/1.1
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: Server certificate:
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: subject: C=US; postalCode=85287; ST=AZ; L=Tempe; street=502 East Orange Street; O=Arizona State University; OU=University Technology Office; CN=numberfields.asu.edu
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: start date: Aug 23 00:00:00 2016 GMT
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: expire date: Aug 23 23:59:59 2019 GMT
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: subjectAltName: numberfields.asu.edu matched
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: issuer: C=US; ST=MI; L=Ann Arbor; O=Internet2; OU=InCommon; CN=InCommon RSA Server CA
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: SSL certificate verify ok.
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: POST /NumberFields_cgi/file_upload_handler/ HTTP/1.1
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: Host: numberfields.asu.edu
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.9.2)
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: Accept: */*
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: Accept-Encoding: deflate, gzip
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: Referer: http://numberfields.asu.edu/NumberFields_cgi/file_upload_handler/
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: Content-Type: application/x-www-form-urlencoded
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: Accept-Language: en_GB
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: Content-Length: 721
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server:
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: s>
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: </file_info>
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: <nbytes>254</nbytes>
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: <md5_cksum>fdc751521d8ae36a49ae3affc462c760</md5_cksum>
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: <offset>0</offset>
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Sent header to server: <data>
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: We are completely uploaded and fine
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Received header from server: HTTP/1.1 200 OK
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Received header from server: Date: Wed, 04 Apr 2018 08:26:30 GMT
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Received header from server: Server: Apache/2.4.18 (Ubuntu)
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Received header from server: Content-Length: 64
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Received header from server: Content-Type: text/plain
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Received header from server:
04/04/2018 09:26:20 | NumberFields@home | [http] [ID#15238] Info: Connection #21337 to host numberfields.asu.edu left intact
04/04/2018 09:26:20 | NumberFields@home | Finished upload of wu_sf4_DS-13x271-4_Grp106163of200000_0_r122425676_0
If the problem is widespread, one possible cause is the security certificate verification you see here. I'm using a test version of BOINC (v7.9.2) which includes an updated ca-bundle.crt file: the latest test versions (currently v7.9.3, soon to be v7.9.4) are available at https://boinc.berkeley.edu/download_all.php
5) Message boards : News : Minor server overload problems (Message 1973)
Posted 27 Feb 2018 by Richard Haselgrove
Post:
I was in a conference call this evening with a couple of *very* experienced BOINC server administrators. One thought that on a lightly-loaded project, a database connection might well time out between active requests: the other had never seen such a thing. On reflection, both thought there might be some useful information about a possible database server stoppage in the MySQL logs.

They suggested I copy your question to the boinc_projects mailing list, both to remind them to look again, and to get some broader responses from the community. OK if I do that in the morning?
6) Message boards : News : Minor server overload problems (Message 1971)
Posted 27 Feb 2018 by Richard Haselgrove
Post:
The server has gone into the 'feeder not running' maintenance mode again - 1 hour backoff on scheduler requests, no reporting and no new work.

Edit - sorry, working now. Panic over.
7) Message boards : News : Database crash (Message 1939)
Posted 8 Jan 2018 by Richard Haselgrove
Post:
Yes, things getting back to normal here too - reported completed work and got a few new ones.

I'm getting some of those "permanent http error" too. They seem to be data files missing from the download storage area (HTTP 404), so of course the downloads fail for all replications of the workunit. You'll have a few jobs to re-issue when all this is over.
8) Message boards : News : Minor server overload problems (Message 1929)
Posted 31 Dec 2017 by Richard Haselgrove
Post:
It's fairly well known that a stopped feeder puts the server into a form of maintenance mode - nothing gets done, volunteer hosts are backed off for 1 hour (as my log shows). I presume this is deliberate to stop the situation getting worse until it can be inspected.

What that doesn't say is why the feeder stopped in the first place - your second post about the lack of temp file space seems as good an explanation as any.

At intervals throughout the day, I've seen the project come back up fully (so I could report and refill): then go into full maintenance mode with these boards down as well: then return to normal working (the current state: just reported and received new tasks). Which sounds like a good moment to go out and start celebrating the new year...
9) Message boards : News : Minor server overload problems (Message 1926)
Posted 31 Dec 2017 by Richard Haselgrove
Post:
Probably related to the other problems, I'm currently unable to report completed tasks: the messages I get are

31/12/2017 09:27:54 | NumberFields@home | Server error: feeder not running
31/12/2017 09:27:54 | NumberFields@home | Project requested delay of 3600 seconds
Not a problem - they can sit here until you're ready for them.
10) Message boards : Number crunching : No work 'ready to send' (Message 1903)
Posted 6 Nov 2017 by Richard Haselgrove
Post:
Yes, I'm topping up nicely. I guess the new batch ran more quickly than our machines are used to after DS12x271.
11) Message boards : Number crunching : No work 'ready to send' (Message 1901)
Posted 5 Nov 2017 by Richard Haselgrove
Post:
Although you opened up new search DS13x270 for us last week, we seem to be getting ahead of you ;-)
12) Message boards : News : implementing SSL on the server (Message 1784)
Posted 16 Nov 2016 by Richard Haselgrove
Post:
I tried out version 5.10.45, as it can be used portable on my USB drive, and received the message:
16-Nov-2016 08:51:21 [NumberFields@home] Scheduler request failed: Peer certificate cannot be authenticated with known CA certificates

I suppose that means no more portable operating for me.

You could try extracting the file 'ca-bundle.crt' from a newer BOINC download and replacing your old one with that. No promise that it will work, though.
13) Message boards : News : implementing SSL on the server (Message 1778)
Posted 11 Oct 2016 by Richard Haselgrove
Post:
I'd be interested to hear if someone found a simpler way to connect using an older machine/manager.

I see that Vitaly also has a Windows 7 machine attached to the project.

If he finds the file 'account_numberfields.asu.edu_NumberFields.xml' in the root of the BOINC data directory on that machine, and copies it to the equivalent location in the data directory of the Ubuntu machine, it *may* attach that machine to the project. You may need to restart the BOINC client/service.

I've used that method to attach a new Windows machine in the past, but I can't be sure that Linux will accept the Windows file format (CRLF line endings, instead of *nix LF only). Also, I did it before SSL came into widespread use: in fact, my own account file still has the http:// master url, and I keep getting nagged to detach and re-attach 'when convenient'. But since it's currently still working, I haven't bothered.
14) Message boards : Number crunching : Upload problems? (Message 1771)
Posted 1 Sep 2016 by Richard Haselgrove
Post:
I had that all day (starting about 07:00 UTC): the "transient HTTP error" in question was a server timeout, according to the http_debug log.

I had the same timeout error when attempting to access this website, but when this site came back to life again (about an hour and a half ago), the uploads resumed too. Worth giving them a prod with the Transfers::Retry Now button.
15) Message boards : Number crunching : Qsqrt421_DS3x8 15+ hour running time? (Message 1752)
Posted 6 Jul 2016 by Richard Haselgrove
Post:
I reckon I've got four of those:

wu_Qsqrt421_DS1x8_CV1_S1000_N2_21_N1_-6868to-4346 (the original _0 dated 14 June, still running on a slower machine)
wu_Qsqrt421_DS1x8_CV1_S1000_N2_41_N1_-7063to-4131
wu_Qsqrt421_DS1x8_CV1_S1000_N2_10_N1_-4391to-1809 (I'll abort that one, someone else has completed it)
wu_Qsqrt421_DS1x8_CV1_S1000_N2_28_N1_-4197to-1862

wu_Qsqrt421_DS1x8_CV1_S1000_N2_49_N1_-3971to-1938 had already slipped through - not as slow as the others - before I saw your post.
16) Message boards : Number crunching : Qsqrt421_DS3x8 15+ hour running time? (Message 1749)
Posted 27 Jun 2016 by Richard Haselgrove
Post:
Please don't feel you have to. I had

wu_Qsqrt421_DS1x8_CV1_S1000_N2_49_N1_-7142to-4045_0 for several days last week, but it finished fine and within the original deadline.

There does seem to be a class of workunit which spends a lot of time on the first few %age steps at the start of computation, and a similar slow phase at the end, but which runs through the middle section very quickly.
17) Message boards : News : Bounded App Final Tally of Results (Message 1731)
Posted 23 May 2016 by Richard Haselgrove
Post:
Yes. The original Hobbes link gave me, and is still giving,

This site can’t be reached

hobbes.la.asu.edu refused to connect.

ERR_CONNECTION_REFUSED

The use of the word "refused" implies something more specific than a simple failure.

The locally-hosted copy displays fine, though.
18) Message boards : Number crunching : Upload problems May 12 (Message 1723)
Posted 12 May 2016 by Richard Haselgrove
Post:
LOL - posting about it did the trick. All cleared now.
19) Message boards : Number crunching : Upload problems May 12 (Message 1722)
Posted 12 May 2016 by Richard Haselgrove
Post:
I'm getting upload failures on three machines so far.

12/05/2016 08:58:23 | NumberFields@home | [http] [ID#108] Info:  Connected to numberfields.asu.edu (129.219.51.76) port 80 (#191)
12/05/2016 08:58:23 | NumberFields@home | [http] [ID#108] Sent header to server: POST /NumberFields_cgi/file_upload_handler/ HTTP/1.1
12/05/2016 08:58:23 | NumberFields@home | [http] [ID#108] Sent header to server: Host: numberfields.asu.edu
12/05/2016 08:58:23 | NumberFields@home | [http] [ID#108] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.6.22)
12/05/2016 08:58:23 | NumberFields@home | [http] [ID#108] Sent header to server: Accept: */*
12/05/2016 08:58:23 | NumberFields@home | [http] [ID#108] Sent header to server: Accept-Encoding: deflate, gzip
12/05/2016 08:58:23 | NumberFields@home | [http] [ID#108] Sent header to server: Content-Type: application/x-www-form-urlencoded
12/05/2016 08:58:23 | NumberFields@home | [http] [ID#108] Sent header to server: Accept-Language: en_GB
12/05/2016 08:58:23 | NumberFields@home | [http] [ID#108] Sent header to server: Content-Length: 722
12/05/2016 08:58:23 | NumberFields@home | [http] [ID#108] Sent header to server:
12/05/2016 08:58:23 | NumberFields@home | [http] [ID#108] Info:  We are completely uploaded and fine
12/05/2016 08:59:28 | NumberFields@home | [http] [ID#108] Info:  Operation too slow. Less than 10 bytes/sec transferred the last 60 seconds
12/05/2016 08:59:28 | NumberFields@home | [http] [ID#108] Info:  Closing connection 191
12/05/2016 08:59:28 | NumberFields@home | [http] HTTP error: Timeout was reached
12/05/2016 08:59:28 | NumberFields@home | Temporarily failed upload of wu_sf3_DS-11x271_Grp158560of614400_0_0: transient HTTP error
12/05/2016 08:59:28 | NumberFields@home | Backing off 01:51:48 on upload of wu_sf3_DS-11x271_Grp158560of614400_0_0

The upload appears to complete, but I don't get the necessary acknowledgement that all is well.
20) Message boards : Number crunching : Long running wu_Qsqrt421_DS1x5 units - how long to let them run? (Message 1694)
Posted 19 Apr 2016 by Richard Haselgrove
Post:
This was explained to me once, many years ago - by Rom Walton, I think.

The action was

Exit status	203 (0xcb) EXIT_ABORTED_VIA_GUI

- user clicked the abort button.

They (the BOINC developers) deliberately coded it to throw a breakpoint on abort, to get some state debug information and feed it back to the developer. Perhaps, like in this thread, the user aborted it because it seemed to be in an infinite loop? If so, we'd like to know which code segment had the bug so it could be fixed.

There aren't many developer aids in the standard BOINC client: why they chose this one, I don't know. Maybe there were a lot of infinite loops in their test code?


Next 20


Main page · Your account · Message boards


Copyright © 2018 Arizona State University