268 replies to this topic

[Sparks]

To elaborate, specifically: Varnish froze and stopped accepting connections, but the process was still running (which means the server's watchdog program did not try to fix it, or page me to wake me up to deal with it).  When I did get up and heard from Ran, I ran Varnish's admin tool... and watched Varnish silently crash rather than giving me any info.  Woo.

I've restarted Varnish, and I'm going to probably rewrite the watchdog later today to make sure Varnish is actually answering queries rather than checking if the process is running actively.

As Ran notes, there was a scripted attack the other day against the entire data center where this machine is hosted, which caused some issues; I suspect this may be sort of the aftermath, but I'm going to look into stuff today.  In the meantime, everything's back up.

[Florina Stark]

I was wondering what was going on with the 503 errors at a certain time. Is this the server readjusting itself or something?

[Ran]

You know, not yet sure what's up with that. There seems to be something that's hit the server around the same time two evenings running that leads to excessive lag. Or could be coincidence.

[Light a wight tonight]

Just want to say that this is one of the best-running sites I'm on. Thank you, Sparks. That there are outages is irritating mostly because I miss it.

[Florina Stark]

Ran, on 29 November 2012 - 03:39 AM, said:

You know, not yet sure what's up with that. There seems to be something that's hit the server around the same time two evenings running that leads to excessive lag. Or could be coincidence.

I get 503 errors and time outs. Its not unusual for such a thing to occur however, when the server is like recalculating or backing itself up. But anyway, if it happens again tonight, ill let ya know. Cuz its weird.

[Sparks]

If you give me a specific window of times, I'll take a look.  However, right now, the only 503s I see in Varnish's log are when PHP is returning an error that prevents the page being cached.

Which is an entirely different issue; when that one happens, it's specific to when there's a compressed page being returned and PHP tosses an uncompressed warning into the middle of the compressed stream, thus preventing the page from being uncompressed.  Were we not going through Varnish, that sort of error would usually cause the browser to reload once, but Varnish instead goes 'What the hell I don't even OKAY YOU KNOW WHAT FORGET THIS, erroring out.'  That one seems to go away after one reload, but I'm still trying to track down *why* PHP is periodically tossing an error.

(The other place this can happen is if PHP returns an erroneous content-length for a compressed page; most browsers are somewhat more willing to accept Random Incorrect Junk than Varnish is; that's the other aspect I'm currently looking into.)

Edited by Sparks, 29 November 2012 - 04:22 PM.

[K.C.]

Sparks, on 29 November 2012 - 04:19 PM, said:

If you give me a specific window of times, I'll take a look.

2:44am your time (10:44am GMT) this time. There was a message about database issues, so presumably someone knew about it. Just trying to help if I can.

Thanks for all the headaches you go through for us.

[Ran]

I think I figured out what the problem was. For some reason, a regularly scheduled process to clean out the content cache of the forum (the forum caches about 30% of its posts in a special table to make accessing them quicker -- it decides what to cache based on what's been visited in the last X days) has in the last four days decided that it need to ram up load to over 100 for awhile until it laboriously un-choked. The spike of lag at 2:44 that Cacti reports was when I manually ran the task, having become suspicious that it was our problem (which it was).

The downtime was when I took things down to repair and optimize the offending table, and then I ran the scheduled task again to see if things changed. This time, it was done very quickly, so I think it's all good now. There was a smaller spike of lag afterward which I think may have been because the reputation tables (the source of many headaches for awhile) may have gotten finicky after the earlier lag, so I fixed those too.

Edited by Ran, 30 November 2012 - 07:15 AM.

[Eyron I]

The problems I've noticed have occured almost at the same time every time. Sometimes it was a 500 error and sometimes 503, and at times there was a cloudflare message about the downtime.

From what I gathered by other boarders notes:

November 14th, 18th, 19th, 27th, 28th and 29th (probably some of the days inbetween as well but these are the ones that were noted) at appr. 04.00 GMT+1.

ETA: crosspost!

Edited by Eyron I, 30 November 2012 - 07:21 AM.

[Ran]

Yep, that's when the task was regualrly scheduled. Hopefully the repair and optimize will sort that out from now on. I also took the precaution of rescheduling the task to 8 AM, which is a quieter time than the 4 AM that it generally ran.

[Florina Stark]

Awesome! Thanks guys!

[Pebbles]

Yay,  Just reporting that the Malware false positive thing has cleared again, and I can now bum arround at work once more.

Thankyou for everything yuo have done for us.

[mirielredux]

Hello~ I have an odd problem and am hoping a moderator can help. I have been a member of them board for some time (original use name: Miriel).  Well, for some reason suddenly I could not log in. Of course, I'd forgotten my password but when I attempted to do the password retrieval it would not send me the email to reset it. So I waited a while then tried to contact somebody but my emails would not go through. The mailer daemon said "message undeliverable.  Sender address rejected: unverified address."  Anyway, so I take it that my email is persona non grata here for some reason.  

So, I joined again with a new user name and a gmail account (with the intent of posting in a help thread since I cannot seem to connect through email). I'd like to just go back to my old user name, etc if possible.

Any help appreciated.  Sorry to be any trouble.

Edited by mirielredux, 04 December 2012 - 06:11 PM.

[Stubby]

Hi miriel.

If you haven't been here since last year some time, this thread may help you.

Hope that is the fix for you.

[mirielredux]

Thank you for the help Stubby.  But I don't think that is the issue in my case--I was able to log on as recently as last week. I think the problem began in my computer somehow--maybe a problem w/ my cookies? I had to log into other sites as well when this happened.

The problem here with this board is it will not send me emails (for whatever reason).  And it is the email I need to allow me to reset my password to get back into my miriel account.

Perhaps it is unfixable? I am fine w/ just using this new account.  Maybe a mod could  bump my  post count back up north of 750so I could have my custom title back.  I'm a very slow poster and it is the little things in life you know

[Stubby]

Dunno what the issue might be with the email.

It would be better to get your old account back, so we'll see what Ran suggests in due course.

[Ran]

Miriel,

I have emailed you.

[Miriel]

Ran, on 05 December 2012 - 06:36 AM, said:

Miriel,

I have emailed you.

All fixed!  Thank you Ran!!!  And thank you Stubby!

[Alia of the knife]

Think I'm starting to have e-mail notification issues again?

And thanks as always Sparks.

[kissdbyfire]

Alia of the knife, on 07 December 2012 - 02:58 PM, said:

Think I'm starting to have e-mail notification issues again?

And thanks as always Sparks.

Me too. Either that or everyone is ignoring me - not that I'm paranoid or anything...

Sparks, you're virtually a goddess... Or a virtual goddess? Both?