Sparks Posted November 18, 2012 Share Posted November 18, 2012 To elaborate, specifically: Varnish froze and stopped accepting connections, but the process was still running (which means the server's watchdog program did not try to fix it, or page me to wake me up to deal with it). When I did get up and heard from Ran, I ran Varnish's admin tool... and watched Varnish silently crash rather than giving me any info. Woo.I've restarted Varnish, and I'm going to probably rewrite the watchdog later today to make sure Varnish is actually answering queries rather than checking if the process is running actively.As Ran notes, there was a scripted attack the other day against the entire data center where this machine is hosted, which caused some issues; I suspect this may be sort of the aftermath, but I'm going to look into stuff today. In the meantime, everything's back up. Link to comment Share on other sites More sharing options...
Florina Laufeyson Posted November 29, 2012 Share Posted November 29, 2012 I was wondering what was going on with the 503 errors at a certain time. Is this the server readjusting itself or something? Link to comment Share on other sites More sharing options...
Ran Posted November 29, 2012 Share Posted November 29, 2012 You know, not yet sure what's up with that. There seems to be something that's hit the server around the same time two evenings running that leads to excessive lag. Or could be coincidence. Link to comment Share on other sites More sharing options...
Light a wight tonight Posted November 29, 2012 Share Posted November 29, 2012 Just want to say that this is one of the best-running sites I'm on. Thank you, Sparks. That there are outages is irritating mostly because I miss it. Link to comment Share on other sites More sharing options...
Florina Laufeyson Posted November 29, 2012 Share Posted November 29, 2012 You know, not yet sure what's up with that. There seems to be something that's hit the server around the same time two evenings running that leads to excessive lag. Or could be coincidence.I get 503 errors and time outs. Its not unusual for such a thing to occur however, when the server is like recalculating or backing itself up. But anyway, if it happens again tonight, ill let ya know. Cuz its weird. Link to comment Share on other sites More sharing options...
Sparks Posted November 29, 2012 Share Posted November 29, 2012 If you give me a specific window of times, I'll take a look. However, right now, the only 503s I see in Varnish's log are when PHP is returning an error that prevents the page being cached.Which is an entirely different issue; when that one happens, it's specific to when there's a compressed page being returned and PHP tosses an uncompressed warning into the middle of the compressed stream, thus preventing the page from being uncompressed. Were we not going through Varnish, that sort of error would usually cause the browser to reload once, but Varnish instead goes 'What the hell I don't even OKAY YOU KNOW WHAT FORGET THIS, erroring out.' That one seems to go away after one reload, but I'm still trying to track down *why* PHP is periodically tossing an error.(The other place this can happen is if PHP returns an erroneous content-length for a compressed page; most browsers are somewhat more willing to accept Random Incorrect Junk than Varnish is; that's the other aspect I'm currently looking into.) Link to comment Share on other sites More sharing options...
K.C. Posted November 30, 2012 Share Posted November 30, 2012 If you give me a specific window of times, I'll take a look.2:44am your time (10:44am GMT) this time. There was a message about database issues, so presumably someone knew about it. Just trying to help if I can.Thanks for all the headaches you go through for us. Link to comment Share on other sites More sharing options...
Ran Posted November 30, 2012 Share Posted November 30, 2012 I think I figured out what the problem was. For some reason, a regularly scheduled process to clean out the content cache of the forum (the forum caches about 30% of its posts in a special table to make accessing them quicker -- it decides what to cache based on what's been visited in the last X days) has in the last four days decided that it need to ram up load to over 100 for awhile until it laboriously un-choked. The spike of lag at 2:44 that Cacti reports was when I manually ran the task, having become suspicious that it was our problem (which it was).The downtime was when I took things down to repair and optimize the offending table, and then I ran the scheduled task again to see if things changed. This time, it was done very quickly, so I think it's all good now. There was a smaller spike of lag afterward which I think may have been because the reputation tables (the source of many headaches for awhile) may have gotten finicky after the earlier lag, so I fixed those too. Link to comment Share on other sites More sharing options...
Eyron Posted November 30, 2012 Share Posted November 30, 2012 The problems I've noticed have occured almost at the same time every time. Sometimes it was a 500 error and sometimes 503, and at times there was a cloudflare message about the downtime.From what I gathered by other boarders notes:November 14th, 18th, 19th, 27th, 28th and 29th (probably some of the days inbetween as well but these are the ones that were noted) at appr. 04.00 GMT+1.ETA: crosspost! Link to comment Share on other sites More sharing options...
Ran Posted November 30, 2012 Share Posted November 30, 2012 Yep, that's when the task was regualrly scheduled. Hopefully the repair and optimize will sort that out from now on. I also took the precaution of rescheduling the task to 8 AM, which is a quieter time than the 4 AM that it generally ran. Link to comment Share on other sites More sharing options...
Florina Laufeyson Posted November 30, 2012 Share Posted November 30, 2012 Awesome! Thanks guys! Link to comment Share on other sites More sharing options...
Pebble thats Stubby Posted December 3, 2012 Share Posted December 3, 2012 Yay, Just reporting that the Malware false positive thing has cleared again, and I can now bum arround at work once more.Thankyou for everything yuo have done for us. Link to comment Share on other sites More sharing options...
mirielredux Posted December 4, 2012 Share Posted December 4, 2012 Hello~ I have an odd problem and am hoping a moderator can help. I have been a member of them board for some time (original use name: Miriel). Well, for some reason suddenly I could not log in. Of course, I'd forgotten my password but when I attempted to do the password retrieval it would not send me the email to reset it. So I waited a while then tried to contact somebody but my emails would not go through. The mailer daemon said "message undeliverable. Sender address rejected: unverified address." Anyway, so I take it that my email is persona non grata here for some reason. So, I joined again with a new user name and a gmail account (with the intent of posting in a help thread since I cannot seem to connect through email). I'd like to just go back to my old user name, etc if possible.Any help appreciated. Sorry to be any trouble. Link to comment Share on other sites More sharing options...
Stubby Posted December 5, 2012 Share Posted December 5, 2012 Hi miriel.If you haven't been here since last year some time, this thread may help you.Hope that is the fix for you. Link to comment Share on other sites More sharing options...
mirielredux Posted December 5, 2012 Share Posted December 5, 2012 Thank you for the help Stubby. But I don't think that is the issue in my case--I was able to log on as recently as last week. I think the problem began in my computer somehow--maybe a problem w/ my cookies? I had to log into other sites as well when this happened.The problem here with this board is it will not send me emails (for whatever reason). And it is the email I need to allow me to reset my password to get back into my miriel account.Perhaps it is unfixable? I am fine w/ just using this new account. Maybe a mod could bump my post count back up north of 750so I could have my custom title back. I'm a very slow poster and it is the little things in life you know :) Link to comment Share on other sites More sharing options...
Stubby Posted December 5, 2012 Share Posted December 5, 2012 Dunno what the issue might be with the email.It would be better to get your old account back, so we'll see what Ran suggests in due course. Link to comment Share on other sites More sharing options...
Ran Posted December 5, 2012 Share Posted December 5, 2012 Miriel,I have emailed you. Link to comment Share on other sites More sharing options...
Miriel Posted December 5, 2012 Share Posted December 5, 2012 Miriel,I have emailed you.All fixed! Thank you Ran!!! And thank you Stubby! Link to comment Share on other sites More sharing options...
Alia of the knife Posted December 7, 2012 Share Posted December 7, 2012 Think I'm starting to have e-mail notification issues again?And thanks as always Sparks. :) Link to comment Share on other sites More sharing options...
kissdbyfire Posted December 8, 2012 Share Posted December 8, 2012 Think I'm starting to have e-mail notification issues again?And thanks as always Sparks. :)Me too. Either that or everyone is ignoring me - not that I'm paranoid or anything... :wideeyed:Sparks, you're virtually a goddess... Or a virtual goddess? Both? :) Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.