The Latest News
Connect with Us

Notable Releases
From the Store
Game of Thrones Stark Metallic Ink V-Neck T-Shirt
Stark Metallic V-neck T-Shirt
HBO US
Featured Sites
License Holders

Jump to content


Photo

Board Issues 2


  • This topic is locked This topic is locked
403 replies to this topic

#41 Sparks

Sparks

    Sysadmin of Ice and Fire

  • Members
  • Pip
  • 19 posts

Posted 18 November 2012 - 02:34 PM

To elaborate, specifically: Varnish froze and stopped accepting connections, but the process was still running (which means the server's watchdog program did not try to fix it, or page me to wake me up to deal with it). When I did get up and heard from Ran, I ran Varnish's admin tool... and watched Varnish silently crash rather than giving me any info. Woo.

I've restarted Varnish, and I'm going to probably rewrite the watchdog later today to make sure Varnish is actually answering queries rather than checking if the process is running actively.

As Ran notes, there was a scripted attack the other day against the entire data center where this machine is hosted, which caused some issues; I suspect this may be sort of the aftermath, but I'm going to look into stuff today. In the meantime, everything's back up.

#42 Florina Laufeyson

Florina Laufeyson

    Heišinn

  • Members
  • PipPipPipPipPipPipPipPip
  • 6,061 posts

Posted 28 November 2012 - 11:57 PM

I was wondering what was going on with the 503 errors at a certain time. Is this the server readjusting itself or something?

#43 Ran

Ran

    King o' the Board

  • Administrators
  • PipPipPipPipPipPipPipPip
  • 30,582 posts

Posted 29 November 2012 - 03:39 AM

You know, not yet sure what's up with that. There seems to be something that's hit the server around the same time two evenings running that leads to excessive lag. Or could be coincidence.

#44 Light a wight tonight

Light a wight tonight

    I bid you to burn!

  • Members
  • PipPipPipPipPipPipPipPip
  • 3,830 posts

Posted 29 November 2012 - 01:53 PM

Just want to say that this is one of the best-running sites I'm on. Thank you, Sparks. That there are outages is irritating mostly because I miss it.

#45 Florina Laufeyson

Florina Laufeyson

    Heišinn

  • Members
  • PipPipPipPipPipPipPipPip
  • 6,061 posts

Posted 29 November 2012 - 04:10 PM

You know, not yet sure what's up with that. There seems to be something that's hit the server around the same time two evenings running that leads to excessive lag. Or could be coincidence.

I get 503 errors and time outs. Its not unusual for such a thing to occur however, when the server is like recalculating or backing itself up. But anyway, if it happens again tonight, ill let ya know. Cuz its weird.

#46 Sparks

Sparks

    Sysadmin of Ice and Fire

  • Members
  • Pip
  • 19 posts

Posted 29 November 2012 - 04:19 PM

If you give me a specific window of times, I'll take a look. However, right now, the only 503s I see in Varnish's log are when PHP is returning an error that prevents the page being cached.

Which is an entirely different issue; when that one happens, it's specific to when there's a compressed page being returned and PHP tosses an uncompressed warning into the middle of the compressed stream, thus preventing the page from being uncompressed. Were we not going through Varnish, that sort of error would usually cause the browser to reload once, but Varnish instead goes 'What the hell I don't even OKAY YOU KNOW WHAT FORGET THIS, erroring out.' That one seems to go away after one reload, but I'm still trying to track down *why* PHP is periodically tossing an error.

(The other place this can happen is if PHP returns an erroneous content-length for a compressed page; most browsers are somewhat more willing to accept Random Incorrect Junk than Varnish is; that's the other aspect I'm currently looking into.)

Edited by Sparks, 29 November 2012 - 04:22 PM.


#47 K.C.

K.C.

    Noble

  • Members
  • PipPipPipPipPipPipPip
  • 680 posts

Posted 30 November 2012 - 06:29 AM

If you give me a specific window of times, I'll take a look.


2:44am your time (10:44am GMT) this time. There was a message about database issues, so presumably someone knew about it. Just trying to help if I can.

Thanks for all the headaches you go through for us.

#48 Ran

Ran

    King o' the Board

  • Administrators
  • PipPipPipPipPipPipPipPip
  • 30,582 posts

Posted 30 November 2012 - 07:15 AM

I think I figured out what the problem was. For some reason, a regularly scheduled process to clean out the content cache of the forum (the forum caches about 30% of its posts in a special table to make accessing them quicker -- it decides what to cache based on what's been visited in the last X days) has in the last four days decided that it need to ram up load to over 100 for awhile until it laboriously un-choked. The spike of lag at 2:44 that Cacti reports was when I manually ran the task, having become suspicious that it was our problem (which it was).

The downtime was when I took things down to repair and optimize the offending table, and then I ran the scheduled task again to see if things changed. This time, it was done very quickly, so I think it's all good now. There was a smaller spike of lag afterward which I think may have been because the reputation tables (the source of many headaches for awhile) may have gotten finicky after the earlier lag, so I fixed those too.

Edited by Ran, 30 November 2012 - 07:15 AM.


#49 Eyron I

Eyron I

    TEAM CAPSLOCK

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,253 posts

Posted 30 November 2012 - 07:18 AM

The problems I've noticed have occured almost at the same time every time. Sometimes it was a 500 error and sometimes 503, and at times there was a cloudflare message about the downtime.

From what I gathered by other boarders notes:

November 14th, 18th, 19th, 27th, 28th and 29th (probably some of the days inbetween as well but these are the ones that were noted) at appr. 04.00 GMT+1.

ETA: crosspost!

Edited by Eyron I, 30 November 2012 - 07:21 AM.


#50 Ran

Ran

    King o' the Board

  • Administrators
  • PipPipPipPipPipPipPipPip
  • 30,582 posts

Posted 30 November 2012 - 07:51 AM

Yep, that's when the task was regualrly scheduled. Hopefully the repair and optimize will sort that out from now on. I also took the precaution of rescheduling the task to 8 AM, which is a quieter time than the 4 AM that it generally ran.

#51 Florina Laufeyson

Florina Laufeyson

    Heišinn

  • Members
  • PipPipPipPipPipPipPipPip
  • 6,061 posts

Posted 30 November 2012 - 04:03 PM

Awesome! Thanks guys!

#52 Pebbles

Pebbles

    Not Innocent

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,958 posts

Posted 03 December 2012 - 05:23 AM

Yay, Just reporting that the Malware false positive thing has cleared again, and I can now bum arround at work once more.

Thankyou for everything yuo have done for us.

#53 mirielredux

mirielredux

    Commoner

  • Members
  • Pip
  • 0 posts

Posted 04 December 2012 - 06:10 PM

Hello~ I have an odd problem and am hoping a moderator can help. I have been a member of them board for some time (original use name: Miriel). Well, for some reason suddenly I could not log in. Of course, I'd forgotten my password but when I attempted to do the password retrieval it would not send me the email to reset it. So I waited a while then tried to contact somebody but my emails would not go through. The mailer daemon said "message undeliverable. Sender address rejected: unverified address." Anyway, so I take it that my email is persona non grata here for some reason.

So, I joined again with a new user name and a gmail account (with the intent of posting in a help thread since I cannot seem to connect through email). I'd like to just go back to my old user name, etc if possible.

Any help appreciated. Sorry to be any trouble.

Edited by mirielredux, 04 December 2012 - 06:11 PM.


#54 Stubby

Stubby

    The night is dark and full of Dropbears

  • Board Moderators
  • PipPipPipPipPipPipPipPip
  • 8,017 posts

Posted 04 December 2012 - 08:04 PM

Hi miriel.

If you haven't been here since last year some time, this thread may help you.

Hope that is the fix for you.

#55 mirielredux

mirielredux

    Commoner

  • Members
  • Pip
  • 0 posts

Posted 05 December 2012 - 12:22 AM

Thank you for the help Stubby. But I don't think that is the issue in my case--I was able to log on as recently as last week. I think the problem began in my computer somehow--maybe a problem w/ my cookies? I had to log into other sites as well when this happened.

The problem here with this board is it will not send me emails (for whatever reason). And it is the email I need to allow me to reset my password to get back into my miriel account.

Perhaps it is unfixable? I am fine w/ just using this new account. Maybe a mod could bump my post count back up north of 750so I could have my custom title back. I'm a very slow poster and it is the little things in life you know /smile.png' class='bbc_emoticon' alt=':)' />

#56 Stubby

Stubby

    The night is dark and full of Dropbears

  • Board Moderators
  • PipPipPipPipPipPipPipPip
  • 8,017 posts

Posted 05 December 2012 - 12:52 AM

Dunno what the issue might be with the email.

It would be better to get your old account back, so we'll see what Ran suggests in due course.

#57 Ran

Ran

    King o' the Board

  • Administrators
  • PipPipPipPipPipPipPipPip
  • 30,582 posts

Posted 05 December 2012 - 06:36 AM

Miriel,

I have emailed you.

#58 Miriel

Miriel

    more prone than you to believe in magical mumbo jumbo

  • Members
  • PipPipPipPipPipPipPipPip
  • 1,134 posts

Posted 05 December 2012 - 09:50 AM

Miriel,

I have emailed you.

All fixed! Thank you Ran!!! And thank you Stubby!

#59 Alia of the knife

Alia of the knife

    Council Member

  • Members
  • PipPipPipPipPipPipPipPip
  • 4,694 posts

Posted 07 December 2012 - 02:58 PM

Think I'm starting to have e-mail notification issues again?

And thanks as always Sparks. /smile.png' class='bbc_emoticon' alt=':)' />

#60 kissdbyfire

kissdbyfire

    aka Swims with Dolphins

  • Members
  • PipPipPipPipPipPipPipPip
  • 2,889 posts

Posted 07 December 2012 - 07:21 PM

Think I'm starting to have e-mail notification issues again?

And thanks as always Sparks. /smile.png' class='bbc_emoticon' alt=':)' />


Me too. Either that or everyone is ignoring me - not that I'm paranoid or anything... /wideeyed.gif' class='bbc_emoticon' alt=':wideeyed:' />

Sparks, you're virtually a goddess... Or a virtual goddess? Both? /smile.png' class='bbc_emoticon' alt=':)' />