Jump to content

Shit's Borked


RPM

Recommended Posts

1 hour ago, Vertigo said:

Yep, cert just expired. Honest question but why don’t you guys host the site on a PaaS? Seems like so many of these types of issues are auto managed for you on a platform like Heroku or Azure. 

Because it would turn the site from profitable to a money loser

1 hour ago, Captainant said:

ffs guy, use a certificate manager or some shit to update your certs automatically. Use AWS ACM you wouldn't have this issue anymore and the SSL certs are free. Don't need anything on AWS to use it, not even domain registration on route53 

EDIT: Huh, their SSL provider, Let's Encrypt, says they do automatic cert rotation. No idea why it's broken, but cmon man

The automation works fine for a single server site . When you have multiple servers running it's not just a one liner.

To renew the cert I had to:

  • change nginx config
  • turn off haproxy
  • restart nginx
  • run certbot script
  • cat keys together and move to proper location
  • change nginx config
  • restart nginx
  • start up haproxy

Yeah, I could write a script to do this but I'm fucking lazy

 

Link to comment
Share on other sites

34 minutes ago, blacklab said:

The automation works fine for a single server site . When you have multiple servers running it's not just a one liner.

To renew the cert I had to:

  • change nginx config
  • turn off haproxy
  • restart nginx
  • run certbot script
  • cat keys together and move to proper location
  • change nginx config
  • restart nginx
  • start up haproxy

Yeah, I could write a script to do this but I'm fucking lazy

 

Or you could stick the whole site behind a load balancer and then only have to worry about certing that single point instead of every backend server. Shit, are we load balancing at the DNS level instead of routing to a load balancer box?

Link to comment
Share on other sites

29 minutes ago, blacklab said:

haproxy is our load balancer, it does take care of the https, the hosting servers do not do https

haproxy is running on a box that is also running nginx for hosting, no point in having that server sit at 1% usage

Yeah we terminate at the LB level. LB was misconfigured and didn't acknowledge and update the certs so shit got booked. 

Won't happen again. Please give me your neg rep. 

Link to comment
Share on other sites

  • 2 weeks later...
  • 2 weeks later...

Just as a datum - I was having problems writing replies in Chrome last night.  The reply box was very short - clicking in it would drop it down enough for the "insert other media" box and maybe the submit reply box, but no space for the body/would not place a cursor.  I manually updated Chrome (which should be set to autoupdate, but... meh) and it functions normally now.  Other websites seemed fine, Firefox was working normally, based on the lack of anyone else saying anything about the same problem it's probably just something wonky in Chrome... or actually it may have been related to a flash update that I'm now remembering installed last night.  Fuckin adobe.

Link to comment
Share on other sites

9 hours ago, blacklab said:

are you on a browser or tapatalk? If a browser which browser?

There are no ads on the site for people who are signed in.

 

2 hours ago, immamac said:

Strange there is no mechanism to show ads to anyone that is signed in and we don't run "hijack" ads. You should consider doing a virus/malware scan. 

just sent you both a pm with a screen grab video of what i am talking about.

Link to comment
Share on other sites

iPad Pro running iOS 12.1 Safari, private browsing:  hijack ads when trying to sign in at https://www.surlyhorns.com/board/index.php 

Happened a couple of times this morning.  This time I was able to sign in before getting hijacked.

Edited by solamente73
Sign in, log in, 220, 221....
Link to comment
Share on other sites

  • 5 weeks later...

Just an update for those curious about the downtime this morning that was unplanned:

Our main database server basically shit itself this morning and is effectively dead. Catastrophic failure, luckily we didn't even have to restore from backup and we lost no data from just failing over to our slave database host. Unfortunately I had the management software set to manual instead of automatic otherwise there would have been no disruption in service. The 503 errors that were happening afterwards were just due to the server still being in our load balancing rotation - I fixed that and everything should be good to go. No worries we have another slave and off site backups in the event of another failure on the current main database.

Link to comment
Share on other sites

I may have spotted what's causing the double post problem. If you reply to the thread before the last page it looks unresponsive. But open a new tab and check the thread it has the post. People keep clicking until they get a response or give up.

Link to comment
Share on other sites

  • 4 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.



×
×
  • Create New...