Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Site Migration - 2021-02-09 at 05:30 EST [Site offline]
[Migration postponed until tomorrow, February 9, at 05:30 AM --admin LadyGeek]
We've enjoyed a excellent service at our current hosting facility under very generous terms. The facility and their service has been flawless.
I intend to migrate the website from that facility, running on "bare metal" machines that I own, into a hosting facility using virtual servers. The primary reason for this migration is my concern about the long term reliability and sustainability of my machines. Should hardware ever fail, recovery is faster and easier in a virtual environment.
The availability of virtual servers has also allowed me to design a clustered configuration which should adapt more easily to our growth.
The new cluster is being hosted in a facility in Dallas, TX, with a highly regarded provider. My interactions with the provider have been exemplary. They have a good team and the technical staff is both eager and well equipped to answer questions and solve problems.
LadyGeek and I have reviewed the new system and are confident that we're ready to move.
So, Feb 8, 2021, at 06:30 EST I will shut down the current site and begin the migration.
My target is to have the website online at the new facility by 08:00 AM.
We'll be very attentive to the site tomorrow -- we'll be watching it like hawks.
Larry Auton
We've enjoyed a excellent service at our current hosting facility under very generous terms. The facility and their service has been flawless.
I intend to migrate the website from that facility, running on "bare metal" machines that I own, into a hosting facility using virtual servers. The primary reason for this migration is my concern about the long term reliability and sustainability of my machines. Should hardware ever fail, recovery is faster and easier in a virtual environment.
The availability of virtual servers has also allowed me to design a clustered configuration which should adapt more easily to our growth.
The new cluster is being hosted in a facility in Dallas, TX, with a highly regarded provider. My interactions with the provider have been exemplary. They have a good team and the technical staff is both eager and well equipped to answer questions and solve problems.
LadyGeek and I have reviewed the new system and are confident that we're ready to move.
So, Feb 8, 2021, at 06:30 EST I will shut down the current site and begin the migration.
My target is to have the website online at the new facility by 08:00 AM.
We'll be very attentive to the site tomorrow -- we'll be watching it like hawks.
Larry Auton
Re: Site Migration - 2021-02-08 at 06:30 EST
This thread is in the Forum Issues and Administration forum, but appears in all forums.
I clarified the thread title to mean that the site will be unavailable until we show up at our new home.
I clarified the thread title to mean that the site will be unavailable until we show up at our new home.
- Cheez-It Guy
- Posts: 4007
- Joined: Sun Mar 03, 2019 3:20 pm
-
- Posts: 1071
- Joined: Tue May 15, 2018 10:14 pm
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
Thanks for the heads-up.
"Good luck, we're all counting on you."
"Good luck, we're all counting on you."
Just because you're paranoid doesn't mean they're NOT out to get you.
-
- Posts: 205
- Joined: Tue Jun 07, 2016 5:33 pm
- Peculiar_Investor
- Site Admin
- Posts: 2445
- Joined: Thu Oct 20, 2011 12:23 am
- Location: Calgary, AB 🇨🇦
- Contact:
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
I would suggested that MediaWiki:Sitenotice - Bogleheads should also be used to advise wiki editors and readers of the planned downtime.
There have been other instances where software updates of the forum have taken the website offline. Using the wiki's sitenotice as well as posting announcements here might be considered as part of the standard operating procedure. It is quite possible that wiki editors and/or readers might not visit the forum so could might any announcement there.
There have been other instances where software updates of the forum have taken the website offline. Using the wiki's sitenotice as well as posting announcements here might be considered as part of the standard operating procedure. It is quite possible that wiki editors and/or readers might not visit the forum so could might any announcement there.
Normal people… believe that if it ain’t broke, don’t fix it. Engineers believe that if it ain’t broke, it doesn’t have enough features yet. – Scott Adams
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
Good idea, thanks. The site notice is now in the wiki. See: MediaWiki:Sitenotice
FYI - The site notice does not display in a mobile device.
FYI - The site notice does not display in a mobile device.
-
- Posts: 437
- Joined: Sun Oct 21, 2018 1:56 am
- Location: Sandwich Islands
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
The dreaded Y21K prophecy that has long been foretold is upon us...
Jerry Garcia: If I knew the way...I would take you home.
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
Good luck with the migration
"I started with nothing and I still have most of it left."
-
- Posts: 9479
- Joined: Sun Oct 08, 2017 7:16 pm
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
Maybe Vanguard should hire you guys to fix their site.
- WoodSpinner
- Posts: 3504
- Joined: Mon Feb 27, 2017 12:15 pm
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
Great move! Working in a Virtual Server environment is sooo much easier!
Thanks for all the hard work!
WoodSpinner
Thanks for all the hard work!
WoodSpinner
WoodSpinner
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
what is the plan for the nameservers, TTL, etc?
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
^^^ DNS, etc. will be taken care of by mingstar.
Bumping for visibility.
Bumping for visibility.
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
See you on the other side! (I hope)
Retired life insurance company financial executive who sincerely believes that ”It’s a GREAT day to be alive!”
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
Due to unforeseen circumstances, we need to postpone until tomorrow.
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
Oh well -- unforeseen circumstances have scrubbed the mission for today. I'm resetting for 2021-02-09 at 0530 EST.
If you want to make God laugh, tell Him your plans.
Larry
If you want to make God laugh, tell Him your plans.
Larry
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
Wow, that's unexpected, since it's 18,979 years early!BionicBillWalsh wrote: ↑Sun Feb 07, 2021 7:32 pm The dreaded Y21K prophecy that has long been foretold is upon us...
Who wouldda thunk!
(Y21K = year 21,000.)
"Happiness Is Not My Companion" - Gen. Gouverneur K. Warren. |
(Avatar is the statue of Gen. Warren atop Little Round Top @ Gettysburg National Military Park.)
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
I confirmed the time change with mingstar and have updated the thread title.
The migration will start at 05:30 (5 AM Eastern, 10:30 UTC) - 1 hour earlier.
The migration will start at 05:30 (5 AM Eastern, 10:30 UTC) - 1 hour earlier.
Re: Site Migration - 2021-02-08 at 06:30 EST [Site offline]
The one-day delay provided an opportunity to fix the wiki.LadyGeek wrote: ↑Sun Feb 07, 2021 7:32 pm Good idea, thanks. The site notice is now in the wiki. See: MediaWiki:Sitenotice
FYI - The site notice does not display in a mobile device.
Mobile devices will now see the wiki's site notice.
- asset_chaos
- Posts: 2629
- Joined: Tue Feb 27, 2007 5:13 pm
- Location: Melbourne
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Thank you LadyGeek and mingstar for your exemplarary service to our Bogleheads community.
Regards, |
|
Guy
- nisiprius
- Advisory Board
- Posts: 52215
- Joined: Thu Jul 26, 2007 9:33 am
- Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Good luck, mingstar and LadyGeek.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Thanks! Bumping for visibility.
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Here's hoping all goes well. Thanks as always for providing such a fantastic forum!
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
We're back, all OK.
Welcome to the cloud!!!
Welcome to the cloud!!!
- Harry Livermore
- Posts: 1937
- Joined: Thu Apr 04, 2019 5:32 am
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Seems a bit slow. (do I mean latency? not sure)
I click a link and it's quite a while for the thread to load.
Other websites seem to behave as expected. But it could be on my end I suppose...
Just a data point for you.
Cheers
ETA: it seems hit or miss. Now it's working as usual. So maybe disregard...
I click a link and it's quite a while for the thread to load.
Other websites seem to behave as expected. But it could be on my end I suppose...
Just a data point for you.
Cheers
ETA: it seems hit or miss. Now it's working as usual. So maybe disregard...
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Expect things to run slow for a while, we're starting on a new server. Things have to come up to speed.
- oldcomputerguy
- Moderator
- Posts: 17934
- Joined: Sun Nov 22, 2015 5:50 am
- Location: Tennessee
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Glad to see the forum back. For what it's worth, I haven't noticed any particular slowness here, but I've only been on a few minutes.
There is only one success - to be able to spend your life in your own way. (Christopher Morley)
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
FWIW, I see that Lady Geek proclaimed the site “open” at 17 minutes after the hour.
But I was not able to access the site until almost 50 minutes after the hour.
Oh, well ....
But I was not able to access the site until almost 50 minutes after the hour.
Oh, well ....
Retired life insurance company financial executive who sincerely believes that ”It’s a GREAT day to be alive!”
- oldcomputerguy
- Moderator
- Posts: 17934
- Joined: Sun Nov 22, 2015 5:50 am
- Location: Tennessee
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
It might have taken a bit more time for the DNS changes to migrate out to the servers in use at your ISP. I wasn't involved at all with this migration, so that's just a guess, but I've done some similar tasks in my working career.
There is only one success - to be able to spend your life in your own way. (Christopher Morley)
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
This thread no longer appearing at the top of every forum.
mingstar and I are 'watching this forum like a hawk'. We're fine-tuning the clustered configuration.
mingstar and I are 'watching this forum like a hawk'. We're fine-tuning the clustered configuration.
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Site is working well for me! Thanks
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Like itworks's hawk (slightly cross-eyed)?
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Everything is running flawlessly.
Once again precision execution.
Once again precision execution.
"..the cavalry ain't comin' kid, you're on your own..."
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Great work team!
While the site was down, I authorized a donation to the Bogle Center from our DAF. Check is in the mail.
Thanks so much for all you do.
While the site was down, I authorized a donation to the Bogle Center from our DAF. Check is in the mail.
Thanks so much for all you do.
Retired life insurance company financial executive who sincerely believes that ”It’s a GREAT day to be alive!”
- nisiprius
- Advisory Board
- Posts: 52215
- Joined: Thu Jul 26, 2007 9:33 am
- Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.
- Peculiar_Investor
- Site Admin
- Posts: 2445
- Joined: Thu Oct 20, 2011 12:23 am
- Location: Calgary, AB 🇨🇦
- Contact:
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Notification emails do not appear to be going out. LadyGeek has been advised.
Normal people… believe that if it ain’t broke, don’t fix it. Engineers believe that if it ain’t broke, it doesn’t have enough features yet. – Scott Adams
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
I'm able to reach https://www.bogleheads.org , but not http://bogleheads.org . Not sure what's going on there.
-
- Posts: 163
- Joined: Mon Sep 23, 2019 3:55 pm
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
I unfortunately haven't automatically received "Topic Reply Notifications" where the name of the post is also in the subject line of the email since that software update a few months ago. The only way I receive these emails is if I go into the Board Index, open a topic and use the drop down to subscribe to a topic, and I'll only receive them then if someone replies to a post after I subscribe it. I really wish I could still receive them automatically in my email Inbox like I used to. I am subscribed to receive them, but something happened when they did that update.Peculiar_Investor wrote: ↑Tue Feb 09, 2021 8:52 am Notification emails do not appear to be going out. LadyGeek has been advised.
- Peculiar_Investor
- Site Admin
- Posts: 2445
- Joined: Thu Oct 20, 2011 12:23 am
- Location: Calgary, AB 🇨🇦
- Contact:
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
We know, viewtopic.php?f=3&t=330270. This is different, since the server migration this morning there are NO notification emails going out to anyone. LadyGeek and mingstar are aware and are working on it.
Normal people… believe that if it ain’t broke, don’t fix it. Engineers believe that if it ain’t broke, it doesn’t have enough features yet. – Scott Adams
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Thank you to mingstar and LadyGeek!
-
- Posts: 163
- Joined: Mon Sep 23, 2019 3:55 pm
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Thanks for your reply because it made me realize that I also haven't received any email notifications since 5:22 am.Peculiar_Investor wrote: ↑Tue Feb 09, 2021 12:42 pmWe know, viewtopic.php?f=3&t=330270. This is different, since the server migration this morning there are NO notification emails going out to anyone. LadyGeek and mingstar are aware and are working on it.
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Mail notifications have been fixed.
Anything sent before the time of my post won't be received.
Anything sent before the time of my post won't be received.
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Well done on the migration!
"Never underestimate one's capacity to overestimate one's abilities" - The Dunning-Kruger Effect
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Thanks, much appreciated.
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
What a merry chase! LadyGeek and I had a lot of fun with this. We both learned a lot in a fairly short time.
The site is now running a mostly vanilla instantiation of a distributed, load-balanced, web cluster:
The web servers are CPU-centric machines with very little disk i/o; pretty much all they do with their local disk is write logs
(Yes... we will set up a log server soon.)
The file and database server has a bunch of RAM to keep as much as possible in core.
(hint: RAM is faster than disk -- "Important safety tip. Thanks Egon")
There were a few really interesting aspects of trying to run the software behind a load balancer but once they were well understood, the solutions were straightforward. I will be making some suggestions to the Wordpress development team.
One of the key goals of this project was to make things as "ordinary" as possible. I wanted the both the code base and the procedures to be found in the simple set of instructions that you get in the first ten hits on your favorite search engine. I think we've done a pretty good job.
Another goal was that I wanted the system to be able to scale quickly and easily and to accommodate a wide variety of computing resources. We started this morning with five web servers, all small, all the same. By noon, we realized that they were not going to be able to keep up when we hit the evening peak load so, within about ten minutes, we scaled up to ten small web servers. By mid-afternoon, we had enough data to see that different parts of the site have very different performance constraints and characteristics. By evening, we had scaled back down to four web servers but with different parts of the site handled by different classes of machine. All of this was done with the site "live" and without significant service interruption.
We will continue to watch and learn. For performance tuning, we believe that it's better to measure than to guess. This infrastructure decouples many of the components of the site in ways that make getting accurate measurements possible. It's a rare and fascinating opportunity.
Larry
The site is now running a mostly vanilla instantiation of a distributed, load-balanced, web cluster:
Code: Select all
(Internet) ---> [load balancer] ---> [multiple web servers] ---> [file and database server]
(Yes... we will set up a log server soon.)
The file and database server has a bunch of RAM to keep as much as possible in core.
(hint: RAM is faster than disk -- "Important safety tip. Thanks Egon")
There were a few really interesting aspects of trying to run the software behind a load balancer but once they were well understood, the solutions were straightforward. I will be making some suggestions to the Wordpress development team.
One of the key goals of this project was to make things as "ordinary" as possible. I wanted the both the code base and the procedures to be found in the simple set of instructions that you get in the first ten hits on your favorite search engine. I think we've done a pretty good job.
Another goal was that I wanted the system to be able to scale quickly and easily and to accommodate a wide variety of computing resources. We started this morning with five web servers, all small, all the same. By noon, we realized that they were not going to be able to keep up when we hit the evening peak load so, within about ten minutes, we scaled up to ten small web servers. By mid-afternoon, we had enough data to see that different parts of the site have very different performance constraints and characteristics. By evening, we had scaled back down to four web servers but with different parts of the site handled by different classes of machine. All of this was done with the site "live" and without significant service interruption.
We will continue to watch and learn. For performance tuning, we believe that it's better to measure than to guess. This infrastructure decouples many of the components of the site in ways that make getting accurate measurements possible. It's a rare and fascinating opportunity.
Larry
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
Thanks for the progress reports, Larry & LadyGeek. The BH community is fortunate to have dedicated admins keeping the site going!
Is it safe to say that a successful migration means (1) lowered overall costs - presumably having virtual servers is cheaper than physical servers? and (2) easier maintenance/less time required for the BH admins?
Is it safe to say that a successful migration means (1) lowered overall costs - presumably having virtual servers is cheaper than physical servers? and (2) easier maintenance/less time required for the BH admins?
Just curious what part of bogleheads uses Wordpress - just the blogging feature?
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
(1) Cost was one of the things we optimized for. After all, this is a financial forum. Our main goal was to come as close as possible to (or better than) what we had before. It's classic system trade study. The standard criteria apply: Speed - complexity - cost. Pick any two.
We tested as much as we could beforehand, but it can't replace doing this with live traffic.
(2) Yes, absolutely easier and much faster than recovering to a hardware server. Once everything is setup, all we need to do is restore from an archived snapshot.
Yes, our Blog is WordPress.
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
As I said earlierIs it safe to say that a successful migration means (1) lowered overall costs - presumably having virtual servers is cheaper than physical servers? and (2) easier maintenance/less time required for the BH admins?
I love this site and how it continues to help people. It has helped me so very much. However, as LadyGeek and I have often commiserated, we often wake up with thoughts about "what would we do if Something Bad (R) happens to the hardware." I finally got to the point where the worry was costing me more than I wanted to pay. This move increases my cost but lets me wake up thinking about other things.The primary reason for this migration is my concern about the long term reliability and sustainability of my machines. Should hardware ever fail, recovery is faster and easier in a virtual environment.
As for the idea that virtual servers are cheaper than physical servers, that's not the case. Virtual servers run on physical servers. Somebody bought those physical servers and has to pay staff and buy hardware to maintain them. Typical virtual server pricing pays for the hardware on which it runs in less than a year.
If your application need only a small portion of a physical server, as when running a simple web site with low to moderate traffic, you can save money by buying that small capacity as a small virtual server versus buying a physical machine and paying for the rack space and bandwidth in a colocation facility.
However, bogleheads.org is not a simple web site with low to moderate traffic. Its load demands significant resources and the virtual servers offered by most providers are not adequate for the task. As I mentioned in the previous post, the file and database server, in particular, needs a lot of RAM. That's the golden ticket in VPS hosting: it comes with a price.
The machines I bought a few years ago are still going strong. They're high capacity, bigger than the biggest physical machines offered by this hosting facility, they're high end and they're well built. They could last another decade ... but they might not, hence my move.
As we learn more, we will be constantly evaluating options but our current facility has both fair prices and excellent technical support. For example, while building this cluster, when I had questions I could get to a technically adept human in under thirty seconds. Without exception, they could understand my questions. That, by itself, is rare but, even better, they could usually provide an answer right away. If they couldn't get the answer immediately, they would quickly escalate to higher tiers of support. In one instance, I was trading e-mail with Tier 4 support in under thirty minutes. That is a precious commodity.
Yes - the forum is phpBB, the wiki is MediaWiki, and the blog is Wordpress.Just curious what part of bogleheads uses Wordpress - just the blogging feature?
Larry
-
- Posts: 136
- Joined: Tue Aug 04, 2020 2:43 am
Re: Site Migration - 2021-02-09 at 05:30 EST [Site offline]
It probably doesn't matter much, but in case you weren't aware, the new IP for bogleheads.org (23.254.167.168) is on a couple blacklists.
https://mxtoolbox.com/SuperTool.aspx?ac ... n=toolpage
I only noticed this because I have a firewall rule that blocks IPs that query certain well known exploitable ports, and apparently that rule had added 23.254.167.168 to my own personal blacklist a couple years ago.
Most blacklists don't keep blocked IPs around for that long, and given that the IP appears clean on the majority of public blacklists, it's probably fine.
Just a heads up in case this somehow ends up being an issue.
https://mxtoolbox.com/SuperTool.aspx?ac ... n=toolpage
I only noticed this because I have a firewall rule that blocks IPs that query certain well known exploitable ports, and apparently that rule had added 23.254.167.168 to my own personal blacklist a couple years ago.
Most blacklists don't keep blocked IPs around for that long, and given that the IP appears clean on the majority of public blacklists, it's probably fine.
Just a heads up in case this somehow ends up being an issue.