WeChat ID
aaronpk_tv
#ops
-
Several Days of Server Migrations Later...
I just spent two days finishing migrating the last of my old websites and services off my old server and into new homes. This project has been on my todo list for over two years, and I've been making slow progress since then, mostly by spending a day on it here and there.I finally finished moving the last things off the server today. The last two things left were my Twitter streaming search script and a really old MediaWiki website. Both were running on a 6-year old operating system, and under deprecated versions of Ruby and PHP. The MediaWiki instance had been EOL since 2012. The OpenSSL library was so old the Heartbleed bug hadn't even been written yet. Other things on that server were a Gitlab install (which I replaced with Gogs in February), my own SSL root authority (which is significantly less useful now that Letsencrypt makes certificates free and easy), an old web app I haven't used since 2009, and many cobbled together scripts.This was quite a painful process all told, and I would like to avoid getting into this this situation in the future. Here are some things I'm planning to do that will help make the upgrade process easier in the future.Upgrade regularlySpend the time it takes to upgrade the whole server (OS as well as application dependencies) every 3-6 months, instead of letting the machine get to be several years old. The whole reason I got into this situation in the first place is the OS was so old incremental upgrades were no longer possible. Updating the OS, PHP/Ruby versions, and applications regularly means smaller incremental changes rather than getting into a situation where your app no longer runs because the scripting language changed so much since the last upgrade.Separate the web and database serversI've had a separate database server for a long time. The main benefit this gives me is I'm able to move the source code for a website to a new server without having to also worry about migrating the database at the same time. It's one less moving part in the migration process, and I will continue to do this in the future.Productize everythingSome of these projects and websites were so old that I didn't actually have them in Git repos yet. Some of them were in Subversion, some were just piles of files on disk. A few of them shared a common set of files on the server, rather than using any sort of package management and proper dependency chain. Of course my more recent projects are much better packaged, but I had forgotten I had not always done things this way. Making each project a self-contained application with its own Git repo makes it easier to move them around to new servers later.Avoid using servers in the first placeThese days, we have many options for running applications without dealing with the underlying operating system. Google AppEngine (on which I currently run indiewebify.me and xray.p3k.io), Amazon Lambda (where I run ca3db), and Heroku to name a few. Once a service is running in infrastructure such as these, you can forget worrying about the underlying operating system updates, and focus on your application logic. The service providers will handle everything under the hood and you can forget that upgrading OpenSSL is a thing that needs to happen.continue reading... -
Monitoring CPU and memory usage from Postgres
Let's say you have a cluster of Postgres database servers and you want a way to easily monitor their CPU and memory usage. You could install any of the system monitoring daemons like Nagios, Collectd, Munin, etc, but that means opening up another port in your firewall, and maintaning another running process. If all you're after is basic info like CPU and memory usage, there's an easier way!continue reading... -
Knightmare: A DevOps Cautionary Tale – Doug Seven (dougseven.com)
This is the story of how a company with nearly $400 million in assets went bankrupt in 45-minutes because of a failed deployment.
-
furbo.org · Fear China (furbo.org)
The first course of business was to regain control of the server. Every service on the machine was unresponsive, including SSH. The only thing to do was perform a remote restart and wait for things to come back online.
-
What should a website operator do about the Heartbleed OpenSSL exploit? (security.stackexchange.com)
-
How to test and confirm OpenSSL is updated for Nginx and Ruby on Ubuntu 12.04
A quick guide to updating OpenSSL on Ubuntu 12.04 running Nginx and Ruby and verifying that your system is safe from the Heartbleed bug.continue reading... -
SSH to EC2 Servers by Instance ID
I often know an EC2 instance ID and need to SSH to the server. In order to avoid manually looking up the IP address or hostname, I wrote a small script to let myself SSH to an EC2 given an instance ID.continue reading... -
Add nginx as a ubuntu service stop and reload doesn't work (stackoverflow.com)
Service started well with the init file, but stop or restart didn't. In the end the cause was a different path to the pid file in the nginx.conf and the init script. Make sure they both point to the same location.
