Musings of Dan

Recently I had to increase the available space on a virtual machine running RedHat Linux 6.   The filesystem was configured with LVM, so the quick answer that came to mind was to add another virtual hard drive and expand.  While this would work, it was “messy” as future expansion in this fashion would get out of hand.

Thankfully the solution was easier:

1. Confirm your starting state to verify these steps work
   1. Check the free LVM space assuming “/dev/sdb” is your disk in that volume group, and “MyVG” is the name of the volume group.
      1. vgdisplay MyVG | grep Free
2. Power down your virtual machine.
   1. In my experimenting, I could did this while the OS was running but it wasn’t 100% consistent.
3. Expand the HDD size within the virtual system host environment
   1. In my case I am using VMware vSphere and increased from 60GB to 300GB.
4. Boot the virtual machine
   1. It should boot without any difference than before.
   2. Login as root
5. Run the “pvresize” command to tell the LVM subsystem to look at the new size of the HDD
   1. Assuming the drive in question is “/dev/sdb”, run this command:
      1. pvresize /dev/sdc
6. You can confirm your LVM has additional space by noting the “Free” section of the output of the “vgdisplay” for the volume group that /dev/sdb is associated with
   1. vgdisplay MyVG | grep Free

Happy New Year!

My RedHat rep sent me a link to this Ansible on-line training.  It’s not the standard 60-90 minute live walkthrough, this is a set of pre-recorded training videos.

Title: “Ansible Essentials: Simplicity in Automation Technical Overview”

Link: https://www.redhat.com/en/services/training/do007-ansible-essentials-simplicity-automation-technical-overview

As I just received this information today I haven’t had time to look at them, but the chapter titles look like a good overview of the entire Ansible suite.

A couple weeks ago I posted about converting some Subversion repositories to Git.

Since then I’ve found a different write-up that seemed to work a bit better:

http://john.albin.net/git/convert-subversion-to-git

My VMware support engineer forwarded on the current VMware knowledge base weekly digest, and one of the new KB article titles caught my eye:

Upgrading VMware Tools using PowerCLI or vSphere API (2147641)

Hey!  That sounds like something to look over and possibly provide to my operations team to help them upgrade older VMs that were now running on newer VMware hosts!  Clicking on the URL for that KB article (KB 2147641) brought up the usual Details and Solution sections, but they were strangely lacking.

The “Details” section usually explains the subject of the KB article in detail.  This one just said:

VMware Tools can be upgraded using vSphere PowerCLI, vSphere API and other methods. For optimum performance, upgrade VMware Tools in not more than five virtual machines simultaneously.

And the “Solution” section was even less helpful:

None

Talking with my support engineer, he thinks that the article may have been posted to note that it is possible to upgrade the VMware tools using either of those methods, just the actual steps how to do this were not documented.

I know from a quick search there are plenty of examples from non-VMware.com sites on how to do this:

• http://notesofascripter.com/2016/04/05/using-powercli-set-ip-update-vmware-tools/
• https://ryanmangansitblog.com/2014/06/04/upgrading-vmware-tools-in-batch-using-powercli/
• http://www.vmspot.com/using-powershell-to-update-vmware-tools-on-vms-2/
• http://virtualize-automate.com/wp/index.php/2015/07/15/mass-upgrade-vmwaretools-in-vcenter-environment-using-powercli/

Ironically, there are some links to VMware.com pages addressing this:

• https://blogs.vmware.com/vsphere/2016/03/six-methods-for-keeping-vm-tools-up-to-date.html
  • #5 and #7 discuss the PowerCLI and vSphere API methods
• http://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.wssdk.apiref.doc%2Fvim.VirtualMachine.html&path=7_1_0_2_5_20_82
  • The vSPhere 6.0 API call
• https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1028291
  • The PowerCLI example

I’ve seen odd VMware KB articles in the past – hopefully the addition of a “Tip of the Week” flag or at least a sentence in the Solution field denoting the article is not fully fleshed out solution would save a lot of confusion.

Well, it had to happen.  I finally got my new site up and planned to restore the blog posts to the new site.  My backup files from various past sites were all in place – I had setup a backup script to dutifully collect the data monthly (I didn’t update the sites all that often), and also clean up after itself and only keep three months of backups.

The script ran, the backup files appeared and automatically cleaned themselves up after 90 days.  When I first ran it I verified that the files were complete – I didn’t restore them anywhere, but the blog text was there.  Success!  Add it to cron on my desktop and let it run.

And run.   And run.   And run.  Unattended.  For the past couple years.

I had been lax and wasn’t blogging much so I let my SquareSpace site go away a year ago.  Recently I decided to resume my ramblings..er, um, blogging, so I installed WordPress on my site and went looking through the backups.

The good news, my cron job continued to work dutifully backing up all the blog posts.  Except when I canceled my SquareSpace account, it continued to “backup” the site – except this time the files it saved were essentially “the site does not exist” messages.  (Insert sad face here…)  Thankfully I was able to restore some of the older text using Archive.org and I’m still combing through other old sources.  But much of it is a loss.

So, what did I learn (or re-learn) from all this?

1. A single copy of a backup is not a backup.  Use the “Three/Two/One” rule.
2. Don’t cleanup archives when it’s not necessary.  The backup files were small enough (less than a megabyte after compressions), so I could have kept many years in couple gigabytes on my server.
3. Keep track of services and the processes associated with it – I didn’t need to keep the backup script running after cancelling the service. This didn’t have a real expense associated with it, but how often have we looked at our budget and realized that we’ve continued to pay for something well after we stopped using it.

I have a Subversion project that I’m migrating to use Git, but I don’t want to loose the history if possible.  I found these steps and they mostly worked for me with one exception (below):

https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git

The only problem was during the export I got the error message:

Author: (no author) not defined in authors file

After a bit of searching I found this workaround:

Author: (no author) not defined in authors file

In short I had to add this line to my users.txt file:

(no author) = no_author <no_author@no_author>

Let me start off by saying that I didn’t come up with this backup mnemonic, rather Peter Krogh first wrote this up (to my knowledge) in this blog post.

As I recently re-learned, backups even done right are hard to do well.  In my case there’s a chance that I still would have lost my data, but there’s no accounting for human error in every case.

The “Three, Two, One” backup strategy is pretty simple:

• Three – A file isn’t backed up until there are at least three copies of it, the original and two other copies not on that machine.
• Two – The backups must be on two different media types.  For example, a hard drive and a DVD drive, or a tape backup.
• One – Finally, one of those copies should be stored off-site or at least off-line.  A cloud storage service such as Carbonite, Amazon Cloud Drive, Google Drive, or even storing it at a friends house.

In my case (backing up my website), one version would have been the site itself, a second would have been a copy stored on my home computer, and a third would have been stored on a DVD (probably not every month, but probably once every six months or so) or I would have copied it up to my Google Drive.

Sadly, I didn’t take those precautions and now I’m paying the price (thankfully a small one).

And I’ll add one more thing – be sure to VERIFY the backup you created periodically.  It does you no good if the restore process fails or isn’t documented for someone else to perform.

In a previous post I documented the use of ddrescue to recover the data from a failing hard drive.  I follow-up a few months later noted that the second drive had started failing but this time I was able to copy the data before needing to resort to the rescue tools.  As I promised, here’s a follow-up.

After using the second replacement drive for just a couple weeks, I started noticing the same errors creeping into the “dmesg” output.  Though I know some manufacturers have a bad line, I’ve never experienced any that failed that rapidly, especially when the production runs and the differences were getting to be substantial.

My first thought was that the motherboard might be failing, unfortunately I wasn’t able to find an inexpensive SATA disk controller so I did the next best thing and move the disk to a different SATA port.  The move helped a bit but the errors still came back after a bit of hard disk activity.  On a whim I decided to change out the SATA cable with a different one in my collection.  Neither cable was especially “high quality” compared to the other, but when I put the new cable in on Dec 2 I haven’t seen an error since.

I’m at a loss as to what has happened to the old cable – the drive and cable are well inside the case and not touching anything so I don’t think it’s a problem with wear, but it’s possible there is some oxidation/rust on the cable that I can’t see.

I hope this is my last update on this issue, but I’ll continue the saga if it’s necessary.