Categories
Uncategorized

When is a disk space problem not a disk space problem?

A co-worker setup an Ansible playbook to update some packges but it kept erroring out. The error that Ansible reported from “yum” was “No space left on device“. He had jumped onto the system and saw that this partition had plenty of space left so asked if I could look into it.

I got on and confirmed that when I ran a simple “yum update” it showed this:

[root@linux5237 ~]# echo n | yum update

Loaded plugins: product-id, rhnplugin, search-disabled-repos, security, subscription-manager

[Errno 28] No space left on device: ‘/var/run/rhsm/cert.pid’

This system is receiving updates from RHN Classic or RHN Satellite.

Could not create lock at /var/run/yum.pid: [Errno 28] No space left on device: ‘/var/run/yum.pid’

Hmm, no disk space still. Looking at the “df /var” output looks good:

[root@linux5237 ~]# df /var

Filesystem           1K-blocks   Used Available Use% Mounted on

/dev/mapper/rootvg-varlv

                       2514736 914948   1468716  39% /var

Suspecting other resource issues I checked the inode availability using “df -i:

[root@linux5237 ~]# df -i /var

Filesystem           Inodes  IUsed IFree IUse% Mounted on

/dev/mapper/rootvg-varlv

                     163840 163840     0  100% /var

A ha! No inodes left. I’ll let you use your favorite search engine to look up details, but an easy way to think of “inodes” is as space on the first few pages of a book dedicated to being the “table of contents.” If you have a book with a few chapters, you only need a single page for the table of contents (the inodes). If you have a book with lots of chapters and sub-chapters, you might need a lot of pages (more inodes). By default Unix systems have a forumla on how much of the filesystem to dedicate to being “inodes” and how much is left for actual data storage. Usually this is fine for most systems.

To find them we want to look for directories which have chewed up the 163K files:

for i in /var/*; do echo $i; find $i |wc -l; done

This pointed to the “/var/spool/app01/” directory – it has over 160K small files.  The owner of the system was able to clean up some old files there and the “yum update” worked as expected.

It’s possible to override the inode settings when the filesystem is formatted, so if you know this ahead a time you can do this. If you run into this after the fact, the usual resolution is to backup the data, reformat the filesystem with more inodes allocated, then restore from backup.

Categories
Uncategorized

SELinux and NFS $HOME directories

Recently we re-installed a common server with RHEL-7 and that went well.  But after a couple days I noticed that I was unable to login with my personal ssh key but I had before. It was a minor annoyance and didn’t pursue it … until today.

It turns out that the /home/ directory on this system is an NFS mount, and in RHEL-7 we have set SELinux to default to enforcing.  There is an SELinux boolean flag, “use_nfs_home_dirs” that needed to be set to “1” (true).  Running the “setsebool -P use_nfs_home_dirs 1” on this system was the fix and now we/I can resume logging in with the SSH key instead of typing in my passwordeach time.

Some were reluctant to fix this as they always typed in their password. While typing in your password over the SSH login connection is encrypted, but it does present the possibility that your password could get copied given a compromised endpoint, plus we are trying to use longer passwords so typing this in multiple times per day was frustrating and slowed workflow.  Using SSH keys eliminates this risk and provides for other features such as scheduled/scripted command execution and file transfers.

Categories
Uncategorized

Delete yourself from Medium

I’ve been on the Internet for years. The openness of the whole environment was refreshing. Want to find out something obscure or highly technical, you could head over to a university website and find an article, or over to a vendors site to get details on something you were looking for..

Then came web-rings and blogs. A webring was a simple “next page” link on a site that would take users from one persons page to another, usually the pages had a common theme (a hobby, an interest, etc). Later there were blogs (like this WordPress site) that were more dynamic. You could write multiple pages on varying topics, or you could work with others to share access and create a source of information about your topics.

This wasn’t free, but many of us kept our sites up out of the love for the art we were discussing, or out of the feeling of providing back to the wider world. For times when sites got too big to support by one persons budget, there were advertisers who would trade some of the blog page real estate to host adds in return for a small bit of money back to the owner of the site. And for some sites, they turned to user supported options so that people who paid a small periodic fee could in turn get access to other articles earlier, or possibly in-depth that weren’t public. Many newspapers have turned to this – the general public gets the first paragraph of a story, but subscribers could get the entire article and additional features.

But over time the increase of the web as a “social medium” platform took root. After a while, the need to drive more and more eyeballs to a website took on a more “closed off” approach. I’m sure there are many more that I are out there, but the one that finally got to me and made me say “enough is enough” is the website “Medium.com“.

They are a hosted blogging site and that’s nice. They also have good search-engine-optimization features so a well written article gets found easily on Google and other search engines. Many times when I’m searching for some additional information on a news topic, I will come across something hosted on a Medium.com page and click the link. Or at least only up to a threshold of free posts per month then you have to wait until the next month. Or you can sign up for a monthly subscription to access more articles.

All that is good and I will not begrudge them for collecting a fee to offset their designers and staff who keep the website running smoothly, as well as the curation of articles they are performing. But I can’t justify spending another $5 a month on yet another blogging site.

And something about the whole “pay to see anything” mentality seems to be antithetical to what really makes the Internet ‘work’, and honestly what was the groundwork that permitted sites such as Medium, Facebook, Google, Amazon, etc. to thrive. It feels like all the good works that were poured into the initial Internet (open networking standards, operating systems, email, web servers and browsers) and kept free all this time are being clear-cut by these new digital locusts.

But they aren’t listening. Instead they are using the subscription fees to lock more and more content behind their closed doors. If this continues, we’ll have an entire Internet made up of toll roads and not the wide-open digital universe we have today.

If you agree, please take some time and remove yourself from the Medium website. It’s very easy – under your profile, the very bottom option is “delete from Medium”. I don’t need their constant barrage of articles that don’t provide back to the greater good, at least not without a fee and tracking to better serve up my time and attention to their article writers for another piece of fluff.

Do it – it felt good!
Categories
Uncategorized

RedHat Summit 2019

Just a quick shout-out to anyone who is going to the RedHat Summit in Boston May 7-9 (https://www.redhat.com/en/summit/2019). I’m a co-presenter in a session Thursday at 2PM (“West Corporation—a digital transformation journey”) – if you’re there stop up afterwards and introduce yourself.

Categories
Uncategorized

The humble check-list

Driving home last night I listened to the next episode of the NPR “Hidden Brain” podcast titled “Check Yourself”. The topic of this was around the humble “checklist” that we’ve all made but never gave much thought to.

Two parts of the story surprised me:

  • They came to be a requirement in the airline industry after a crash of a new and highly complex aircraft in 1939.
  • The very recent addition of checklists to hospital procedures.

Link to text of this podcast: https://www.npr.org/2018/08/27/642310810/you-2-0-check-yourself

General podcast link with option to listen to MP3 in browser or download: https://www.npr.org/podcasts/510308/hidden-brain

Categories
Uncategorized

Come’on RedHat!

Come’on RedHat – give us the MSDN version for home labs and training.

I recently signed up for the RedHat Developer Program and have setup my lab system with a fully updated RedHat Enterprise Linux 7 operating system with all of the bells-and-whistles the OS provides.

The Developer Program not only gives you a fully licensed RHEL system, it also provides access to JBoss, and the RedHat Container Development Kit which contains a set of development tools and additional resources (Python, PHP, Ruby, OpenJDK, etc).  All great stuff, especially for developers looking to hone their OpenSource development practices.

Unfortunately, for those of us on the systems side of the world (not hard-core developers) the package omits a couple of great RedHat products: Cloudforms and RedHat Virtualization.

I know that I could attempt to knit together the upstream versions of each of these (ManageIQ and oVirt), but there is a reason that the Cloudforms and RHV exist to save us from the complexity of configuring all of the components together.  As a learning platform, the effort of researching numerous blog posts and reams of documentation for each project (and their pre-requisites) might be worthwhile, but for some one like me who is interested in learning more about Cloudforms/ManageIQ itself (automations, hooking into a lab VM environment, etc) it severely skews away from the objective I’m trying to educate myself on.

Microsoft has their MSDN – Microsoft Developer Network – that provides a license for a nominal fee (at the low end) that provides licensing for Windows OS installations (five licenses, many versions, both standard and Server), SQL, Exchange, SharePoint, Office and Office365, Azure, etc.  Windows Server has Hyper-V as an option – it’s answer to RHV/oVirt.  (I don’t believe they have a Cloudforms/ManageIQ alternative, though one might exist.)

I understand that MSDN is not free ($0.00 expense) like the RedHat Developer Program is, so it’s not a perfect comparison but Microsoft has a range of offerings price points.  If a small ($99/year) cost would expand the RedHat option to include Cloudforms, RHV, and multiple server subscriptions (not the single RHEL instance), I could greatly expand my experience with the options RedHat provides.

So RedHat, what do you say?  Expand the program for those of us who want to learn about the full breadth of RedHat products!

Categories
Uncategorized

So, what you’re saying is…

Had Chinese with my family last night and I got this in my fortune-cookie:

 

Fortune: :) You are soon going to change your present line of work. :)
Interesting fortune…

I’m wondering if this is foreshadowing upcoming events?  What do the fates have in store for me?

Categories
Uncategorized

When precision comes naturally

My wife has some wonderful traits, one of my favorite is her joy of cooking fits well with my joy of eating.  She is also very logical and efficient in her work so it didn’t surprise me that when we had to bake cookies, so she used the recently cleaned kitchen (weekend cleaning spree) to make the job easier.

I entered the kitchen and saw our recently cleaned counter looking like this:

That’s a photo of our kitchen counter with the recipe card laid in the geometric center of the work area.  It struck me as how symmetric it was and I joked about it with my wife.  After looking at it a second time I realized that it was VERY close to the center along both axis of the counter.  So, to satisfy my curiosity I grabbed a tape measure and put my life (and happy marriage) on the line.

 

I measured the depth of the counter and it was 26″.  I then measured the center of the recipe card and it was exactly 13″ from each edge.

Both my wife and I laughed at this, the precision of the placement even though she had just “laid it down” while she was getting other things ready in the kitchen.

 

 

Well, I had gone this far, I might as well measure the other axis.  (If you know my wife, you won’t be surprised with the results.)

Yup, you guessed it.  The width of the working area of the counter is 61″, and the center of the card is just under 30″ from the edge.  While not a perfect 30.5″, we decided that the corner area that wasn’t readily usable for baking, and her intrinsically logical mind accounted for this and the ‘usable center’ of the counter is slightly skewed to the left away from the cluttered end.

The human mind amazes me all the time, and the cookies were great too!

Edit 1: Fixing math.

Categories
Uncategorized

An open letter to my daughters.

In case I never get around to saying this, or in the event that something happens and I cause you to think otherwise I want you to always remember that you are the most beautiful women in the world and deserve to have every chance you can take.

As much as I think I am “making you”, as a parent I’m really only pointing you in a general direction for your life in the best direction as I see it. The direction may not always be the best, it might even lead to a dead-end in some cases and require you (us actually) to back-track and try something else. I realize that there are some times you’re doing something only because “dad said so”, but I’m hopeful that over time our directions will continue to become more and more parallel. I only want what is best for you even if “best” isn’t the direction we appear to be going.

I also want you to grow up to be someone who can and will make good decisions on your own. There will be a time when feel alone – it is those times as a parent I’m most afraid for the outcome. It might be a time when you’re truly alone and have to rely on your own instinct and reasoning to persevere, or it might be a time when you’re feeling pressured to do or say something that you feel in your gut isn’t right. I hope you’ll have the strength to act on what you feel is right at that time.

Your gut feeling is made up partially by what I have helped instill in you over time, but also part of what you have learned and have taken from what I and others have said and done. I can’t pour in a lifetime of knowledge into you in the few short years you’re living under my roof, but I can try to lead as good as an example as I can every day.

Was I perfect? I’m the first to admit no, not by a long shot. There are some events that we both could point to as a failing, and there are some that you saw as a failing that I did not. There are also points along the way where I feel that I failed you but you made it through and probably didn’t notice. The ones where we acknowledge a failure together are acceptable because we can both learn and hopefully move on. The ones where you saw a failure and I didn’t are harder for me to correct but at the same time I’m proud that you are able to adapt and grow from it. The ones that I’ve kept inside are probably some of the hardest – I know I have failed in some extent but somehow either you adapted or you assumed that I made that choice. If you adapted I’m proud of you accomplishment – it definitely wasn’t by me helping at that moment. If you suffered and thought it was because I made the choice to have you suffer, I apologize and I can never make it up to you. Those moments will live in my mind forever.

All of that is taking place while you’re still on your way to adulthood. After you become an adult, your directions and actions are your own. I’m hopeful that the direction I set you on is a good one, and I hope that all those life events (good and bad) help provide you with adequate background to continue making good decisions when you’re on your own.

Remember this: As long as you continue to make good decisions based on what you know and stay true to yourself, family, and friends, you’ll always have my love and respect.

I can’t tell what is going to happen tomorrow any more than I can ten years from now. If you choose a path then that isn’t a faint option today, as long as you make that choice by staying true to yourself I’ll respect and love you for it irregardless. There may be some decisions you make that I don’t understand, and you might hear me sigh or see me roll my eyes I still love you deeply.

Once you’ve are on your own making adult decisions and accepting the consequences for yourself, I will do my best to stay out of your personal business until I’m asked. I hope that you will feel you can come to me for any advice – “words of wisdom” is way too conceited – but I’ll understand when you make your own way even if it isn’t what I’d advise. I may not fully support every action you might take, but I will promise to do my best to understand your decisions and support you.

As you become an adult, I’ll do my best to let you shine in the spotlight for all the accomplishments you will have. I will brag about being your parent when you’re graduating or accepting awards, but everything it took to get up to that point was you and I won’t take credit for that. Your decisions – whether I agreed with them or not – were yours and got you to where you will be. I can only hope that the direction I set you on many years ago were the ones that helped you become the “you” you will be.

Categories
Uncategorized

Net Neutrality

On the topic of “Net Neutrality”, this is one case where I believe that a true neutral Internet is something that the US should mandate. I won’t write a lot here – use your favorite search engine for “net neutrality” and look at some of the sites that are backing it (Google, Wikipedia, NetFlix, Amazon, etc).

Without it, our Internet service providers can slow or block access to sites on the Internet that either don’t pay them directly, or have a competing product. You might have noticed this in your cell phone advertisements – notice how many offer “free music streaming” if you use their service? Cell companies aren’t held under the current net neutrality rules and can do that; they can pick-and-choose the music that doesn’t use up your data allowance for the month and by extension are choosing what music services “win”. Now image that Google pays your ISP to make searching their site the only search engine that your home computer can access.
Or worse yet, the local WalMart pays the local ISPs to block access to Amazon.com? They probably couldn’t pay your ISP block it, but I’m sure for the right sized check they could get the ISP to make Amazon.com take minutes to come up, but WalMart.com would be instant. Who would you buy from then?

I don’t like the degree of government oversight especially in rapidly evolving technical topics, but this is one that a basic law that says “treat all data as equal” has won me over.
If you’re interested in taking action, head over to “Battle for the Net” at https://www.battleforthenet.com/ and sign up to call your congress men and congress women to ask them to stand with us.