blogs

So, I’ve realized that I made a mistake in splitting out this blog into two more (my working with teenagers one and my scriptures one).  The reason being that, the other two felt like I had to have these nicely crafted blog posts put together.  That kinda sucks.  It puts pressure on me to come up with something nice, and more importantly, it doesn’t allow me to explore at all.  In other words, make mistakes, and talk about stuff I’m researching versus delivering a final draft.

I think I’m gonna retain my other two blogs, but just put revised posts there, and go back to the old way here.

Leave a comment

Filed under Uncategorized

gentoo, openrc, apache and monit – proper starting and stopping

I regularly use monit to monitor services and restart them if needed (and possible).  An issue I’ve run into though with Gentoo is that openrc doesn’t act as I expect it to.  openrc keeps it’s own record of the state of a service, and doesn’t look at the actual PID to see if it’s running or not.  In this post, I’m talking about apache.

For context, it’s necessary to share what my monit configuration looks like for apache.  It’s just a simple ‘start’ for startup and ‘stop’ command for shutdown:

check process apache with pidfile /var/run/apache2.pid start program = “/etc/init.d/apache2 start” with timeout 60 seconds stop program = “/etc/init.d/apache2 stop”

When apache gets started, there are two things that happen on the system: openrc flags it as started, and apache creates a PID file.

The problem I run into is when apache dies for whatever reason, unexpectedly.  Monit will notice that the PID doesn’t exist anymore, and try to restart it, using openrc.  This is where things start to go wrong.

To illustrate what happens, I’ll duplicate the scenario by running the command myself.  Here’s openrc starting it, me killing it manually, then openrc trying to start it back up using ‘start’.

# /etc/init.d/apache2 start
# pkill apache2
# /etc/init.d/apache2 status
* status: crashed
# /etc/init.d/apache2 start
* WARNING: apache2 has already been started

You can see that ‘status’ properly returns that it has crashed, but when running ‘start’, it thinks otherwise.  So, even though an openrc status check reports that it’s dead, when running ‘start’ it only checks it’s own internal status to determine it’s status.

This gets a little weirder in that if I run ‘stop’, the init script will recognize that the process is not running, and reset’s openrc’s status to stopped.  That is actually a good thing, and so it makes running ‘stop’ a reliable command.

Resuming the same state as above, here’s what happens when I run ‘stop’:

# /etc/init.d/apache2 stop
* apache2 not running (no pid file)

Now if I run it again, it checks both the process and the openrc status, and gives a different message, the same one it would as if it was already stopped.

# /etc/init.d/apache2 stop
* WARNING: apache2 is already stopped

So, the problem this creates for me is that if a process has died, monit will not run the stop command, because it’s already dead, and there’s no reason to run it.  It will run ‘start’, which will insist that it’s already running.  Monit (depending on your configuration) will try a few more times, and then just give up completely, leaving your process completely dead.

The solution I’m using is that I will tell monit to run ‘restart’ as the start command, instead of ‘start’.  The reason for this is because restart doesn’t care if it’s stopped or started, it will successfully get it started again.

I’ll repeat my original test case, to demonstrate how this works:

# /etc/init.d/apache2 start
# pkill apache2
# /etc/init.d/apache2 status
* status: crashed
# /etc/init.d/apache2 restart
* apache2 not running (no pid file)
* Starting apache2 …

I don’t know if my expecations of openrc are wrong or not, but it seems to me like it relies on it’s internal status in some cases instead of seeing if the actual process is running.  Monit takes on that responsibility, of course, so it’s good to have multiple things working together, but I wish openrc was doing a bit more strict checking.

I don’t know how to fix it, either.  openrc has arguments for displaying debug and verbose output.  It will display messages on the first run, but not the second, so I don’t know where it’s calling stuff.

# /etc/init.d/apache2 -d -v start
<lots of output>
# /etc/init.d/apache2 -d -v start
* WARNING: apache2 has already been started

No extra output on the second one.  Is this even a ‘problem’ that should be fixed, or not?  That’s kinda where I’m at right now, and just tweaking my monit configuration so it works for me.

12 Comments

Filed under Gentoo

freebsd, quick deployments, shell scripts

At work, I support three operating systems right now for ourselves and our clients: Gentoo, Ubuntu and CentOS.  I really like the first two, and I’m not really fond of the other one.  However, I’ve also started doing some token research into *BSD, and I am really fascinated by what I’ve found so far.  I like FreeBSD and OpenBSD the most, but those two and NetBSD are pretty similar in a lot of ways, that I’ve been shuffling between focusing solely on FreeBSD and occasionally comparing at the same time the other two distros.

As a sysadmin, I have a lot of tools that I use that I’ve put together to make sure things get done quickly. A major part of this is documentation, so I don’t have to remember everything in my head alone — which I can do, up to a point, it just gets really hard trying to remember certain arguments for some programs.  In addition to reference docs, I sometimes use shell scripts to automate certain tasks that I don’t need to watch over so much.

In a typical situation, a client needs a new VPS setup, and I’ll pick a hosting site in a round-robin fashion (I’ve learned from experience to never put all your eggs in one basket), then I’ll use my reference docs to deploy a LAMP stack as quickly as possible.  I’ve gotten my methods refined pretty well so that deploying servers goes really fast — in the case of doing an Ubuntu install, I can have the whole thing setup close to an hour.  And when I say “setup” I don’t mean “having all the packages installed.”  I mean everything installed *and* configured and ready with a user shell and database login and I can hand over access credentials and walk away.  That includes things like mail server setup, system monitoring, correct permissions and modules, etc.  Getting it done quickly is nice.

However, in those cases of quick deployments, I’ve been relying on my documentation, and it’s mostly just copy and paste commands manually, run some sed expressions, do a little vim editing and be on my way.  Looking at FreeBSD right now, and wanting to deploy a BAMP stack, I’ve been trying things a little differently — using shell scripts to deploy them, and having that automate as much as possible for me.

I’ve been thinking about shell scripting lately for a number of reasons.  One thing that’s finally clicked with me is that my skill set isn’t worth anything if a server actually goes down.  It doesn’t matter if I can deploy it in 20 minutes or three days, or if I manage to use less memory or use Percona or whatever else if the stupid thing goes down and I haven’t done everything to prevent it.

So I’ve been looking at monit a lot closer lately, which is what I use to do systems monitoring across the board, and that works great.  There’s only one problem though — monit depends on the system init scripts to run correctly, and that isn’t always the case.  The init scripts will *run*, but they aren’t very fail-proof.

As an example, Gentoo’s init script for Apache can be broken pretty easily.  If you tell it to start, and apache starts running, but crashes after initialization (there’s specifics, I just can’t remember them off the top of my head) the init script thinks that the web server is running simply because it managed to run it’s own commands successfully.  So the init system thinks Apache is running, when it’s not.  And the side effects from that are that, if you try to automatically restart it (as monit will do), the init scripts will insist that Apache is already running, and things like executing a restart won’t work, because running stop doesn’t work, and so on and so forth.  (For the record, I think it’s fair that I’m using Apache as an example, because I plan on fixing the problem and committing the updates to Gentoo when I can.  In other words, I’m not whining.)

Another reason I’m looking at shell scripting as well is that none of the three major BSD distros (FreeBSD, NetBSD, OpenBSD) ship with bash by default.  I think all three of them ship with either csh or tcsh, and one or two of them have ksh as well.  But, they all have the original Bourne shell.  I’ve tried my hand and doing some basic scripting using csh because for FreeBSD, it’s the default, and I thought, “hey, why not, it’s best to use the default tools that it ships with.”  I don’t like csh, and it’s confusing to try and script for, so I’ve given up on that dream.  However, I’m finding that writing stuff for the Bourne shell is not only really simple, but it also adds on the fact that it’s going to be portable to *all* the distros I use it on.

All of this brings me back to the point that I’m starting to use shell scripts more and more to automate system tasks.  For now, it’s system deployments and system monitoring.  What’s interesting to me is that while I enjoy programming to fix interesting problems, all of my shell scripting has always been very basic.  If this, do that, and that’s about it.  I’ve been itching to patch up the init scripts for Gentoo (Apache is not the only service that has strange issues like that — again, I can’t remember which, but I know there were some other funky issues I ran into), and looking into (more) complex scripts like that pushes my little knowledge a bit.

So, I’m learning how to do some shell scripting.  It’s kind of cool.  People always talk about, in general, about how UNIX-based systems / clones are so powerful because of how shell scripting works .. piping commands, outputting to files, etc.  I know my way around the basics well enough, but now I’m running into interesting problems that is pushing me a bit.  I think that’s really cool too.  I finally had to break down the other day and try and figure out how in the world awk actually does anything.  Once I wrapped my head around it a bit, it makes more sense.  I’m getting better with sed as well, though right now a lot of my usage is basically clubbing things to death.  And just the other day I learned some cool options that grep has as well, like matching an exact string on a line (without regular expressions … I mean, ^ and $ is super easy).

Between working on FreeBSD, trying to automate server deployments, and wanting to fix init scripts, I realized that I’m tackling the same problem in all of them — writing good scripts.  When it comes to programming, I have some really high standards for my scripts, almost to the point where I could be considered obsessive about it.  In reality, I simply stick to some basic principles.  One of them is that, under no circumstances, can the script fail.  I don’t mean in the sense of running out of memory or the kernel segfaulting or something like that.  I mean that any script should always anticipate and handle any kind of arbitrary input when it’s allowed.  If you expect a string, make sure it’s a string, and that it’s contents are within the parameters you are looking for.  In short, never assume anything.  It could seem like that takes longer to write scripts, but for me it’s always been a standard principle that it’s just part of my style. Whenever I’m reviewing someone else’s code, I’ll point to some block and say, “what’s gonna happen if this data comes in incorrectly?” to which the answer is “well, that shouldn’t happen.”  Then I’ll ask, “yes, but what if it *does*?”  I’ve upset many developers this way. :)  In my mind, could != shouldn’t.

I’m looking forward to learning some more shell scripting.  I find it frustrating when I’m trying to google some weird problem I’m running into though, because it’s so difficult to find specific results that match my issue.  It usually ends up in me just sorting through man pages to see if I can find something relative.  Heh, I remember when I was first starting to do some scripting in csh, and all the search results I got were on why I shouldn’t be using csh.  I didn’t believe them at first, but now I’ve realized the error of my ways after banging my head against the wall a few times.

In somewhat unrelated news, I’ve started using Google Plus lately to do a headdump of all the weird problems I run into during the day doing sysadmin-ny stuff.  Here’s my profile if you wanna add me to your circles.  I can’t see a way for anyone to publicly view my profile or posts though, without signing into Google.

Well, that’s my life about right now (at work, anyway).  The thing I like the most about my job (and doing systems administration full time in general) is that I’m constantly pushed to do new things, and learn how to improve.  It’s pretty cool.  I likey.  Maybe some time soon I’ll post some cool shell scripts on here.

One last thing, I’ll post *part* of what I call a “base install” for an OS.  In this case, it’s FreeBSD.  I have a few programs I want to get installed just to get a familiar environment when I’m doing an install: bash, vim and sometimes tmux.  Here’s the script I’m using right now, to get me up and running a little bit.  [Edit: Upon taking a second look at this -- after I wrote the blog post, I realized this script isn't that interesting at all ... oh well.  The one I use for deploying a stack is much more interesting.]

I have a separate one that is more complex that deploys all the packages I need to get a web stack up and running.  When those are complete, I want to throw them up somewhere.  Anyway, this is pretty basic, but should give a good idea of the direction I’m going.  Go easy on me. :)

Edit: I realized the morning after I wrote this post that not only is this shell script really basic, but I’m not even doing much error checking.  I’ll add something else in a new post.

#!/bin/sh
#
# * Runs using Bourne shell
# * shells/bash
# * shells/bash-completion
# * editors/vim-lite

# Install bash, and set as default shell
if [ ! -e /usr/local/bin/bash ] ; then
	echo "shells/bash"
	cd /usr/ports/shells/bash
	make -DBATCH install > /dev/null 2>&1
	if [ $? -ne 0 ]; then
		echo "make install failed"
		exit 1
	fi
else
	echo "shells/bash - found"
fi
if [ $SHELL != "/usr/local/bin/bash" ] ; then 
	chsh -s /usr/local/bin/bash > /dev/null 2>&1 || echo "chsh failed"
fi

# Install bash-completion scripts
if [ ! -e /usr/local/bin/bash_completion.sh ] ; then
	echo "shells/bash-completion"
	cd /usr/ports/shells/bash-completion
	make -DBATCH install > /dev/null 2>&1
	if [ $? -ne 0 ]; then
		echo "make install failed"
		exit 1
	fi
else
	echo "shells/bash-completion - found"
fi

# Install vim-lite
if [ ! -e /usr/local/bin/vim ] ; then
	echo "editors/vim-lite"
	cd /usr/ports/editors/vim-lite
	make -DBATCH install > /dev/null 2>&1
	if [ $? -ne 0 ]; then
		echo "make install failed"
		exit 1
	fi
else
	echo "editors/vim-lite - found"
fi

# If using csh, rehash PATH
cd
if [ $SHELL = "/bin/csh" ] ; then
	rehash
fi

6 Comments

Filed under Computers, Gentoo, Programming, Uncategorized

freebsd

I’ve started looking at FreeBSD at work this week, because I was reading some blog posts about how MySQL performs well on a combination of that and ZFS together.  I haven’t gotten around to getting ZFS setup yet, but I have been looking into FreeBSD as an OS a lot, and so far, I like it.

This makes the second distro in the past year that I’ve really started to seriously look into, the other one being Ubuntu.  I’m still trying to wrap my head around the whole FreeBSD design structure and philosophy, and for now I’m having a hard time summing it up.  In my mind, it kind of feels like a mashup of functionality between Gentoo and Ubuntu.  I like that there is a set group of packages that are always there, kind of like Ubuntu, but that you can compile everything from source, like Gentoo.

What has really surprised me is how quickly I’ve been able to pick it up, understand it, and already work on getting an install up and running.  I think that having patience is probably the primary reason there.  Figuring out how things work hasn’t really been that hard, but I say that because of past Linux experience that has helped me figure out where to look for answers more easily.  That is, when I get stuck on something, I can usually figure it out just by guessing or poking around with little effort.

Years ago, if I would have looked at any BSD, I would have been asking “why?”  I still don’t know why I’m looking at it, other than I believe it’s not a good idea to put all your eggs in one basket.  At work we already support CentOS, Gentoo and Ubuntu, and it’d be awesome to add FreeBSD to the list.

I’m really enjoying it so far.  It’s easy to install packages using the ports system.  I tried going the route of binary packages at first, but that wasn’t working out so well for me.  Then I tried mixing ports and packages, and that wasn’t doing too great either, so I switched to just using ports for now.

The only thing I don’t like so far is how it’s kind of hard to find what I’m looking for.  I totally chalk that up to me being a noob, and not as any real flaw of the distro or it’s documentation — I just don’t know where to look yet.  Fortunately, ‘whereis’ has saved me a lot of time.

The system seems familiar enough and easy to use for me, coming from a Linux background.  In fact, I really can’t find many differences.  The things I have noticed are that it uses much less memory, even on old underpowered boxes, and that it is relatively quick out of the box.  I never would have guessed that.

I’m curious to see how ZFS integrates into the system, if at all.  I like the filesystem, and it’s feature set, but that’s about it for now (I got to play with it a bit as we had a FreeNAS install for a few months).  If it’s a major pain to integrate it, I’m probably not going to push for it right now — I’m content with riding out the learning curve until I feel more comfortable with the system.

So, all in all, it’s cool to find something different, that doesn’t feel too different, but still lets me get my head in there and figure out something new.

If you guys know of any killer apps to use on here, let me know.  I’m kind of wishing I had an easier way to install stuff using ports aside from tromping through /usr/ports manually looking for package names.

8 Comments

Filed under Computers

what i’m reading: “real boys”

Summer is rough for me.  I take fewer classes, I have lots more free time, and things are generally a lot less unstructured.  This means my life is full of chaos.

One thing I’ve noticed about school, recently, is that if I’m not taking any psychology courses, I become indifferent about working towards a degree.  It’s hard slogging through generals for any student, but in my case, where there’s limited amounts of time and money to spend on pursuing an education, it just feels like it’s not worth the hassle.

So, summer is a little rougher for me, and I’m looking forward to Fall and Spring semesters again.

In the meantime, and I’ve been doing this for awhile, I always have one book about psychology or counseling that I’m reading.  Right now, I’m making my way through a great book called “Real Boys.”

If I could summarize the book, it’s basically documenting the effects of boys not expressing their feelings.  I was going to expound on that, but that’s just about how it goes.  I also use the term ‘boys’ here from the author’s context, not mine. He tends to cover a large age group, from about eight to sixteen.

Flipping it open tonight, the page I started on perfectly expressed the “why” I want to work with youth so much — or, that is, the kinds of problems I want to encounter and help people out with:

When boys become hardened, they become willing to endure emotional and physical pain–even to risk their lives–if it means winning the approval of their peers.  Boys can become so thoroughly hardened that they literally anesthetize themselves against the pain they must cope with.  And they are often left unsupervised at an earlier age than girls and are usually discouraged by adults from engaging in help-seeking behaviors at their time of greatest vulnerability or need, boys learn to remain silent despite their suffering.

Incredibly sad commentary, of course, but also accurate.

I suppose that the solution could be summed up in “love your kids,” but what I see happening is that culture is a strong influence of how to love them — when to cut them loose, when to have them “man up,” and so on.  Culture is a poor guide for determining personal milestones.

I’ve been learning more about counseling and people not just with what I read, but as I casually observe people and realize how simple things are.  The realization is dawning on me that humans are alike emotionally, wanting the same basics subsets of love and caring: respect, communication, validation, correction and instruction.  Things that people do that are “weird” or “out there” are most times going to be tied back to some fundamental need that is unaddressed.  And in the cases where that is the case, there can be a check for internal chemical imbalances (depression, schizophrenia, OCD, mood disorders etc.) where medication can do a lot of good in providing more stability.

On a personal level, not an academic one, from helping out others, I’ve noticed how important it is that people have someone that will look them in the eye and listen to them.  I’ve noticed that just looking at someone directly often times can slightly startle someone, since it is so unexpected.  I’ve seen though, how talking calmly and directly to someone will both relax them and engender some trust.  People just want to be listened to.

Anyway, it’s all fascinating stuff, and I love reading up on it, and discovering new things.  In a lot of ways, I’m finding that counseling is based on really simple principles of caring and communicating.

1 Comment

Filed under Psychology

rebooting my mini-itx

It’s been a long time since I’ve worked on much anything computer-related as a hobby.  Things have changed quite a lot in the past year.  I moved to a much smaller apartment in Salt Lake, which is about a third the size of my old place.  The idea was to trim the fat and focus on going back to school, which is my major direction in life these days.  When I moved in, I didn’t have room for setting up a desktop computer anywhere, so it’s been just my netbook and me.  That suits me plenty fine, though, I wasn’t really using it that much either.  I had just upgraded to a six-core so I could rip DVDs much faster, and now it was sitting headless wherever I could find room, and even then, only used occasionally.

It’s not just at home that things have been changing.  At work I got to make the transition from programmer to full-time sysadmin, and I’m absolutely loving it.  I knew I was getting tired of coding, and I had always enjoyed just taking care of servers, and now I get to do that all day long. When I initially started as a sysadmin, I didn’t think our small company would have enough work for me to do after a few months.  In actuality, I’m kept busy all the time.  The part I like the most is that part of my job is doing research, how to do things better, more efficiently, anything to make the workload easier.  It’s fun.

On top of all that, my school attendance is starting to ramp up more, and I’ve been consistently drifting to adding more classes to my workload.  All this stuff has basically booted Linux out of my life as a hobby, and so now I need things to “just work” without hassle, so I leave my installations alone.

One thing I’d been neglecting a little bit was my entire HTPC setup.  I hadn’t been using it much lately just because I would mostly stream some Netflix (yay, Doctor Who!).  My setup has been a beast though, normally running for months on end without the slightest hiccup.  What started to happen though is that I would come back to using it, switching my HDMI input over, and the box would be powered off for some reason.  Most of the time, I would either power it back on and go on with life or just ignore it.  Until one day it wouldn’t power on at all, and I just shrugged it off and determined to look at it later.

Well, later turned out to be finals week, when my brain has been working overtime, and I seriously needed a hobby.  I pulled out my main frontend and started looking at it to see what was going on.  It was plugged in properly and everything looked legit, but when I hit the power, the CPU fan would start up for a second and then everything would stop.  After fiddling with it for a bit, I started to notice that something was smelling burnt.  Once that happened, I abandoned my diagnosis.  Even if I did manage to get it working, I didn’t want it to catch everything on fire.

At the same time, my external USB drive enclosure died on me.  So even if I could have gotten it working, I still wouldn’t have had a way to watch my shows.  Them giving out on me hasn’t bothered me in the least — the entire setup has been running flawlessly for years, and I’d managed to get a lot of mileage out of them.

Now I had to decide what I was going to do.  I have a lot of hardware, but in pieces.  I have four mini-ITX boards altogether, two of them are VIA C7 chipsets, and the other two are Zotac boards both running low-powered Celeron CPUs (around 35W if I remember correctly).  The power supplies for the VIA boards use 20-pin connectors and only run at about 80W, and aren’t enough to handle the Zotac boards which use 24-pin connectors.  So I have this mix of hardware, and nothing powerful enough to act as a frontend.

There are some great packaged systems out there now where for between $200 to $300 you can get an entire package in one go that does exactly what I’m putting together myself. I considered the idea of just starting over, but I decided that it’d be cheaper to just salvage what I could.

So this week I ordered a new USB HDD enclosure, and I also ordered a new power supply for the main Zotac board.  I found a site that sells really small power supplies for mini-ITX boards, called picoPSU.  The design eliminates a lot of the hardware that I would normally need to get all the power to my box.  I was really skeptical about them when I first heard of it, but did some looking around and it looks like it’s exactly what I need.

In the meantime, I ripped out my motherboard out of my desktop, and put both Zotac boards in there to make sure they still work, and thankfully they do.  I got the old setup pieced together using my desktop case, and fired up the old system to play around with it.

I had started to forget how much time I put into this thing.  I forgot that I had put countless hours stitching this thing together, running a custom build of Gentoo suited to run on small environments.  On top of that I made hacks to mythvideo and got those working to polish off some rough edges.  It just started to come back to me how much I’d worked on this … and how much fun it was. :)

I played around with my frontend a little bit, and fired up a few movies just to try out the surround sound.  It was awesome.  I’d forgotten how nice it was to have that huge library on demand, too.

So I’m excited now to get things up and running.  It’s been a good little while.

Leave a comment

Filed under Hardware, Multimedia

working with teenagers … the blog!

Okay, so I decided to start yet. another. new. blog. It’s called “working with teenagers”. I’m reproducing … at least, in some fashion. I wonder if my parents are proud of me. Late at night, they can stay up and say, “this is about as close to grandkids as we’ll get! Pass me some Wheat Thins.” Seems reasonable.

Really, though, since I’m going to school to, you know, do this full-time, I thought it’d be cool to archive my old posts about working with them, and just post stuff to it whenever I feel like it. Like tonight, I just added another one, and I figured, “I should probably go to bed. And also write a blog post and my other blog!” And then my mind went blank after that.

In addition to the archives of stuff on here that you’ve already taken the time to memorize, I’ve added two new posts over there since then. You’ll notice that I’ve refrained from shamelessly using my blog to do some cross-posting mojo to do some self-promotion … at least until tonight. To make it seem like this blog post has actual content, I’ll throw in something slightly more interesting.

I found out recently that I really enjoy bowling. Me and my cousin have been going for a few weekends in a row. We’ve mastered the art of playing 4 games in a row for $10. That’s not bad, considering it’s late Saturday nights. Good times. I’m actually getting better at (since it’s impossible to be worse). The hardest part is getting people to ignore that I’m using an 8 pound ball because I’ll throw out my wrists if I use anything heavier.

Where was I going with all of this. I remember I was playing Skyrim tonight (level 60, yo!), and I was fighting a dragon and trying to eat cheesy nacho goodness at the same time. I kept having to pause my game so I could eat, and I thought to myself, “I can’t pause a nacho.” Words to live by.

In other entertainment, I present to you, the best picture on the internet:

It’s totally legit. They have their own domain and everything: http://thebestpictureontheinternet.com/

I think it’s time to go to bed.

2 Comments

Filed under Psychology