Quick Bash One-Liner

Well, I'm here trying to wrap up things for the day and just found a very useful technique to get a directory without the filename from the listing if using find.

So, I'm sure you were all thinking, "well why don't you just use `find` with `-type d`?" Well, for this specific task I will be using `find` to get a list of all files and then I need whatever that directory is for another script. So booyah, in your face! (hehe jk)

So, apparently if I have a directory structure string with a file at the end of it I can drop the end part like so:

f1=/mnt/www/tmpdrin/dir1/dir2/fileindir.txt
f2=${f1%/*};
## $f2 will now be /mnt/www/tmpdrin/dir1/dir2


Sweet as!


And if you were curious how I was using it as a one-liner, here ya go:

for f in `find /mnt/www/tmpdrin/ -type f`; do f2=`echo "$f" | sed -e 's!/mnt/!!g'`; f3=${f2%/*}; echo "$f3"; done


Obviously, I'm doing a lot more here than just echoing, but this is just an example mmmkay? ;)

Cheers!

Post Script: It's been a while for posts, but I've just been working on proprietary stuff lately and nothing would be too useful for ya'll. Sorry!

Reset FreeBSD root Password

I have various different VMs for testing purposes. I needed to test something on FreeBSD and luckily I already had a VM setup. But, hmmm...I forgot the root password!

So, if you happened to ever forget the root password for one of your FreeBSD servers, fear not, you can reset it! Well, of course if you have access to boot the server into single user mode of course! ;)

The process is quick and painless; select the default shell once in single user mode (/bin/sh). After that just enter these commands to change the root password:


# mount -u /
# mount -a
# passwd


Obviously, if you've used passwd before it'll prompt you for the new password and to confirm. After that, all you need to do is exit or sync;sync; reboot.

That's it. Cheers!

Check MySQL Tables Across Multiple Databases

If you manage multiple databases on a MySQL server that are identical in architecture, but unique in data well I'm sure you may come across this issue at some point in time. This issue being that, the MySQL err log shows an error on a table, but it doesn't specify which database!

(Error looks like this: # 090217 18:38:22 [ERROR] /usr/local/bin/mysqld: Can't open file: 'wp_comments.MYI' (errno: 144)
)

Now sure, you can just run a mysqlcheck on all-databases' tables, but obviously that may take a while, especially if you manage a lot of databases.

So, remember just a few blog posts before, I mentioned a way of monitoring MySQL's err log. Well, to be honest, that was actually a product of my original idea; that idea being:
- monitor the MySQL err log for table errors
- if there is a table error, run a mysqlcheck for that specific table against all databases
- email the results

Well, why only search for one specific error if you can search for them all? So, I broke my script down into two; an err log monitor and a mysqlcheck. Same as always, tailor the code to fit your environment. For example in this line: if ($dbdata->[0] =~ m/^wp_.+/) { I'm only going to search through our Wordpress databases and skip everything else. You can change the regex so that you will only check databases that begin with the letter 'a' if that fits your situation (/^a.+/).

I'm sure you get the point! Eventually (when I'm less busy), I'll modify some of my code so that this will run more of an automated process. --All you need to do is have the err log script write to a file with the tables giving errors and then write a wrapper to process the err log script, then process the mysqlcheck script. But, that's for another day kids!

Cheers!


#!/usr/bin/perl -w
#
# chkmysqltables.pl v1.0
# 2009-02-17
# Adrian J. Cruz
#

use strict;
use DBI;

if (!$ARGV[0]) {
print "Usage: $0 \" [db_table]\"\n";
print "Note: Use quotes if you are checking multiple tables.\n";
exit;
}

my $db_table = $ARGV[0];

my $mysqlu = "dbuser";
my $mysqlp = qw(dbpass);
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) =
localtime(time);
my $yy = sprintf("%02d", $year % 100);
$year += 1900;
$mon += 1;
if ($mon < 10) { $mon = 0 . $mon; }
if ($mday < 10) { $mday = 0 . $mday; }
my $todaylog = $year . $mon . $mday . $hour . $min;
my $runlog = "/root/log/chkmysqltables.$todaylog.log";

my @tables;
push @tables, $db_table;
# we only want the unique tables
my %sorthash;
@sorthash{@tables} = ();
my @error_tables = keys %sorthash;

my @dbs;
my $dbh = DBI->connect("DBI:mysql:mysql",$mysqlu,$mysqlp)
|| die "Unable to connect: $!\n";
my $query = "SHOW DATABASES";
my $sth = $dbh->prepare($query)
|| die "Could not prepare: $!\n";
$sth->execute || die "Could not execute: $!\n";
while (my $dbdata = $sth->fetchrow_arrayref) {
if ($dbdata->[0] =~ m/^wp_.+/) {
push(@dbs,$dbdata->[0]);
}
}
$sth->finish;

open(LOG, ">>$runlog");
print LOG "$todaylog\n";
for my $t (@error_tables) {
for my $db (@dbs) {
system "/usr/local/bin/mysqlcheck --password=$mysqlp $db $t >> $runlog\n";
}
}
print LOG "--END LOG--\n";
close(LOG);

my $email_message;
open(MSG, "<$runlog");
$email_message = do {local $/; };
close(MSG);

sendMail( "you\@email.com", "mysqlerr\@email.com",
"mysqlerr: $db_table", "$email_message" );

sub sendMail {
my ($to, $from, $subject, $message) = @_;
my $sendmail = "/usr/sbin/sendmail";
open(MAIL, "|$sendmail -oi -t");
print MAIL "From: $from\n";
print MAIL "To: $to\n";
print MAIL "Subject: $subject\n\n";
print MAIL "$message\n";
close(MAIL);
}


Processing a Mailbox With POP and Perl

My company sends out newsletters. Of course, for whatever reason sometimes a user no longer wishes to receive these newsletters; and sometimes a user will continue to receive newsletters and instead of using the "unsubscribe" links in the newsletter, they will just mark the email as junk. So, then their ISP, email provider, etc. will then do their job to send us an email saying that there has been a complaint about us spamming. The routine continues and the emails go round and round.

Here is where our "Auto Unsubscribe" mailbox comes to play. We have a mailbox setup where email providers forward emails of complaints. They pretty much always keep the original email message intact, so the auto unsubscribe job is easy: search through the mailbox for the data you need; the unsubscriber's email address and the newsletter.

Finding the correct data is simple on our end as long as the original newsletter email is intact. At the bottom of all of our newsletters is a link to an unsubscribe page. Now, in plain text this is obviously just going to be a line with a URI and parameters. For example, http://www.server.com/unsubscribe.htm?newsletter=ABC&e=youremail@mail.com

You see where I'm going here? You've got all the data you need and now you can do what you want with it.

Cheers!

NOTE: I'm using the Mail::POP3Client module because I personally needed to use SSL when connecting to the mailbox. You can easily use Net::POP3, Net::IMAP, etc. just the same.


#!/usr/bin/perl
#
# unsub-pop.pl v1.0
# Adrian J. Cruz
# 2009-02-19

use strict;
use Mail::POP3Client;

my $mail_server = qw(mail.server.com);
my $mail_user = qw(user@server.com);
my $mail_pass = qw(pass123);
my $pop = new Mail::POP3Client( USER => $mail_user,
PASSWORD => $mail_pass,
HOST => $mail_server,
USESSL => "true");
$pop->Connect
or die "couldn't connect: $!\n";

# find newsletter and email
my @ecunsubs;
for (my $i = 1; $i <= $pop->Count(); $i++) {
foreach ($pop->Body($i)) {
if (/unsubscribe\.htm\?newsletter\=(.+\&e\=.+)\"/) {
push(@ecunsubs,$1);
}
}
# mark msg to be deleted
$pop->Delete($i);
}
$pop->Close;

# remove duplicates so now we process only unique entries
my %sorthash;
@sorthash{@ecunsubs} = ();
my @unsubs = keys %sorthash;
...
...
etc.
...


Monitoring the MySQL ERR Log

If you've ever wanted a quick and easy way to monitor MySQL for errors, well, you've stumbled onto the right place.

At work we've been getting these corrupt tables that go unnoticed until an end-user notifies us. Which, of course, for any website is utterly disastrous since some part of your site (even if it is just a minuscule widget on the site) is now broken. So, being the proactive technologists that we are we decided to monitor the MySQL err log daily. You can find your err log in the "data" directory where you installed MySQL (it defaults to the name: hostname.err).

What this perl script does is parse through the err log and find today's ERRORs. After it finds the errors, it then emails them. So, obviously go through and change the variables to your likings and set up a cron to have this run at the end of the day.

Cheers!


#!/usr/bin/perl -w
#
# chkmysqlerrlog.pl v1.0
# 2009-02-17
# Adrian J. Cruz
#

use strict;

my $dbserver = "dbserver.domain.com";
my $email_to = "email\@domain.com";
my $email_from = "mysqlerr\@domain.com";
my $errlog = "/usr/local/mysql/data/dbserver.domain.com.err";
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) =
localtime(time);
my $yy = sprintf("%02d", $year % 100);
$year += 1900;
$mon += 1;
if ($mon < 10) {
$mon = 0 . $mon;
}
# set $today in date format: YYMMDD to get today's errors
# e.g.:
# 090217 18:38:22 [ERROR] mysqld: Can't open file: 'ts.MYI' (errno: 144)
my $today = $yy . $mon . $mday;
my $todaylog = $year . $mon . $mday . $hour . $min;

# parse through the mysql err log for ERRORs
my @errors;
open(IN, $errlog);
while () {
chomp;
if ($_ =~ m/$today.+(\[ERROR\].+$)/) {
push(@errors, $1);
}
}
close(IN);

# we only want the unique errors
my %sorthash;
@sorthash{@errors} = ();
my @error_msgs = keys %sorthash;

my $email_message = "$todaylog\n";
for my $e (@error_msgs) {
$email_message .= $e . "\n";
}

if ($email_message && @error_msgs) {
sendMail( $email_to, $email_from,
"$todaylog $dbserver ERRORS", $email_message );
}

sub sendMail {
my ($to, $from, $subject, $message) = @_;
my $sendmail = "/usr/sbin/sendmail";
open(MAIL, "|$sendmail -oi -t");
print MAIL "From: $from\n";
print MAIL "To: $to\n";
print MAIL "Subject: $subject\n\n";
print MAIL "$message\n";
close(MAIL);
}


NOTE: with a little bit of work you can even include the Warnings; we weren't too worried about those.

Happy 1234567890 Epoch Time (EST)

"Wait for it... wait for it... come on... getting close... aha! There we go!"

Yay, we had an "epoch" of a time here on the east coast. hehehe

Cheers!

Using a Script to Edit a conf File

Well, because I have been uber busy here at work, I haven't been able to write any blog posts. But, since I've been super productive at work, I get to write up a few posts about some more tools I've written to add to my arsenal. w00t!

So my latest task involved editing some of our web server's Apache confs. It consisted of adding a new alias so that we can call in some new app someone else wrote. Okay cool. Alias? Sure not a problem. Er...wait! The alias requires the site's directory name as part of the path! Oh no! That means I can't create one global alias!

Ah, see that's when I thought of writing this script that will look for a specific line in the vhost entries and will enter the new line after it. Pretty nifty eh? :)

So basically, what we have here is a list of all sites in a text file. Now we can use the site's directory name read in as a variable. It will then use awk to search for a specific line, and then print the new line you need right below it!

So if you take a look at the code, I want to add this line:
"Alias /newalias /mnt/www/$site" (where $site is the site directory name variable)
right after each line that fits the regex that searches for any line that has "Alias" and "reports" in one line followed by the site dir name at the end of the line. Obviously if you want to utilize this script you should edit the regex to your needs.

And that's it! Now think of all of the time saved rather than adding a new line manually to EACH vhost entry on EACH server! whoa....scary...


#!/bin/bash

exec < sites.txt
while read site ; do
echo "processing $site..."
awk '{print $0} '"/Alias.+reports.+\/$site"$/'{print " Alias /newalias /mnt/www/'"$site"'"}' httpd.conf > newhttpd.conf && mv newhttpd.conf httpd.conf
echo "done!"
done


Shell Scripting to Read Data From a File

Okay, so here's your problem: you have a file that contains data that you'd like to parse through a script. It's a lot of data and you don't want to sit there and continuously do the same thing over and over!

The data can be set up as one long column of data or multiple columns of data. In this case for simplicity I'll only use one column of data.

So let's say your data file is as follows:

dog
cat
cow
...
...
(long list of data)


These represent parts of filenames that you need to clear out. You don't want to sit there typing cat /dev/null > dog
cat /dev/null > cat
... So you come up with a master plan to script the process. The script will be short, sweet, and absolutely time saving!

You first need to read in the file; so this line does the work for you: exec < files.txt. Obviously, in this example my "files.txt" is the filename that contains the data I want to read. After you've opened up the file for reading, you can now parse through the file with a while loop:
while read file ; do
[do some work here]
done


Now note, "file" is the variable that we're assigning to the column. You can call the variable whatever you want. Also remember that I am being simple with this example and only have one column of data. If I wanted to do multiple columns of data, I would just assign more variables to each column. e.g. while read col01 col02 col03 ; do where $col01, $col02, and $col03 will be my new variables.

That's it! I'll leave the customization to you. Cheers!

Here is the full script example:
#!/usr/bin/bash

exec < files.txt

while read file ; do
if [ -s /mnt/$file ] ; then
cat /dev/null > /mnt/www/vdir/z/ec/$file
else
echo "/mnt/$file already empty!"
fi
done


xargs Is a Very Useful Tool

I was recently asked how to remove specific files of a specific extension (.tmp in this case). Well, of course you can find them by doing a find . -type f -name "*.tmp". Easy eh? But how to delete them if they span deep into the directory tree? Well, I suppose you can just append the output of your find command into a text file and then have a script do the rm'ing of the files. But, what if you can do all of that work in one command line?

So, I mentioned xargs and mentioned what it can do and gave a brief example and lo and behold, he saved a good 5 minutes of extra work. Awesome!

Now, to exemplify what I just said:
find . -type f -name "*.tmp" | xargs rm

Just think of all the possibilities now that you have all of this functionality! You can cp, tar, mv, etc. etc.

Enjoy! Cheers!