Python: Reading and Running Commands From A File

So I recently came across a broken process that needed some manual re-running. I was able to search through the log files and then see what part needed to be re-run. I ended up with a new data file (that I modified) to be a list of each command that failed for each iteration. So, the file looked something like this:
/path/to/program -option x -site a
/path/to/program -option x -site b
...
/path/to/program -option x -site z


Using a bash for line in `cat command.file`; do $line; done would not work since I would end up with a
-bash: -option: command not found
-bash: x: command not found
-bash: -site: command not found
error. So, here's some quick Python that can get the job done:

import os

f = open('/tmp/tmp-fix.commands')
for line in f:
    print line.rstrip()
    os.system(line.rstrip())


Yay for saving me some time! Cheers!

Python: Getting Line Counts of Files and Reading Directories

I have a bunch of little scripts to help out with day-to-day tasks (don't we all?). For some reason, (probably because I've been busy) I never posted anything tagged with "python". Whoa! Sharing is caring, so here it is.


import sys
import os

def lines_in_file(f):
    return len(open(f).readlines())

files = os.listdir(sys.argv[1])
totalcount = 0
for f in files:
    totalcount += lines_in_file(sys.argv[1] + f)

print totalcount


Why would this be helpful?

Well, it may or may not to you, but maybe you're just looking for a python example on how to get a line count of a file:
def lines_in_file(f):
    return len(open(f).readlines())

Or maybe, you're just looking for a python example to list files in a directory:
files = os.listdir(sys.argv[1])
for f in files:
    print f ## modified from the original script, to show you an example


How I Use It In Case You Were Curious...

So, we have this one cron job that parses through a directory that contains CSV files. So essentially, each line is a python run. Or I can also think of it as, X number of accounts are participating in Job Y.

That's it! Cheers!

Printing a Specific Range With sed

Issue

You want to print only a specific range of lines from a file/input.

Solution

Use sed! Say for example you want to print out lines 15 through 30, you can do that like so:
sed -n '15,30p' [file]

Real World Scenario

Okay, so here's how I actually ended up using it; I have a process to publish files across hundreds of sites. I have a PHP script that prints out site information (one site per line). I also have another PHP script that handles publishing that takes in the site name as an argument. See where I'm going here? ;)

So what I need to do is loosely pseudo-coded as follows:

for site in siteinfo.php
publish.php $site


Where does the sed command come in handy? Well, like I said I have hundreds of sites to publish to, so a sleep comes in handy to give the servers a little break. So, I ended up breaking it up as follows:
for s in `/usr/local/bin/php sites.php|sed -n '1,100p'`; do /usr/local/bin/php publish.php $s; done | tee -a $log;
sleep 300;

for s in `/usr/local/bin/php sites.php|sed -n '101,200p'`; do /usr/local/bin/php publish.php $s; done | tee -a $log;
sleep 300;

etc...
etc...


Quick Bash One-Liner

Well, I'm here trying to wrap up things for the day and just found a very useful technique to get a directory without the filename from the listing if using find.

So, I'm sure you were all thinking, "well why don't you just use `find` with `-type d`?" Well, for this specific task I will be using `find` to get a list of all files and then I need whatever that directory is for another script. So booyah, in your face! (hehe jk)

So, apparently if I have a directory structure string with a file at the end of it I can drop the end part like so:

f1=/mnt/www/tmpdrin/dir1/dir2/fileindir.txt
f2=${f1%/*};
## $f2 will now be /mnt/www/tmpdrin/dir1/dir2


Sweet as!


And if you were curious how I was using it as a one-liner, here ya go:

for f in `find /mnt/www/tmpdrin/ -type f`; do f2=`echo "$f" | sed -e 's!/mnt/!!g'`; f3=${f2%/*}; echo "$f3"; done


Obviously, I'm doing a lot more here than just echoing, but this is just an example mmmkay? ;)

Cheers!

Post Script: It's been a while for posts, but I've just been working on proprietary stuff lately and nothing would be too useful for ya'll. Sorry!

Getting Unique Entries From Two Files

Okay, so you have data in one file and then you have updated data in another file. Let's say you have a list of directories that have been searched and they all have a filename, "0.txt". Another file lists all of the directories that have this "0.txt" file but they are actually empty. If you diff the files that will only give you changes being made and won't necessarily give you the entries NOT in the updated file. But, if you get all of the directory entries that are NOT in the new list, then you have a complete list of directories that have a valid file!

*(Well, that was just my example, my real life scenario involved MySQL data.)

Anyhow, with this PHP code you can basically go through both files and utilize in_array to see if certain entries exist in one file or not.

**Note: I wanted to use $argv to handle the arguments, but for some reason file didn't like it. Curious...

$f1 = "/tmp/original.txt";
$f2 = "/tmp/modified.txt";

$a = file($f1);
$b = file($f2);

foreach ($a as $k => $v) {
if (!in_array($v, $b)) {
print "$v";
}
}


Cheers!

Processing a Mailbox With POP and Perl

My company sends out newsletters. Of course, for whatever reason sometimes a user no longer wishes to receive these newsletters; and sometimes a user will continue to receive newsletters and instead of using the "unsubscribe" links in the newsletter, they will just mark the email as junk. So, then their ISP, email provider, etc. will then do their job to send us an email saying that there has been a complaint about us spamming. The routine continues and the emails go round and round.

Here is where our "Auto Unsubscribe" mailbox comes to play. We have a mailbox setup where email providers forward emails of complaints. They pretty much always keep the original email message intact, so the auto unsubscribe job is easy: search through the mailbox for the data you need; the unsubscriber's email address and the newsletter.

Finding the correct data is simple on our end as long as the original newsletter email is intact. At the bottom of all of our newsletters is a link to an unsubscribe page. Now, in plain text this is obviously just going to be a line with a URI and parameters. For example, http://www.server.com/unsubscribe.htm?newsletter=ABC&e=youremail@mail.com

You see where I'm going here? You've got all the data you need and now you can do what you want with it.

Cheers!

NOTE: I'm using the Mail::POP3Client module because I personally needed to use SSL when connecting to the mailbox. You can easily use Net::POP3, Net::IMAP, etc. just the same.


#!/usr/bin/perl
#
# unsub-pop.pl v1.0
# Adrian J. Cruz
# 2009-02-19

use strict;
use Mail::POP3Client;

my $mail_server = qw(mail.server.com);
my $mail_user = qw(user@server.com);
my $mail_pass = qw(pass123);
my $pop = new Mail::POP3Client( USER => $mail_user,
PASSWORD => $mail_pass,
HOST => $mail_server,
USESSL => "true");
$pop->Connect
or die "couldn't connect: $!\n";

# find newsletter and email
my @ecunsubs;
for (my $i = 1; $i <= $pop->Count(); $i++) {
foreach ($pop->Body($i)) {
if (/unsubscribe\.htm\?newsletter\=(.+\&e\=.+)\"/) {
push(@ecunsubs,$1);
}
}
# mark msg to be deleted
$pop->Delete($i);
}
$pop->Close;

# remove duplicates so now we process only unique entries
my %sorthash;
@sorthash{@ecunsubs} = ();
my @unsubs = keys %sorthash;
...
...
etc.
...


Monitoring the MySQL ERR Log

If you've ever wanted a quick and easy way to monitor MySQL for errors, well, you've stumbled onto the right place.

At work we've been getting these corrupt tables that go unnoticed until an end-user notifies us. Which, of course, for any website is utterly disastrous since some part of your site (even if it is just a minuscule widget on the site) is now broken. So, being the proactive technologists that we are we decided to monitor the MySQL err log daily. You can find your err log in the "data" directory where you installed MySQL (it defaults to the name: hostname.err).

What this perl script does is parse through the err log and find today's ERRORs. After it finds the errors, it then emails them. So, obviously go through and change the variables to your likings and set up a cron to have this run at the end of the day.

Cheers!


#!/usr/bin/perl -w
#
# chkmysqlerrlog.pl v1.0
# 2009-02-17
# Adrian J. Cruz
#

use strict;

my $dbserver = "dbserver.domain.com";
my $email_to = "email\@domain.com";
my $email_from = "mysqlerr\@domain.com";
my $errlog = "/usr/local/mysql/data/dbserver.domain.com.err";
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) =
localtime(time);
my $yy = sprintf("%02d", $year % 100);
$year += 1900;
$mon += 1;
if ($mon < 10) {
$mon = 0 . $mon;
}
# set $today in date format: YYMMDD to get today's errors
# e.g.:
# 090217 18:38:22 [ERROR] mysqld: Can't open file: 'ts.MYI' (errno: 144)
my $today = $yy . $mon . $mday;
my $todaylog = $year . $mon . $mday . $hour . $min;

# parse through the mysql err log for ERRORs
my @errors;
open(IN, $errlog);
while () {
chomp;
if ($_ =~ m/$today.+(\[ERROR\].+$)/) {
push(@errors, $1);
}
}
close(IN);

# we only want the unique errors
my %sorthash;
@sorthash{@errors} = ();
my @error_msgs = keys %sorthash;

my $email_message = "$todaylog\n";
for my $e (@error_msgs) {
$email_message .= $e . "\n";
}

if ($email_message && @error_msgs) {
sendMail( $email_to, $email_from,
"$todaylog $dbserver ERRORS", $email_message );
}

sub sendMail {
my ($to, $from, $subject, $message) = @_;
my $sendmail = "/usr/sbin/sendmail";
open(MAIL, "|$sendmail -oi -t");
print MAIL "From: $from\n";
print MAIL "To: $to\n";
print MAIL "Subject: $subject\n\n";
print MAIL "$message\n";
close(MAIL);
}


NOTE: with a little bit of work you can even include the Warnings; we weren't too worried about those.

Rename Multiple Files With Perl

So here's the problem: I have a directory full of files that have special characters. In this particular case they were spaces and pounds. So of course, for some reason I can't find a working way to do a quick bash script. I was trying to do a for x in `ls` ; do ... but because of the special characters my filenames weren't being read properly. Wonderful!

So, after maybe a good 20 minutes of trying to find the right shell commands to use, I gave up and wrote this quick perl script.

So...anybody wanna give me a hint on how I'd be able to do this in a shell script?


#!/bin/perl
#
$dir="/home/drin/tmp/";
opendir(DIR, $dir) || die "Couldn't open $dir";
@files = grep {/.doc/ && -f "$dir/$_"} readdir(DIR);
closedir(DIR);

for $x (@files) {
$y = $x;
$y =~ s/\#//g;
$y =~ s/\ /_/g;
print "Renaming $x to $y\n";
rename ($x, $y);
}


Cheers!

Quick Perl for Bash Boolean Woes

My task is simple; delete files that are in one data center's SAN but not in any of the others. Though, to be sure that the files really do not exist in the other data centers, I want to do a little logic to only delete the file if they do not exist in any of the data centers.

So, maybe my brain isn't working properly or I just knew that perl would've been quicker, but I couldn't find the right way to use the -a (AND) bash boolean operator. If anyone knows how I could use bash's booleans to test if a file does not exist in location a, b, c, ..., n+1, then I would be more than curious to know!

This did NOT work: if [ ! -f $x -a ! -f $y -a ! -f $z ] ; then ...

I ended up doing the following in perl fyi:


#!/usr/bin/perl -w

open(FILE, "files_to_del.txt") || die("Could NOT open!");
@cfdata=;
close(FILE);

$ny1="/mnt/ny1/www/";
$ny2="/mnt/ny2/www/";
$sj1="/mnt/sj1/www/";
$tx1="/mnt/tx1/www/";

$count=0;
foreach $filepath (@cfdata) {
chomp($filepath);
$ny1file=$ny1 . $filepath;
$ny2file=$ny2 . $filepath;
$sj1file=$sj1 . $filepath;
$tx1file=$tx1 . $filepath;
if (! -e $ny1file && ! -e $sj1file && ! -e $tx1file) {
$count++;
print "DELETING: $count $ny2file\n";
unlink($ny2file);
}
}


Cheers!

Winmail.dat Attachments For Linux Thunderbird

This is only going to be a rant and a link to Andrew Beacock's blog post solving that pesky winmail.dat attachment issue for linux users.

I never had this issue until this morning when a project manager sent over an email with an attachment.

Sheesh Microsoft, can we stop making the proprietary apps please!?

Using a Script to Edit a conf File

Well, because I have been uber busy here at work, I haven't been able to write any blog posts. But, since I've been super productive at work, I get to write up a few posts about some more tools I've written to add to my arsenal. w00t!

So my latest task involved editing some of our web server's Apache confs. It consisted of adding a new alias so that we can call in some new app someone else wrote. Okay cool. Alias? Sure not a problem. Er...wait! The alias requires the site's directory name as part of the path! Oh no! That means I can't create one global alias!

Ah, see that's when I thought of writing this script that will look for a specific line in the vhost entries and will enter the new line after it. Pretty nifty eh? :)

So basically, what we have here is a list of all sites in a text file. Now we can use the site's directory name read in as a variable. It will then use awk to search for a specific line, and then print the new line you need right below it!

So if you take a look at the code, I want to add this line:
"Alias /newalias /mnt/www/$site" (where $site is the site directory name variable)
right after each line that fits the regex that searches for any line that has "Alias" and "reports" in one line followed by the site dir name at the end of the line. Obviously if you want to utilize this script you should edit the regex to your needs.

And that's it! Now think of all of the time saved rather than adding a new line manually to EACH vhost entry on EACH server! whoa....scary...


#!/bin/bash

exec < sites.txt
while read site ; do
echo "processing $site..."
awk '{print $0} '"/Alias.+reports.+\/$site"$/'{print " Alias /newalias /mnt/www/'"$site"'"}' httpd.conf > newhttpd.conf && mv newhttpd.conf httpd.conf
echo "done!"
done


Shell Scripting to Read Data From a File

Okay, so here's your problem: you have a file that contains data that you'd like to parse through a script. It's a lot of data and you don't want to sit there and continuously do the same thing over and over!

The data can be set up as one long column of data or multiple columns of data. In this case for simplicity I'll only use one column of data.

So let's say your data file is as follows:

dog
cat
cow
...
...
(long list of data)


These represent parts of filenames that you need to clear out. You don't want to sit there typing cat /dev/null > dog
cat /dev/null > cat
... So you come up with a master plan to script the process. The script will be short, sweet, and absolutely time saving!

You first need to read in the file; so this line does the work for you: exec < files.txt. Obviously, in this example my "files.txt" is the filename that contains the data I want to read. After you've opened up the file for reading, you can now parse through the file with a while loop:
while read file ; do
[do some work here]
done


Now note, "file" is the variable that we're assigning to the column. You can call the variable whatever you want. Also remember that I am being simple with this example and only have one column of data. If I wanted to do multiple columns of data, I would just assign more variables to each column. e.g. while read col01 col02 col03 ; do where $col01, $col02, and $col03 will be my new variables.

That's it! I'll leave the customization to you. Cheers!

Here is the full script example:
#!/usr/bin/bash

exec < files.txt

while read file ; do
if [ -s /mnt/$file ] ; then
cat /dev/null > /mnt/www/vdir/z/ec/$file
else
echo "/mnt/$file already empty!"
fi
done