Tuesday, February 24, 2009

Processing a Mailbox With POP and Perl

My company sends out newsletters. Of course, for whatever reason sometimes a user no longer wishes to receive these newsletters; and sometimes a user will continue to receive newsletters and instead of using the "unsubscribe" links in the newsletter, they will just mark the email as junk. So, then their ISP, email provider, etc. will then do their job to send us an email saying that there has been a complaint about us spamming. The routine continues and the emails go round and round.

Here is where our "Auto Unsubscribe" mailbox comes to play. We have a mailbox setup where email providers forward emails of complaints. They pretty much always keep the original email message intact, so the auto unsubscribe job is easy: search through the mailbox for the data you need; the unsubscriber's email address and the newsletter.

Finding the correct data is simple on our end as long as the original newsletter email is intact. At the bottom of all of our newsletters is a link to an unsubscribe page. Now, in plain text this is obviously just going to be a line with a URI and parameters. For example, http://www.server.com/unsubscribe.htm?newsletter=ABC&e=youremail@mail.com

You see where I'm going here? You've got all the data you need and now you can do what you want with it.

Cheers!

NOTE: I'm using the Mail::POP3Client module because I personally needed to use SSL when connecting to the mailbox. You can easily use Net::POP3, Net::IMAP, etc. just the same.


#!/usr/bin/perl
#
# unsub-pop.pl v1.0
# Adrian J. Cruz
# 2009-02-19

use strict;
use Mail::POP3Client;

my $mail_server = qw(mail.server.com);
my $mail_user = qw(user@server.com);
my $mail_pass = qw(pass123);
my $pop = new Mail::POP3Client( USER => $mail_user,
PASSWORD => $mail_pass,
HOST => $mail_server,
USESSL => "true");
$pop->Connect
or die "couldn't connect: $!\n";

# find newsletter and email
my @ecunsubs;
for (my $i = 1; $i <= $pop->Count(); $i++) {
foreach ($pop->Body($i)) {
if (/unsubscribe\.htm\?newsletter\=(.+\&e\=.+)\"/) {
push(@ecunsubs,$1);
}
}
# mark msg to be deleted
$pop->Delete($i);
}
$pop->Close;

# remove duplicates so now we process only unique entries
my %sorthash;
@sorthash{@ecunsubs} = ();
my @unsubs = keys %sorthash;
...
...
etc.
...

Wednesday, February 18, 2009

Monitoring the MySQL ERR Log

If you've ever wanted a quick and easy way to monitor MySQL for errors, well, you've stumbled onto the right place.

At work we've been getting these corrupt tables that go unnoticed until an end-user notifies us. Which, of course, for any website is utterly disastrous since some part of your site (even if it is just a minuscule widget on the site) is now broken. So, being the proactive technologists that we are we decided to monitor the MySQL err log daily. You can find your err log in the "data" directory where you installed MySQL (it defaults to the name: hostname.err).

What this perl script does is parse through the err log and find today's ERRORs. After it finds the errors, it then emails them. So, obviously go through and change the variables to your likings and set up a cron to have this run at the end of the day.

Cheers!


#!/usr/bin/perl -w
#
# chkmysqlerrlog.pl v1.0
# 2009-02-17
# Adrian J. Cruz
#

use strict;

my $dbserver = "dbserver.domain.com";
my $email_to = "email\@domain.com";
my $email_from = "mysqlerr\@domain.com";
my $errlog = "/usr/local/mysql/data/dbserver.domain.com.err";
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) =
localtime(time);
my $yy = sprintf("%02d", $year % 100);
$year += 1900;
$mon += 1;
if ($mon < 10) {
$mon = 0 . $mon;
}
# set $today in date format: YYMMDD to get today's errors
# e.g.:
# 090217 18:38:22 [ERROR] mysqld: Can't open file: 'ts.MYI' (errno: 144)
my $today = $yy . $mon . $mday;
my $todaylog = $year . $mon . $mday . $hour . $min;

# parse through the mysql err log for ERRORs
my @errors;
open(IN, $errlog);
while () {
chomp;
if ($_ =~ m/$today.+(\[ERROR\].+$)/) {
push(@errors, $1);
}
}
close(IN);

# we only want the unique errors
my %sorthash;
@sorthash{@errors} = ();
my @error_msgs = keys %sorthash;

my $email_message = "$todaylog\n";
for my $e (@error_msgs) {
$email_message .= $e . "\n";
}

if ($email_message && @error_msgs) {
sendMail( $email_to, $email_from,
"$todaylog $dbserver ERRORS", $email_message );
}

sub sendMail {
my ($to, $from, $subject, $message) = @_;
my $sendmail = "/usr/sbin/sendmail";
open(MAIL, "|$sendmail -oi -t");
print MAIL "From: $from\n";
print MAIL "To: $to\n";
print MAIL "Subject: $subject\n\n";
print MAIL "$message\n";
close(MAIL);
}


NOTE: with a little bit of work you can even include the Warnings; we weren't too worried about those.

Tuesday, February 17, 2009

Don't Panic Filesystem, It'll Be Okay

"Ouch! My G1 isn't reading my microsd card! Nooooo! I need to listen to something on the train! Oh wait, let me just use iMeem for now."

That's not a good way to start off the first day back from a long weekend. But unfortunately, that's how I started it off. So now, as I'm finally having time to sit down and troubleshoot why my microsd card was not reading, I can tell you all about how I fixed it!

So, I plug in my G1 to my laptop here at work. Here's my dmesg output:

acruz@acruz:~$ dmesg | tail
[28938.531732] sd 6:0:0:0: [sdc] Attached SCSI removable disk
[28938.531824] sd 6:0:0:0: Attached scsi generic sg3 type 0
[28938.969366] FAT: Filesystem panic (dev sdc1)
[28938.969387] fat_get_cluster: invalid cluster chain (i_pos 0)
[28938.969392] File system has been set read-only
[28941.483468] FAT: Filesystem panic (dev sdc1)
[28941.483493] fat_get_cluster: invalid cluster chain (i_pos 0)
[29038.322352] FAT: Filesystem panic (dev sdc1)
[29038.322365] fat_get_cluster: invalid cluster chain (i_pos 0)
[29038.322372] File system has been set read-only


Whoa...a panic is never any good in linux. Hmmm...all of my data is intact. Okay, no problem then. All I need to run is a quick fsck.vfat -a.


acruz@acruz:~$ sudo fsck.vfat -a /dev/sdc1
dosfsck 2.11, 12 Mar 2005, FAT32, LFN
FATs differ but appear to be intact. Using first FAT.
/albumthumbs
Contains a free cluster (4). Assuming EOF.
/music/P/mp3/Pimsleur01.mp3
Contains a free cluster (43543). Assuming EOF.
Reclaimed 1552 unused clusters (50855936 bytes) in 17 chains.
Performing changes.
/dev/sdc1: 1105 files, 127709/244416 clusters


Well, seems like I had a few files that were probably corrupted when I transferred them last. Sheesh! Well, at least now fsck fixed it! Now, all I needed to do was umount/mount and then all looked better again!

Here's the healthy dmesg output:

acruz@acruz:~$ dmesg | tail
[30213.598301] sd 7:0:0:0: Attached scsi generic sg3 type 0
[30222.667142] sd 7:0:0:0: [sdc] 15659008 512-byte hardware sectors (8017 MB)
[30222.669098] sd 7:0:0:0: [sdc] Write Protect is off
[30222.669107] sd 7:0:0:0: [sdc] Mode Sense: 03 00 00 00
[30222.669112] sd 7:0:0:0: [sdc] Assuming drive cache: write through
[30222.673118] sd 7:0:0:0: [sdc] 15659008 512-byte hardware sectors (8017 MB)
[30222.675136] sd 7:0:0:0: [sdc] Write Protect is off
[30222.675147] sd 7:0:0:0: [sdc] Mode Sense: 03 00 00 00
[30222.675151] sd 7:0:0:0: [sdc] Assuming drive cache: write through
[30222.675164] sdc: sdc1


That looks much better! w00t!

Cheers!

Android Phone 2: HTC Magic

Finally, a second Android-based phone! You can read (and view photos!) CNet UK's first look at the phone here.

Well, right now I'm just curious if the battery life is better than the HTC Dream (G1). Other than that, well I could care less for a touchscreen-only phone. (One of the main reasons I didn't get an iPhone.) Other than that, the HTC Magic looks pretty slick and seems to be a nice addition to the Android family.

Cheers!

Monday, February 16, 2009

Rename Multiple Files With Perl

So here's the problem: I have a directory full of files that have special characters. In this particular case they were spaces and pounds. So of course, for some reason I can't find a working way to do a quick bash script. I was trying to do a for x in `ls` ; do ... but because of the special characters my filenames weren't being read properly. Wonderful!

So, after maybe a good 20 minutes of trying to find the right shell commands to use, I gave up and wrote this quick perl script.

So...anybody wanna give me a hint on how I'd be able to do this in a shell script?


#!/bin/perl
#
$dir="/home/drin/tmp/";
opendir(DIR, $dir) || die "Couldn't open $dir";
@files = grep {/.doc/ && -f "$dir/$_"} readdir(DIR);
closedir(DIR);

for $x (@files) {
$y = $x;
$y =~ s/\#//g;
$y =~ s/\ /_/g;
print "Renaming $x to $y\n";
rename ($x, $y);
}


Cheers!

Friday, February 13, 2009

Happy 1234567890 Epoch Time (EST)


"Wait for it... wait for it... come on... getting close... aha! There we go!"

Yay, we had an "epoch" of a time here on the east coast. hehehe

Cheers!

Wednesday, February 11, 2009

Quick Perl for Bash Boolean Woes

My task is simple; delete files that are in one data center's SAN but not in any of the others. Though, to be sure that the files really do not exist in the other data centers, I want to do a little logic to only delete the file if they do not exist in any of the data centers.

So, maybe my brain isn't working properly or I just knew that perl would've been quicker, but I couldn't find the right way to use the -a (AND) bash boolean operator. If anyone knows how I could use bash's booleans to test if a file does not exist in location a, b, c, ..., n+1, then I would be more than curious to know!

This did NOT work: if [ ! -f $x -a ! -f $y -a ! -f $z ] ; then ...

I ended up doing the following in perl fyi:


#!/usr/bin/perl -w

open(FILE, "files_to_del.txt") || die("Could NOT open!");
@cfdata=;
close(FILE);

$ny1="/mnt/ny1/www/";
$ny2="/mnt/ny2/www/";
$sj1="/mnt/sj1/www/";
$tx1="/mnt/tx1/www/";

$count=0;
foreach $filepath (@cfdata) {
chomp($filepath);
$ny1file=$ny1 . $filepath;
$ny2file=$ny2 . $filepath;
$sj1file=$sj1 . $filepath;
$tx1file=$tx1 . $filepath;
if (! -e $ny1file && ! -e $sj1file && ! -e $tx1file) {
$count++;
print "DELETING: $count $ny2file\n";
unlink($ny2file);
}
}


Cheers!

Tuesday, February 3, 2009

Helix3 2008R1

So, I recently was asked to get data off of a laptop for a friend. I took the laptop home with me and noticed that I left my old Helix CD at my parents' house. "Not a big deal, I can just download the latest iso!"

So first thing I noticed was the use of Gnome, but wait, look further, it's actually Ubuntu! Cool! But wait, I just want to recover some data and I don't need Gnome to be running, it costs too much overhead! Bah!

I was not even able to append a boot parameter to boot in console-only mode (or at least I wasn't able to find one). Fine, I got the files off anyway!

I also noticed an "install" option. That's cool! Maybe I'll be more curious and even test it out.

Cheers!

UPDATE
Ok, silly me! To go about booting into console-only mode the answer was really just in my face! If you hit F6 at the Helix menu, you can edit the boot parameters. Just delete splash and that's it! Though, Gnome still loads so you'll just have to do a Ctrl+Alt+[F1-F6] to get to one of those console screens.