A Sendmail replacement in Python

I know I blog about odd things, but it's just that those odd things happen to me! The problem we have now is the following: we need to have a sendmail-like command that will send email through an external SMTP server. Why? Well its a long story that can be resumed to the fact that I just don't want to install Sendmail into my VoIP server ;).

So instead of doing what everyone would do (apt-get install sendmail) I wrote a simple script in Python that works somewhat like Sendmail does, with the exception that it just forwards everything to another server. I *could* make it work without an external server, but that would require to resolve the DNS of the destination server and to find the MX registers... and I'm lazy. Plus that would require an external Python Module to do the DNS resolution, and I wanted my script to be pure-Python with no dependencies other than the Python library. Anyway if anyone, ever, needs a script to do the DNS thing, ask and I'll do it *sigh*.

So here comes the script. The script accepts the sendmail -t parameter so the "To" header from the Email is used to specify the recipients, otherwise recipients should be specified by command line.

#!/usr/bin/python

import email
import smtplib
import sys

SMTP_SERVER="YourMailServer.com"
SMTP_USER="YourUser"
SMTP_PASS="Ssh-Sekrat!"

def sendmail(from_addr, to_addrs, msg):
    smtp = smtplib.SMTP(SMTP_SERVER)
    smtp.login(SMTP_USER, SMTP_PASS)
    smtp.sendmail(from_addr, to_addrs, msg)
    smtp.quit()

def main():
    msgstr = sys.stdin.read()

    msg = email.message_from_string(msgstr)
    if sys.argv[1] == "-t":
        to_addrs = msg["To"]
    else:
        to_addrs = sys.argv[1:]
    sendmail(msg["From"], to_addrs, msgstr)

main()


Thats it! To test it:

./sendmail.py someone@somewhere.com
From: your@friend.com
Subject: Hello!

Hello! Hows life?
<CTRL+D>

I hope you liked it, happy hacking!

The Holy Grail of Binary Data in PostgreSQL and Python

I'm not sure if you've ever had this problem. The thing is
that I've had it a lot of times and I suppose I'm not alone. The
thing is the following, say you want to store binary data in
PostgreSQL. For those who use MySQL, I've heard it's a very
simple task, however I don't care since I never use MySQL :). On
the other hand, to do this from Python to a PostgreSQL database,
can be quite tricky. What I used to do is to Base64 encode
everything and store it on the database, that worked but it was
slow and bloated. There's another way, the "bytea" data type. The
problem with bytea is that PostgreSQL wants things escaped and it
is quite unclear how to do it. However, today I found it out and
I was so happy I decided to blog it! It is actually simple once
you know the trick. First you have to create a table with a bytea
field (where you store your bytes), for example:

CREATE TABLE IMAGES (ID SERIAL, NAME VARCHAR, DATA BYTEA);

This is the easy part and I think we all have gotten this far.
Now the thing is in the Python side. The secret is to escape
things properly. To insert binary data to a PostgreSQL database
you need to follow this scheme: E' + data in octal + '::bytea For
example:

INSERT INTO IMAGES VALUES(default, 'PARIS.JPG', E'\\001\\002\\003\\031\\313'::bytea);

Notice the double "\" since the bar is un-escaped by the
database. Easy? Yeah! But hard to find out. Now the thing is just
to write a simple Python function to do the escaping:

def octize(data):
    out = "E'"
    for char in data:
        octdata = oct(ord(char))[1:].zfill(3)
        out += "\\\\" + octdata
     return out + "'::bytea"

And now we could do something like:

conn = psycopg.connect()
cur = conn.cursor()
cur.insert("INSERT INTO IMAGES VALUES(default, 'PARIS.JPG', %s)" % octize(image_data))
cur.close()
conn.commit()

Hope you liked it! Happy hacking!

Website statistics: Roll your own real-time Google Analytics in 5 minutes!

Hello my fellow hackers! I want to talk about something we all freak out about at least once in life: website statistics.

Website statistics have been a headache for webmasters since Internet became Inter-net. It is fundamental to know how many people see your site every day, what they do there, from where they come, what OS and browser they are using, etc. This is so for a lot of reasons, for optimizing your site to get traffic, for bandwidth optimization and (in my case) just to watch endlessly, frenetically and insanely how the visit counter crawls up ;).

Though in the early beginnings of Internet these solutions weren't so easily available, in this ever-changing, over-connected and wonderful world there is a huge number of options for us to choose. The most popular and widely used (IMO) is Google Analytics.

So why bother making our own then? I won't say I don't like Google Analytics because I really like it, I think its a wonderful solution and a must-have tool for any webmaster. However there is something I personally don't like: the fact it is not real-time. Although this is perfectly justified (they must index and analyze millions of sites and real-timeness is difficult to achieve at such scales), I still wanna see my lil' counter crawl up!!!

The question is, do YOU want to see your lil counter crawl? From a practical point of view, I'd say probably not, since usually having the information available by the next day is already enough to do all your work. But actually I think you probably do. Why? Well, I guess we techies are just like that :) I decided therefore to make my own lil' analytics and set it up in my blog today. So lets get to it.

First of all I'd like to start with how statistics work and how do we get them. As most of you already know, in the ancient times only log statistics were available. That is, you'd just walk through your http server's log and you would build statistics from it. That was great since it was simple and totally passive (no modification needed to the site) and you could see it real-time (a tail -f access.log would show it all), the cons: some things you may find interesting about the users such as the resolution of their screens were not logged in the server and hence out of your stats.

Now how Google Analytics work. You basically embed a little piece of Javascript code in every page in your site you want to analyze and you are set up. Then all you need to do is to log into your Google account and you have your stats there. The con: as I said, it's usually updated after 24 or 48h. And how do they do it? Well basically, this little script makes your browser point to one of their servers while the site is loading, posting all the necessary information to the stats collector.

So lets do the same!

First we will start with the Javascript code, you can save this as stats.js:

data = [ document.referrer,
navigator.userAgent,
screen.width + "x" + screen.height,
screen.colorDepth]
query ="";
for (val in data) {
query+=data[val] + "&";
}
img = document.createElement("img");
img.setAttribute("src", "http://yoursite.com/analyz0r.gif|" + query);

Ok this little monster does all the magic. To embed this in your site you'd simply:

<script type="text/javascript" src="stats.js"></script>

What will this do? It will simply make the browser try to load an image called analyz0r.gif in your server, sending all the information we want about the client. The image can just be missing, we don't really care. We are combining here log analysis with Javascript. We will get something like this in the server's log:

exe@melange:~/workz/stats$ cat /var/log/apache2/access.log
127.0.0.1 - - [23/Dec/2008:01:21:29 +0100] "GET /analyz0r.gif|http://localhost/&Mozilla/5.0%20(X11;%20U;%20Linux%20i686;%20en-US;%20rv:1.9.0.5)%20Gecko/2008121622%20Ubuntu/8.10%20(intrepid)%20Firefox/3.0.5&1680x1050&24& HTTP/1.1" 404 334 "http://localhost/" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.5) Gecko/2008121622 Ubuntu/8.10 (intrepid) Firefox/3.0.5"

Although it looks ugly, this is easily parseable into meaningful data. We can for example use this simple script to export it to CSV and then you can load it with your preferred spreadsheet.

exe@melange:~/workz/stats$ cat /var/log/apache2/access.log |grep analyz0r| sed s/%20/" "/g|cut -d"|" -f2 |cut -d"&" -f1-4
http://localhost/&Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.5) Gecko/2008121622 Ubuntu/8.10 (intrepid) Firefox/3.0.5&1680x1050&24

The format is simple: referrer & user agent & resolution & depth. You can modify the script to add all the fields you would like to.

And it works with other browsers too! Look:

http://localhost/&Opera/9.61 (X11; Linux i686; U; en) Presto/2.1.1&1680x1050&24

Happy analyzing!

From Illegal Immigrant to IT Entrepreneur - The beginning

I don't know if this has ever happened to you, but haven't you ever stopped and said "If this ever turns out right, I will write a book!"?

In my case, I have said it a lot of times. And I thought, maybe, in this Web 2.0 (or 3.0? or 25.2?) world, and since I call myself a programmer and a geek, maybe I shouldn't write a book but, instead, a blog.

In this space I want to speak openly and frankly. I may say things some may not like, please don't take it like that. I don't intend to hurt anyone, it's just my personal point of view, my experiences and the story of a short life that's just beginning and yet has a lot to say. And special thanks to my wife Olya and my friend Dan for helping me with the English redaction!

The title is true, mostly. I came to this country 6 years ago and I was an illegal immigrant. I spent 3 months eating spaghetti since it was the only thing we could afford. I had no bed (slept on the floor and then on the sofa for weeks). I had to teach Chemistry to pay for food (and I don't know Chemistry), if I ever needed to take a train I had to do inspector-surfing (got caught once), and the list goes on... Right now I lead a team of 15 programmers, designers and electronic engineers, we are developing an innovative solution (I will talk about it later), in my team there are people of all the world and all specialties, from (dark) black hat hackers to computer scientists. And I may go back to eating spaghetti at any moment :).

I've actually never written a blog, I don't have the patience nor the perseverance required to do it. I also tend to switch quickly from one subject to another and I quickly lose interest in almost everything, for example my fractals: http://thepeak.deviantart.com (I may keep posting but they suck anyway :). In spite of all this, I feel I finally have something interesting to tell the World, something I would read myself if I had the chance, and since I'm far from being unique, I hope you'll like it too.

My story isn't over yet, maybe that's the interesting part, if things turn out right or wrong, I will be writing them just as they happen. That's more fun, to say things how I felt them at that precise moment. My story isn't the most interesting nor heroic, and it's far from being the best, but it's a story after all.

So, let's get to it. My name is Martin E. Zulliger. The "E" of my name is "Exequiel" which translates basically to Ezekiel in English as you probably guessed. My family and friends have always called me "Exequiel", but I usually sign "Martin" since it's easier for most people and sounds more international :). I was born in Argentina (yes that's in South America, you know, Tango and football, that's all they know us for), to a Swiss-Spanish-Italian-Dunno-What-Else family. I currently live in Madrid, Spain. I am 24 years old and I am a geek. I spend almost all my time at the computer. I love programming in Python, Java and C++ and I love starting odd/really hard projects which I rarely ever finish, they mostly end on the Huge Pile of Unfinished Stuff.

My story begins 6 years ago, by then I was starting to study Computer Engineering at University of Mendoza, in my home town Mendoza. By then Argentina was facing a lot of economical problems and basically major mess (something sadly usual in my beloved country). My family had a "business" there, a school. I say "business" since my mother has a huge heart and hence most of the students there were granted full or partial scholarships, in other words, nobody paid the tuition.

Most private schools in Argentina (as in many other places) receive funding from the government, which is essential for their survival, and it was also essential for my mom's school. The problem was that she didn't get any government funding, nor any funding whatsoever. I am not an economist but that situation and the fact that most people going there didn't pay a dime either made the business highly unprofitable.

She wasn't doing it for the money as you can guess, actually, she was doing it for us. She is an educator and she wanted us to have the best possible education. Instead of just home-schooling us, she decided to build a school out of nothing and with no money. It naturally ended as it should obviously. But it lasted more than 10 years and most of the guys who studied there still long for the years at our St. Bernard's College.

As the years went by and the school started accumulating a huge debt and a lot of enemies. The debt was due to the poor economical situation of the country and especially Mendoza's one. If the school was done in Buenos Aires (the capital) the story could have been really different. The enemies due to, well, I have no idea why. My parents only wanted to do things right, to help everyone, as idealistic as it sounds. But for some reason the government, specially the education ministry didn't like it at all. Maybe it was envy, maybe it was misunderstanding, maybe hypocrisy or ignorance, I guess I will never know, but it doesn't matter anymore. Thanks to them I am here, happily married and doing cool things, I owe them a lot.

Only the sheer work and sacrifice of my parents kept it working so long. My dad had to work overtime to get enough money to pay for the school's urgent debts. Looking back from now, I don't understand how they made it. But all the good things come to an end and so did the St. Bernard's. When it crushed, the debt was so huge we had to sell everything, including the college itself to pay for the debt. After selling it all we had nothing, just plain nothing. We had sold our house to keep the college working so we didn't even have a place to stay.

In the middle of desperation, rage and impotence my parents took a hard decision, to leave their homeland. So we had to pick a country to move to. I would call this moment of my life the archaeological rush. Yes, archaeological rush. I know many of you have been in this situation so you get me, for those who haven't let me explain.

My gramma was born to Spanish parents in Argentina. She was raised as a Spanish rather than as Argentinean and she always wanted to see her parents land. My gramma raised my mother as a Spanish immigrant and helped my mom a lot with the school and taking care of us. So I was somehow a bit of a spanish immigrant in my own country, but I was happily mostly unaware. Due to all this, the obvious choice was Spain. However though my gramma had Spanish citizenship, neither my parents neither us had it due to some obscure and odd fact that Spanish citizenship couldn't be transferred by the mother but only by the father unless you were born after 1980 or something like that. The thing is that we were screwed, and afraid to go there as illegals since we heard lots of creepy stories of how people were hunted by the police and sent back ruthlessly to their countries, that you would never find a job as an illegal immigrant and that all you would be able to do is to clean bathrooms. I later found out isn't actually like that, but it's not easy at all anyway, I will have time to explain later.

The thing is my grampa (mom's side) was born in an Italian family, though his parents were not born in Italy, Italian laws apparently allowed him to recover the citizenship and then pass it on to my mom. As Italy is part of the EU, well, we would be citizens as well in Spain, with no need to do any special paper nor permit. All we needed was the birth certificate of some guy "Viadana" who was supposed to be my Italian great-great-grandfather. And there is where you can start to hear Indiana Jones theme. Oh yes, we went from one civil registry to another, from one embassy to another, consulates, public offices, we went even to the graveyard (several times) to find out about this strange figure (may he rest in peace) of my past. All this just with the pathetic hope to find a remote link that would give us the Italian citizenship and hence open Europe's doors. But Europe's doors were apparently closed very tightly for us.

But we were not going to let Old Continent slip away. So we tried something else. As you see my lastname is of Swiss origin. So we though, thats it! Switzerland is more permisive with it's people and just being a Zulliger may give us chance to get hold if the European dream! And what comes next somehow this makes me think of Tomb Raider. We needed the birth certificates of my Swiss family to go and talk with the consulate. Unlike the Italian side, the Swiss side did have all the papers required to initiate the process, however, my grampa (from dad's side this time) didn't really like the idea of us going anywhere, so well, let's say we were totally unable to get these papers and there was no other way to recover them anytime soon. In the meantime we visited the Swiss tombs as well, just for fun.

Our perseverance knew no limits, and we went on trying, each time something more pathetic and desperate. But cool and odd things kept happening! The next thing was to try to find fellow Zulligers over the would who would help us. Yes, just as it sounds, pathetic, but survival was at stake. I don't want my distant relatives to feel offended by this, but we naturally got no reply, I don't know if they died laughing or just freaked out, in any case we were just as we began. And this is when something funny happened. After the failure of the Zulliger project, the last thing was to write randomly over the Internet with my dad's curriculum to European companies with the hope some would reply. One did. RAD, in Israel. They were offering my dad work and a house for us as soon as we reached Israel. We freaked out a lot, and I think we will never understand how they contacted us since we didn't write them but going to Israel was like hard for us, especially not knowing the language.

After a lot of frustration and really hard work we concluded the only way to make it was to take a risk, my dad had to go to Spain with nothing more than a couple Euros. It was then or never, we had to try or die and, like Julius Caesar said, Alea iacta est!

I have tried to make it as short as possible, sorry if I have diverged too much during my story. The next episode of my story will be: From Illegal Immigrant to IT Entrepreneur - Starting From Scratch.

Thanks a lot and stay tuned!!!