Strace to the Rescue!

For some unfathomable reason, a mailing list that I was running, decided to stop working last week. Whenever anyone sent an email to the mailing list, they would get a bounced error and fail. I was forced to reboot the server two weeks ago, but I had not changed any settings since. So, I was rather puzzled as to what was going on.

The first thing I did was to check the mail server logs and the error surprised me:

Jul 2 15:41:23 localhost postfix/pipe[10572]: 70CB7401A4: to=, orig_to=, relay=mlmmj, delay=1.5, delays=0.96/0.02/0/0.49, dsn=5.3.0, status=bounced (Command died with signal 11: "/usr/bin/mlmmj-recieve")

This did not make any sense to me. MLMMJ was segfaulting without any reason. I tried googling around for a solution but there was none. MLMMJ is not a very well documented mailing list manager either. So, I was left to find my own solution to the problem.

The usual causes for a segfault are usually memory related. So, I thought that the server might be running out of RAM. To test this, I stopped running some other services to free up RAM. Then, I sent a test email to the list to see the same segfault error. So, it’s not a case of “lack of memory”. Then, I thought that it might be a file permissions problem so I set the test mailing list to be world read-write-able. This didn’t work either.

So, time to dig deeper into the “mlmmj-receive” programme. Reading up on the man pages, I realised that it was only a front-end software that calls two other back-end programmes to do the dirty work. It calls “mlmmj-process” to process the email and “mlmmj-send” to send the email. So, it was possible that any one of these three things were failing.

The next thing that I tried was to create a blank mailing list with no subscribers and sent a test email to it. Surprisingly, the “mlmmj-receive” completed without much ado. So, this indicated to me that the problem was likely with the “send” stage. But, it got me no closer to an answer as I couldn’t think of any reason why this stage was failing at all as the main SMTP server was running perfectly.

So, it was time to roll up the sleeves and engage STRACE. I have only used it once for debugging purposes and now it was time for me to see what was going on behind the scenes. But to deploy this, I would first need to intercept all the messages being sent to and returned from MLMMJ. So, I wrote a wrapper script “/usr/bin/mlmmj-receive” to wrap around “/usr/bin/mlmmj-recieve” (note the spelling difference).

#!/bin/sh
cat - > /tmp/mlmmj.in
cat /tmp/mlmmj.in | strace -f -o /tmp/mlmmj.log /usr/bin/mlmmj-recieve $@ 1>/tmp/mlmmj.out
cat /tmp/mlmmj.out

This simple script first intercepts all the data piped to MLMMJ and saves it in “/tmp/mlmmj.in”. The next line is the magic that uses strace to trace the actual operations, which dumps all the system calls to “/tmp/mlmmj.log”. Studying this log revealed, the interesting inner workings of MLMMJ and also provided a clue as to what was going wrong.

socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 5
connect(5, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("xx.xx.xxx.xxx")}, 28) = 0
fcntl64(5, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
gettimeofday({1215041316, 932534}, NULL) = 0
poll([{fd=5, events=POLLOUT, revents=POLLOUT}], 1, 0) = 1
send(5, "\v\352\1\1\nlocalhost\t\1\1", 28, 0) = 28
poll([{fd=5, events=POLLIN, revents=POLLIN}], 1, 5000) = 1
ioctl(5, FIONREAD, [28]) = 0
recvfrom(5, "\v\352\205\203\1\nlocalhost\t\1\1", 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("xx.xx.xxx.xxx")}, [16]) = 28
close(5) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++

This bit showed that just before segfaulting, the programme (in this case, mlmmj-send) was polling the DNS server for the IP address of “localhost”. This did not make sense as a computer should not be polling a DNS server for “localhost”, which is defined in “/etc/hosts”. In fact, at an earlier part of the programme, it actually opens up “/etc/hosts” and tries to find “localhost” in it.

Seems like there was a formatting problem with the file which I didn’t realise. But none of the other programmes seemed to have any problems connecting to localhost except MLMMJ. So, I fixed the problem and voila, the mailing list was working again.

Anyway, I now think that STRACE is a god-like tool. It gives me great insight into the inner workings of any software application. This would be very useful for reverse engineering purposes as well as debugging and optimisation tasks. It has an option to output timestamps, including how long each system call takes. I will look into this application more, and learn how to exploit it thoroughly. This tool is one to keep in the tool bag.

Enemy of the State

I love Battlestar Galactica. it has, once again, taught me something useful in life. According to an article in TheStar:

The armed forces will be called in to assist police in maintaining public order if the security situation of the country warrants it.

In light of this comment, I am reminded of the following quote by Commander Adama:

There’s a reason we separate military and the police: o­ne fights the enemy of the state, the other serves and protects the people. When the military becomes both, then the enemies of the state tend to become the people.

I do not want to see the military in our streets. If that ever happens, we are just one step away from Zimbabwe.

Sexy Machine

Man, this is a machine that is aiming straight for my heart! Ignoring for a moment, the blatantly cheesy name “Eee Monitor”, it is still a svelte and sexy machine. Photos of it are just making the rounds around the Internet. It’s not just a monitor, but carries a full blown PC inside it. Plus, it will come standard with Linux for under $500!

This machine is definitely going to give the iMac a run for it’s money. Some people may complain about the fact that these machines are a little underpowered compared to the iMac. That is true as the Eee will only come with an Atom processor, which is the new line of low-powered mobile focused processor chips from Chipzilla. It is a processor that is essentially a Pentium inside the body of a i386. So, it is certainly underpowered for computational fluid dynamics. But it’s more than enough for your everyday applications.

Furthermore, people should stop thinking of computers as singular entities whose computing powers are limited to what’s inside the box. Think clusters. If someone were to be setting up a small office, for example, networking 10 of these machines with a single quad core computer server, would actually be more efficient and powerful than having 10 top end boxes individually. Not to mention, it would look far cooler than having 10 regular beige boxes sitting on the desk.

Personally, I think that ASUS has been doing a great job recently, innovating like crazy and opening up new market niches where no one has gone before, which is forcing everyone else to follow along. They started with the Eee PC, which has opened up a market that is now known as the “NetBook” market. This has forced everyone else, from HP, Dell, Acer, MSI to play catch-up. Then, ASUS released it’s Eee Box, which is trying to give the Apple mini, a run for it’s money. And now this!

Whomever says that the PC computing market is dead, is obviously wrong. Although the Eee PC may be a novelty that appeals mainly to road-warriors and hacker-like people, I can potentially see an Eee Monitor in every home and office. I’d like to see Acer come up with an equivalent device at a $400 price point instead. That would be just snazzy and I’ll grab one or more of these for myself.

PS: They should fire their marketing department though, for coming up with such lame names as Eee PC, Eee Box and Eee Monitor. What will they think of next?

Mobiles and I

Typical reactions about my mobile phone and I:

  • 1998: Wah, you got mobile phone!
  • 1999: Eh, pinjam your phone.
  • 2000: What’s your number?
  • 2001: Why your battery die so fast ah?
  • 2002: Eh, cannot call u lar!
  • 2003: Why your phone so simple one, ah?
  • 2004: Eh, phone so cheap, get new one lar!
  • 2005: Why don’t you ever pick up?
  • 2006: So, how can I contact you?
  • 2007: Why don’t you use your phone?

The thing is, I bought myself a mobile a decade ago. At the time, it was rather a novelty and I did use it a lot as I had just gone to university and the public phones there were never in working order. Then, phones became common and I realised that I began to use it less and less, as it was a device for people to get in touch with me, not the other way round. I tend to prefer face-to-face communications. Then, phones became so packed with features that I ended up buying the simplest phone on the market.

But now, I’m actually thinking on spending close to £300 for a new mobile phone (and it’s not the iPhone!). The phone that I am planning to get is the OpenMoko, a fully open sourced mobile phone. I love the whole idea of a open sourced mobile stack. It would not only be a phone to me, but a fun new toy for me to build interesting apps on. They have also worked hard to secure hardware documentation for the community. That changes the whole dynamic of the relationship.

The only problem is that it’s still in a very alpha stage. Although the hardware platform is mostly fixed, the software is still under heavy development. At the moment, it is not guaranteed to be able to make a stable call but they are constantly releasing new firmware revisions each day. I might wait till it stabilises a little and at least allows me to make phone calls and send text messages, before buying it for myself.

The hardware is quite elegant, except for one extremely rookie mistake. They must have left that part to an intern to do, or something. They had actually connected the graphics accelerator onto a slow SPI bus. That’s like fitting a Ferrari with bicycle wheels. But other than that, everything else looks quite nice. As I will be leaving the UK soon, I would need to buy this thing soon. It isn’t easily available in Malaysia.

Graduation 2008

Another year and another batch passes through, into the real world. This is the batch that I am particularly close to, among the different batches of students here. There are some friends whom I will miss dearly, and some whom I hope never to see again for the rest of my life. Such is the way of things.

I will try my best to keep in touch with at least some of them. There are a few whom I would like to take away as lifelong friends. As I may be going home soon, it should not be too difficult for me to maintain a friendship with the ones who go home, as we will all inevitably end up working in KL. The only way to keep in touch with the ones working elsewhere would be by email/MSN.

To my friends, I would like to see them be the best that they can be and realise their full potentials. It is wonderful to see everyone starting off on their next chapters in life, with so much enthusiasm and drive. I pray that life treats them well and hope that they will breeze through any curve balls that life will inevitably throw their way. I wish that they succeed in finding the happiness in life that so eludes the rest of us.

Well, I hope that I will get my chance to walk through the same doors, one day. But I’ll need to finish writing up my thesis first! It’s so hard to focus on work when there are so many other things happening in and around me at the moment.

Screwed from Behind

I just read that the police are going to arrest Anwar Ibrahim soon. It seems that one of Anwar’s aides has gone missing for the last two days. The aide has now surfaced and filed a police report against Anwar for sodomy (wtf??!!). In addition, Anwar had recently received evidence that implicates the a former IGP of police and a former AG, for fabricating evidence against him during his trial, 10 years ago. This rumour is just too big, fat, and juicy to ignore.

Incidentally, some friends and I were just chatting about this on our way home from dinner. We mentioned that our former PM, had locked Anwar up for good and essentially killed his political career beyond any hope of recovery. That was until our present PM decided to set Anwar free and claim that the charges were false. This let lose an avenging force, which culminated in the severe losses suffered during the last general election.

I have one advice for our present government: Once you let the genie out of the bottle, you cannot put it back in.

Regardless of the truth of the matter, we are most certainly now at political crossroads. Anwar Ibrahim is the main strategist that is pulling the opposition together. As a former DPM, he has dirt on a lot of people and he knows all the tricks that the government has up its sleeve. Now, he claims to have 50 government MPs in his pocket and he plans to use them by September to topple the government and claim the throne of power.

I am hoping that things will turn out for the best, and praying that it does not deteriorate for the worst.

If Anwar had seriously committed the crimes that they claim that he has, I certainly hope that he gets punished for it. However, I also hope that every corrupt politician and police, also get their dues some day. But I am a realist, and I know that the world is not a fair and perfect place. So, it is very likely that the political dullards will get away with things unless something is done to stop them, and stop them for good.

I have no idea what needs to be done though, and although some people are calling for the rakyat to stand up and take to the streets, I am not sure if it would be effective as we have all seen what happened in 1998. But I still think that something needs to be done, less we end up like Zimbabwe.

Parliament Stunts

I like to read TheStar when they report on the silliness of our parliamentarians on both sides of the divide. Recently, there was a minor stunt by our opposition MPs, who have decided to politicise my idea of cycling to work. According to the article:

Four Parti Keadilan Rakyat (PKR) MPs – Tian Chua (Batu), Hee Loy Sian (PJ Selatan), N. Gobalakrishnan (Padang Serai) and Zulkifli Nordin (Kulim Bandar Baru) – cycled to Parliament Monday as a mark of protest against the recent fuel hike.

Tian Chua said move was also to “appeal to the Cabinet Ministers to understand the difficulty in using bicycles or public transport” in the country. “Changing of lifestyles is not as easy as what the Government claims,” he told reporters.

He said that he and the three others had taken about 20 minutes to cycle from Bank Negara to the Dewan Rakyat. “It’s a difficult task because the road transport system here is not designed for bicycle use and cyclists will face problems cruising around town,” he said.

The opposition MPs should not have called it a protest. As protests go, this is a very silly and gimmicky stunt. They used to do really good protests but this is certainly not one. They should have positively spun it as an act of support for the government’s call to ubah gaya hidup (change lifestyles). It can also be spun as being an environmentally friendly move on their part. But no, they had to spin it as a lame attempt at a protest against the price hike. Silly.

Then, we have another MP from the ruling coalition, who calls the stunt, a cheap gimmick that is suitable for kids. He could have come up with something better, and praise the opposition MPs for supporting the government’s position on fuel prices and changing their lifestyle. He could even say that the ruling coalition MPs would lead by example and cycle over short distances. But instead, he claims that he isn’t stupid and he isn’t going to cycle, which implies that only stupid people follow the government’s call of ubah gaya hidup.