Software

I've spent my evenings this week tracking down a website hack. It's not on this server, but on another some friends and I maintain. A message came in on the trouble ticket system, "you have a hack". Someone had noticed pages full of links to "blue pill" sites stuck up on our site. I volunteered to look into it. I've had to do this sort of forensic/detective work a few times in the past; it's pretty fun and challenging.

The first clue was that the files were all owned by the web server process, indicating a high likelihood that a vulnerability in the web site had been exploited (as opposed to some other type of hack masquerading as a web sploit). The second clue was that there was a zip file, plus all the files had the same timestamp, indicating that they were extracted from the zip (versus uploaded individually). That meant that the hacker had been able to execute code on our server. Arbitrary code execution is a severe vulnerability, it means the hacker can pretty much do anything (limited only by the privileges of the web server process).

This all pointed to PHP. There's a ton of PHP code on this system. PHP itself was notoriously crackable early on, but by about age ten they'd rooted out all the major security flaws (it's still a case study in how not to design, or grow, a programming language). But since PHP has such a low barrier to entry, the code that web programmers write with it is often flawed.

We're using third-party software like mediawiki (wiki system), vbulletin (forums), gallery (images) and wordpress (blogging). Some of those have a pretty bad track record, and I'm looking at wordpress when I say that. And there's a variety of WP vintages on this system. But there's also a lot of custom code written by people I don't even know, with a variety of skill and attention to security concerns. But there's also some legacy Perl CGI code, some of it written by me, that could be at fault....

Read More

The problem: Apple movie trailers won't play on your KDE3 Linux web browser (such as Firefox on Kubuntu 8.04). The solution: get the backport of mozilla-mplayer. Of course, you could upgrade to the latest Kubuntu, 8.10, but then you're forced to switch to the KDE 4 desktop, and I'm not going there... yet (maybe in a few more releases).

Figuring out the problem took a little while, viewing the source of a typical trailer page and digging around in the JavaScript. Google searches turned up sub-optimal solutions. The issue turns out to be a browser detection JavaScript that Apple is using, and the fact that the older version of the mozilla-mplayer plugin advertises itself as quicktime "6 / 7" compatible (the script needs to see "7" as the first character).

A new browser release (Firefox 3.1 beta 2), big whoop, right? But Firefox is charging ahead, leading the way to a better web. Here's some of the new features that will be important to web designers, developers and even the average surfer.

• Embedded fonts and text-shadowing styles. Maybe designers will finally stop rendering text in images? I won't hold my breath, but it's nice to know the capability is there.
• Inline, in-browser audio and video without plugins (part of the HTML5 spec), using the free/open Ogg format. I suspect Apple and Microsoft will "embrace and extend" this, but it could mean that finally, finally (after a decade or more) there will be at least one video format that all platforms support.
• Support for n-th child CSS selectors, a cleaner way to do things like render table rows with alternating colors (I can't tell you how many times I've inserted a new row and had to change all the "evens" to "odd" and vice versa).
• A new, even faster JavaScript JIT compiler/interpreter (the one in 3.0 was itself a big improvement). Also a threading capability for JavaScript. This will enable smoother running, more fully-featured web applications.
• Geolocation, now you can grant sites permission to use/track your location. (Uh, might want to be careful with that one).
• Better security for AJAX (cross-site authorization control) and a JSON parse method, a secure way to load data and prevent code injection.
• Switchable 'private browsing' mode... which they suggest is useful for hiding your online xmas shopping from your spouse (!).

There's more, that's just the stuff that seemed significant to me. I think Safari and Internet Explorer are working on, or already have implemented, some of this stuff, and there's a new Opera browser in the works too. This reminds me of the long-lost days of yore, when browser competition fostered innovation. This difference this time is that they all seem committed to common baseline standards.

XUL is an application framework from Mozilla (Firefox and Thunderbird are XUL applications with C++ extensions). I recently learned that you can run command-line programs from within script sections of XUL, which makes it significantly more useful for me. Of course, there are security restrictions; the XUL file needs to load from a local file (or be signed).

What I really like about it is that the UI is declarative, you can just type it up, preview it in the browser and hit reload to see changes. With the error console and other developer tools, you can use the same development tools and techniques from the web. Add the flexibility of JavaScript and you've got a really powerful base. There are some issues with particular 3rd party JavaScript libraries (such as jQuery), but it's not a big stumbling block.

Here is a toy application, yet another Twitter client. You'd run it with (for instance) firefox -chrome test.xul. If it executed commands, it would need a line that said netscape.security.PrivilegeManager.enablePrivilege ("UniversalXPConnect"); (the example does output with AJAX instead).

<?xml version="1.0"?>
<?xml-stylesheet href="chrome://global/skin/" type="text/css"?>

<!--

    Twitter updater implemented in XUL

-->
<window
    id="win"
    title="XUL Twitter"
    orient="vertical"
    xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
    onload="updateLabel()"
>
    <script>
        var max = 140;
        var uid = 'guymac';
        var pwd = '******';

        var twitUpdate = 'http://twitter.com/statuses/update.xml';
        var twitCloser = 'http://twitter.com/account/end_session';

        function updateLabel()
        {
            var lbl = document.getElementById('lbl');
            var txt = document.getElementById('txt');
            var rem = max - txt.textLength;
            lbl.textContent = rem + ' char' + (rem > 1 ? 's' : '' ) + ' remaining';
            txt.disabled = !(rem >= 1);
        }

        function postTweet()
        {
            var lbl = document.getElementById('lbl');

            try
            {
                var req = new XMLHttpRequest();
                req.open("POST", twitUpdate, false, uid, pwd);
                req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');

                req.send('status=' + document.getElementById('txt').value);

                if (req.status != 200) throw req.statusText;

                lbl.textContent = 'Twitter Updated!';
            }
            catch (ex)
            {
                lbl.textContent = 'Update failed (' + ex + ')'; 
            }

            var  req = new XMLHttpRequest();
            req.open("GET", twitCloser, false, uid, pwd);
            req.send(null);
        }
    </script>

    <textbox
        id="txt"
        maxlength="140"
        size="140"
        multiline="true"
        oninput="updateLabel()"
    />

    <label id="lbl"/>

    <hbox>
        <button label="Post" oncommand="postTweet()"/>
        <button label="Close" oncommand="window.close()"/>
    </hbox>

</window>

A few weeks back, I posted a couple of examples of trivial (but graphical) Twitter clients. Here's a slightly more featured example, which uses the PyQt library to hook into the powerful cross-platform Qt toolkit from Python.

Here's a screenshot. It shows how many characters you have remaining, and disables posting if the character count is over the limit. Below is the code. To run on a Debian system you'd install the python-qt4 and python-httplib2 packages. I think it is a good example that for simple applications, the Python-Qt framework is pretty nice, though in lines of code, it would probably be about the same in Java (and probably less in JavaFX). Another nice thing is that you can put together the UI with Qt Designer.

Zetcode's PyQt4 tutorial and the official PyQt documentation were indispensable.

#!/usr/bin/python

import sys, urllib, httplib2
from PyQt4 import QtCore, QtGui

class PTwitter(QtGui.QWidget):

    max = 140
    uid = 'guymac'
    pwd = '********'

    twitUpdate = 'http://twitter.com/statuses/update.xml'
    twitCloser = 'http://twitter.com/account/end_session'

    def postTweet(self):
        mesg = self.txt.toPlainText()
        post = urllib.urlencode({ 'status' : mesg })

        try:
            http = httplib2.Http()

            # override the default (no exceptions, status in response)
            http.force_exception_to_status_code = False

            http.add_credentials(self.uid, self.pwd)

            resp, content = http.request(self.twitUpdate, 'POST', post)

            if resp and resp.status == 200:
                self.lbl.setText('Twitter updated!')
            else:
                raise httplib2.HttpLib2Error, 'No response'
        except httplib2.HttpLib2Error, ex:
            print ex
            self.lbl.setText(ex.__str__())

        http.request(self.twitCloser)

    def updateLabel(self):
        len = self.txt.toPlainText().size()
        self.lbl.setText('%d char(s) remaining' % (self.max-len))

        # disable the post button if chars > max
        self.pb1.setEnabled(len <= max)

    def __init__(self):
        QtGui.QWidget.__init__(self)

        # Button to post
        self.pb1 = QtGui.QPushButton('Post')

        # Button to close
        self.pb2 = QtGui.QPushButton('Close')

        # TextField for input
        self.txt = QtGui.QTextEdit()

        # Label to show characters remaining
        self.lbl = QtGui.QLabel()

        tab = QtGui.QGridLayout()
        tab.setSpacing(5)

        tab.addWidget(self.txt, 0, 0, 1, 2)
        tab.addWidget(self.lbl, 1, 0, 1, 2)
        tab.addWidget(self.pb1, 2, 0)
        tab.addWidget(self.pb2, 2, 1)

        self.setLayout(tab)
        self.setWindowTitle('PyQt4 Twitter')

        self.updateLabel()

        # connect text change to number of chars message
        self.connect(self.txt, QtCore.SIGNAL('textChanged()'),
            self.updateLabel)

        # connect post click to send message
        self.connect(self.pb1, QtCore.SIGNAL('clicked()'),
            self.postTweet)

        # connect close click to quit
        self.connect(self.pb2, QtCore.SIGNAL('clicked()'),
            QtGui.qApp, QtCore.SLOT('quit()'))

# application
app = QtGui.QApplication(sys.argv)

# window
win = PTwitter()

# display
win.show()

# execute
sys.exit(app.exec_())

I'm disappointed with O'Reilly's Python Cookbook (Second Edition). It's unfortunate because I've found their "Cookbooks" for other languages to be A) a good way to pick up the fundamentals while B) learn recipes for common programming tasks. The Python Cookbook skimps on the language fundamentals while, despite its huge bulk, having few recipes for task types that I can imagine needing to know.

For instance, there are no examples of an XPath API, which I'd consider indispensable. One of the first things I wanted to do was take the Netflix Queue XHTML and simply extract the titles. I dug up some old Java code to do it.

The chapter on XML doesn't even have any examples of creating XML! And every Python example I've seen so far does it by simply spitting out strings. Where are the equivalents of StAX, DOM, JDOM, or (wishful thinking) E4X?

I appreciate Python's attempt to be clean, consistent, and object-oriented... in marked contrast to Perl's hodge-podge gumbo of pre-OOP stuff like C, Bourne shell, and so on. But the OOP chapter is just bizarre, full of things that I've never needed to worry about in Java.

The chapters on network and web programming don't show how to post to a web server or do authentication. The online library reference is useful, but very short on complete examples. I could go on, but suffice it to say that without Google searches, I wouldn't have gotten very far with Python.

So I figured out a reason to use twitter. No, not publishing random thoughts--anything worth a tweet is too infrequent. And not describing my daily whereabouts--there's little utility in that. What I'm doing instead is a fitness activity log. Not that anyone will care, but it does a) provide me a log which is useful for training and b) give some extra motivation vis-à-vis the mere possibility that my friends and co-workers might check it out.

Hook graphical widgets to shell scripts and you have a powerful but under-utilized class of applications. Systems programmers are typically writing code that does not need user interaction. Software developers are typically writing larger-scale programs in languages with rich widget libraries. But the domain in between does have some important uses, for instance small-scale programs that need input from non-technical operators, or need to alert them as tasks succeed (or fail).

I was reminded of this the other day when looking for a really simple Twitter client, just something I could click from the desktop or taskbar, input a string of text and be done. The one I found was a single-line shell script that used kdialog. I added a line to catch errors. Here is the complete script.

#!/bin/bash
curl -u username:password -s -F status="`kdialog --inputbox 'what are you doing right now?'`" http://twitter.com/statuses/update.xml http://twitter.com/account/end_session
if [ $? != 0 ]; then kdialog --error 'Twitter could not be updated'; fi

The next day I happened to read about a Python module that provides similar functionality for Mac OS X; it's called EasyDialogs. So I made a Mac version (my first real Python program). Here it is.

#!/usr/bin/pythonw
import EasyDialogs
import os
ret = os.system('curl -u username:password -s -F status="' + EasyDialogs.AskString('What are you doing right now?') + '" http://twitter.com/statuses/update.xml http://twitter.com/account/end_session')
if ret:
     EasyDialogs.Message('Twitter could not be updated.')

Either can be made executable and linked from the dock/taskbar for quick launch. Check out the tutorials for kdialog or EasyDialog for a better idea of what the possibilities are (complete with many graphical examples) using the standard widgets such as file choosers, checkboxes, etc.

Last week I upgraded my laptop to the latest and greatest version of Linux, Kubuntu 8.04 (code-named the Hardy Heron; each new version of Ubuntu alphabetically increments an alliterative adjective and animal).

The only issue was that when resuming from hibernation, the screen resolution would change. I filed a bug report and asked a question on a Ubuntu support page. The response was pretty quick... if I had been checking my email! The simple fix was removing a package obsoleted by the new version of X Windows.

I'm not brave enough to try the new KDE (and, apparently, neither are the Kubuntu developers, it's still optional). KDE 3.5 with a few tweaks is just about perfect IMHO, and still leagues ahead of OS X or Vista, at least in terms of functionality. The screencap shows a feature that has yet to be built into other OS's, the ability to drag-n-drop with sftp:// locations, which I find vital for website management.

There's cool and useful features like this all over the place. It's a near perfect OS!

I'm looking for a lightweight scripting language that can run from the command-line or desktop and with a rich set of standard widgets. Here's the catch: it should be a modern language like JavaScript/ActionScript/ECMAScript. It sounds simple, but I'm not finding anything.

A long time ago I whipped up a Tcl/tK application; the language was weird and I don't want to go down that road again. Even further back, I made a small utility in REXX which was also kind of frustrating. I've done a little Applescripting, which is truly bizarre if you're doing anything more than driving existing applications.

Hopefully, this would run on any UNIX/Linux that has the interpreter installed. But if it's just OS X or just Linux it would be a good start.

It would also use be ideal if it used many of the same mechanisms that I'm familiar with as a web developer. Events fired from the user interface and such. An API to manipulate the user interface such as the Document Object Model, or libraries that layer on top of the DOM to make it less verbose to work with.

So, ideally the UI would be rendered in HTML, but communication would not be through a web server. The script would not be in a "sandbox", so could run with all the privileges of the user. So you'd have a standard UNIX script starting with the shebang (#!), a pile of interpreted code, and a graphical user interface.

Here are the things I know of that come close to fitting the bill.

Read More

I've re-written this simple utility a number of times over the years, this is the latest incarnation. It lets you instantly view a slideshow of the images in a folder on a web folder. All you have to do is copy the file into the directory. Only images within that directory can be viewed, so it is safe (can't be used to get info about other directories).

An example of what I'm talking about.

All of the functionality is contained within a single file. It uses JavaScript to control the slideshow and PHP to build it.

Syntax-highlighted source code

Copy and paste the text into a file called index.php in the folder. Or...

Raw source code

Save this as a file called index.php in the folder. Finally, open the URL to the folder in a browser.

It's dead simple, but sure is handy.

Instead of paying $100 per year (typically) for a secure certificate, many websites use a "self-signed" certificate, meaning that one of the big network agencies hasn't vouched for their credentials. The benefit to the web developer (and to users) is that a secure (encrypted) web connection can be created. The downside is that the user will be prompted upon a first visit to accept the certificate, and the risk that the site and its certificate might be fraudulent. This risk is minimal for low-profile sites (who may not have the money for a certificate) who have little risk of being spoofed via spam emails for instance. Some examples [from my own web dev experience] would be webmail for a small non-profit or a protected wiki for a small to medium sized science team. In these cases, users can be informed in advance of what to expect on the first visit, and one can be reasonably sure that hackers have not set up sites with similar URLs and content to trap the unwary.

And, for what it's worth, self-signed certs are not significantly less secure or less risky than normal certificates. A hacker can obtain a minty-fresh cert for a spoof site just as easily as a legitimate site--it just means the hacker was willing to part with $100 (or whatever it is) in pursuit of far more valuable rewards. [Update: This was just plain wrong, as indicated by the comment below, self-signed certs are only as secure as the method used to distribute them.]

But I just noticed in the new Firefox (3.0 beta 2 for Mac), that accepting a self-signed certificate is now a four-step process with messages designed to scare you away from accepting such a cert. Read the rest of this entry for screenshots. I think this will be sufficient to dissuade many users from accepting a self-signed cert, no matter what they have been told up-front.

Read More

Have you ever done the following?

1. Logged in to a web site, let's say for the sake of example, any web-based email service
2. While still logged in, visited another site, perhaps just clicking on a link in an email, or viewing an HTML email

Now, you have just opened the doors wide for an attack. A malicious script could (for instance) grab your email or do anything else the programming interface (the web API) of the site you are logged into, allows. This chilling story indicates the extent of your vulnerability. A script used the GMail API to insert a rule that forwarded all email concerning domain names to a site hijacker. The hijacker then sent an email to a domain registrar, and quickly and easily had control of the person's business web site. Most likely the exploit was a script uploaded to a web forum or other insecure page, where upon it attempted the same for everyone who viewed that page. It could even be in an image tag in a HTML spam email--no action required on the victims part.

I've known about this style of attack (now called a XSRF or cross-site request forgery) for many years. How do you ensure that a request coming in is legitimate? After all, it is the web, legitimate use could be coming in from any Internet Protocol address. One barricade that can be placed in the way of hackers is to use what is called the referrer or "HTTP referer" (mis-spelled in the original specification). This is information that the web browser provides indicating which URL the request came from. For example, only Google URLs should be accessing GMail settings.

This information however can be spoofed, for example by writing a program that emulates a browser (and this is much easier than it may sound to the non-programmer). A larger problem is then distributing the program so that victims inadvertently use it, which is where well-known techniques of code injection such as viruses come into play. Even simpler is to use a Flash applet stuck on a web page, which you will probably not even be aware of running when you stumble over it. But apparently Google's GMail did not have even this low barricade, allowing simple scripts to directly access the settings (of any GMail user who was currently logged in while being subjected to the evil script).

So, the question is, is there any way to safely browse? After all, banking, shopping, and many forms of communicating now take place primarily via the web.

Read More

Here's my top-12 list of Mac annoyances, things that suck about Mac OS X. I bought a Mac Mini in the fall of 2005, a 1.2 GHz model with 512 MB of RAM. Around the same time my work machine became a G5 quad (actually a dual dualie) with 2GB of RAM (since upgraded to 4). Last year I got a Dell Inspiron B130 laptop, which is now my main machine at home, relegating the mini to a server for music. The laptop runs Kubuntu, a derivative of Debian Linux with the KDE desktop, which supports it very well.

1. Poor Performance

OS X is beyond dog slow, it is slug slow. Windows XP or Linux on a ten-year-old Pentium II will be snappier than OS X. Even on high-end hardware, it is not discernably more responsive. In general, most applications are poorly threaded and the spinning beach ball busy cursor is always around the corner. For instance in Firefox, when launching a tab, you'll frequently be unable to do anything with the Finder menu while the tab is connecting.

2. iPhoto

iPhoto is the all-time champion when it comes to poor performance, taking a minute to load up. It is such a memory hog that minimizing it to switch to other applications, or maximizing it again, takes almost another minute.

3. Lack of package management

Debian has apt-get or the graphical versions Adept or Synaptic, which make installing (or un-installing) any one of over ten thousand applications and libraries a one-step procedure, complete with finding and installing all their dependencies, synchronized with an always up-to-date online catalog. The technique is used in nearly all Linux distributions and was perfected in Debian years ago. In contrast, OS X has the concept of dragging an application bundle to a folder. Each bundle (usually) includes its own copy of those libraries that it depends on, leading to bloat. If you want to upgrade to the latest version, there is no automated way to do it.

4. Poor previewing

In KDE, you can set the icon size to be different than the size of image previews (when viewing a folder). In OS X, you cannot. Either you have nicely sized icons and tiny previews, or giant icons and nice previews. What's more if you select a set of images for a slideshow, they are in random order, with no apparent way of changing the order! Preview.app is also flawed; you can't view a selected set of images at their actual size, instead you must hit 'actual size' on each individual one.

Read More

I tuned up the Javascript / DOM code for the AJAX version of Joggle. Someday I will get around to a network, multi-player version, a cell phone version, and more. Sadly, I can barely get onto the high scores list of my own game. It's tough!

I couldn't find a really good guide to typing in extended characters in Linux (KDE specifically), so I've compiled what I've found into a table. Each OS does this a different way. I remember on Windows and OS/2 it was ALT + numeric id of character, so for the 'é' in résumé you'd have to remember that 'é' is 233 is extended ASCII. Or you'd just use the charmap application, which involved a lot of clicking. Mac OS X has it's own way of doing things as far as character combinations go, and there's always a little pull-down menu you can get to for obscure characters.

KDE is very nicely designed and has an elegant way of entering in extended characters. You type the compose key, which by default is the right 'Win' key. My laptop doesn't have a right 'Win' key, so I've configured caps lock as the compose key; I never use caps lock otherwise... it is an anachronism. Anyways, you hit & release the compose key, then press the keys which are shown in my chart. For 'é', it would be <compose> then apostrophe and e. The mnemonics seem to be easier to remember than Mac OS. I.e. e + ' = é.

Here is the table.

In my experience, O'Reilly press has consistently made the best technical books. I've owned or at least borrowed (heavily) more than a few, particularly those in the Java, Perl, and UNIX lines.

So when I wanted to learn ActionScript (Flash), I was pleased to find ActionScript 3.0 Cookbook. I immediately found section titles specifically geared for what I wanted to do, like loading images via URL, scrolling, passing variables from HTML pages. Furthermore, the first few chapters got me up to speed on language basics very quickly. The "cookbook" style text works extremely well for learning a new language when you are already familiar with the concepts, such as event handlers or associative arrays. I had written a nearly complete version of what I envisioned that evening, and finished it in another evening. Glancing at other sections, it looks like this book will continue to work for me up the learning curve for quite some time. All in all, this is one of the best titles from O'Reilly, which means it is really quite exceptional. Coupled with the language reference and SDK, it's all you need to get started.

ActionScript 3.0 combines the best of Java (strongly typed, object-oriented, namespaces, compiled) and JavaScript (simplicity, syntax). Actually, it looks like ActionScript 3.0 and JavaScript 2.0 will be one and the same; AS3 will be formalized as the next version of the JavaScript standard. Adobe has donated the virtual machine to the Mozilla project for use in future versions of Firefox and other applications.

Read More

Here's my first Flash "movie" (I prefer the term "applet"). Since our (HiRISE) images typically have unusual aspect ratios (like 1:4 or 1:8), this Scroller allows the thumbnail versions to be embedded in a web page like a more commonly sized image (1:1 or 4:3). It also displays a scale bar, which may be dragged around.

The sourcecode is available.

I've been staunchly opposed to Flash for many a year; now, oddly, I find that it is my new favorite programming language.

You see, I had this idea for a small project that seemed best written in Flash. The following table outlines what has changed that has made me reconsider it.

Old, Cold Flash	Hot Flash (Today)
Required proprietary development environment	Free development tools
Required graphical development environment	Compiler can be called via Makefile
Linux version was not available, or out-of-date	Linux player (almost) a first-class citizen
All-Flash sites in vogue	Web Standards prevail
ActionScript not a "real" programming language	ActionScript 3.0

RoundCube... the name may be an oxymoron... but this new webmail application ain't. I just got through installing it here on my server and it seems to work well. The nice thing about it is the support for the PosgreSQL database engine, which is more powerful and reliable than MySQL. It has a nice interface, closer to GMail and the newer hosted webmail applications, using AJAX and all that web-2.0-ish goodness; less page-to-page navigation & reloading than the venerable king of open source webmail, SquirrelMail. Plus, it uses IMAP internally, so you can connect it to any IMAP or IMAP-over-SSL host.