Sysadmin, security, billing: SNAFU

Today's subject is a SNAFU in one of my consulting projects, and what can be learned from it. As the acronym suggests, this is Situation Normal... perfectly typical of projects involving software. I'll get into a few technical details, but I'm writing this is for owners/managers too.

The project is an online store for a successful distribution company with several million dollars in revenue. This is the company's first foray into online retail; 99% of its business is by phone/fax/mail. The client is not tech-savvy.

So far I'm marginally involved in this project, merely bridging the gap between web designer and the wacky world of e-commerce. I'm about 5 rungs down the chain. I made some recommendations, not the final decisions. So don't laugh :-)

Ultimately this is a story of unpredictable costs and a... failure of positive thinking. But first I need to cover some technical details for my core audience of geeks. Skim down to the next section if you're not tech-savvy.

Technical details

The payment provider is Authorize.net -- the same one I dealt with on my first little e-commerce project over 10 years ago.

The e-commerce software is Magento (the open-source "community" version). I had recommended using the "cloud service" version, Magento Go, instead. That way, the company that developed the software would be fully responsible for server maintenance, upgrades, SSL certificates, etc (which all tend to be tedious things) and more importantly they'd be liable for security breaches... yes, "indemnity" is the killer feature.

Magento runs on a LAMP server platform (Linux, Apache, MySQL, PHP)

We started with $10/mo shared hosting, but moved to a $60/mo dedicated server, because

1. Magento is resource-hungry, and...
2. The hosting company (1and1) doesn't allow multiple IP addresses for shared hosting accounts. Others do (e.g. Webfaction, which I've been happy with) but the client did not want to switch.
3. We needed another IP address for a second SSL certificate for a second (related) online store.
4. Alternatively, we could have used something called SNI, which allows a (costlier) multi-site SSL certificate on a single IP... except SNI requires browser support, which is lacking in every version of Internet Explorer that runs under Windows XP. That's half the customers.
5. As another alternative, we could have used Authorize.net's new DirectPost API, which makes a lot of sense -- it lets customers submit their credit card info directly to Authorize.net via secure HTTP POST using Authorize.net's SSL certificate, without shunting them off of the store website, without requiring SSL certificates on store websites! There's one catch: users wouldn't see the stupid padlock icon, and the merchant service company wouldn't approve a store without it (I called and asked).
6. Ultimately, we needed SSL certificates because the payment card industry says so.

So I installed the new SSL certs on the dedicated server, and tested the store. It was fine... until the checkout page. Nothing. Blank.

Server log reveals that PHP 'mcrypt' module is not installed. Ahhh, I remember that from the last time I dealt with a PHP e-commerce site, in 2009.

Ok, how do I SSH into this server and become root? I poke around the 1and1 control panel and FAQs... Answer: 'Install PuTTY and log in to root@YOURSITE.com using your (weak) initial password.' And they don't even say "THIS IS VERY BAD" - they're tacitly encouraging customers to LEAVE THE DOOR WIDE OPEN.

So anyway, I'm logged in as root, the wrong way. It's CentOS Linux, similar to Fedora (my desktop OS), so I type yum list '*mcrypt*' and... php-mcrypt is available but not installed. Ok, yum install php-mcrypt? No go, there's a version conflict: we're running PHP 5.3, but there's only a php-mcrypt package for PHP 5.1. Hmm... will Magento work with 5.1? Nope, 5.2 minimum. I do some googling... this has been an issue for at least 3 years. Long story short, I ended up downloading the source code for the version of PHP packaged with this CentOS from museum.php.net and building the 'mcrypt' extension. It's a stopgap solution which does not inspire confidence.

Oh, and there are dozens of updates that need to be installed. Shouldn't automatic updates be the default?

Back to the business side

When this project began, the scope was very small. Being unfamiliar with Magento, I estimated a full day's work just to be on the safe side, but I'd be paid hourly. It only ended up costing $300. Client happy.

Months pass. Client returns to me with phase 2: setup a second Magento store. Should be easy, maybe only $200, but a multi-site SSL cert would add ~$300, and there could be complications. Sure enough, there are. The total first-year cost is now $1,000 for my time, $1,000 for the dedicated server, and at least $3,000 for the other (sub)contractors.

Let's say $10,000 total cost to the end client, who has several millions in revenue. (To put that in perspective, I know several geeks who briefly worked for a local web design firm that charges $50,000 to $100,000 a year to maintain e-commerce sites. Top-flight? Not really: just a 3-4 person dev team with high turnover, and a 10-person marketing department.)

As the project unfolded, and in talking to other people, it became apparent that there's a major shortage of Unix sysadmin skills in the backwater Western Mass web-design industry, and this project is no exception. Some firms have a developer or two acting as part-time sysadmins, but the firms that focus stuff like WordPress and Drupal and graphic design, they're in over their heads. They got along fine without sysadmin skills for 10 years, when the Web was just static HTML and Flash files. Then came Web 2.0, i.e. dynamic backends, i.e. custom software running on a Unix web server in a faraway data center. You kinda need sysadmin skills to secure and maintain backend software, even if it's just a WordPress site that a total noob can install in 20 minutes, nevermind an e-commerce site that handles credit card numbers!

I have a mind to charge $3,000 up front for future e-com projects. That's double what I expect to bill for this one. It's high, but I can stick to it.

However, I may not get any work.

Fortunately, I've got a few interviews lined up for full-time Node.js/html5 development gigs, so I may not have time for e-com consulting. Even then, I'm still interested in bringing together a regional economy of designers, developers, sysadmins, e-commerce/finance specialists, honest marketers, etc.

What's a sensible billing arrangement?

Clients don't want to deal with internet-related service providers piecemeal. I do insist that clients register their own domains in their own names, both to demonstrate their initiative and to show that I don't want to hold them hostage. But I don't expect them to buy their own hosting, bandwidth, storage, CDN, SSL, etc. I don't even want them to. I won't give them a choice between $10 shared hosting and $100 dedicated hosting. I'll rent my own damn virtual servers running whatever OS I'm prefer... maybe CentOS, maybe FreeBSD with its efficient "Jail" VM... I'll move client websites around depending on how many resources they actually need.

Just pay me a generous (but predictable) flat fee, and I'll make the decisions that minimize my total cost -- in terms of enjoyment vs aggravation, not just dollars!

I also don't want clients calling me freaking out "my site's down! I'm out of business!" because they forgot to pay a $10 hosting fee. I'll handle all that. Except for domain renewals... but I will harp on them if they keep putting it off. Hell, I'll barge into their office and hold their hand through the whole process, if necessary.

Yesterday I stumbled upon a book on the office shelf: Value-Based Fees by Alan Weiss, 2008. It's geared toward consultants of all stripes, including those who aspire to be obnoxious $5000/hr talking heads. But he confirms what I've been thinking. Consulting (advice-giving), like programming and design, has negligible marginal value. There's a big research/development cost up front, but related projects cost less and less, approaching zero. There's a ton of shared value there (and intellectual property laws don't have much to say about it, by the way). Hourly billing is unfair to client #1. It's also part of the old "bean-counter fallacy". Furthermore, it's a conflict of interest: paying consultants hourly creates an incentive to waste time. It's a recipe for mistrust. Much better to build a healthy long-term consulting relationship

So I'm planning to get away from hourly billing. I'll quote a high lump sum price and ask for a deposit. The rest will be due upon successful completion (in phases, as warranted). Success will be defined entirely by the client. In case of failure we can renegotiate the contract, or I can just give up; the client loses their deposit, I lose some time, and we all learn some lessons. Hopefully we stay on friendly terms.

Examples:

Small website: $1,000 for setup, design, first year (or so) of hosting. And I'll pay the designer a good chunk of that.

E-commerce store: $5,000. That includes a big pain-in-the-ass surcharge. Last spring I told a few prospects here in Greenfield "$3,000 minimum" and they're still "thinking about it" last I checked. Fine with me... I'd rather that, than lure them in with lowball estimates.

Frankly, e-commerce is too expensive for mom-and-pop businesses... sorry, our banking system sucks!

A few footnotes...

Who is responsible for security?

Bosses are responsible for

• Maintaining basic hands-on and theoretical knowledge of the technology the business depends on

• Budgeting for security (weighing the risks of downtime, bad publicity, financial liability, and criminal prosecution for extreme negligence)

• Encouraging honest and forthright behavior (by encouraging criticism, rewarding those who admit mistakes, not "shooting the messenger"... while avoiding crippling indecision, somehow :-)

Not-so-technical staff (clerical, marketing, management, finance, etc)

• Choosing strong, unique passwords -- and keeping them secret

• Maintaining basic technical skills

Application programmers (and designers, architects, etc)

• Focusing on clean, simple, understandable designs

• Writing good code

• Choosing solid foundations (OS, tools, libraries, frameworks, etc)

• Resisting client/boss demands for superfluous features

System administrators

• Choosing good hosting companies and system software

• Configuring servers properly

• Updating software to plug known security holes

• Monitoring for security breaches (amongst other problems)

• Prodding programmers and bosses to clean up software that's stale or unnecessary (it leads to a cascade of IT complications)

Hosting companies

• Physical security at data centers

• Network firewalls

• Choosing good equipment and software

• Configuring it sensibly

• Providing sensible advice and instructions

People and institutions beyond our control (end-users, open-source coders, software companies, equipment makers, banks, regulators, legislators, voters, etc.)

• All share the blame for the current economic/polical environment, which is detrimental to effective computer security AND magnifies the resultant damage. I'm just saying.

What everyone needs to know about SSL ("https websites")

1. SSL encryption is an essential part of online security. It prevents hackers from intercepting communications BETWEEN your computer and the web server.

2. SSL is only a small part of security. A lot of computers (including PCs, mobile devices, and web servers) are loaded with viruses and backdoors, which sit there skimming data for passwords, credit card numbers, and other juicy tidbits.

3. SSL Certificate Authorities (the "trusted third parties") are an extortion racket offering a false sense of security. For a small fee, they will "certify" anyone, without so much as a background check or security audit. The indirect costs of this racket are enormous.

The good news is, the certification racket is falling apart. In 2011, a few big CAs were hacked and lost credibility, effectively shutting them down. Comodo's credibility is hanging by a thread. And, Verisign and Network Solutions have bought up most of the remaining CAs, raising the prospect of antitrust action. Or, a few more hacks and the entire industry will be wiped out. When that happens, more sensible web-of-trust systems are waiting in the wings.

Password advice

Courtesy of xkcd :)

The RIGHT WAY to become root on your server

1. Disallow remote root logins. Make sure /etc/ssh/sshd_config contains the line PermitRootLogin no (and not another line saying 'yes'!)
2. Create regular user accounts for day-to-day use. None of these should be in the 'wheel' (administrators) group, which lets you become root by typing sudo -i root and entering the same password you logged in with -- that, as they say, would be very bad.
3. Set very strong passwords for all accounts, especially root. Write them down.
4. Generate public/private SSH key pairs for regular users. Set good passwords on their private keys. Put each user's public key in their own .ssh folder on the server(s).
5. To login to the server from a Unix machine, run ssh-add and enter the password on your private key to 'unlock' it. (Some Unixes, like Ubuntu, automatically prompt for the password when necessary, in which case you can skip the ssh-add step.) Then run ssh YOURNAME@YOURSITE.COM and you're in, no password necessary.
6. Only become root when you need to. Don't do anything as root that you can do as a regular user... within reason.
7. To become root, run su - and enter the root password.