In Search of the True Value of Linux

NOTE: this document subject to change without notice. It is a work in progress.

29 October 2003
Matt Warner

Background
I'm admittedly one who favors Unix. Much of my professional experience is with Solaris, with additional work with IRIX and the BSD-based Mac OS X. Everywhere you look, Linux is coming on strong. But many established Unix folk don't understand where the value proposition lies with Linux, but we do understand that we'd be fools to just ignore it. This is my search for enlightenment.

Assumptions
In a given part of the world, a good admin, one who really knows their stuff, will cost the same regardless of the platform, be it Windows, a commercial Unix, or a free Unix.

Hardware
Much of the argument for Linux is based on the inexpensive, commodity hardware on which it can run. Every conversation I hear assumes that Intel-based x86 systems offer the biggest bang for the buck, bar none. This largely ignores inexpensive, higher-performing offerings such as AMD's Opteron and IBM's own Power4 and Power5 systems. It also ignores Intel's own Itanium 2, which in a meager dual CPU configuration is expected to cost more than $100K. Why does the Linux world seem to focus almost exclusively on x86? The signs are that the aging x86 architecture is reaching its end. What then? This makes Linux a great short-term solution that will hit the brick wall of obsolescence if they don't more vigorously adopt other hardware platforms. Other hardware platforms, however, are better supported by their respective vendor's OS. Catch 22.

My experience is that Intel-based hardware becomes obsolete more quickly than other vendors' stuff. Cheap or not, in a data center environment, swapping out equipment on a 6 month cycle isn't feasible in terms of cost or in administrator overhead.

When Linux first came out, there was speculation that it would displace Windows since it runs on the same hardware. This has gradually died down as people have come to realize that replacing Windows is harder than first thought and offers no real savings since it involves replacing only the OS--other costs such as infrastructure, training, etcetera all dominate. Now the Linux camp has turned its focus on the commercial Unix installations, citing the fact that Linux is really just a free Unix-like OS, which implies that it should easily integrate with and even replace existing Unix installations.

Speed or Scalability? Fewer, bigger CPUs or more, smaller CPUs?
This reflects our current programming model where many applications are, in effect, single threaded. Very few applications efficiently subdivide their tasks such that performance is unchanged when moved from a single CPU to two CPUs that run at only half the speed. Read-only multiuser applications tend to scale well across multiple CPUs, even on different systems. But read-write typically does not.

Based on clock speed alone, a 3GHz CPU should be able to service 3 times as many requests as a 1GHz CPU. But that's not the way it works out. Take the case of a single CPU running at a high clock speed. It services requests very quickly and will perform context switching so that concurrent applications appear to run in parallel. As the number of concurrent applications goes up, context switching becomes a bigger part of the system's task list. Compare that with a 72 CPU system where you have true concurrency. Many CPUs in a single system is sometimes called a Single System Image (SSI). A given load consumes much more time on 1 CPU than on 72. But the 72-CPU system costs far more than the single CPU. So what about 72 single-CPU systems? If you're talking about Intel-based x86 systems, the cost for 72 single-CPU systems is far less than the single, 72-CPU system. So why would anyone purchase the 72-CPU system? The answer is parallelization and very large memory access. Larger quantities of cheaper, single CPU systems are easy and inexpensive to fabricate. It requires far more complex hardware and OS programming in an SMP system to ensure cache coherency, processor affinity, localized memory access, and so on. It's horizontal scaling versus vertical scaling. Does your application scale across multiple systems? Read-only content such as many web sites provide is a good candidate. Updating that content, especially on a busy site, though, can be challenging. Highly transactional sites typically require a vertically-scalable system (lots of CPUs in a single system), at least for the data layer (often a database).

Clustering attempts to emulate a larger single system image (SSI) by connecting two separate systems and using software to approximate the performance of the larger system. Oracle's Real Application Cluster (RAC) uses this approach. How do you handle two systems trying to write to the same place? What about locking? You must also face these issues with an SSI, but even with gigabit links, you cannot approach the communication speed found across the backplane of an SSI. The only possible exception I've heard would be SGI's Craylink they used years ago to interconnect multiple Origin 2000 systems into an SSI. Not exactly commodity material, this. Even Oracle, with all its resources, had serious issues with its first implementation of OPS (Oracle Parallel Server). In short, it's very complex to do inter-system processing, and slow. Your scalability and performance both take a bullet.

So is the Linux value in cheap farms, or is it that it's "faster?" You will hear both arguments from its proponents.

It is one dimensional, but unfortunate or not, it's reality. Will it change? Are we getting more efficient at parallelization?

Can you get something for nothing?
One of the laws of thermodynamics is often paraphrased as "you can't get something for nothing; you can't break even either." So if the OS is free and the hardware is cheap and fast, what's not to like? It turns out that you can't get something for nothing. Support and application availability are two crucial areas.

The cost of purchasing a commercial OS and support is an easily-identified dollar amount. So easy, in fact, that advocates of Linux frequently use it when making the case for Linux. So what does Linux cost? That's where the waters get murky. Red Hat "licenses" you their OS, and includes support. Some wonder how they can license an open source offering, and where Red Hat's value lies. Does Red Hat provide software? Yes, they reportedly employ many of the core contributors to the OS. But the majority of what you receive for your money is support services.

Application availability sounds crazy to Linux folk. Everything worth running is either open source or has been ported to Linux, right? True, but it's too easy to ignore the true cost of deploying these applications. Open source is great because you get the source code. Open source sucks because you have to deal with the source code, including figuring out how to compile it and all its dependencies. You can download precompiled versions, and various package managers make getting the dependencies right a much simpler task. At the end of the day, it all comes down to infrastructure. If you're a dedicated Linux shop with thousands of systems all built identically, building additional systems doesn't even require dusting off any brain cells. But if you're deploying many different applications or have very few Linux systems, it's more challenging. Keep in mind too that if the vendor isn't providing the support, then you're providing it, whether you realize it or not. The cost hasn't disappeared, it's just migrated. Whether or not it's reduced will depend on you and your organization.

In light of the two recent worms that exploit Microsoft RPC vulnerabilities in two weeks (AKA MSBlaster, Nachi, etc), it's clear that it's all about you and your infrastructure. Otherwise companies would have dumped Windows. But they can't--there's billions invested in that infrastructure, so eliminating it isn't a feasible option for most companies. Instead, companies look harder into making their Windows infrastructure more resilient--automated patch deployment, anti-virus software updates, forced standard OS images, and so on. That Linux offers fewer viruses/worms for a much-decreased admin load seems to matter little to most managers. Companies employ administrators for only one reason: to support the code that makes money. So as long as developers continue to churn out code that runs only on Microsoft products, Windows will continue to exist in the data center, flaws and all. I see this as the primary reason why Linux had to ease off on the attack against Microsoft and turn on its Unix cousins instead. Will it find the same resistance? Get ready for irony. The commercial Unix vendors all agreed on open standards such as POSIX (portable operating system interface) many years ago, and that made it easier for developers to compile their code on various Unix flavors. So because the commercial Unix vendors comply with open standards, because they're not a proprietary, completely closed from end-to-end system (from development frameworks to OS to "integrated" applications), they're much more interchangeable than trying to move from Windows to Unix. This is why Unix flavors are sometimes called Open Systems. Being this open is what allows Linux the opportunity to come in and steal away the business. Here's the evil twist, though: Linux breaks with POSIX compliance in dozens of areas as documented by opengroup.org. We're even starting to see Linux-only code that will not compile on other flavors of Unix. Things such as non-standard variable types (u_int_32t versus uint32_t) contribute to porting nightmares.

Another issue with open source software is that much of it falls under the 80/20 rule. With some notable exceptions, most open source software is 80% complete, but providing the remaining 20% will consume 80% of the total time, and that's tough to do when your paying job consumes 8-12 hours of your day.

Per a September 1, 2003 Infoworld article, The Real Cost of Linux stated, "Because Linux developers have unfettered access to the Linux OS, fewer administrators are needed to manage more machines and greater workloads." Huh? What does developer access to the source code have to do with administration? While there are developers who genuinely know their stuff, the vast majority completely fail to understand systems administration. In more disciplined shops, developers aren't even allowed access to production equipment for this and other reasons. Larger companies also tend to have more rigid discipline--developers get paid to develop, not administer or play at Systems Engineer/Admin.

What of the argument that having access to the source makes it easier to identify problems? This assumes that developers will A) spend the time to track a problem, B) identify the specific piece of code causing the trouble, and C) will have the time and expertise to fix the issue. Identifying a problem is only half the battle. You still have to fix it. What about your timeline to deploy your custom code that will make your company money? Are you that diligent in fixing bugs in your own code? If it's a fundamental flaw with an open source project code, you've just bought a huge addition to your workload. It's true that with commercial software you aren't guaranteed a fix either. But there's a chance the vendor will (eventually) address it. After all, their developers get paid to write code, and ultimately, no matter how altruistic you may be, you and I require money to provide for our needs and wants.

References

POSIX defined
http://search390.techtarget.com/sDefinition/0,,sid10_gci214309,00.html

The Real Cost of Linux
http://www.infoworld.com/pdf/special_report/Linux_feature_2003.pdf