There are a wide variety of techniques out there for clustering storage appliances. The question is: what problem are you really trying to solve? If you look at Isilon’s clustered storage appliances (where I was chief architect), you’ll see that the clustering is done at the block level. The block addresses are generalized into a generic (node, drive, block_num) tuple and the on-disk data structures simply use that generalized address everywhere a block address would normally be used (plus a bunch of details I’m glossing over). The communication on the back end of an Isilon cluster is block reads and writes, transaction messages, and lock messages (plus some other miscellaneous bits). Each read or write operation is controlled by the initiator, and the smallest granularity of locking is at the block level. Cache lives both at the disk and at the initiator. If you were to put it into an architecture category, you’d call it an Infiniband SAN (Storage Area Network). This is perfect for a file system. This architecture lends itself to zero-copy, extremely high-performance file access for streaming files, very low CPU utilization on the nodes holding the disks (which allows the addition of the accelerator nodes for high speed FibreChannel and 10GbE), infinite scalability, and extremely low latency for operations on cached data.
However, it doesn’t support high read/write concurrency on a single file. Imagine if you ran an OLTP database with a high write load using an architecture like that. With the locking done at the block level, you can never expect to get high concurrency for items smaller than a block. Every node that wants to write to a block would have to get an exclusive lock on that block, which invalidates other nodes’ caches. If you had an active table with massive read/write load sitting on top of a cluster like this, performance would tank, dominated by lock contention. Then why do some databases take this approach to scale? How can you possibly make a shared-backend cluster resembling a SAN and expect it to scale with a database workload like some have done? How can you make an expandable storage engine plug-in and expect the entire database to scale? What works extremely well for a file system does not work at all for a database. We need a new approach.
Clustrix has a new approach. Rather than shipping the data blocks on the back end, we ship the queries. That may sound like an innocuous statement, but really it has a far-reaching impact on the architecture. To learn more, read the white paper A New Approach: Sierra Distributed Database Engine that I wrote on this subject. It shows how Clustrix has taken a novel approach to solve the clustered database problem, resulting in a database system that can handle high concurrency at any scale.
Surprising Advice For Startups
Clustrix just exhibited at two good shows: Structure and Velocity. I spent my time at Structure. One of the highlights of the show for Clustrix was the The Future of SQL In the Cloud panel that Paul was on. There was some good discussion about SQL vs. NoSQL and different products available for scaling your database. The title is a bit of a misnomer, however. Database challenges are orthogonal to the cloud. The database is a critical challenge no matter how the application is deployed.
But this isn’t what surprised me the most at Structure. That came in the How Does a Company Scale in Real Time? panel. This panel had representatives from PayPal, Engine Yard, Yahoo!, Facebook, and Zynga. These panelists are all responsible for ensuring their products continue to function smoothly. These companies are all recognizable and successful. I was really hoping to hear some good insight into operating large scale web properties and perhaps some good advice for new web startups. I think the money question came at around 16:30 in the video. Jonathan asks the panelists to rewind the clock and say what they would do differently and to give advice to startups for what they should focus on to avoid application bottlenecks. There was a lot of advice offered. There was talk of establishing a single-signon infrastructure. There was emphasis on avoiding SQL entirely, saying it is far and away the biggest scalability problem in web architectures today. There was a suggestion you should invest up front in separating out your architecture. You should think hard about the abstraction layers in key parts of your system and get the infrastructure right first. Another talks about getting the right instrumentation and metrics in your application. You should put in the right levels of abstraction, and the right levels of caching. You need to force yourself to run your application on two servers. Think about sharding and replicating at the beginning and force your scaling challenges to come up in advance before you have to scale.
None of this advice resonated with me. Startups have enough challenges finding the right business model and securing the initial set of customers. Why would you spend your very limited money, and more importantly, time on infrastructure and architecture of the back-end before you’ve proved your business? Matthew Mengerink, VP of Customer Quality, Engineering Services, and Site Operations at PayPal finally stepped in and made some sense. He says he would do nothing different. To him, a startup working on architecture is a waste of a dollar. He advises spending your time and money on making the business model work. Thank you, Matthew!
This discussion reminded me of a post titled Start In the Middle I read a while ago. In there, it offers a bit of advice that should be obvious that is so often not: solve the interesting bit of the problem first. Prove there’s value in the core idea, and then flesh out the infrastructure around it. Do what makes your business unique first and put the majority of time into that. Delay building the rest of the surrounding architecture until its really needed or when possible, just buy those bits. The rest of that BS is not what your customers see and not what makes you money.
So what’s my advice? Make the code and architecture as simple as you possibly can but no simpler. Don’t abstract too early and don’t make premature optimizations. Rewriting code and adding abstractions later is not a sign of coding failure, rather it’s a sign the product is successful. Choose the data model that fits your problem. That is frequently SQL. Full transactional and relational SQL is exceptionally expressive and it fits many problems so well. Why wouldn’t you use it? Don’t re-invent things that are not part of your core business. When the product becomes successful and the load starts to ramp, collect hard data to find the bottlenecks and fix them when they become a problem. When your data shows that your database is that bottleneck, perhaps Clustrix can help.