Leaders in checkouts‐per‐hour for Magento
During peak season or fast growth, the last thing you want is to lose orders because your e-commerce site buckles under the load and can’t keep up with demand. The engine of your e-commerce site – the technical component that’s most critical to moving at the speed of revenue – is your database. Your database engine is involved at all stages of e-commerce, and as it approaches its limits, your site can slow down or crash. With ClustrixDB powering your Magento application, that never has to happen.
To establish that ClustrixDB can scale Magento solutions to a scale MySQL cannot, Clustrix has executed benchmarks that prove industry-leading performance for Magento applications hosted on cloud infrastructure. In benchmark tests simulating real-world workloads during periods of peak demand, ClustrixDB averaged more than 25,000 checkouts per hour. With conditions more typical of a flash sale, that number rose to over 44,000 checkouts per hour – more than 12 checkouts per second. ClustrixDB demonstrated it handles site traffic generated by over 300,000 visitors per hour.
The purpose of this benchmark is to highlight that ClustrixDB, with it’s patented architecture that distributes work across multiple servers, can scale far beyond MySQL. Because ClustrixDB can add power by adding systems to the database, the benchmark was focused on scale alone. We did not run the additional tests that would be necessary to make price‐performance comparisons.
Our technical objective was to push Magento application servers to levels not obtainable with a MySQL server. While MySQL results might be improved by breaking the database into pieces (‘sharding’) or replicating the database (‘read slaves’), there are problems with these approaches. Sharding is a complex and, in our view unsupportable, and ‘read slaves’ only add value to read transactions. It does not help with scaling write transactions. The tradeoff for increased performance-which may be marginal-is increased fragility, higher administrative costs and a high probability of data inconsistencies. In the end, these approaches cannot address the fundamental problem that MySQL was not architected to distribute writes across multiple servers and therefore cannot scale effectively beyond a single server. For scale and ease of use, these workarounds can never match what ClustrixDB provides.
What is ClustrixDB?
ClustrixDB is a turbocharged scale‐out database engine that delivers all the power you need when you need it. ClustrixDB scales by distributing SQL reads and writes across a large number of servers or ‘nodes’. Unlike MySQL, ClustrixDB was architected from the ground up to harness the power of all cores – and nodes – available to it, and it does so easily and seamlessly. The Magento application doesn’t need to know that it’s working with a cluster or that the cluster is growing. And because ClustrixDB nodes can be added to a cluster easily, it’s as if your engine can go from 4- to 6- to 8- cylinders while your car is racing down the road.
In fact, because ClustrixDB is able to add and remove nodes in a matter of minutes, administrators can ‘flex up’ and ‘flex down’ their hosting resources to match short term needs – growing hosting resources before a seasonal peak period, such as holidays, and shrinking to a more cost – effective solution afterward.
To prove our scalability, we tested our ClustrixDB database engine using tools provided by Magento that simulated an e-commerce workload – specifically, we used Magento Performance Toolkit rev 1.14. That allowed us to run the exact same test on ClustrixDB and MySQL so the results can be compared.
The Magento toolkit populates a database based on a small number of input parameters and generates a corresponding workload.
In our benchmark, we used JMeter, a popular Java tool for load testing, to simulate the web site’s visitors – browsing, sometimes adding items to a cart and then either abandoning the cart or checking out – according to the plan created by the Magento toolkit.
The standard benchmark environment developed by Magento was not designed for benchmarks of this scale, so we adapted it in minor ways to better simulate a very large application in the real world. For example, to simulate 300,000 visitors, multiple JMeter instances were required. Their web traffic was fed to a bank of 18 Magento systems, which in turn acted as clients to the database engine – either ClustrixDB or MySQL. Access from the Magento clients to products in the database was segmented to more realistically simulate a database load across multiple storefronts. The number of customer accounts was increased from 200 to 5000.
The workload generated by the Magento toolkit simulated customer activity on an e-commerce site with these characteristics:
|1000||Configurable products – 3 options each|
|300||Categories, which can be nested 3 deep|
|20||Catalog price rules|
|5||Catalog target rules|
|20||Cart price rules|
|2||Cart price rule floors|
Requests to the simulated e-commerce site were generated by visitor sessions scheduled to arrive over four-minute segments. The number of visitors per four-minute segment varied with system size. ClustrixDB tests were assigned 24,000 visitors per four-minute segment; MySQL tests were assigned 9000. MySQL was unable to exceed 9000 visitors without the error rate experienced by users climbing above 1%. In contrast, ClustrixDB had an error rate of less than 0.1% with over 2.5x the visitors.
Two conversion rates for customers were tested: 8% and 20%. An 8% rate simulated the conversion rate that might be experienced during an extended period of peak demand. A 20% conversion rate was used to simulate a flash sale.
Each user visit was assigned one of four shopping profiles: browsing, adding items to a cart, etc. The distribution of user requests across these profiles follows:
|Visitor Shopping Profile||Conversion||Rate|
|Browsing, adding items to cart and abandoning cart||62%||50%|
|Browsing, adding items to cart and checking out as guest||4%||10%|
|Browsing, adding items to cart and checking out as registered user||4%||10%|
Benchmark Hardware and Software
The servers represent hardware typically found in a hosting or cloud environment. Linux CentOS 6.5 Operating System ran on all servers. SSD and spinning disks were available on each server, but only SSD disks were used. 10Gb Ethernet networking was used throughout. All tests were run in a cloud environment.
Clustrix Database Environment
Nine 16-core servers were used as nodes within the ClustrixDB database. Each server was provisioned with 64GB of RAM and 600GB of SSD storage. ClustrixDB software version 6.0.1 was used.
Magento requests were sent to a simple load balancer that randomly distributed them to ClustrixDB instances. The load balancer was VM-aware and part of the cloud.
MySQL Database Environment
MySQL was tested on a 32-core server with 458 GB RAM and 1200GB of SSD storage. MySQL software 5.6.24-3.el6 was used in both tests. Since a single server was used, no load balancer was between the Magento clients and the MySQL server.
Magento Client Environment
Magento clients were spread across eighteen 16-core servers provisioned with 64GB RAM and 800GB of SSD storage. Load balancing of requests and static content of the Magento application was handled by 4 nginx servers, each of which ran on a 4-core server.
Redis cache servers were used by Magento clients to cache three types of data: full pages, Magento ‘standard cache’ data, and session data. 8-core servers were used.
|Magento Enterprise Edition||1.14.1|
|PHP-fpm||5.5.23||Fast CGI process manager|
|PHP||5.5.23||Executes Magento code|
|nginx||1.0.15||Serves static content to the Magento app and
balances the active load among all Magento clients
|Redis||2.8.19||Cache for key-value pairs used for hash tables|
Requests were fed to the Magento clients by 16 instances of JMeter running on 4-core servers. One instance of JMeter was assigned the additional task of coordinating the execution of all user streams across all 16 JMeter servers. Each JMeter instance routed its traffic through a simple load balancer that randomly distributed traffic to the 4 nginx servers, which in turn passed them on to the Magento clients.
Multiple benchmark tests were run. The first test compared ClustrixDB against MySQL using a conversion rate of 8%. This represents a demanding load that might be sustained during a period of peak sales. The second set of tests boosted the conversion rate to 20%, which might be seen during a flash sale. In all cases, the ability of ClustrixDB to scale was impressive with over 300K visitors/ hour and 7 checkouts/s.
No two real-world customer environments are the same, and this is even more true for simulated customer environments. For that reason, Clustrix does not recommend extrapolating these results to your environment. These results are intended for use in comparing ClustrixDB and MySQL in a controlled way that can be repeated by others.
Throughput-ClustrixDB vs MySQL
The following results were delivered by MySQL and ClustrixDB at the 8% conversion rate:
|Environment||Conversion Rate||Visitors/hr||Checkouts/hr||Checkouts/s||Page Views/s|
As can be seen, ClustrixDB, at 318,820 visitors per hour, was able to scale far beyond the capacity of MySQL, at 135,000 visitors per hour. A 32-core server was used for MySQL because that’s the largest server that is commercially available from most hosting providers and that server was running close to maximum capacity – allowing little room for growth. While a server with more cores might offer more capacity, finding that server could be problematic.
The Magento PHP server had a 60% utilization rate. At 35% system utilization, the ClustrixDB engine had power to spare. We believe that still more throughput would have been seen if more Magento client nodes had been added to our test environment.
|Environment||Conversion Rate||Visitors/hr||Checkouts/hr||Checkouts/s||Page Views/s|
With this test, we boosted the conversion rate to 20% to see how it affected the performance of ClustrixDB. The peak in check outs 12/s is outstanding.
System utilization of the database server was 31% for the test with a 20% conversion rate versus 35% for the test with the 8% conversion rate. As before, the PHP server had a 60% utilization rate.
Below, we see the effect on checkouts per hour over a spectrum of conversion rates.
The growth rate drops somewhat after the 10% conversion rate. The reason for this is not known, but most likely either the database engine is approaching a capacity constraint within the cluster of servers assigned to it or the Magento clients are approaching a limit of the servers assigned to them.
Because we saw somewhat lower system utilization at the higher conversion rates, we believe that the gains we saw at lower conversion rates would have been sustained if we had added more Magento client nodes.
In our next chart, we see a continuous increase in checkouts per hour as conversion rates increase. Above the 10% conversion rate, visits per hour falls off at a faster rate, but the cause was not determined. As above, this is most likely due to a constraint on the number of servers used for either the Magento clients or the ClustrixDB engine. Checkouts are database intensive, so it’s possible that each additional checkout is displacing multiple page views and cart updates in the database engine, but the cause might be resolved by more Magento clients as well. In any case, the number of visits per hour remains impressive – well in excess of 200,000.
Our final chart shows the effect of a higher conversion rate -with more checkouts- on page views. Here, the fall-off in page views parallels the decline in visits per hour seen in the previous graph.
Response times were captured for the ClustrixDB test showing 318,820 visitors per hour at an 8% conversion rate – the test used to compare with MySQL results. Looking at response times in milliseconds, we see that the simulated user experience was within bounds for a real-world workload for the vast majority of users:
|Task||Response Time Window||% of Responses within Window|
|Add Product to Cart||1500ms||98.3%|
|Checkout as Guest||2000ms||98.1%|
|Checkout as Registered User||2000ms||98.2%|
Another important measure of user experience is the error rate experienced by the end user. When ClustrixDB was used, fewer than 0.1% of user operations involved errors. The error rate with MySQL was 1.02%.
These benchmark tests have proven that ClustrixDB can scale far beyond the limits of MySQL. Tests run with a standard toolkit developed by Magento showed that the ClustrixDB database engine has the power to scale to more than 300,000 visitors per hour with a high conversion rate (8%), acceptable response times and very low error rates (<0.1%). And, in a test simulating flash sales, ClustrixDB demonstrated 44,500 checkouts per hour – more than 12 checkouts per second.
ClustrixDB is able to achieve these remarkable results because of its patented distributed database architecture, which enables the database to grow by adding cores and nodes. In contrast, MySQL coordinates all writes through a single node, so it is no surprise that, even on a server with 32 cores – the highest core count offered by most hosting providers – MySQL was unable to match the performance of ClustrixDB running on multiple smaller nodes.
Your database needs to fire on all cylinders to keep up with growth and meet peak seasonal demand. Running on a database system that coordinates all writes through a single server can hold you back, and that’s costly. Choose the ClustrixDB engine with its distributed architecture to turbocharge your Magento site for growth and performance.