Load Impact Load Testing and Performance Testing

We're happy to introduce Load Impact 2.3!

Load Impact 2.3 contains a new and improved proxy recorder that automatically detects pages and creates page load time result metrics for each of your web pages. The recorder also allows you to insert code comments in the generated user scenario, which can be useful in order to find where in your user scenario code a certain page is being loaded.

Behind the scenes, Load Impact 2.3 also includes a lot of optimizations that result in a much faster reporting interface, especially for large tests that generate a lot of results data these optimizations will make a huge difference to how snappy the "view test" page feels. And for live tests, the reporting page will also be a lot smoother. In fact, Load Impact 2.3 is a major rewrite of the underlying storage subsystem and how data is being accessed by the user interface code. More things are loaded on-demand now (i.e. as/when needed) and this results in a page that is much lighter on the client computer. You should now be able to view even the largest tests on the flimsiest of laptops.

Other improvements you will find in 2.3 include:

• Graphical editor support for data stores, custom metrics and other new API functionality 
• Several API updates - http.page API functions, named parameters, etc.
• You can now plot graphs of load generator CPU and memory usage during the test!
• The URL list on the report page now displays bytes received and compression ratio
• Content type classification now uses the Content-Type header
• Click the pie charts to highlight different objects in the URL list on the test report page
• Many bug fixes...

Big data - what is that?

You might have heard the term, but its actual meaning depends a bit on who you are. Big data is essentially a name for data sets that are much larger than they used to be. It is a term that describes how modern applications often generate enormous amounts of data, as compared to their fairly recent predecessors. It can be the Large Hadron Collider, a weather satellite or a popular social networking site but they all tend to generate huge amounts of data. Data that needs to be stored somewhere.

The rapid development of harddrive technology and network communications has enabled data sets to also grow very rapidly. This means that transferring the bits and bytes, and storing them on some kind of physical media is not a big problem. What is a bit of a problem, on the other hand, is the software that is used to store and retrieve data. Like SQL database applications. These were excellent when all you wanted to do was store the name and address of your company's 1,000 customers in an organized manner, so that you can then perform searches for all customers located in a certain city, etc. However, today there are companies like Facebook that has to store huge amounts of data - data that they want fast access to as they need to quickly dig up those 14 pictures that are yours from a collection of over 40 billion uploaded pictures. This was not what the old SQL databases were designed for, and in general, many of the old software tools and applications for handling data simply are not up to working with these new and huge data sets.

Enter technologies such as NoSQL-databases - Cassandra, Voldemort, CouchDB, MongoDB, Neo4J, Dynamo, Redis, Memcached etc, etc. There are now a wide range of different systems that can store large amounts of data in an efficient manner. You always get some kind of trade-off, of course, and the end result is always a more complex design of your application, but it allows you to scale the size of your data sets to previously unimaginable levels. The development of these systems is progressing very rapidly, and data sets are growing at the same furious pace as a result. Load Impact uses Apache Cassandra to store load test results, providing us with a flexible way to scale our system as the number of users and the amount of test data we store increases. Currently, our test result database is growing at a rate of several gigabytes per day.

Cloud computing is another enabler of big data. While previously it was difficult to scale your infrastructure to be able to handle large data sets without making huge upfront investments in your infrastructure, today you can rent the infrastructure as, and when, you need it. An application that quickly has to perform a complex calculation on a large set of data, but do it only occasionally, would previously have been too costly to run because of the infrastructure cost. Today you can rent a thousand Amazon EC2 servers for an hour and pay only a couple of tens of dollars to do so.

For us, big data is a positive development as it increases the demand for large-scale load testing. Online services get larger and more resource-intensive, and there is money to be saved on optimizing your solution to use the least resources possible. Even more important, in many cases, is optimizing for speed so that people (or machines!) using your online application, site or service, will not choose a competitor over you. Speed is becoming a critical competitive advantage, and load testing is an important test method for those who want to ensure that their site or application is fast under all circumstances. Traditional load testing solutions are often unable to scale up the load levels to what is required to properly stress a large site or application out there, so we see a clear trend that people are becoming more and more interested in cloud-based, online load testing.

If you want to know more, a good start is the Wikipedia article on big data:

http://en.wikipedia.org/wiki/Big_data

We, are happy to introduce two new, major features in Load Impact that many users have asked for: parameterized data ("data stores") and custom metrics.

Parameterized data is when you're able to provide data in bulk using some common format - often a CSV (comma-separated) file that you upload, and which you can then access from your load script. The typical example is when you have e.g. 10,000 login names and passwords that you want to use in your load test. Entering these by hand into your load script code is prohibitively time-consuming, but with parameterized data you just upload a text file with all the usernames and passwords, and are then able to access these from inside your load script.

Custom metrics is a feature that allows you to store arbitrary result metrics during a load test. A typical use-case would be to store the load time for a certain page on your site (as opposed to just storing the load time for individual URLs/resources on the page). A more advanced use-case would be to fetch server monitoring data (via HTTP) from the web servers that are being tested, and log e.g. their CPU load along with the standard response time data collected by Load Impact. Any metrics stored with our custom metric feature will be visible in the test results interface, and can be plotted as graphs for easy correlation with the standard metrics.

Parameterized data in Load Impact

Parameterized data in Load Impact is implemented using something we call data stores. A data store is basically a two-dimensional array (a "table") with data that can be shared by multiple clients in a load test. The usage is simple: You create a new data store in the user scenario configuration interface, assign a name to it, then upload a text file with the data you want to insert into it. The data file should be in CSV format (comma-separated values) and be a two-dimensional table, but can contain any number of rows and columns. When you have a data store assigned to a user scenario your load test clients can then use the data store API functions to access the data store, and retrieve data from it.

Further reading: FAQ: How do I use parameterized data?

Custom metrics

Post-2.0 updates and plans for 2012

We released Load Impact 2.0 at the end of October, and the reception has been really, really good. We see an increased number of user registrations and more user activity and what is especially fun is to see that people are starting to use Load Impact for really complex load testing of a wide range of different platforms and applications. It seems all the new functionality of Load Impact 2.0 has been very well received and that people are starting to realize its potential, which is great to see for us who have worked so hard in 2011 on getting it out the door.

Right after release we had a number of issues with the payment system, as well as numerous small bugs that only manifested themselves in production, but overall it was a successful release without any major malfunctions. There are still small annoyances left to iron out, but we are making the service better by the day, and also adding new functionality. Here is a list of some things we have done post-release:

• Support for new load zone in south america - Sao Paolo, Brazil - and new US West load zone - Portland, Oregon.
• New chart/graph component implemented, providing even more advanced graphing capabilities (e.g. instant zoom)
• Improved help/documentation - customer case studies, load scripting tutorial and example scripts
• Data export functionality added (export to CSV)
• Several problems related to the payment system have been fixed. AMEX support was added.
• Several problems related to test startup have been fixed.
• Credit refund logic for aborted or failed tests has been improved. You now get partial or full refunds when a test fails, for some reason.
• Anonymous tests are run from random load zones.
• Numerous small UI bugfixes/improvements.
• HTTP basic Auth now supported for automatic load script generation.

Load Impact 2.0 was released at the end of October (27th). The first few days after release were pretty chaotic, with lots of minor issues and some major ones, but having been involved in many big releases during my career I have to say that this one went pretty well actually. The system was up and functional most of the time, the first few days post-release, and that isn't bad at all :-)

Still, there were some difficulties, of course. We had problems first with AMEX payments due to contractual reasons (AMEX payments have been removed for now, until we manage to get through the AMEX bureaucracy) and then with VISA/MC payments. Then there was occasional problems with internal queueing systems that caused some load tests to either fail, "freeze" (get stuck in some state), or never get started. All these issues should have been resolved by now, but there are likely smaller things that will pop up from time to time, so we urge everyone to get in touch with us if you see anything strange happening on the site. Don't hesitate to get in touch with us even if you're unsure whether something is a problem on our side or not, we want to know about all situations where someone has any kind of problem using our service. No issue is too small.

In general, the system is starting to get very stable now, however, and we see more activity than before the release, with more user registrations and more tests being executed. We also see more advanced usage of our service - more people are writing advanced load scripts and running both larger and more complex load tests than ever before. It is all very encouraging and tells us that we are moving in the right direction!

So what is so great about 2.0 then?

Some people may see Load Impact 2.0 as simply an upgrade, but it's more like the launch of a whole new service. It is that much different from 1.0. We have kept some 1.0 key elements that we (and hopefully everyone else) liked such as the ability to run small, simple tests anonymously from our front page, the ability to watch other such anonymous tests that are being run, and the scripting language and scripting API, but behind the scenes most of the code base is new and 2.0 includes a lot of new functionality that didn't exist in 1.0. Here is a small list:

• Large-scale load tests
  As we are now using public cloud infrastructure (Amazon) to generate load test traffic, we have the ability to scale up a load test to a very large size at any of the geographic locations where there are cloud servers available (currently California, Oregon and Virginia in the US, plus Ireland, Japan and Singapore outside the US).
  
  
  
• Multiple user scenarios in a single test
  In 2.0 we introduce "user scenarios". A user scenario defines a certain simulated user category and what that category should be doing on your site. An example can be an e-shopping site that has two types of visitors - one type that just browses the site without buying anything, and another type that registers a user account on the site and then goes on to actually buy products on the site. In Load Impact 1.0 you could not easily combine these two different user categories in a single load test, but with Load Impact 2.0 it is easy - you just create two different user scenarios, that run different load scripts, then you configure your load test to use these two scenarios.
  
  
• Multiple geographical traffic sources
  With Load Impact 2.0 you can now choose to have your traffic originate from more than one physical place, if you want. You can specify any number of combinations of user scenarios (described above) and geographical locations where that particular user scenario should be executed, and create very complex load test configurations where you define that e.g. 10% of the total number of simulated users during the load test should run user scenario X from geographical location Y.
  
  
• More performance metrics
  We now collect more performance metrics than in 1.0, such as "requests per second", and we collect many more sampling points that are all time-based rather than client level-based. This results in more performance data available at higher resolutions than before.
  
  
  
• Much more advanced chart/graph capabilities
  We provide a very dynamic test report page where you can create your own charts and graphs, plotting a wide range of parameters and correlate data with a certain user scenario or test results from a certain geographical region.
  
  
  
• Text-based script editor
  
  For expert users, a text-based scripting editor is usually the best choice, and in Load Impact 2.0 we provide the option to choose between our graphical script editor (LILE) and a text editor that allows easy copy-and-paste and faster code entry for the experienced programmer. Load script programmers now have much more choice in how they create their load scripts.
  
  
• Continuous tests
  Load tests are now executed continuously, which means that a simulated client thread is never shut down as long as the load level is meant to increase. Old simulated clients will just continue execution, reiterating their load script again and again, while more clients are being added. The result is a smoother and more time-efficient ramp-up process than was offered in Load Impact 1.0.
  
  
• Credit based pricing model
  Load Impact 2.0 introuces the credit based model that means there is no difference between one user and the next as regarding them being a "premium" user or not. All users are the same, they just have different amounts of credits, and the ones that have more credits can run larger and longer tests than those who have few credits. This provides several advantages - first of all it allows us to skip all the old limits on how many tests you can run per 24 hours, etc. Now, every test you run consumes credits and only the number of credits you have affects the number of tests you can run. Secondly it means we don't have to restrict access to some functionality to premium users - everyone can do everything on the system, so it is easy to "try before you buy". Thirdly, it makes our product much simpler in general as we only sell one single thing now - Credits - while as earlier we sold access to different premium levels for different amounts of time, making everything a lot more complex. The drawback, however, is that it can be difficult for people to understand exactly how many credits they need to do the testing they want to do. All in all, though, we think the upsides with the credit model are much bigger than the downsides.